# De Haas-van Alphen effect and a giant temperature peak in heavy fermion SmB6
compound

Vladimir A. Zyuzin  L.D. Landau Institute for Theoretical Physics, 142432,
Chernogolovka, Russia

###### Abstract

In this paper we give a possible explanation of the giant temperature peak in
the temperature dependence of the amplitude of the dHvA oscillations
experimentally observed in heavy fermion SmB6 compound. We propose a
theoretical model when two species of fermions, one much heavier than the
other and both with electron-like dispersing energy bands, although hybridize
to open up an energy gap at their degeneracy point, but nevertheless keep the
system metallic with a heavy fermion hybrid at the Fermi energy. It is shown
that the heavy fermion hybrid oscillates in the dHvA effect not only with its
own Fermi surface, but also with a surface which does not correspond to any
existing fermion state. The experiment suggests that the values are such that
the surface of the cross-section of the fermion energy bands before they
hybridize approximately equals to the Fermi surface of this unexisting fermion
state after the hybridization. It is exactly this heavy fermion hybrid which
results in the giant temperature peak. A number of new signatures for the
experiments to look for in order to confirm or rule out our explanation is
given.

Recent experiments where quantum oscillations in Kondo insulator SmB6 were
observed LiScience2014 ; SuchitraScience2015  still pose questions to the
theory. It is claimed this material becomes insulating below
$T_{\mathrm{c}}=40\mathrm{K}$ due to the gapping of the Fermi surface.
Resistivity grows below this temperature and saturates at very small
temperatures. The observed quantum oscillations occur only in the
magnetization, i.e. it is de Haas - van Alphen (dHvA) effect which is
observed, while the resistivity does not show any oscillations.

In our point of view, there are three main experimental facts drawn from
LiScience2014 ; SuchitraScience2015  a successfull theory should address.
First of all, the observed frequency of dHvA oscillations of the insulator is
that of the metallic phase of the system LiScience2014 ; SuchitraScience2015 ,
i.e. before it turned insulating via the gapping mechanism. In experiments it
was proved by comparing frequencies of a material LaB6 which has a similar
band struture to SmB6, but which never turns insulating. Proposed theories
KnolleCooperPRL15 ; AlisultanovJETP2016 ; PalPRB2016 ; PalArxiv2022  do
explain this fact in a various models of insulators.

Secondly, another main outstanding question is the giant temperature peak
observed at small temperatures SuchitraScience2015  in the temperature
dependence of the amplitude of the dHvA oscillations of SmB6. This is a
drastic deviation from the standard Lifshits-Kosevich formula of the amplitude
and there is no theory so far which explains the peak. In this paper we give a
possible explanation of this giant temperature peak. Based on our model and
predictions, we will try to address the third issue of the difference of the
experimental results of the two groups LiScience2014 ; SuchitraScience2015  in
the discussion part of this paper. In addition, certain corollaries on the
nature of SmB6 system can be drawn from our theory. Then, in order to verify
our explanation of the peak, based on the corollaries, we provide new
experimental signatures to look for in the samples of experiments
SuchitraScience2015 .

One of us in Zyuzin2023  have obtained original theoretical details of the
dHvA in correlated insulators. There the occurance of the hybridization of the
dispersive band corresponding to itinerant $d-$ electrons with the localized
flat band of $f-$ electrons leading to the insulating gap in the system was
studied. The case when the hybridization gap is treated self-consistently was
considered. It was found that the dHvA oscillations are possible, in accord
with KnolleCooperPRL15 ; AlisultanovJETP2016 ; PalPRB2016 ; PalArxiv2022 , and
that instead of a standard Lifshits-Kosevich formula for the amplitude there
are quantum oscillations of the amplitude with inverse temperature which show
characteristic local maxmima and minima. However, this result doesn’t explain
any giant temperature peak similar to the one in SuchitraScience2015  in the
expression for the amplitude of oscillations.

We will generalize the theoretical model of Zyuzin2023  to the case when $f-$
electrons are not localized, but rather have electron-like dispersion with a
large mass. The Hamiltonian of our new model is given by

$\displaystyle H=\int_{\bf k}\bar{\psi}\left[\begin{array}[]{cc}\xi_{\bf
k}&\theta\\\ \bar{\theta}&\alpha\xi_{\bf k}\end{array}\right]\psi,$ (3)

where $\alpha>0$ and $\alpha\ll 1$ is a measure of how heavy the $f-$
electrons are, $\xi_{\bf k}=\frac{{\bf k}^{2}}{2m}-\mu$ is chosen to
conveniently describe the cross-section of the two electrons spectra, and
$\theta$ corresponds to the hybridization between $d-$ and $f-$ electrons and
in principle should be considered self-consistently just like it is done in
Zyuzin2023 . The Fermi energy is set to the cross-section of the two electron
spectra. Resulting dispersion assuming mean field ansatz for the
hybridization, $\theta=\mathrm{const}(t;{\bf r})$ and
$\theta^{*}=\bar{\theta}$, is

$\displaystyle\epsilon_{{\bf k};\pm}=\frac{1+\alpha}{2}\xi_{\bf
k}\pm\sqrt{\left(\frac{1-\alpha}{2}\xi_{\bf
k}\right)^{2}+\bar{\theta}\theta}.$ (4)

A zero of $\epsilon_{{\bf k};-}$ is at $\xi_{\bf
k}=\sqrt{\frac{\bar{\theta}\theta}{\alpha}}$, which is the Fermi surface of
the heavy fermion $d-f$ hybrid. See Fig. (1) where on the left the system is
before the hybridization while after the hybridization is on the right.

Figure 1: Schematics of a $k_{y}=0$ slice of the spectrum of the heavy fermion
model given by the Hamiltonian Eq. (3) for $\alpha=1/700$, $2m=1$, $\mu=4$,
and $\bar{\theta}\theta=1.5$. The numerical values (not directly related to
the experimental values) of the parameters are chosen to only picture the
Fermi surface after the hybridization equal to approximately $3$ times larger
than that before. Left: before the hybridization. Right: after the
hybridization.

Let us now add magnetic field and study the dHvA effect. Just like it is done
in Zyuzin2023  we will only focus on the self-consistent equation on the
hybridization, as it will already contain all necessary temperature dependence
of the dHvA effect. In magnetic field $B$ the equation is

$\displaystyle 1=UB\sum_{n}\frac{{\cal F}_{\epsilon_{n,+}}-{\cal
F}_{\epsilon_{n,-}}}{\sqrt{(1-\alpha)^{2}(Bn+\frac{B}{2}-\mu)^{2}+4\bar{\theta}\theta}},$
(5)

where ${\cal F}_{x}=\tanh\left(\frac{x}{2T}\right)$ is the Fermi-Dirac
distribution function, and $\epsilon_{n;\pm}$ are obtained from Eq. (4) by
$\xi_{\bf k}\rightarrow B\left(n+\frac{1}{2}\right)-\mu$, where $n$ is the
Landau level index, and where we count $B$ in cyclotron frequency
$\omega_{B}$, and finally where $U>0$ is the repulsive interaction. We again
consider two-dimensional case, while all of the results can be easily
generalized to the three-dimensions straightforwardly, and we again utilize
the Poisson summation formula to sum up over the Landau levels,

$\displaystyle\sum_{n=0}g(n)=\int_{0}^{\infty}g(x)dx+\sum_{p\neq
0}\int_{0}^{\infty}e^{i2\pi px}g(x)dx,$ (6)

where $g(n)$ is some function. Details of the dHvA effect can be carried out
from the $p=\pm 1$ terms of the sum. They read as

$\displaystyle{\cal
R}^{(+1)}_{\mathrm{osc}}(T)=2e^{i2\pi\left(\frac{\mu}{B}-\frac{1}{2}\right)}$
(7) $\displaystyle\times\mathrm{Re}\int_{-\infty}^{+\infty}\frac{e^{i2\pi
z}}{\sqrt{z^{2}+b^{2}}}\left[\frac{1}{2}-\frac{1}{e^{\frac{-z+\sqrt{z^{2}+b^{2}}}{2T/B}}+1}\right]dz,$

where have we applied ${\cal F}_{x}=-{\cal F}_{-x}$ identity in the
distribution function ${\cal F}_{\epsilon_{n;-}}$, and in the integral
containing ${\cal F}_{n;+}$ we have made $n\rightarrow-n$ change of variables
and as a result obtained $\mathrm{Re}$ part as shown. All we will do is
analyze the residues of the distribution function, which are defined by

$\displaystyle z_{m;\pm}=\frac{i}{\beta^{2}-1}\left[\beta
T_{m}\pm\sqrt{T_{m}^{2}-\left(\beta^{2}-1\right)b^{2}}\right],$ (8)

where $T_{m}=\frac{4\pi T}{(1-\alpha)B}(2m+1)$ is the dimensionless fermionic
Matsubara frequency, $b=\frac{2\sqrt{\bar{\theta}\theta}}{(1-\alpha)B}$, and
we will be calling $\beta=\frac{1+\alpha}{1-\alpha}$. When $\beta=1$, i.e.
$\alpha=0$, only $z_{m;-}$ works and we recover the case of Zyuzin2023 ,
indeed

$\displaystyle\lim_{\beta\rightarrow
1}z_{-}=\frac{i}{2T_{m}}(T_{m}^{2}+b^{2}).$ (9)

This is also the case for $\alpha>0$ and $\alpha\ll 1$ at large tempeartures,
when $T_{0}^{2}\gg\left(\beta^{2}-1\right)b^{2}$, and we expect that the
results of Zyuzin2023  hold at these temperatures. In the $T\rightarrow 0$
limit, we simply put $T=0$ under the square root in Eq. (8),

$\displaystyle
z_{m;\pm}=\mp\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}\frac{1}{B}-i0\mathrm{sign}(2m+1),$
(10)

and the oscillation of the heavy fermion $d-f$ hybrid become perfectly
coherent in this limit. The result in this limit is similar to the standard
Lifshits-Kosevich formula.

The analysis above means that at large temperatures there is only an
oscillation with $F_{1}\equiv\mu$ frequency, i.e.
$\cos\left(2\pi\frac{\mu}{B}\right)R(T)$, where $R(T)$ is the Dingle factor in
an insulator (for example studied in Zyuzin2023 ). At zero temperature the
Dingle factor transforms in to
$R(T)\rightarrow\cos\left(2\pi\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}\frac{1}{B}\right)$.
Therefore, the resulting oscillation will be composed of

$\displaystyle\cos\left(2\pi\frac{\mu}{B}\right)\cos\left(2\pi\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}\frac{1}{B}\right)$
(11) $\displaystyle=$
$\displaystyle\frac{1}{2}\left[\cos\left(2\pi\frac{\mu+\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}}{B}\right)+\cos\left(2\pi\frac{\mu-\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}}{B}\right)\right],$

and if oscillation with
$F_{2}\equiv\mu+\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}$ frequency
corresponds to the oscillation with the Fermi sirface of the heavy fermion
$d-f$ hybrid, oscillation with
$F_{3}\equiv|\mu-\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}|$ does not
correspond to any existing Fermi surface. This is the main result of the
present paper. We propose that the giant temperature peak in the amplitude
observed experimentally in SuchitraScience2015  is due to the oscillation of
the heavy fermion $d-f$ hybrid. It can only be possible if and only if
$F_{1}\approx F_{3}$, then
$F_{2}=\mu+\sqrt{\frac{\bar{\theta}\theta}{|\alpha|}}\approx 3\mu$. We indeed
see in SuchitraScience2015  that in addition to the main frequency
$F^{\mathrm{exp}}_{1}=330\mathrm{T}$ there is also
$F^{\mathrm{exp}}_{2}\approx 1010\mathrm{T}$ (top panel of Fig. 2a and Fig. 3a
in SuchitraScience2015 ), though with a rather small amplitude at the
temperature slightly above the peak (we will come back to that below). Then,
clearly, if we set $\mu=330\mathrm{T}\equiv F^{\mathrm{exp}}_{1}$, then
$3\mu=990\mathrm{T}\approx F^{\mathrm{exp}}_{2}$, and our proposal is
supported.

At temperatures above the peak dHvA oscillations are with $F_{1}$ frequency
with the amplitude being suppressed by the insulating Dingle factor. At very
low temperatures the heavy fermion $d-f$ hybrid sets in, an oscillation with
$F_{1}$ frequency vanishes, and instead oscillations with $F_{2}$ and $F_{3}$
frequencies appear. Since the heavy fermion $d-f$ hybrid is metallic, it does
not have a strong suppression due to the insulating Dingle factor, and it
results in a giant temperature peak in the temperature dependence of the
amplitude of the oscillation with frequencies $F_{2}$ and $F_{3}$. If
$F_{1}\approx F_{3}$, i.e. $F_{1}$ and $F_{3}$ agree within uncertainty, then
$F_{3}$ can be mistaken for $F_{1}$ and both be plotted on one temperature
dependence of the amplitude like the one shown in Fig. 2.

Figure 2: Temperature dependence of the amplitude of the dHvA oscillations of
a system described by Eq. 3. For the illustration we chose $B=35\mathrm{T}$,
$2\theta_{0}=35\mathrm{K}$, $\alpha=0.025$. The giant temperature peak
observed in SuchitraScience2015  in our model is explained if and only if
$F_{1}\approx F_{3}$, then $F_{2}\approx 3F_{1}$. The experiment
SuchitraScience2015  sees such $F_{1}$ and $F_{2}$.

Let us now check whether the quantum oscillations with inverse tempearture
proposed in Zyuzin2023  do exist. We look at the Fig. (4) in
SuchitraScience2015 , and associate a first bump occuring at $T^{*}\approx
8\mathrm{K}$ with the predicted in Zyuzin2023  position of the largest peak,
$T_{\mathrm{peak}}=\frac{\theta_{0}}{\pi}$, then we extract the value of the
hybridization gap to be $2\theta_{0}=2\pi T^{*}\approx 48\mathrm{K}$ which is
reasonable since the transition temperature is about $40\mathrm{K}$.
Furthermore, with that value we can estimate the flatness of the $f-$ fermions
in our models to be
$\alpha=\frac{\bar{\theta}\theta}{4\mu^{2}}=\frac{48^{2}}{16*330^{2}}\approx
0.0013$. Then, we can estimate the temperature $T_{\mathrm{g}}$ at which the
giant temperature peak starts to grow by setting
$T_{m=0}^{2}-\left(\beta^{2}-1\right)b^{2}=0$ which reads as
$T=\frac{\sqrt{\alpha}}{2\pi}\sqrt{4\bar{\theta}\theta}$, and get
$T_{\mathrm{g}}\approx 0.27\mathrm{K}$. This number is in good agreement with
the Fig. 4 in SuchitraScience2015 .

If one believes our theory indeed explains the experiment SuchitraScience2015
, then there is a number of immediate suggestions to the experiment we can
give. Firstly, SuchitraScience2015  should see a substantial increase of the
amplitude of the $F_{2}\approx 3\mu$ frequency of the oscillation below
$0.35\mathrm{K}$, i.e. at temperatures where the giant temperature peak
emerges. In SuchitraScience2015  the $F_{2}$ is already seen at temperatures
slightly higher than the peak with a rather small amplitude. One may think
this is the third harmonic of the $F_{1}=\mu$ frequency. However, there is no
clear evidence for the second harmonic, with $2\mu$ frequency, observed in
SuchitraScience2015 .

Secondly, since our theory proposes an existence of the $d-f$ heavy fermion
hybrid, which becomes visible in dHvA oscillations at very low temperatures,
we speculate that Shubnikov-de Haas effect with $F_{2}$ and $F_{3}$
frequencies should be observed at these very low temperatures.

Third claim is that SmB6 samples in experiments SuchitraScience2015  are not
insulating, but rather systems with a heavy fermion at the Fermi energy.
Although the hybridization between itinerant $d-$ electrons and localized $f-$
electrons happens at the transition temperature and the system seemingly
establishes a collective gap for the $d-$ electrons at the Fermi energy, a
possible slight electron-like dispersion of the $f-$ electrons results in a
large Fermi surface of the heavy fermion $d-f$ hybrid which is never gapped
and is thus metallic. It is quite likely that the observed saturation of the
resistivity in SmB6 at very low temperature is due to the residual resistivity
of this heavy fermion $d-f$ hybrid. If so, it rules out the proposal that SmB6
is a topological insulator DzeroSunGalitskiColeman .

Next, a region in which quantum oscillations as a function of temperature were
predicted by the author in Zyuzin2023  needs to be scanned in details to
confirm that the small heights observed in Fig. 4 of SuchitraScience2015  at
$T\approx 8\mathrm{K}$ and $T\approx 5\mathrm{K}$ are not just the deviations
within the uncertainties but rather unique physical features. We stress that
these oscillations are guaranteed if the giant low temperature peak is
observed.

Finally, condition $F_{3}\approx F_{1}$ in reality may not be precisely
satisfied. A possible difference in $F_{1}$ and $F_{3}$ frequencies should be
visible in the experiment.

Crystals of SmB6 used in the experiments SuchitraScience2015  are grown by the
floating zone technique are pristine and very clean. Crystals of SmB6 used in
the experiments LiScience2014  are grown with the aluminum fluxes. Although,
to the best of our knowledge, it is claimed that the aluminum does not
drastically change electronic structure of SmB6, it apparently does. Either it
alters the structure of $f-$ electrons, such that $\alpha$ changes sign, or it
reduces the value of $\alpha$ such that the temperatures at which the giant
peak emerges reduce to even lower ones such that it is hard to observe, or it
makes the heavy fermion $d-f$ hybrid to oscillate with $F_{2}$ way different
from $F_{1}$. We think that it is the very former case when the dispersion of
$f-$ fermions becomes hole-like, the hybridization will occur with the gap
opening, but there will be no giant temperature peak. This is partially
supported by the claims of LiScience2014  that the dHvA signal is due to the
surface states which could have been mistaken with the bulk response with the
amplitude being suppressed by the insulating Dingle factor, while having
regular Lifshits-Kosevich temperature dependence.

## References

  * (1) G. Li, Z. Xiang, F. Yu, T. Asaba, B. Lawson, P. Cai, C. Tinsman, A. Berkley, S. Wolgast, Y.S. Eo, D.-J. Kim, C. Kurdak, J.W. Allen, K. Sun, X.H. Chen, Y.Y. Wang, Z. Fisk, and L. Li, Science 346, 1208 (2014).
  * (2) B.S. Tan, Y.-T. Hsu, B. Zeng, M.C. Hatnean, N. Harrison, Z. Zhu, M. Hartstein, M. Kiourlappou, A. Srivastava, M.D. Johannes, T.P. Murphy, J.-H. Park, L. Balicas, G.G. Lonzarich, G. Balakrishnan, and S.E. Sebastian, Science 349, 287 (2015).
  * (3) J. Knolle and N.R. Cooper, Phys. Rev. Lett. 115, 146401 (2015).
  * (4) L. Zhang, X.-Y. Song, and F. Wang, Phys. Rev. Lett. 116, 046404 (2016)
  * (5) Z.Z. Alisultanov JETP Letters 104, 187 (2016).
  * (6) H.K. Pal, F. Piechon, J.-N. Fuchs, M. Goerbig, and G. Montambaux, Phys. Rev. B 94, 125140 (2016).
  * (7) G. Singh and H. Pal, arXiv: 2210.10475 (2022).
  * (8) V.A. Zyuzin, arXiv:2302.13923 (2023).
  * (9) M. Dzero, K. Sun, V. Galitski, and P. Coleman, Phys. Rev. Lett. 104, 106408 (2010).


# LuSEE ’Night’: The Lunar Surface Electromagnetics Experiment

Stuart D. Baleref1ref2  Neil Bassettref3  Jack O. Burnsref3  Johnny Dorigo
Jonesref3  Keith Goetzref4  Christian Hellum-Byeref5  Sven Herrmannref6
Joshua Hibbardref3  Milan Maksimovicref7  Ryan McLeanref2  Raul
Monsalveref2ref8ref9  Paul O’Connorref6  Aaron Parsonsref5  Marc Puluparef2
Rugved Pundref10  David Rapettiref3ref11ref12  Kaja M. Rotermundref13  Ben
Saliwanchikref6  Anže Slosarref14  David Sundkvistref2  and Aritoki
Suzukiref13  ref1Physics Department, University of California, Berkeley, USA;
e-mail<EMAIL_ADDRESS>ref2Space Sciences Laboratory, University of
California, Berkeley, USA ref3Center for Astrophysics & Space Astronomy,
University of Colorado, Boulder, CO, USA ref4School of Physics and Astronomy,
University of Minnesota, Minneapolis, USA ref5Department of Astronomy,
University of California, Berkeley, USA ref6Instrumentation Division,
Brookhaven National Laboratory, Upton, NY, USA ref7LESIA, Observatoire de
Paris, Université PSL, CNRS, Sorbonne Université, Université de Paris, Meudon,
France ref8Facultad de Ingeniería, Universidad Católica de la Santísima
Concepción, Concepción, Chile ref9School of Earth and Space Exploration,
Arizona State University, Tempe, USA ref10Department of Physics and Astronomy,
Stony Brook University, Stony Brook, NY, USA ref11NASA Ames Research Center,
Moffett Field, CA, USA ref12Research Institute for Advanced Computer Science,
Universities Space Research Association, Washington, DC, USA ref13Lawrence
Berkeley National Laboratory, Berkeley, CA, USA ref14Physics Department,
Brookhaven National Laboratory, Upton NY, USA

###### Abstract

The Lunar Surface Electromagnetics Explorer ’LuSEE Night’ is a low frequency
radio astronomy experiment that will be delivered to the farside of the Moon
by the NASA Commercial Lunar Payload Services (CLPS) program in late 2025 or
early 2026. The payload system is being developed jointly by NASA and the US
Department of Energy (DOE) and consists of a 4 channel, 50 MHz Nyquist
baseband receiver system and 2 orthogonal $\sim$6m tip-to-tip electric dipole
antennas. LuSEE Night will enjoy standalone operations through the lunar
night, without the electromagnetic interference (EMI) of an operating lander
system and antipodal to our noisy home planet.

##  1 Science Motivation

The radio sky below 20 MHz is not well-studied, primarily due to the
$\mathcal{O}$(10 MHz) plasma frequency of the F-layer peak of the terrestrial
ionosphere [1], below which the ionosphere becomes opaque, but also due to the
presence of intense shortwave radio transmissions [2] and military over-the-
horizon (A-band) radar.

At these frequencies, the ambient sky is dominated by the synchrotron spectrum
of cosmic ray electrons spiraling in the galactic magnetic field [3, 4, 5].
This galactic synchrotron spectrum is brighter in the galactic plane above a
few MHz [6, 5] but becomes attenuated in the plane by free-free absorption at
lower frequencies [7, 8]. Indeed, recent measurements by the Parker Solar
Probe/FIELDS instrument [9, 10] suggest that modifications to the state-of-
the-art low frequency sky models [8] are required in the form of small-scale
angular structure, hemispheric asymmetries, and modifications to the galactic
electron density map [11, 5].

The brightest discrete radio sources below 50 MHz are likely to be the Sun
[12] and Jupiter [13]; however there are other known astrophysical sources Cas
A, Cyg A [14, 15] present and plenty of ’discovery space’. Nonthermal solar
and Jovian radio sources will be dynamic and in some cases strongly polarized.
The known positions of the Sun, Jupiter and other sources will allow
occultation studies as they set below the lunar horizon, potentially allowing
studies of thermal emission from the extended solar corona.

Characterizing the global spectrum of this low frequency foreground is a first
step in a measurement strategy to access the high redshift ($z\sim$ 100) 21 cm
signature associated with the so-called ’Dark Ages’ [16, 17, 18]. The Dark
Ages signal arose as the universe recombined and the cosmic microwave
background (CMB) interacted with neutral hydrogen via the 21 cm spin-flip
transition. In principle, this absorption feature should be dominated by the
thermal physics of the early universe - any striking deviations may represent
new physics. However, the Dark Ages spectral feature is hidden more than 5
orders of magnitude below the galactic foreground such that any spectral
structure (real and/or system chromaticity) in the foreground measurement will
make recovery of the cosmological signal very challenging. The LuSEE Night
measurements will be an important step in this longer-term program.

##  2 LuSEE Night System

The LuSEE payload was selected by NASA in 2019 under the Lunar Surface
Instrument and Technology Payloads (LSITP) program, which is itself in support
of the Commercial Lunar Payload Services (CLPS) program. As proposed, LuSEE
was largely heritage instrumentation from the ’FIELDS’ experiment [9] on the
NASA Parker Solar Probe mission, comprised of magnetometers, electric field,
and low frequency radio and plasma waves measurements. In early 2020, NASA
entered into a partnership with the DOE to promote LuSEE to a low frequency
radio pathfinder and the LuSEE Night payload was separated from the original
payload (now called ’LuSEE Lite’ and destined for the Schrödinger Basin near
the lunar South Pole, in early 2025).

LuSEE Night is designed to make low frequency ($\leq$ 50 MHz) full-Stokes,
spectral density measurements of the radio sky in the pristine radio
environment of the lunar farside. To that end, LuSEE Night is required to
operate continuously through the lunar night ($\sim$14 Earth days) with the
Sun below the lunar horizon. Furthermore, LuSEE Night requires that the CLPS
lunar lander spacecraft cease all operations before the first nightfall and
remain powered off for the duration of the mission. This gives LuSEE Night
full control of the electromagnetic environment and eliminates the risk of
spacecraft-generated EMI (which bedeviled the Chang’e 4 radio astronomy
experiment [19]).

Figure 1: A CAD drawing of the LuSEE Night system showing the dipole antennas
on the carousel, solar arrays (blue) and the large battery (gold box below).
The dipole antennas are $\sim$6m tip-to-tip, the upper deck is approximately
1m x 1m and the system is $\sim$70 cm tall, but will be mounted atop the
lander.

Figure 1 is a CAD drawing of the LuSEE Night payload system showing the dipole
antennas fully deployed. LuSEE Night uses 3m BeCu stacer antenna elements [20]
configured in orthogonal, co-linear $\sim$6m tip-to-tip dipole pairs; a stacer
is a cold-rolled, helical pitch spring that deploys by its own stored spring
energy and forms a rigid tube of roughly 1 cm diameter. The LuSEE Night
antennas are mounted on a motor-driven carousel so that the antenna
orientation can be rotated in the plane of the lunar surface. This allows for
a degree of freedom that can be used to understand the electrical coupling of
the antenna system to the lunar lander structure and to the dielectric
regolith below [21]. The antenna carousel is only operable during the lunar
day, when LuSEE Night solar arrays are illuminated and the system is power-
positive.

The LuSEE Night structure occupies a cube (1m x 1m by 70cm high). The top
surface will support the four electric monopoles on a rotating platform as
well as horizontal solar cells. In addition the East and West sides of the
cube will also include some vertical solar cells to improve the charging at
dawn and dusk. The LuSEE electronics package will be located within the cube
and will be thermally isolated such that self-heating from the nighttime
operations power will keep the electronics payload warm during the night. One
side of the cube will include south-facing radiator panels equipped with
Parabolic Reflector Radiators (PRR ) developed at JPL [22] to help reject day-
time heating.

An instrument block diagram is shown in Figure 2. LuSEE Night will make a
voltage measurement using a high impedance, low-noise JFET front end on each
monopole antenna. While a voltage measurement provides less gain than a
balanced system, it can be more robust against impedance mismatch and internal
reflections, which are difficult to correct after launch and landing.

Figure 2: A block diagram of the LuSEE Night payload showing the instrument
components - spectrometer, antennas, preamps, C&DH computer, and power
systems. The LuSEE Night computer also operates the S-band communications
system and motor drive to reorient the antenna carousel during daytime.

A spectrometer samples the four single-ended antenna voltages at 102.4
Msamples/sec and uses an FPGA to process the waveforms into auto- and cross-
correlation spectra. Stokes parameters can be computed using only cross-
correlation products if desired, avoiding the antenna shotnoise present in the
autocorrelations. The spectrometer design is based on the Radio Frequency
Spectrometer (RFS) system [10] from the Parker Solar Probe/FIELDS instrument
[9]. A DC-DC power converter system will operate at well-defined, stable
(crystal-controlled) switching frequencies, which allow digital signal
processing to mask power supply noise. This technique was used very
successfully on Parker Solar Probe. LuSEE Night requires a large ($\sim$40 kg)
battery to provide power through the lunar night. At sunrise, the system
becomes power-positive allowing the battery to recharge, the S-band
communication system to send science data back to Earth, and the antenna
carousel to be operated.

The LuSEE Night ’Data Controller Board’ (DCB) is a computer that interfaces to
the other subsystems, including the spectrometer, the S-band communication
system (provided by Vulcan Wireless), and the lander’s Command and Data
Handling (C&DH) system. The DCB also controls the motor drive for the antenna
carousel.

At the date of this writing (late January 2023), NASA is selecting a CLPS
vendor to provide the LuSEE Night launch and landing system. This mission is
designated ’CS-3’ and will also carry the European Space Agency (ESA) Lunar
Pathfinder (LPF) communications relay satellite into lunar orbit. LuSEE Night
will assist in commissioning of the LPF satellite and will use LPF as its
S-band uplink/downlink asset.

##  3 Far-field Calibration Source

NASA has released a Request For Proposals to the CLPS vendors to provide a
far-field calibration source (hereafter FFCS) for LuSEE Night (designated as
the ’CS-4’ mission). As envisioned, the CS-4 FFCS will be a payload
accommodated on another CLPS lunar orbiter or perhaps a dedicated system (e.g.
a CubeSat) that carries an instrument that transmits a known pseudo-random
waveform that is repeated contiguously with a flux density that corresponds to
10-22 to 10-18 W/m2/Hz (to $\sim$1% knowledge) over 10 seconds at the LuSEE
Night site on the lunar surface. This signal corresponds to a frequency comb
in the LuSEE Night band (to 51.2 MHz) and the LuSEE Night spectrometer will
correlate against this signal as CS-4 passes overhead horizon-to-horizon. The
CS-4 FFCS will be required to provide at least 30 passes over LuSEE Night and
to operate for 50 Earth days and then cease operation. This FFCS will provide
LuSEE Night a known signal over a range of altitudes and azimuths that will be
used to calibrate the antenna pattern and system voltage response and
chromaticity.

##  4 Landing Site

The LuSEE Night landing site is a 100m ellipse centered on (23.813∘S,
182.258∘E) on the lunar farside near the anti-meridian, shown in Figure 3.
This site was chosen to be near to the antipode and to therefore minimize
terrestrial radio frequency interference [2], recognizing that at lower
frequencies some noise will be refracted by the tenuous lunar ionosphere [23].
The landing site was also chosen to have a relatively flat horizon. The
aforementioned spatial variations of the galactic power spectrum (brighter
plane at high frequencies, brighter poles below 3 MHz) may induce spectral
chromaticity as the horizon occults galactic structure and the relatively flat
horizon minimizes uncertainties due to this effect. Finally, the landing site
was chosen to have well-mixed regolith to avoid asymmetric dielectric
properties below the LuSEE Night antenna system. The requirement that LuSEE
Night operate through the lunar night requires an integrated thermal design,
as described above. To optimize the thermal management, the LuSEE Night
radiator requires a view of the dark sky, therefore the requirement to land
either North or South of the lunar equator.

Figure 3: The LuSEE Night landing site (within green circle) on the lunar
farside at (23.813∘S, 182.258∘E). This site was chosen to minimize terrestrial
RFI, provide relatively uniform sky coverage, optimize payload thermal design,
and provide favorable downlink with the relay satellite. The landing site of
the Chang’e 4 mission is shown in the red circle and yellow vertical line is
the anti-meridian.

##  5 Summary

The LuSEE Night payload is in development and funded by NASA and the DOE. The
system will be delivered and integrated to a lander vehicle for a late 2025 -
early 2026 launch and landing on the lunar farside. LuSEE Night will make
unique measurements of the global radio sky below 50 MHz, in the absence of
terrestrial and lander EMI, and be a pathfinder for larger, more ambitious
lunar radio astronomy projects.

## Acknowledgements

The LuSEE Night program is funded under NASA Contract 80MSFC22CA018 and DOE
Office of Science Contract DE-SC0012704. We acknowledge the superb technical
support of Dr Ahmed Fadl of the NASA Goddard Space Flight Center.

## References

  * [1] Bilitza, D., Pezzopane, M., Truhlik, V., et al. 2022, Reviews of Geophysics, 60, e2022RG000792. doi:10.1029/2022RG000792
  * [2] Kaiser, M. L., Desch, M. D., Bougeret, J.-L., et al. 1996, Geophys. Res. Lett., 23, 1287. doi:10.1029/95GL03665
  * [3] Novaco, J. C. & Brown, L. W. 1978, Astrophys. J., 221, 114. doi:10.1086/156009
  * [4] de Oliveira-Costa, A., Tegmark, M., Gaensler, B. M., et al. 2008, MNRAS, 388, 247. doi:10.1111/j.1365-2966.2008.13376.x
  * [5] Page, B., Bassett, N., Lecacheux, A., et al. 2022, Astron. Astrophys., 668, A127. doi:10.1051/0004-6361/202244621
  * [6] Manning, R. & Dulk, G. A. 2001, Astron. Astrophys, 372, 663. doi:10.1051/0004-6361:20010516
  * [7] Peterson, J. D. & Webber, W. R. 2002, Astrophys. J., 575, 217. doi:10.1086/341258
  * [8] Cong, Y., Yue, B., Xu, Y., et al. 2021, Astrophys. J., 914, 128. doi:10.3847/1538-4357/abf55c
  * [9] Bale, S. D., Goetz, K., Harvey, P. R., et al. 2016, Space Science Rev., 204, 49. doi:10.1007/s11214-016-0244-5
  * [10] Pulupa, M., Bale, S. D., Bonnell, J. W., et al. 2017, Journal of Geophysical Research (Space Physics), 122, 2836. doi:10.1002/2016JA023345
  * [11] Bassett, N., Rapetti, D., Nhan, B., et al. 2023, submitted to Astrophys. J., https://arxiv.org/abs/2301.09612
  * [12] Sasikumar Raja, K., Maksimovic, M., Kontar, E. P., et al. 2022, Astrophys. J., 924, 58. doi:10.3847/1538-4357/ac34ed10.48550/arXiv.2110.10935
  * [13] Panchenko, M., Rucker, H. O., & Farrell, W. M. 2013, Planetary Space Sci., 77, 3. doi:10.1016/j.pss.2012.08.015
  * [14] Helmboldt, J. F. & Kassim, N. E. 2009, Astronom. J., 138, 838. doi:10.1088/0004-6256/138/3/83810.48550/arXiv.0903.5010
  * [15] de Gasperin, F., Vink, J., McKean, J. P., et al. 2020, Astron. Astrophys, 635, A150. doi:10.1051/0004-6361/20193684410.48550/arXiv.2002.10431
  * [16] Loeb, A. & Zaldarriaga, M. 2004, Phys. Rev. Lett., 92, 211301. doi:10.1103/PhysRevLett.92.21130110.48550/arXiv.astro-ph/0312134
  * [17] Furlanetto, S. R., Oh, S. P., & Briggs, F. H. 2006, Phys. Reports, 433, 181. doi:10.1016/j.physrep.2006.08.00210.48550/arXiv.astro-ph/0608032
  * [18] Burns, J. O., MacDowall, R., Bale, S., et al. 2021, Planetary Sci. J., 2, 44. doi:10.3847/PSJ/abdfc310.48550/arXiv.2102.02331
  * [19] Su, Y., Li, C., Li, J., et al. 2019, EPSC-DPS Joint Meeting 2019
  * [20] Bale, S. D., Ullrich, R., Goetz, K., et al. 2008, Space Science Rev., 136, 529. doi:10.1007/s11214-007-9251-x
  * [21] Grimm, R. E. 2018, Icarus, 314, 389. doi:10.1016/j.icarus.2018.06.007
  * [22] Bugby, D. C., Rivera, J. G., and Britton, S. R., 2022, 85, Cryocoolers 22.
  * [23] Bassett, N., Rapetti, D., Burns, J. O., et al. 2020, Advances in Space Research, 66, 1265. doi:10.1016/j.asr.2020.05.050


# Vision-LSTM: xLSTM as Generic Vision Backbone

Benedikt Alkin$~{}^{1,2}$ Maximilian Beck$~{}^{1,2}$ Korbinian
Pöppel$~{}^{1,2}$
Sepp Hochreiter$~{}^{1,2,3}$ Johannes Brandstetter$~{}^{1,2,3}$

$~{}^{1}$ ELLIS Unit Linz, Institute for Machine Learning, JKU Linz, Austria
$~{}^{2}$ NXAI Lab, Linz, Austria
$~{}^{3}$ NXAI GmbH, Linz, Austria
<EMAIL_ADDRESS>
###### Abstract

Transformers are widely used as generic backbones in computer vision, despite
initially introduced for natural language processing. Recently, the Long
Short-Term Memory (LSTM) has been extended to a scalable and performant
architecture – the xLSTM – which overcomes long-standing LSTM limitations via
exponential gating and parallelizable matrix memory structure. In this report,
we introduce Vision-LSTM (ViL), an adaption of the xLSTM building blocks to
computer vision. ViL comprises a stack of xLSTM blocks where odd blocks
process the sequence of patch tokens from top to bottom while even blocks go
from bottom to top. Experiments show that ViL holds promise to be further
deployed as new generic backbone for computer vision architectures.
Project page: https://nx-ai.github.io/vision-lstm/

##  1 Introduction

Language modeling architectures — such as Transformers [56, 1, 50] or more
recently State Space Models [24, 25] such as Mamba [23] — are commonly adapted
to the domain of computer vision to make use of their powerful modeling
capabilities. However, in natural language processing, an input sentence is
typically encoded into tokens that represent words or common subwords [8] via
a discrete vocabulary. To encode images into a set of tokens, Vision
Transformer [18] (ViT) proposed to group an input image into non-overlapping
patches (of e.g. 16x16 pixel), linearly project them into a sequence of so-
called patch tokens and add positional information to these tokens. This
sequence can then be processed by language modeling architectures.

The Extended Long Short-Term Memory (xLSTM) family [5] was recently introduced
as a new architecture for language modeling. It demonstrates the resurgence of
LSTM in the LLM era, performing favorably against the likes of Transformers
and State Space Models (SSMs). Analogous to existing vision versions of
Transformers or State Space Models, e.g., ViT [18] or Vision Mamba [67], which
have produced great results in various computer vision tasks [48, 37, 44, 46,
3], we introduce Vision LSTM (ViL) – a generic computer vision backbone that
uses xLSTM blocks as its core components. To adjust xLSTM (an autoregressive
model) to computer vision (an often non-autoregressive domain), we employ a
stack of alternating mLSTM blocks [5] where odd blocks process patches row-
wise from top left to bottom right and even blocks go from bottom right to top
left. This simple alternating design allows ViL to efficiently process non-
sequential inputs, such as images, without introducing additional
computations.

Similar to vision adaptions of SSMs [40, 67, 58], ViL can exhibit linear
computational and memory complexity w.r.t. sequence length which makes it
appealing for tasks that benefit from high-resolution images such as medical
imaging [9, 26, 55, 62], segmentation [37, 10], or physics simulations [6, 43,
7, 2]. In contrast, ViT’s computational complexity scales quadratically due to
the self-attention mechanism, rendering them costly to apply to high-
resolution tasks.

##  2 Method

Vision-LSTM (ViL) is a generic backbone for computer vision tasks, which is
residually built from xLSTM blocks, as visualized in Figure 1. Following ViT
[18], ViL first splits an image into non-overlapping patches via a shared
linear projection, then adds learnable positional embeddings to each patch
token. At the core of ViL are alternating mLSTM blocks, which are fully
parallelizable and equipped with a matrix memory combined with a covariance
update rule. Odd mLSTM blocks process patch tokens from top left to bottom
right while even blocks go from bottom right to top left.

Figure 1:  Schematic overview of Vision-LSTM (ViL). Following ViT [18], an
input image is split into patches and linearly projected. Then, a learnable
vector is added per position to the patches, producing a sequence of patch
tokens. This sequence is then processed by alternating mLSTM blocks where even
blocks flip the sequence before and after the mLSTM layer. For classification,
ViL uses the concatenation of the first and the last patch as input to a
linear classification head.

##  3 Experiments

We pre-train models on ImageNet-1K [17], which contains 1.3M training images
and 50K validation images where each image belongs to one of 1000 classes. ViL
models are trained for 800 epochs (tiny) or 400 epochs (small, base) on
192x192 resolution with a learning rate of 1e-3 using a cosine decay schedule.
Afterwards, the model is fine-tuned on 224x224 resolution for 20 epochs using
a learning rate of 1e-5. Detailed hyperparameters can be found in Appendix
Table 12.

As ViL is an isotropic architecture (like ViT [18]), we compare it against
other isotropic architectures and leave exploration of hierarchical ViL
architectures to future work.

Figure 2:  Performance overview of ImageNet-1K pre-trained models in relation
to pre-training compute. ViL shows strong performances across classification
and semantic segmentation tasks.

As ViTs are well established in the vision community, they underwent multiple
optimization cycles over the years [18, 51, 53, 52, 54, 27]. Since this work
is the first to apply xLSTM to computer vision, we do not expected to
outperform years of hyperparameter tuning of ViTs in all cases. Even so,
Figure 2 shows an overview of performance metrics in relation to total pre-
training compute where ViL performs favorably against heavily optimized
transformer protocols (DeiT, DeiT-III) and Vision Mamba (Vim) [67] on
ImageNet-1K [17] classification, ADE20K [66] semantic segmentation and VTAB-1K
[64] classification. Detailed results are presented in the following sections.

###  3.1 ImageNet-1K Pre-training

Table 1 relates parameter counts and FLOPS to classification accuracy after
pre-training on ImageNet-1K. ViL outperforms heavily optimized ViT protocols
and other backbones on the tiny and small scale. While ViL does not outperform
all other models on the base scale, evaluations on downstream tasks (as shown
later in Table 3 and Table 2) show that ViL-B still learns strong features.
Additionally, training a ViL-B for 400 epochs takes roughly 600 A100 GPU-hours
or 19 hours on 32 A100 GPUs. Therefore, the hyperparameter configuration is
most likely not optimal as we were not able to extensively optimize
hyperparameters for this scale. Note that training ViL is still quite
expensive to train as there does not exist an optimized hardware
implementation yet, which does exist for other models like ViT [15, 14] or Vim
[23, 67]. Nevertheless, the efficient alternating block design of ViL makes it
already faster than Vim (up to 69% as shown in Appendix B.1) while only using
torch.compile, a generic speed optimization of the PyTorch [45] framework.

Model | Epochs | #Params | FLOPS | IN-1K
---|---|---|---|---
DeiT-T [51] | 300 | 6M | 1.3G | 72.2
DeiT-II-T [53] | 400 | 6M | 1.3G | 73.5
DeiT-III-T (reimpl.) | 800+20 | 6M | 1.3G | 76.2
VRWKV-T [19] | 300 | 6M | 1.2G | 75.1
Vim-T [67] | 300 | 7M | 1.5G | 76.1
Mamba®-T [58] | 280+20 | 9M | - | 77.4
ViL-T | 800+20 | 6M | 1.5G | 78.3
DeiT-S [51] | 300 | 22M | 4.6G | 79.8
DeiT-II-S [53] | 400 | 22M | 4.6G | 80.7
DeiT-III-S [54] | 800+20 | 22M | 4.6G | 81.4
ConvNeXt-S [41] (iso.) | 300 | 22M | 4.3G | 79.7
VRWKV-S [19] | 300 | 24M | 4.6G | 80.1
Vim-S [67] | 300 | 26M | 5.3G | 80.5
Mamba®-S [58] | 280+20 | 28M | - | 81.1
ViL-S | 400+20 | 23M | 5.1G | 81.5
DeiT-B [51] | 300 | 86M | 17.6G | 81.8
DeiT-II-B [53] | 400 | 86M | 17.6G | 82.7
DeiT-III-B [54] | 800+20 | 86M | 17.6G | 83.7
ConvNeXt-B [41] (iso.) | 300 | 87M | 16.9G | 82.0
VRWKV-B [19] | 300 | 94M | 18.2G | 82.0
Mamba®-B [58] | 280+20 | 99M | - | 82.9
ViL-B | 400+5 | 89M | 18.6G | 82.4
Table 1: ImageNet-1K pre-training accuracy. All models use a patch size of
16x16 with 224x224 resolution at most. Models with “+” in their “Epochs”
column pre-train on lower resolution followed by fine-tuning on 224x224
resolution for some epochs. ViL performs favorably against against an
isotropic convolutional architecture (ConvNeXt) and vision adaptions of
transformers (DeiT), RWKV [47] (VRWKV) and Mamba [23] (Vim, Mamba®).

###  3.2 ADE20K Semantic Segmentation

Table 2 shows results for transferring ImageNet-1K pre-trained models to
ADE20K [66] semantic segmentation using UperNet [61]. Also here, ViL shows
strong performances across the board, even outperforming DeiT-III-B despite
the lower ImageNet-1K accuracy of ViL-B.

For this benchmark, we train on 512x512 images and inference could be
conducted on the original images (without resizing), which results in even
larger images. Consequently, the number of patch tokens can become quite
large. As sequence length increases, ViL would ideally use a recurrent or
chunkwise formulation of the mLSTM layer. Shortly summarized, the mLSTM can
operate in parallel, recurrent or chunked modes, each with distinct FLOPS and
runtime characteristics. Given a sequence length $L$ and hidden dimension $D$,
the trade-off between modes can be formulated as $\mathcal{O}(LCD+LD^{2})$
where $C$ denotes the chunksize of the chunked form. $C=1$ (recurrent mode)
yields complexity $\mathcal{O}(LD^{2})$ and $C=L$ (parallel mode) gives
$\mathcal{O}(L^{2}D)$ or $\mathcal{O}(L\frac{L}{2}D)$ when omitting the
causally masked operations. Due to this trade-off and the current lack of an
optimized hardware implementation for the mLSTM, it is difficult to create
runtime or FLOPS estimates that are of practical relevance in this setting and
therefore leave it to future work.

|  | Single-scale | Multi-scale
---|---|---|---
Model | #Params | mIoU | ACC | mIoU | ACC
DeiT-T [51] | 10M | 38.1 | 78.2 | 40.3 | 79.9
DeiT-III-T [54] | 10M | 39.8 | 79.2 | 42.2 | 80.7
Vim-T [67] | 13M | 41.0 | - | - | -
ViL-T | 11M | 41.2 | 80.2 | 43.1 | 81.3
DeiT-S [51] | 41M | 43.1 | 80.7 | 45.2 | 81.8
DeiT-III-S [54] | 41M | 45.2 | 81.5 | 46.3 | 82.3
Vim-S [67] | 46M | 44.9 | - | - | -
Mamba®-S [58] | 56M | 45.3 | - | - | -
ViL-S | 42M | 46.3 | 82.0 | 47.9 | 82.9
DeiT-B [51] | 113M | 45.8 | 82.1 | 47.0 | 82.9
DeiT-III-B [54] | 113M | 47.5 | 82.6 | 49.0 | 83.3
Mamba®-B [58] | 132M | 47.7 | - | - | -
ViL-B | 115M | 48.6 | 82.8 | 49.6 | 83.3
Table 2:  Semantic segmentation results on ADE20K [66] using UperNet [61]. We
report mean intersection over union (mIoU) and pixelwise accuracy (ACC) for
single- and multi-scale evaluation. Models are trained for 160K updates with a
batchsize of 16 on 512x512 resolution. Detailed hyperparameters are listed in
Appendix Table 14.

###  3.3 VTAB-1K Transfer Classification

Table 3 shows transfer classification results for ImageNet-1K pre-trained
models on the VTAB-1K [64] benchmark. VTAB-1K consists of 19 datasets split
into 7 natural datasets (such as CIFAR100 [38], Caltech101 [20] or SUN397
[60]), 4 specialized datasets (medical imaging [57, 34] and remote sensing
[28, 11]) and 8 structured datasets (with tasks such as object counting [33]
or binned depth estimation [21]). We follow common practices and tune the
learning rate per model and dataset on the validation set followed by training
with the best learning rate on the union of train and validation set. The
performance metric is the average testset accuracy over 5 seeds. ViL shows
strong transfer classification performance outperforming all other models on
the average over all 19 datasets. ViL performs particularly well on the
structured datasets where ViL-B outperforms DeiT-III-B despite ViL-B having
lower ImageNet-1K accuracy.

Model | #Params | FLOPS | Natural | Specialized | Structured | Average
---|---|---|---|---|---|---
DeiT-T [51] | 6M | 1.3G | 69.2 | 82.0 | 53.3 | 65.2
DeiT-III-T (reimpl.) | 6M | 1.3G | 71.9 | 82.6 | 55.2 | 67.1
Vim-T [67] | 7M | 1.5G | 68.0 | 80.7 | 47.1 | 61.9
ViL-T | 6M | 1.5G | 73.6 | 83.4 | 56.1 | 68.3
DeiT-S [51] | 22M | 4.6G | 73.3 | 83.8 | 53.2 | 67.1
DeiT-III-S [54] | 22M | 4.6G | 75.0 | 83.2 | 52.3 | 67.2
Vim-S [67] | 26M | 5.3G | 69.6 | 81.7 | 49.4 | 63.6
ViL-S | 23M | 5.1G | 75.3 | 84.3 | 58.3 | 70.0
DeiT-B [51] | 86M | 17.6G | 76.5 | 85.2 | 55.7 | 69.6
DeiT-III-B [54] | 86M | 17.6G | 77.6 | 84.8 | 56.6 | 70.3
ViL-B | 89M | 18.6G | 76.6 | 84.7 | 59.1 | 70.9
Table 3: Transfer classification accuracies on the VTAB-1K [64] benchmark
using ImageNet-1K pre-trained models. VTAB-1K consists of 19 datasets split
into 7 natural, 4 specialized and 8 structured datasets. We show averages per
category and the average accuracy over all 19 datasets (Appendix Table 10
lists all individual accuracies). ViL shows strong transfer classification
performance, outperforming heavily optimized ViT protocols and Vim on the full
VTAB-1K benchmark. Notably, ViL performs exceptionally well on the structured
category. We tune the learning rate for each model and dataset on the
validation set and report the average performance on the test set over 5
seeds. Appendix Table 13 lists further hyperparameters.

###  3.4 Block Design

We study different ways to design ViL blocks in Table 4. The plain, uni-
directional xLSTM block fails to reach competitive performances since the
autoregressive nature of xLSTM is not suited for image classification.
Traversing blocks in a bi-directional manner, i.e., introducing a second mLSTM
layer in each block that traverses the sequence backwards (akin to Vim [67]),
improves performance, but also requires more parameters and FLOPS. Sharing the
parameters of the forward and backward mLSTM makes the model more parameter
efficient, but still requires more compute and overloads these parameters
which leads to performance drops. Using alternating blocks enhances
performance while remaining compute and parameter efficient. We also explore
quad-directional designs (similar to [40]), which refers to traversing the
sequence row-wise (in both directions) and additionally column-wise (in both
directions). Bi-directional traverses the sequence only row-wise (in both
directions). Figure 3 visualizes the different traversal paths.

Due to the increased cost of bi-directional and quad-directional blocks, this
study was conducted in a heavily reduced setting. We train on a subset of
ImageNet-1K that contains only samples from 100 classes for 400 epochs at
128x128 resolution. This was particularly necessary because our quad-
directional implementation is not compatible with torch.compile (a generic
speed optimization method from PyTorch [45]), which leads to longer runtimes,
as indicated last column of Table 4. Due to this technical limitation, we
choose the alternating bi-directional block as our core design.

Direction | Alternating | Param. Sharing | ACC | #Param | FLOPS | Runtime
---|---|---|---|---|---|---
Uni-dir. (xLSTM) | ✗ | ✗ | 72.8 | 6.1M | 1.0x | 1.0x
Bi-dir. | ✗ | ✗ | 74.8 | 6.5M | 1.4x | 1.9x
Bi-dir. | ✗ | ✓ | 73.3 | 6.1M | 1.4x | 1.9x
Bi-dir. (ViL) | ✓ | ✗ | 74.8 | 6.1M | 1.0x | 1.0x
Quad-dir. | ✗ | ✗ | 76.3 | 7.3M | 2.3x | 7.5x
Quad-dir. | ✗ | ✓ | 72.8 | 6.1M | 2.3x | 7.5x
Quad-dir. | ✓ | ✗ | 75.6 | 6.1M | 1.0x | 1.9x
Table 4:  Traversing the sequence of patch tokens in multiple directions
yields better performance than the plain, uni-directional xLSTM block.
However, using multiple directions per block also requires more compute and
parameters. Sharing parameters between directions matches #Params but
overloads them, leading to performance drops. Using multi-directional designs
but alternating between them in consecutive blocks, matches #Params and FLOPS
while achieving good performances. While quad-directional designs yields the
best performances, they comes with drastically increased runtime. As our
current quad-directional implementation is not compatible with torch.compile,
it suffers from increased runtime even when alternating between directions.
Since quad-directional designs are compute intensive, this study was conducted
by training a ViL-T on an ImageNet-1K subset of 100 classes using resolution
128. “Param. Sharing” shares the parameters of the two (bidirectional) or four
(quaddirectional) mLSTM layers in each block.

Figure 3:  Visualization of traversal paths. Bi-directional uses the first two
traversal paths. Quad-directional uses all of the above. Squares represent
individual patch tokens.

###  3.5 Classification Design

In order to perform classification with a ViT, the sequence of tokens is
pooled into a single token, which is then used as input to a classification
head. The most common methods to do this pooling are (i) adding a learnable
[CLS] token at the start of the sequence or (ii) averaging all patch tokens to
produce an [AVG] token. Whether to use the [CLS] or [AVG] token is typically a
hyperparameter, where both variants achieve comparable performances. On the
contrary, autoregressive models often require specialized classification
designs. For example, Vim [67] requires the [CLS] token to be in the middle of
the sequence, suffering heavy performance losses if other classification
designs, e.g., an [AVG] token or two [CLS] tokens at start and end of the
sequence, are employed. Due to its autoregressive nature, we explore different
classification designs for ViL in Table 5. [AVG] averages all patch tokens,
“Middle Patch” uses the middle patch token, “Middle [CLS]” uses a [CLS] token
in the middle of the sequence, “Bilateral [AVG]” uses the average of the first
and last patch token. We find ViL to be relatively robust the classification
design where all performances that use a single token are within 0.6%. We only
add a [CLS] token if it is used for classification, hence the best design
(“Bilateral Concat”) does not use a [CLS] token.

Design | Dim | ACC
---|---|---
[AVG] | 192 | 75.4
Middle Patch | 192 | 75.9
Middle [CLS] | 192 | 75.3
Bilateral [AVG] | 192 | 75.9
Bilateral Concat | 384 | 76.4
Table 5:  How to pool the sequence of tokens for classification? ViL is
relatively robust to the classification design. Using the middle patch token
and averaging the first and last token (“Bilateral [AVG]”) work similarly
well. The good performance of middle patch could be caused by the bias in
ImageNet-1K that objects are typically in the center of the image. We
therefore choose a bilateral design to avoid exploiting this bias.
Additionally, concatenating the first and last token (“Bilateral Concat”)
instead of averaging improves performance and is used as final design for ViL.
We conduct this study by training a ViL-T on ImageNet-1K for 400 epochs.

###  3.6 Vision Adjustments of xLSTM

The xLSTM [5] architecture was designed for language modeling and therefore
includes some design choices that are specific to language modeling. One of
these design choices is a causal 1D convolution (with kernel size 4) to encode
local context into the query and key input of the mLSTM layer. As images are
neither causal nor 1D structures, we replace the causal 1D convolution with a
2D convolution (with kernel size 3). When training a ViL-T for 400 epochs,
this improves ImageNet-1K accuracy from 76.1% to 76.7%.

Another language specific design choice is the absence of bias parameters in
projection and layernorm [4] layers which improves training stability for
larger models [12, 16]. As vision models are typically much smaller than
language models, ViTs commonly use all bias parameters. When adding bias
parameters, a ViL-T trained for 400 epochs improves from 76.7% to 77.1% on
ImageNet-1K.

##  4 Conclusion

Motivated by the success of xLSTM in language modeling, we introduced ViL, an
adaption of the xLSTM architecture to vision tasks. ViL processes a sequence
of patch tokens in alternating fashion. Odd blocks process image patches row-
wise from top left to bottom right and even blocks go from bottom right to top
left. Our new architecture outperforms SSM-based vision architectures and also
optimized ViT models on ImageNet-1K classification, VTAB-1K transfer
classification and ADE20K semantic segmentation. Remarkably, ViL is able to
outperform ViT training pipelines, which are the result of years of
hyperparameter tuning and transformer improvements.

In the future, we see potential in applying ViL when high-resolution images
are needed for optimal performance, such as semantic segmentation or medical
imaging. In these settings, transformers suffer from high computational costs
due to the quadratic complexity of self-attention, where ViL can use a chunked
form to trade-off between its parallel form (quadratic complexity) and
recurrent form (linear complexity). Additionally, improving pre-training
schemes (e.g. via self-supervised learning), exploring better hyperparameter
settings or investigating hierarchical architectures are promising future
directions that could improve ViL even further.

## Acknowledgments

We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at
IT4Innovations, Czech Republic, MeluXina at LuxProvide, Luxembourg, Leonardo
at CINECA, Italy and LUMI at CSC, Finland.

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are
supported by the Federal State Upper Austria. We thank the projects Medical
Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL
(FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA
(FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids (FFG-
899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847),
Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning
Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL
Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch
GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ.
Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV
Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.

## References

  * Achiam et al. [2023] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Alkin et al. [2024a] Alkin, B., Fürst, A., Schmid, S., Gruber, L., Holzleitner, M., and Brandstetter, J.  Universal physics transformers.  _arXiv preprint arXiv:2402.12365_ , 2024a.
  * Alkin et al. [2024b] Alkin, B., Miklautz, L., Hochreiter, S., and Brandstetter, J.  Mim-refiner: A contrastive learning boost from intermediate pre-trained representations.  _arXiv preprint arXiv:2402.10093_ , 2024b.
  * Ba et al. [2016] Ba, L. J., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _CoRR_ , abs/1607.06450, 2016.  URL http://arxiv.org/abs/1607.06450.
  * Beck et al. [2024] Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S.  xlstm: Extended long short-term memory, 2024.
  * Bi et al. [2023] Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q.  Accurate medium-range global weather forecasting with 3d neural networks.  _Nature_ , 619(7970):533–538, 2023.
  * Bodnar et al. [2024] Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al.  Aurora: A foundation model of the atmosphere.  _arXiv preprint arXiv:2405.13063_ , 2024.
  * Bostrom & Durrett [2020] Bostrom, K. and Durrett, G.  Byte pair encoding is suboptimal for language model pretraining.  _arXiv preprint arXiv:2004.03720_ , 2020.
  * Chen et al. [2021] Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., and Zhou, Y.  Transunet: Transformers make strong encoders for medical image segmentation.  _CoRR_ , abs/2102.04306, 2021.
  * Cheng et al. [2022] Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R.  Masked-attention mask transformer for universal image segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pp. 1280–1289. IEEE, 2022.
  * Cheng et al. [2017] Cheng, G., Han, J., and Lu, X.  Remote sensing image scene classification: Benchmark and state of the art.  _Proc. IEEE_ , 105(10):1865–1883, 2017.
  * Chowdhery et al. [2023] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N.  Palm: Scaling language modeling with pathways.  _J. Mach. Learn. Res._ , 24:240:1–240:113, 2023.
  * Clark et al. [2020] Clark, K., Luong, M., Le, Q. V., and Manning, C. D.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * Dao [2023] Dao, T.  Flashattention-2: Faster attention with better parallelism and work partitioning.  _CoRR_ , abs/2307.08691, 2023.
  * Dao et al. [2022] Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  In _NeurIPS_ , 2022.
  * Dehghani et al. [2023] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Ruiz, C. R., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste, S., Elsayed, G. F., Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M., Gritsenko, A. A., Birodkar, V., Vasconcelos, C. N., Tay, Y., Mensink, T., Kolesnikov, A., Pavetic, F., Tran, D., Kipf, T., Lucic, M., Zhai, X., Keysers, D., Harmsen, J. J., and Houlsby, N.  Scaling vision transformers to 22 billion parameters.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 7480–7512. PMLR, 2023.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA_ , pp. 248–255. IEEE Computer Society, 2009.
  * Dosovitskiy et al. [2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Duan et al. [2024] Duan, Y., Wang, W., Chen, Z., Zhu, X., Lu, L., Lu, T., Qiao, Y., Li, H., Dai, J., and Wang, W.  Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures.  _CoRR_ , abs/2403.02308, 2024.
  * Fei-Fei et al. [2006] Fei-Fei, L., Fergus, R., and Perona, P.  One-shot learning of object categories.  _IEEE transactions on pattern analysis and machine intelligence_ , 28(4):594–611, 2006.
  * Geiger et al. [2013] Geiger, A., Lenz, P., Stiller, C., and Urtasun, R.  Vision meets robotics: The KITTI dataset.  _Int. J. Robotics Res._ , 32(11):1231–1237, 2013.
  * Goyal et al. [2017] Goyal, P., Dollár, P., Girshick, R. B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K.  Accurate, large minibatch SGD: training imagenet in 1 hour, 2017.
  * Gu & Dao [2023] Gu, A. and Dao, T.  Mamba: Linear-time sequence modeling with selective state spaces.  _CoRR_ , abs/2312.00752, 2023.
  * Gu et al. [2021] Gu, A., Goel, K., and Ré, C.  Efficiently modeling long sequences with structured state spaces.  _ArXiv_ , 2111.00396, 2021.
  * Gupta et al. [2022] Gupta, A., Gu, A., and Berant, J.  Diagonal state spaces are as effective as structured state spaces.  _ArXiv_ , 2203.14343, 2022.
  * Hatamizadeh et al. [2022] Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B. A., Roth, H. R., and Xu, D.  UNETR: transformers for 3d medical image segmentation.  In _IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022_ , pp. 1748–1758. IEEE, 2022.
  * He et al. [2022] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. B.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022_ , pp. 15979–15988, 2022.
  * Helber et al. [2019] Helber, P., Bischke, B., Dengel, A., and Borth, D.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens._ , 12(7):2217–2226, 2019.
  * Hendrycks & Dietterich [2019] Hendrycks, D. and Dietterich, T. G.  Benchmarking neural network robustness to common corruptions and perturbations.  In _ICLR (Poster)_. OpenReview.net, 2019.
  * Hendrycks et al. [2021a] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _ICCV_ , 2021a.
  * Hendrycks et al. [2021b] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D.  Natural adversarial examples.  _CVPR_ , 2021b.
  * Huang et al. [2016] Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q.  Deep networks with stochastic depth.  In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV_ , volume 9908 of _Lecture Notes in Computer Science_ , pp. 646–661. Springer, 2016.
  * Johnson et al. [2017] Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. B.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pp. 1988–1997. IEEE Computer Society, 2017.
  * Kaggle & EyePacs [2015] Kaggle and EyePacs.  Kaggle diabetic retinopathy detection, July 2015.
  * Kalamkar et al. [2019] Kalamkar, D. D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E., Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., and Dubey, P.  A study of BFLOAT16 for deep learning training.  _CoRR_ , abs/1905.12322, 2019.
  * Kingma & Ba [2015] Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In Bengio, Y. and LeCun, Y. (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Kirillov et al. [2023] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W., Dollár, P., and Girshick, R. B.  Segment anything.  In _ICCV_ , pp. 3992–4003. IEEE, 2023.
  * Krizhevsky [2009] Krizhevsky, A.  Learning multiple layers of features from tiny images.  2009\.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., and Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  In Bartlett, P. L., Pereira, F. C. N., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), _Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States_ , pp. 1106–1114, 2012.
  * Liu et al. [2024] Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y.  Vmamba: Visual state space model.  _CoRR_ , abs/2401.10166, 2024.
  * Liu et al. [2022] Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., and Xie, S.  A convnet for the 2020s.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pp. 11966–11976. IEEE, 2022.
  * Loshchilov & Hutter [2019] Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * Nguyen et al. [2023] Nguyen, T., Brandstetter, J., Kapoor, A., Gupta, J. K., and Grover, A.  Climax: A foundation model for weather and climate.  _arXiv preprint arXiv:2301.10343_ , 2023.
  * Oquab et al. [2023] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M. G., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P.  Dinov2: Learning robust visual features without supervision.  _CoRR_ , abs/2304.07193, 2023.
  * Paszke et al. [2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E. Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , pp. 8024–8035, 2019.
  * Peebles & Xie [2023] Peebles, W. and Xie, S.  Scalable diffusion models with transformers.  In _IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023_ , pp. 4172–4182. IEEE, 2023.
  * Peng et al. [2023] Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., Du, X., Grella, M., GV, K. K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang, X., Wind, J. S., Wozniak, S., Zhang, Z., Zhou, Q., Zhu, J., and Zhu, R.  RWKV: reinventing rnns for the transformer era.  In _EMNLP (Findings)_ , pp. 14048–14077. Association for Computational Linguistics, 2023.
  * Singh et al. [2023] Singh, M., Duval, Q., Alwala, K. V., Fan, H., Aggarwal, V., Adcock, A., Joulin, A., Dollár, P., Feichtenhofer, C., Girshick, R. B., Girdhar, R., and Misra, I.  The effectiveness of MAE pre-pretraining for billion-scale pretraining.  _CoRR_ , abs/2303.13496, 2023.
  * Szegedy et al. [2016] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z.  Rethinking the inception architecture for computer vision.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pp. 2818–2826. IEEE Computer Society, 2016.
  * Team et al. [2023] Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Touvron et al. [2021a] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 10347–10357. PMLR, 2021a.
  * Touvron et al. [2021b] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and Jégou, H.  Going deeper with image transformers.  In _ICCV_ , pp. 32–42. IEEE, 2021b.
  * Touvron et al. [2022a] Touvron, H., Cord, M., El-Nouby, A., Verbeek, J., and Jégou, H.  Three things everyone should know about vision transformers.  In _ECCV (24)_ , volume 13684 of _Lecture Notes in Computer Science_ , pp. 497–515. Springer, 2022a.
  * Touvron et al. [2022b] Touvron, H., Cord, M., and Jégou, H.  Deit III: revenge of the vit.  In _ECCV (24)_ , volume 13684 of _Lecture Notes in Computer Science_ , pp. 516–533. Springer, 2022b.
  * Valanarasu et al. [2021] Valanarasu, J. M. J., Oza, P., Hacihaliloglu, I., and Patel, V. M.  Medical transformer: Gated axial-attention for medical image segmentation.  In de Bruijne, M., Cattin, P. C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., and Essert, C. (eds.), _Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 - 24th International Conference, Strasbourg, France, September 27 - October 1, 2021, Proceedings, Part I_ , volume 12901 of _Lecture Notes in Computer Science_ , pp. 36–46. Springer, 2021.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Veeling et al. [2018] Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M.  Rotation equivariant cnns for digital pathology.  In Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C., and Fichtinger, G. (eds.), _Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II_ , volume 11071 of _Lecture Notes in Computer Science_ , pp. 210–218. Springer, 2018.
  * Wang et al. [2024] Wang, F., Wang, J., Ren, S., Wei, G., Mei, J., Shao, W., Zhou, Y., Yuille, A., and Xie, C.  Mamba-r: Vision mamba also needs registers.  _arXiv preprint arXiv:2405.14858_ , 2024.
  * Wang et al. [2019] Wang, H., Ge, S., Lipton, Z., and Xing, E. P.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems_ , pp. 10506–10518, 2019.
  * Xiao et al. [2010] Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A.  SUN database: Large-scale scene recognition from abbey to zoo.  In _The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010_ , pp. 3485–3492. IEEE Computer Society, 2010.
  * Xiao et al. [2018] Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J.  Unified perceptual parsing for scene understanding.  In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V_ , volume 11209 of _Lecture Notes in Computer Science_ , pp. 432–448. Springer, 2018.
  * Xu et al. [2024] Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al.  A whole-slide foundation model for digital pathology from real-world data.  _Nature_ , pp. 1–8, 2024.
  * Yun et al. [2019] Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., and Choe, J.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pp. 6022–6031. IEEE, 2019.
  * Zhai et al. [2019] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  _arXiv preprint arXiv:1910.04867_ , 2019.
  * Zhang et al. [2018] Zhang, H., Cissé, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond empirical risk minimization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.
  * Zhou et al. [2019] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A.  Semantic understanding of scenes through the ADE20K dataset.  _Int. J. Comput. Vis._ , 127(3):302–321, 2019.
  * Zhu et al. [2024] Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X.  Vision mamba: Efficient visual representation learning with bidirectional state space model.  _CoRR_ , abs/2401.09417, 2024.

##  Appendix A Changelog

###  A.1 Version 1

In the first version of this paper, we trained models with a causal 1D
convolution in the mLSTM layer (instead of a 2D convolution), “Bilateral
[AVG]” pooling (instead of “Bilateral Concat”) and a constant resolution of
224 (instead of pre-training at 192 followed by a short fine-tuning on
resolution 224). This version of the architecture is also used to conduct the
studies in Table 4 and Table 5. For completeness, we report the accuracies of
these models in Table 6. In the first version, we trained slightly deeper
models (ViL-T+ with 29 blocks and ViL-S+ with 26 blocks) to match the
parameter count of Vim-T and Vim-S. However, as the models with a regular
depth of 24 were already better than Vim with less parameters, we omit them in
future versions.

Method | #Params | ACC
---|---|---
ViL-T | 6M | 77.3
ViL-T+ | 7M | 78.1
ViL-S | 23M | 80.7
ViL-S+ | 26M | 80.9
ViL-B | 88M | 81.6

Model | #Params | ACC
---|---|---
Long-sequence fine-tuning
ViL-T+ | 7M | 80.0
ViL-S+ | 26M | 82.2
ViL-B | 88M | 82.7

Table 6:  ImageNet-1K accuracies of models trained in the first version of
this paper.

###  A.2 Version 2

The current version of the paper improves performance by (i) replacing the
causal 1D convolution with a 2D convolution (with kernel size 3 and padding 1)
(ii) using “Bilateral Concat” instead of “Bilateral [AVG]” pooling (iii) using
bias parameters in projection and LayerNorm [4] layers (iv) pre-training
models on 192x192 followed by a short fine-tuning on 224x224 resolution.

##  Appendix B Extended Results

###  B.1 Runtime Comparison of ViL vs Vim

We compare the time it takes to train ViL and Vim [67] for a single
ImageNet-1K epoch in Table 7. We follow the scaling procedure of ViTs, using
192 (T), 384 (S), 768 (B), 1024 (L) as latent dimension where the large scale
additionally doubles the number of blocks.

Model | Optimization | (T)iny | (S)mall | (B)ase | (L)arge
---|---|---|---|---|---
Vim [67] | custom CUDA kernel | 7.3h | 14.0h | 28.2h | 76.4h
ViL | torch.compile | 5.0h | 8.7h | 16.6h | 45.1h
Speedup of ViL compared to Vim | 45% | 61% | 69% | 69%
Table 7:  Runtime comparisons between Vim [67] and ViL. ViL is up to 69%
faster despite the current lack of a custom CUDA kernel. As mLSTM (and ViL)
can be parallelized analogous to FlashAttention [15, 14] via custom hardware
optimizations, ViL will become even faster in the future. Runtimes denote the
training time for 10 ImageNet-1K epochs and are extrapolated from short
benchmark runs on a single A100-80GB-PCIe using float16 precision and 224x224
images.

###  B.2 Impact of Longer Training

We investigate the impact of training for a longer duration in Table 8. We
also attempted to train larger models for longer but did not observe similar
performance gains. One could probably change that by tuning hyperparameters
for an 800 epoch training schedule, which we omit due to the large
computational costs associated with training larger models.

Model | Epochs | IN-1K ACC | VTAB-1K | ADE20K mIoU
---|---|---|---|---
DeiT-III-T | 400 | 75.6 | 67.0 | 39.1
DeiT-III-T | 800 | 76.2 | 67.1 | 39.8
ViL-T | 400 | 77.2 | 67.8 | 40.9
ViL-T | 800 | 78.3 | 68.3 | 41.2
Table 8: Performance comparison of tiny models when training for 400 and 800
epochs. ADE20K mIoU uses single-scale evaluation. All settings follow the ones
used in the main paper.

###  B.3 Long-sequence fine-tuning

Table 9 shows results for the long-sequence fine-tuning setting of Vim [67].

Method | #Params | ACC
---|---|---
DeiT-III-T (reimpl.) | 6M | 78.3
Vim-T [67] | 7M | 78.3
ViL-T | 6M | 80.1
DeiT-S [51] | 22M | 81.4
DeiT-III-S [54] | 22M | 83.2
Vim-S [67] | 26M | 81.6
ViL-S | 23M | 82.7
DeiT-B [51] | 86M | 83.2
DeiT-III-B [54] | 86M | 84.5
ViL-B | 89M | 83.3
Table 9: Fine-tuning ImageNet-1K pre-trained models by using a 50% overlap
when creating patches. This setting increases the number of patch tokens from
196 to 729. ViL easily outperforms Vim but falls short of ViTs on larger
models. Note that ViL FLOPS are calculated using the parallel form of the
mLSTM, which scales quadratically with the sequence length. Using the
chunkwise form instead of the parallel form can reduce FLOPS for longer
sequences (see Section C.2).

###  B.4 VTAB-1K Individual Dataset Results

Table 10 presents accuracies for each individual dataset of the VTAB-1K [64]
benchmark.

| Natural | Specialized | Structured
---|---|---|---
|  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele
DeiT-T [51] | 47.7 | 86.4 | 63.7 | 85.6 | 87.0 | 78.4 | 35.3 | 83.0 | 93.4 | 80.9 | 70.7 | 71.7 | 60.3 | 43.1 | 78.5 | 67.9 | 41.6 | 30.6 | 32.7
DeiT-III-T (reimpl.) | 52.3 | 90.1 | 62.7 | 88.8 | 87.5 | 83.7 | 37.9 | 83.2 | 93.1 | 81.1 | 72.9 | 76.6 | 60.8 | 44.9 | 79.1 | 67.5 | 48.1 | 31.0 | 33.3
Vim-T [67] | 46.7 | 86.3 | 60.7 | 84.0 | 88.8 | 76.1 | 33.7 | 82.2 | 92.9 | 75.2 | 72.6 | 59.8 | 49.9 | 39.3 | 78.2 | 51.2 | 43.9 | 26.9 | 27.2
ViL-T | 54.2 | 90.2 | 67.4 | 90.7 | 89.9 | 81.6 | 41.1 | 83.4 | 94.2 | 82.7 | 73.1 | 80.7 | 61.8 | 49.4 | 81.3 | 57.8 | 51.8 | 31.4 | 34.8
DeiT-S [51] | 57.0 | 88.9 | 68.2 | 90.9 | 90.8 | 75.4 | 42.1 | 83.3 | 94.0 | 83.8 | 74.0 | 74.6 | 58.3 | 45.6 | 78.2 | 61.9 | 47.9 | 27.1 | 31.9
DeiT-III-S [54] | 58.8 | 88.6 | 67.5 | 90.9 | 91.7 | 84.4 | 43.3 | 84.4 | 92.6 | 82.5 | 73.5 | 76.5 | 57.9 | 46.2 | 78.9 | 58.3 | 49.7 | 23.7 | 27.5
Vim-S [67] | 53.0 | 87.2 | 64.1 | 86.8 | 90.3 | 65.8 | 39.7 | 82.4 | 93.4 | 78.0 | 73.1 | 63.1 | 53.2 | 42.3 | 78.2 | 54.1 | 47.6 | 27.1 | 29.3
ViL-S | 61.4 | 89.6 | 69.2 | 92.8 | 91.7 | 78.7 | 43.8 | 85.5 | 93.9 | 84.4 | 73.5 | 84.0 | 63.4 | 51.3 | 83.3 | 61.0 | 55.4 | 32.4 | 35.5
DeiT-B [51] | 61.8 | 89.8 | 67.5 | 93.7 | 92.6 | 84.4 | 45.6 | 85.3 | 95.1 | 86.3 | 74.2 | 77.7 | 59.9 | 47.2 | 81.7 | 61.7 | 51.4 | 30.0 | 36.2
DeiT-III-B [54] | 62.9 | 89.6 | 69.6 | 93.7 | 93.2 | 87.0 | 47.1 | 85.8 | 94.1 | 85.6 | 73.7 | 80.5 | 61.4 | 48.4 | 80.9 | 64.4 | 55.1 | 30.2 | 31.8
ViL-B | 64.3 | 90.0 | 71.1 | 93.4 | 91.4 | 79.6 | 46.6 | 84.7 | 94.3 | 85.4 | 74.4 | 83.7 | 62.1 | 52.7 | 81.0 | 63.1 | 57.6 | 32.6 | 39.9

Table 10: Results on all datasets of the VTAB-1K [64] benchmark.

###  B.5 Robustness

Table 11 presents robustness and OOD evaluations of ImageNet-1K pre-trained
classifiers.

Model | IN-C ($\downarrow$) | IN-A ($\uparrow$) | IN-R ($\uparrow$) | Sketch ($\uparrow$) | Validation ($\uparrow$)
---|---|---|---|---|---
DeiT-T [51] | 69.7 | 7.6 | 32.7 | 19.9 | 72.2
DeiT-III-T | 65.0 | 11.7 | 39.4 | 27.4 | 76.2
Vim-T [67] | 61.8 | 9.6 | 38.8 | 26.9 | 76.1
ViL-T | 59.6 | 15.2 | 42.2 | 30.0 | 78.3
DeiT-S [51] | 54.4 | 19.6 | 41.9 | 29.1 | 79.8
DeiT-III-S [54] | 50.1 | 23.2 | 46.6 | 35.4 | 81.4
Vim-S [67] | 51.5 | 19.7 | 44.8 | 32.5 | 80.5
ViL-S | 50.6 | 23.8 | 47.9 | 35.2 | 81.5
DeiT-B [51] | 48.6 | 27.9 | 44.6 | 32.0 | 81.8
DeiT-III-B [54] | 42.7 | 36.5 | 54.1 | 41.1 | 83.8
ViL-B | 45.3 | 30.9 | 51.9 | 39.0 | 82.4
Table 11: Robustness and OOD evaluations on ImageNet-C(orruption) [29],
ImageNet-A(dversarial) [31], ImageNet-R(endition) [30] and ImageNet-Sketch
[59].. For ImageNet-C, we report the mean corruption error [29] with AlexNet
[39] as baseline.

##  Appendix C Implementation Details

###  C.1 Hardware

We train models on a mix of custom hardware servers (mainly A100 and A40 GPUs)
and public research clusters equiped with either 8xA100 or 4xA100 nodes.

We estimate the total number of A100 GPU-hours used for this project to be 38K
hours. This estimate includes initial exploration, method development,
analysis and evaluations.

###  C.2 FLOPS Calculation

We calculate the FLOPS with the fvcore
library111https://github.com/facebookresearch/fvcore. For ViL, we report the
FLOPS of the mLSTM parallel form. Given sequence length $L$ and hidden
dimension $D$, it has complexity $\mathcal{O}(L\frac{L}{2}D)$ where the
$\frac{L}{2}$ stems from the causal masking.

We report FLOPS for a complexity of $\mathcal{O}(L\frac{L}{2}D)$ even though
our current implementation of the mLSTM has complexity $\mathcal{O}(L^{2}D)$
(i.e., entries of the $QK$ matrix that are later masked are still calculated)
which we justify by the fact that FlashAttention-2 [14] is approximately 1.7x
faster with a causal mask than without. Therefore, an optimized hardware
implementation of the mLSTM could also omit the calculation of values that are
not needed.

As Vim [67] does not report FLOPS and their model makes use of CUDA kernels
(which are not counted as FLOPS by fvcore), we replace all calls to CUDA
kernels with their reference PyTorch implementation and count the FLOPS with
fvcore.

For the total pre-training compute in Figure 2, we consider an efficient
implementation of stochastic depth [32] which omits the calculation of a
dropped block instead of masking it. Therefore, we change the implementation
of ViT [18] to use our efficient stochastic depth implementation. Vim does not
use stochastic depth for training as their models are relatively small.

###  C.3 ViL Hyperparameters

Table 12 shows detailed hyperparameters used to train ViL models.

Parameter | Value
---|---
Epochs | 800 (T), 400 (S/B) $\rightarrow$ 20 (T, S), 5 (B)
Batch size | 2048 $\rightarrow$ 1024
Model |
Patch size | 16x16
Latent dimension | 192 (T), 384 (S), 768 (B)
Depth | 24
Pooling | Bilateral Concat
Stochastic depth [32] |
Peak rate | 0 (T), 0.05 (S), 0.2 (B)
Layer-wise Decay | ✗
Optimizer | AdamW [42, 36]
Base Learning rate | 1e-3 $\rightarrow$ 1e-5
Linear LR Scaling Divisor [22] | 1024
Weight decay | 0.05
Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$
Gradient Norm Clip | 1.0
Precision | mixed bfloat16 [35]
Backend | torch.autocast
Learning rate schedule | cosine decay
Warmup schedule | linear
Warmup epochs | 5 $\rightarrow$ 5 (T, S), 1 (B)
End LR | 1e-6
Label smoothing [49] | ✗
Train Data Augmentation |
RandomResizedCrop | 192 $\rightarrow$ 224
Scale | [0.08, 1.0]
Interpolation | bicubic
RandomHorizontalFlip | $p=0.5$
3-Augment [54] |
Gaussian Blur $\sigma$ | [0.1, 2.0]
ColorJitter | [0.3, 0.3, 0.3, 0.0]
Normalize | ImageNet-1K statistics
Mixup [65] $\alpha$ | 0.8
Cutmix [63] $\alpha$ | 1.0
Test Data Augmentation |
Resize | 192 $\rightarrow$ 224
Interpolation | bicubic
CenterCrop | 192 $\rightarrow$ 224
Normalize | ImageNet-1K statistics
Table 12:  Hyperparameters for training ViL on ImageNet-1K, inspired by DeiT-
III [54]. We follow the best setting from DeiT-III [54] and pre-train on 192
resolution followed by a short fine-tuning on 224 resolution (indicated by
$\rightarrow$).

###  C.4 Fine-tuning on VTAB-1K

For fine-tuning models on VTAB-1K we provide the hyperparameters in Table 13.
We search for the best learning rate for each dataset by fine-tuning the model
25 times (5 learning rates with 5 seeds each) on the 800 training samples and
evaluating them on the 200 validation samples. With the best learning rate, we
then train each model 5 times on concatenation of training and validation
split, evaluate on the test split and report the average accuracy.

Parameter | Value
---|---
Epochs | 50
Batch size | 64
Seeds | 5
Optimizer | AdamW [42, 36]
Learning rate | [1e-3, 7.5e-4, 5.0e-4, 2.5e-4, 1.0e-4]
Layer-wise lr decay [13] | 0.65*
Weight decay | 0.05
Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$
Learning rate schedule | linear warmup $\rightarrow$ cosine decay
Warmup epochs | 5
Precision | mixed bfloat16 [35]
Backend | torch.autocast
Data Augmentation |
Resize |
interpolation | bicubic
size | 224x224
Normalize | ImageNet-1K statistics
Table 13:  Hyperparameters for fine-tuning on VTAB-1K. *For Vim and ViL we
group two consecutive blocks for the layer-wise lr decay similar to how ViT
considers a pair of attention and MLP block as a single “layer” for the decay.

###  C.5 ADE20K Semantic Segmentation Fine-tuning

We fine-tune models on ADE20K [66] using an UperNet [61] head. We follow
common practices and fine-tune on 512x512 resolution, where we interpolate the
absolute positional embedding from 224x224 to 512x512. For ViTs, we add
relative position biases to the attention layers (initialized to 0) [27].
Table 14 lists detailed hyperparameters.

Parameter | Value
---|---
Updates | 160K
Batch size | 16
UperNet |
Auxiliary |
Weight | 0.4
Input Block | 8*
Dimension | 192 (T), 384 (S, B)
Decoder |
Weight | 1.0
Input Blocks | [4, 6, 8, 12]*
Dimension | 192 (T), 384 (S, B)
Stochastic depth [32] |
Peak rate | 0 (T), 0.05 (S), 0.1 (B)
Layer-wise Decay | ✓
Optimizer | AdamW [42, 36]
Learning rate | 5e-4
Linear LR Scaling Divisor [22] | 16
Layer-wise lr decay [13] | 0.65*
Weight decay | 0.05
Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$
Learning rate schedule | linear warmup $\rightarrow$ cosine decay
Warmup updates | 1500
Precision | mixed float16
Backend | torch.autocast
Train Data Augmentation |
RandomResize |
interpolation | bicubic
RandomCrop |
size | 512x512
RandomHorizontalFlip |
ColorJitter | 0.5
brightness | 0.5
contrast | 0.5
saturation | 0.5
hue | 0.25
Normalize | ImageNet-1K statistics
Evaluation |
Stride | 341
Multi-scale |
scale factors | [0.75, 1.0, 1.25, 1.5, 1.75]
flip | [True, False]
Table 14:  Hyperparameters for fine-tuning on VTAB-1K. *For ViL we group two
consecutive blocks into one similar to how a ViT block consists of a pair of
attention and MLP block.

###  C.6 DeiT-III Reimplementation Hyperparameters

Table 12 shows detailed hyperparameters used to train DeiT-III-T (reimpl.)
from Table 1.

Parameter | Value
---|---
Epochs | 800 $\rightarrow$ 20
Batch size | 2048 $\rightarrow$ 1024
Model |
Patch size | 16x16
Latent dimension | 192
Depth | 12
Pooling | [CLS]
Stochastic depth [32] | ✗
Layerscale [52] | 1e-4
Optimizer | AdamW [42, 36]
Base Learning rate | 1e-3 $\rightarrow$ 1e-5
Linear LR Scaling Divisor [22] | 1024
Weight decay | 0.05
Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$
Gradient Norm Clip | ✗
Precision | mixed bfloat16 [35]
Backend | torch.autocast
Learning rate schedule | cosine decay
Warmup schedule | linear
Warmup epochs | 5
End LR | 1e-6
Label smoothing [49] | ✗
Train Data Augmentation |
RandomResizedCrop | 192 $\rightarrow$ 224
Scale | [0.08, 1.0]
Interpolation | bicubic
RandomHorizontalFlip | $p=0.5$
3-Augment [54] |
Gaussian Blur $\sigma$ | [0.1, 2.0]
ColorJitter | [0.3, 0.3, 0.3, 0.0]
Normalize | ImageNet-1K statistics
Mixup [65] $\alpha$ | 0.8
Cutmix [63] $\alpha$ | 1.0
Test Data Augmentation |
Resize | 192 $\rightarrow$ 224
Interpolation | bicubic
CenterCrop | 192 $\rightarrow$ 224
Normalize | ImageNet-1K statistics
Table 15:  Hyperparameters for training our reimplementation of DeiT-III-T
[54] on ImageNet-1K. The most significant change is that we reduce the
learning rate from 3e-3 to 1e-3 as we found this to greatly improve
performance. We make minor changes to the protocol such as using AdamW or no
gradient clipping as models were stable without it. We follow the best setting
from DeiT-III [54] and pre-train on 192 resolution followed by a short fine-
tuning on 224 resolution (indicated by $\rightarrow$).