#  Noise-induced stabilization of dynamical states in a non-Markovian system

T. F. Q. Larson  Department of Physics, Duke University, Durham, NC 27708,
USA.  L. Zhao  Department of Physics, Duke University, Durham, NC 27708, USA.
E. G. Arnault  Department of Physics, Duke University, Durham, NC 27708, USA.
M. T. Wei  Department of Physics, Duke University, Durham, NC 27708, USA.  A.
Seredinski  Department of Physics, Duke University, Durham, NC 27708, USA.  H.
Li  Department of Physics and Astronomy, Appalachian State University, Boone,
NC 28607, USA.  K. Watanabe  T. Taniguchi  Advanced Materials Laboratory,
National Institute for Materials Science, 1-1 Namiki, Tsukuba, 305-0044,
Japan.  F. Amet  Department of Physics and Astronomy, Appalachian State
University, Boone, NC 28607, USA.  G. Finkelstein  Department of Physics, Duke
University, Durham, NC 27708, USA.

###### Abstract

A Josephson junction subject to an external RF excitation may experience the
inverse AC Josephson effect, where the phase across the junction locks to the
drive. The resulting “Shapiro steps” of quantized voltage are currently
utilized in primary voltage standards. The effect has also been a key
technique in the search for topological superconductivity. Here, we study a
particularly interesting regime in which the zero voltage state becomes
unstable at zero DC bias, and the junction spontaneously develops the first
Shapiro step with a voltage $V=\pm\hbar\omega/2e$. The switching time between
the $+\hbar\omega/2e$ and $-\hbar\omega/2e$ states ranges from milliseconds to
hours. Most interestingly, we find a surprising regime with a non-monotonic
temperature dependence, in which the switching time demonstrates a pronounced
minimum at intermediate temperatures. Tuning of the measurement parameters is
also shown to give rise to a tristable switching behavior, showcasing the
potential for further experiments in stochastic physics and quantum
thermodynamics.

Electronic micro and nanoscale systems have recently arisen as a platform for
studies of stochastic effects and quantum thermodynamics Hänggi and Marchesoni
(2009); Pekola (2015). Such systems are often highly tunable, providing high
quality data across a wide range of parameters and system bandwidths, and
allow for the study of both classical and quantum fluctuations. Several recent
works have utilized quantum dots and single electron transistors tuned between
charge states in order to achieve the desired bistability Koski  _et al._
(2014); Wagner  _et al._ (2019) while other works have focused on
superconducting quantum circuits Masuyama  _et al._ (2018); Naghiloo  _et al._
(2018). Here, we investigate switching in an RF-driven Josephson junction,
where bistability is achieved between different phase space trajectories,
corresponding to states which are _dynamic_ in nature.

Josephson junctions subject to an RF radiation demonstrate an inverse AC
Josephson effect: the phase difference across a junction locks to the external
frequency Josephson (1964). As a result, the phase steadily ramps with time,
changing by integer $n$ periods per RF cycle. The $I-V$ curves of the junction
show “Shapiro steps” of quantized voltage, $V=\frac{n\hbar\omega}{2e}$ Shapiro
(1963). The extremely precise voltage quantization of the steps is presently
utilized in primary voltage standards Hamilton (2000). The exact mechanisms
and stability of the phase locking were investigated in detail in the 1980’s,
with an emphasis on stability of closed trajectories in the phase space Kautz
(1996).

In this work, we study stochastic switching between these phase locked
dynamical states in a gate-tunable graphene Josephson junction. Stochastic
switching between Shapiro steps has recently been observed in this system,
with switching times exceeding the time scale of the dynamics by many orders
of magnitude Kalantre  _et al._ (2020). A slight change of the RF amplitude
changes the switching time over an exponential range, from milliseconds to
hours. The tunability and degree of control enabled by the Josephson junctions
make them particularly appealing to study the switching dynamics in a driven-
dissipative system.

Figure 1: a) A map of differential resistance vs RF power and DC bias current.
Dark regions correspond to the quantized steps, while the transitions between
the steps appear as bright lines. b) The transition between the +1 and -1
Shapiro steps, extracted by numerically differentiating an average of 200 DC
sweeps. In this region, stochastic switching between the two quanized voltages
has between observed at 0 DC bias. In the following, we focus on a small range
of applied power corresponding to the region between the dashed white lines.
c) A sample of the output voltage measured at 2.9 dBm over a small interval of
time. The average time between switching events $\tau_{0}$ was extracted from
these and similar traces, and care was taken to operate in the regime where
the distribution of measured voltages remained bimodal. d) The probability
distribution of switching times, taken from the full two minute trace
corresponding to (c).

We have previously showed an unusual pattern of Shapiro steps in graphene
Josephson junctions Larson  _et al._ (2020), rather different from the
conventional Bessel function behavior Tinkham (1996). Notably, at some
intermediate RF power both $n=+1$ and $n=-1$ are stable solutions at zero
current, while $n=0$ is not, resulting in “zero-crossing steps” in the $I-V$
curves Kautz (1996); Kalantre  _et al._ (2020); Larson  _et al._ (2020).

In the tilted washboard picture commonly used to describe Josephson junctions,
zero current corresponds to zero average tilt of the the washboard. The $n=\pm
1$ states correspond to the “phase particle” ratcheting one period per RF
cycle either forward or backward along the washboard, therefore representing a
peculiar example of a spontaneously broken time-reversal symmetry in a driven-
dissipative system. Over the course of many drive cycles, rare fluctuations
knock the phase particle so that it switches the direction of motion. We find
that the temperature dependence of this switching lifetime shows a striking
non-monotonicity and a reduction of the quantized voltage, which necessitates
non-trivial phase space dynamics at long time scales.

The Josephson junctions studied in this paper are made from graphene
encapsulated in hexagonal Boron Nitride. The superconducting contacts are made
from MoRe alloy ($T_{C}\approx 10K$). Such junctions are gate tunable and
mediate supercurrent via Andreev bound states Tinkham (1996). However we
believe the physics presented in this work does not depend on the microscopic
properties of the sample and could be realized in other types of Josephson
junctions. The sample is measured in a dilution refrigerator with a microwave
antenna placed close to the sample for AC biasing. For all figures except for
Figure 4, the applied frequency is 5 GHz.

In Figure 1a, we reproduce the map of differential resistance vs bias and RF
power Larson  _et al._ (2020). The dark regions correspond to Shapiro steps,
where $dV/dI$ vanishes. The map is measured by sweeping bias $I_{DC}$ from
negative to positive. The hysteresis in switching between the plateaus results
in the pronounced top-bottom asymmetry of the map, noticeable as arching of
the transition boundaries. Sweeping the current bias in the opposite direction
would flip this map about the horizontal axis.

Figure 1b shows a zoom-in map of the region in which the system switches
between the $n=-1$ and $+1$ steps. To obtain these maps, multiple $I-V$ curves
are measured by sweeping the bias $I_{DC}$; these curves are then averaged and
numerically differentiated. Therefore, the bright band of high $dV/dI$ in the
middle of the map corresponds to the average value of the switching current.
Furthermore, the cross-section of the band represents the histogram of the
switching currents for the positive sweep direction. As in Figure 1a, sweeping
the current bias in the opposite direction would flip this map vertically.

For applied RF powers in the middle of Figure 1b ( 4.2 dBm), the $n=\pm 1$
states at $I_{DC}=0$ could persist for hours. The lifetimes are much shorter
at the edges of the region, allowing us to observe multiple transitions
between the quantized steps while holding all control parameters constant. The
primary region studied in this paper is just above the first bifurcation
point, $P_{RF}\approx 2.6$ dBm, at which the $n=0$ state first becomes
unstable and $n=\pm 1$ states are observed. The studied range is marked by the
dashed white lines in Figure 1b.

In Figure 1c, we plot a portion of a typical time trace measured at zero bias
and $P_{RF}=2.9$ dB. The voltage stochastically switches between the two
plateaus with an average time of $\tau_{0}\equiv\langle\tau\rangle\approx 10$
ms. The probability for a step to last a given time $\rho(\tau)$, as observed
over the full time trace, is plotted in Figure 1d, demonstrating a clear
exponential dependence with a slope of $\log(\rho)=1/\langle\tau\rangle$ as
expected for uncorrelated random processes.

Figure 2:  a) Switching times from $n=-1$ to $n=+1$ ($\tau_{+}$) and in the
reverse direction ($\tau_{-}$) vs current bias. The rates are exponentially
sensitive to bias, and $\alpha$, the slope of $\log(\tau_{\pm}/\tau_{0})$ vs.
$I$ is further plotted in panel (c). At zero bias, the two times are equal to
each other and correspond to $\tau_{0}$ of Figure 1d. b) Dependence of
$\tau_{0}$ on temperature at several values of $P_{RF}$ in the range marked in
Figure 1b. While the expected steady decrease of lifetime with temperature is
observed at higher power, $\tau_{0}(T)$ is non-monotonic for lower
temperatures. It is important that the raw voltage traces show two well
defined voltage states for all temperatures and powers shown. c) $\alpha$
extracted from data as in panel (a) and plotted vs $T$ for several applied
powers. While at low temperatures the values of $\alpha$ are similar across
applied powers and show relatively small temperature dependence, elevated
temperatures show a rapid increase of $\alpha$, particularly at lower powers.
d) Temperature dependence of $\tau_{-}$ at several values of bias $I$. Note
that in the low temperature regime, where $\tau_{0}$ decreases with
temperature, the $\tau_{-}(T)$ curves are parallel indicating a roughly
constant $\alpha$. The curves start to diverge at higher temperatures, where
$\tau_{0}$ increases with temperature and $\alpha$ is no longer a constant.

Despite the fact that the two states of the system are dynamical in nature, it
remains possible to consider this as a double well problem in an effective
“quasipotential” Kautz (1996); Chan and Stambaugh (2007); Ricci  _et al._
(2017). This double well potential can be tuned with DC bias which can be
observed in Figure 2a. At zero bias, the quasipotential double well is very
close to symmetric, with the lifetime of each state lasting an equal length of
time. By applying a small bias, on the order of 1 nA, it is possible to break
the symmetry, shifting the imbalance in favor of one of the two states. Both
times follow a clear exponential dependence
$\log(\tau_{\pm}/\tau_{0})=\pm\alpha I$, consistent with bias linearly
increasing the activation gap. As a result, the product
$\tau_{+}\times\tau_{-}$ remains nearly constant as a function of bias.

While all the features are so far consistent with a standard double well
potential problem, the temperature dependence of $\tau$ is surprisingly not.
This is shown in Figure 2b for temperatures from 100 mK to 400 mK and for a
range of powers identified in Figure 1b. Considering first the power range
$P_{RF}>3.1$ dBm, $\tau_{0}$ behaves as might be expected: it increases upon
decreasing $T$ and moving away from the bifurcation point, qualitatively
consistent with thermal activation. However, for lower powers, $P_{RF}<3.1$
dBm, $\tau_{0}$ develops a peculiar non-monotonic behavior, where it
_increases_ at higher temperatures.

We have excluded possible trivial explanations for the observed effect. First,
the measured voltages traces remain distinctly bimodal with two well-defined
voltage states throughout the full temperature range. Second, just before the
second bifurcation point (located at $P=5.1$ dBm), the $\tau_{0}(T)$ curves
(supplemental material) show a qualitatively similar behavior to Figure 2a.
Namely the non-monotonicity moves to lower temperatures the closer we approach
the bifurcation point, which here corresponds to _higher applied RF powers._
We believe this excludes the possibility that non-monotonicity of
$\tau_{0}(T)$ arises from a change of the system parameters with temperature.
For example, a gradual reduction of $I_{C}$ would cause the non-monotonicity
to arise at lower powers close both bifurcation points. We return back to the
surprisingly non-monotonic $\tau_{0}(T)$ curves later in the text.

We next explore the combined effect of bias current and temperature on
$\tau_{0}$. In Figure 2c, we plot the dependence of the slope
$\alpha=\frac{d\ln\tau_{+}}{dI}$ on temperature at several values of $P_{RF}$.
While at low temperatures $\alpha$ saturates at roughly the same level for
different $P_{RF}$, at higher temperatures $\alpha$ rapidly increases. This
result is similarly unexpected, as the picture of a particle in a bistable
potential tilted by current would predict an activation behavior of $\alpha$.
We note that for a given RF power, $\alpha$ eventually begins to _increase_ ,
which happens at temperatures just below the onset of non-monotonic behavior
of $\tau_{0}$.

The interplay between $\tau(T)$ and $\alpha(T)$ is best seen in Figure 2d,
which plots $\tau_{+}$ vs. $T$ at $P_{RF}=2.9$ dBm for 5 values of bias. The
middle curve corresponds to zero bias, at which $\tau_{+}=\tau_{0}$. At low
$T$, where $\tau_{0}$ decreases with $T$, all curves are roughly parallel to
each other (on a log scale), indicating a constant $\alpha$. However, at high
temperatures they start to diverge, which indicates that $\alpha$ is no longer
saturated; this occurs just before $\tau_{0}$ starts to grow with $T$.

It has been shown through both analytical studies and numerical simulations
Kautz (1996), that the transition rate between the two distinct steady states
in the traditional RCSJ model is expected to follow an activation temperature
dependence, with the activation energy $E_{A}$ determined by the the most
probable path in the phase space connecting the two states. Clearly, the non-
monotonic $\tau_{0}(T)$ dependence cannot be explained by such activation.

Inspection of the measured voltage traces in the non-monotonic regime reveals
that while the voltage still switches between positive and negative steps,
their values have become significantly reduced from
$\pm\frac{\hbar\omega}{2e}$. One such trace, as well as a low temperature
trace, are presented in Figure 3a for comparison. Here, the temperatures are
selected such that the average switching times are comparable. Figure 3b shows
the corresponding histogram of the measured voltages, which clearly
demonstrates that at high temperature the average voltage corresponding the
$n=\pm 1$ states is reduced.

The map in Figure 3c is made of the voltage histograms similar to panel (b)
taken at different temperatures. The resulting “pitchfork” feature is non-
trivial. As the thermal noise is increased, the system should switch faster.
Eventually, when the switching rate exceeds the measurement bandwidth, an
averaged voltage should be observed Singh  _et al._ (2016).

In our case this would correspond to the appearance of a broad peak centered
at zero, which would grow while the peaks at $n=\pm 1$ shrink. Instead, we
observe that the two peaks are shifting towards each other, before eventually
merging. We argue below that the $n=\pm 1$ states still survive, but the
system rapidly spikes from e.g. $-1$ to $+1$, before returning back to $-1$.
If these spikes are too fast to be experimentally resolved, the average
voltage of both states would appear to be reduced.

To illustrate this behavior, in Figure 4a we show a 600 second time trace
(blue) measured on a second device under a 5.2 GHz RF drive. The data
demonstrates very rapid switching at a rate of at least $1/\tau\geq 200$ Hz.
Surprisingly, by numerically smoothing the signal to remove the high frequency
component, we recover bistable behavior with a very slow switching between two
well-defined but non-quantized states (red curve). Figure 4b plots a 30 ms
segment of the raw trace right around one of the switching points of the
averaged signal. It is clear that there are two regions with different
averaged behavior: on the left, the system mostly stays around $-1$, while on
the right if mostly stays close to $+1$. In both cases, there are many spikes
reaching to the opposite state, which rapidly return back. This indicates that
while the smoothed data in panel (a) appears to be Markovian, the originally
rapidly switching data is certainly not. Instead, the system has memory: the
switching probability depends on how long the system spent in a given state.

We believe this memory effect arises because of additional poles in our
measurement circuit. Physically, the capacitance of the cryogenic RC filters
and cabling is charged by the DC voltage produced via the inverse AC Josephson
effect. When the system attempts to switch between dynamical states, e.g. from
$-1$ to $+1$, the charged capacitor discharges through the junction, biasing
it back to the original $-1$ state.

Figure 3:  a) Time traces measured at $P_{RF}=2.9$ dBm and temperatures of 110
mK (blue) and 380 mK (orange). The 380 mK trace clearly shows the loss of
quantization. b) The probability distribution of measured voltages
corresponding to these traces. c) Voltage distributions at multiple
temperatures represented as a map. High probability region (yellow) shows two
peaks at $\pm\frac{hf}{2e}$ that smoothly merge into one peak at $V=0$.
Figure 4:  a,b) Blue line: trace of the measured voltage as a function of time
on a large scale (a) and zoomed in (b). The trace in (b) shows many very short
jumps between the two steps, which appear quite different from the bistable
behavior e.g. in Figure 1d. The red line in panel (a) shows the same trace
with higher frequencies numerically filtered out, which suppresses the spikes
and restores a clear bistable behavior. c) The histogram of the trace in (a).
While the data is mostly bimodal, there is also a significant weight in the
region between the peaks.

At low temperature, when the escape rate is low, this current will typically
decay before the system switches back to the original state. But at higher
temperatures, the system has a higher probability to return from $+1$ back to
$-1$ while the capacitor has not yet discharged. The additional RC stage
therefore stabilizes both $\pm 1$ states, leading to the highly correlated
switching (spikes) observed in Figure 4b. This effect may be viewed as a form
of noise enhanced metastability Spagnolo  _et al._ (2004), although the
specific model, and the enhancement in a dynamical bistability are both novel.

To quantitatively verify this picture, we have performed simple simulation
which qualitatively reproduces the observed behaviors, including the
correlated spikes of Figure 4 and the original non-monotonic temperature
dependence of Figure 2 (supplementary). One important question which remains
open is the conceptual importance of the $n=0$ state in explaining the non-
monotonic $\tau_{0}(T)$. Numerical simulations of the full differential
equation typically show no direct transitions between the $+1$ and $-1$
states; instead the system transitions by passing through $n=0$.
Experimentally, whenever the $0$ state is visible within our time resolution,
the system tends to switch between $-1$ and $+1$ via that state
(supplementary). Therefore, the triple well potential is both expected in our
system and makes it easier to account for the nonmonotonicity of
$\tau_{0}(T)$. However, the possibility remains that the memory effects could
cause non-monotonic temperature behavior in a double well even without the
metastable $n=0$ state.

To summarize, we have studied switching in an AC driven Josephson junction,
which demonstrates surprisingly complex dynamical behavior. We focus on the
“zero-crossing steps”, in which the zero voltage state is not stable at zero
bias, and the system spontaneously develops a quantized voltage of
$\pm\hbar\omega/2e$. The switching time between these states vary from msec to
ksec. We found unexpected non-monotonic temperature dependence of the
switching time, which reaches minimum value at some intermediate temperature.
We attribute this behavior to the non-Markovian effects induced by the
measurement system, which retains memory of the previous state of the system
for a long time. These effect are combined with the complex structure of the
phase space, in particular with the presence the metastable zero-voltage
state. While we understand the overall features of the data, the nature of the
most probable escape path, both in the Markovian and non-Markovian limits, is
still an open question for this system.

The type of samples as studied here provide a flexible and highly tunable
platform to probe the unexplored aspects of a quantum driven-dissipative
dynamics. While the electron temperature in our system is likely too high for
the phase particle to behave quantum mechanically De Cecco  _et al._ (2016),
it may be possible to engineer a device which undergoes such a dynamical
bistability in the macroscopic quantum tunneling regime Clarke  _et al._
(1988), opening doors to many interesting experiments on driven-dissipative
quantum dynamics.

###### Acknowledgements.

Measurements of graphene samples by T.F.Q.L. and E.G.A, and data analysis by
T.F.Q.L., L.Z. and G.F., were supported by Division of Materials Sciences and
Engineering, Office of Basic Energy Sciences, U.S. Department of Energy, under
Award No. DE-SC0002765. Lithographic fabrication and characterization of the
samples performed by M.T.W. and A.S. were supported by the NSF Award
DMR-2004870. H. Li, and F.A. acknowledge the ARO under Award W911NF-16-1-0132.
K.W. and T.T. acknowledge support from JSPS KAKENHI Grant Number JP15K21722
and the Elemental Strategy Initiative conducted by the MEXT, Japan. T.T.
acknowledges support from JSPS Grant-in-Aid for Scientific Research A (No.
26248061) and JSPS Innovative Areas “Nano Informatics” (No. 25106006). This
work was performed in part at the Duke University Shared Materials
Instrumentation Facility (SMIF), a member of the North Carolina Research
Triangle Nanotechnology Network (RTNN), which is supported by the National
Science Foundation (Grant ECCS-1542015) as part of the National Nanotechnology
Coordinated Infrastructure (NNCI).

## References

  * Hänggi and Marchesoni (2009) P. Hänggi and F. Marchesoni, Rev. Mod. Phys. 81, 387 (2009).
  * Pekola (2015) J. P. Pekola, Nature Physics 11, 118 (2015).
  * Koski  _et al._ (2014) J. V. Koski, V. F. Maisi, J. P. Pekola, and D. V. Averin, Proceedings of the National Academy of Sciences 111, 13786 (2014).
  * Wagner  _et al._ (2019) T. Wagner, P. Talkner, J. C. Bayer, E. P. Rugeramigabo, P. Hänggi, and R. J. Haug, Nature Physics 15, 330 (2019).
  * Masuyama  _et al._ (2018) Y. Masuyama, K. Funo, Y. Murashita, A. Noguchi, S. Kono, Y. Tabuchi, R. Yamazaki, M. Ueda, and Y. Nakamura, Nature Communications 9 (2018), 10.1038/s41467-018-03686-y.
  * Naghiloo  _et al._ (2018) M. Naghiloo, J. J. Alonso, A. Romito, E. Lutz, and K. W. Murch, Phys. Rev. Lett. 121, 030604 (2018).
  * Josephson (1964) B. D. Josephson, Reviews of Modern Physics 36, 216 (1964).
  * Shapiro (1963) S. Shapiro, Physical Review Letters 11, 80 (1963).
  * Hamilton (2000) C. A. Hamilton, Review of Scientific Instruments 71 (2000).
  * Kautz (1996) R. L. Kautz, Reports on Progress in Physics 59, 935 (1996).
  * Kalantre  _et al._ (2020) S. S. Kalantre, F. Yu, M. T. Wei, K. Watanabe, T. Taniguchi, M. Hernandez-Rivera, F. Amet, and J. R. Williams, Phys. Rev. Research 2, 023093 (2020).
  * Larson  _et al._ (2020) T. F. Q. Larson, L. Zhao, E. G. Arnault, M.-T. Wei, A. Seredinski, H. Li, K. Watanabe, T. Taniguchi, F. Amet, and G. Finkelstein, Nano Letters 20, 6998 (2020).
  * Tinkham (1996) M. Tinkham,  _Introduction to Superconductivity_ , 2nd ed. (Dover, 1996).
  * Chan and Stambaugh (2007) H. B. Chan and C. Stambaugh, Phys. Rev. Lett. 99, 060601 (2007).
  * Ricci  _et al._ (2017) F. Ricci, R. A. Rica, M. Spasenović, J. Gieseler, L. Rondin, L. Novotny, and R. Quidant, Nature Communications 8, 15141 (2017).
  * Singh  _et al._ (2016) S. Singh, J. T. Peltonen, I. M. Khaymovich, J. V. Koski, C. Flindt, and J. P. Pekola, Physical Review B 94, 241407 (2016).
  * Spagnolo  _et al._ (2004) B. Spagnolo, N. Agudov, and A. Dubkov, Acta Physica Polonica B 35 (2004).
  * De Cecco  _et al._ (2016) A. De Cecco, K. Le Calvez, B. Sacépé, C. B. Winkelmann, and H. Courtois, Physical Review B 93 (2016), 10.1103/PhysRevB.93.180505.
  * Clarke  _et al._ (1988) J. Clarke, A. N. Cleland, M. H. Devoret, D. Esteve, and J. M. Martinis, Science 239, 992 (1988).
  * Fulton and Dunkleberger (1974) T. A. Fulton and L. N. Dunkleberger, Physical Review B 9, 4760 (1974).

Supplementary Materials

##  S1 Measurement Techniques

In order to minimize distortions, all presented switching data in Figures 1-3
correspond to switching rates slower than 1 kHz, which is 5 times slower than
our sampling rate and 10 times slower than the cutoff of the cryogenic low
pass filters. All data is also inspected to make sure less than 5$\%$ lies
between the voltages $\pm\frac{1}{2}\frac{hf}{2e}$ corresponding to one half
of the quantized values.

Reported values of $\tau_{0}$ are measured from the $\sqrt{\tau_{+}\tau_{-}}$
(Figure 2a). This helps correct small bias imbalances, to which the system is
particularly sensitive at elevated temperatures. This procedure is only used
when both switching rates occur at measurable rates.

Figure S1: a-d) Evolution of the hysteretic boundary between $n=1$ and $n=-1$
steps measured at temperatures of 100, 250, 450 and 730 mK, respectively.
These panels show that the hysteresis decreases with temperature, as expected,
but also that the RF power width of the zero crossing regime gradually
decreases as temperature increases. This helps to rule out any trivial
mechanism for the observed non-monotonicity, as it should lead to a shorter
lifetime at high temperature rather than a longer one. e) The switching
current is extracted from (a-d) and other intermediate temperatures. Lastly,
in panel (f) represents the lifetime $\tau_{0}$ at zero DC bias and the base
temperature across the entire range of $P_{RF}$. Red dots are measured
directly, while blue dots are extracted from the switching current
distribution.

##  S2 Evolution of the $-1\rightarrow+1$ Boundary with Temperature

Figure 1b demonstrates a pronounced hysteresis depending on the bias sweep
direction, which results in the arching of the boundary between the $n=\pm 1$
states. Several copies of this map taken at different temperatures are
displayed in Figure S1a-d. As one might expect, the switching hysteresis is
suppressed at higher temperatures, eventually disappearing above 1 K. A
summary containing the extracted switching currents at different temperatures
is shown in Figure S1e.

It is also possible to plot the transition rate across the entire zero
crossing regime with minor caveats. In the middle of the zero crossing regime
(RF power between 3.5 and 5 dB), the very long lifetime makes it difficult to
measure the switching directly. Instead, we can perform measurements of the
switching current distribution while ramping the DC bias current. This
distribution can be used to infer $\Gamma(I)$ for $I\lesssim I_{C}$Fulton and
Dunkleberger (1974). Given that $\Gamma(I)$ follows an exponential fit in this
range, it is possible to extrapolate to $\Gamma(0)$. This procedure should be
treated with care, owing to the fact that we are using data of $\Gamma(I)$ for
$I\approx I_{C}$ to extrapolate down to $I=0$. However, we find that it
generates a reasonable overlap when matched with the directly measured
switching time (see the RF power around 5dB in Figure S2f).

It is worth noting that some care must be taken when relating $\tau$,
extracted from direct measurements of the bistable regime, and $\Gamma$, which
is extracted by measuring the switching current distribution. Attempting to
stitch together data from the two different methods using
$\gamma=\frac{1}{\tau}$ produces a significant disagreement with the
methodologies, presumably due to the fact that, generically, $\tau\neq
1/(\int_{0}^{\infty}\frac{1}{\tau^{\prime}}P(\tau^{\prime})d\tau^{\prime})$.
To allow for the different datasets to stitch properly, we directly average
$\frac{1}{\tau}$.

##  S3 Elimination of the Trivial Explanations for the Non-Monotonic
$\tau_{0}(T)$

Figure S2:  Switching times $\tau_{0}(T,P)$, plotted vs temperature at fixed
powers (left) and vs power at fixed temperatures (right). The top row is
measured close to the first bifurcation point ($P_{RF}$ between 2.9 and 3.1
dBm). Panel (a) is a reproduction of Figure 2b. In the bottom row,
$\tau_{0}(T,P)$ is measured close to the second bifurcation point at which the
$n=0$ step reappears ($P_{RF}$ between 5.1 and 5.4 dBm). While the same
characteristic behavior is observed, the non-monotonic $\tau_{0}(T)$ is first
observed in panel (a) at the lower range of powers and in panel (c) at the
higher range of powers. As discussed in the text, this observation allows us
to eliminate some trivial scenarios.

We have considered and discarded several possible alternative explanations for
the nonmonotonic dependence of $\tau_{0}(T)$, primarily related to unintended
shifts in control parameters. One such argument is that increasing the
temperature may decrease $I_{C}$, which would move the zero-crossing regime in
Figure 1a toward lower RF power levels. While Figure S1 presents evidence that
the boundary actually shifts in the opposite direction with temperature, such
that this should cause $\tau_{0}$ to further decrease, we also present
additional experimental evidence addressing this concern.

To this end, we perform measurements close to the second bifurcation point, in
the range of RF powers marked by blue lines in Figure S1. (The bifurcation
point itself occurs around 5.6 dBm, just to the right of the frames of Figure
S1.) The measured switching times near the second bifurcation point are shown
on the bottom row of Figure S2. To facilitate the direct comparison, the data
measured near the first bifurcation point is plotted in the top row of Figure
S2. (This set of data was shown in Figure 2b of the main text and corresponds
to the range of RF power marked by white lines in Figure S1.) For both sets,
the same data is plotted vs. temperature (left column) and vs. RF power (right
column).

For the second bifurcation point, we find that temperature non-monotonicity
first sets in for the highest powers (Figure S2c). This is in contrast to the
first bifurcation point, where it occurs first at the lowest powers (Figure
S2a). This difference indicates that the key parameter is the distance from
the center of the zero crossing region. It also rules out many trivial
explanations related to additional parameters changing as a function of
temperature. For example, both bifurcation points would move to the lower RF
powers as the critical current is decreased at higher temperatures; as a
result both would show non-monotonicity at the lowest powers, contrary to our
observations.

##  S4 Behaviour of $\tau_{0}$ at the Higher Temperatures

The non-monotonic curves in Figure S2 have the switching time increasing with
temperatures. However, this enhanced stability should eventually be reversed
at even higher temperatures.

We now would like to illustrate this eventual decrease of $\tau_{0}(T)$.

In practice, as the step voltage values start to decrease at higher
temperatures, the time trace starts to include more values which are not
quantized, which makes it more difficult to define the two states $n=\pm 1$.
However, there were a few parameter regimes for which it was possible to
observe a decrease in $\tau_{0}$ at the high temperature, while the data still
remained quite bimodal. Such behavior is shown in S7.

Figure S3: a) $\tau_{0}$ measured in extended temperature range at
$P_{RF}=3.14$ dBm (the third green curve in Figure S2a). The eventual drop of
the switching times at the highest temperatures is visible. b) The histogram
corresponding to the highest temperature data point in (a). It shows that the
distribution of voltages is still reasonably bimodal at this temperature,
although both steps are reduced compared to the quantized values, presumably
due to the rapid excursions to the $n=0$ state.  Figure S4: The symmetrized
Shapiro maps for (a) 5 GHz, (b) 5.4 GHz, (c) 5.9 GHz, and (d) 6.42 GHz. The
maps represent the difference between the differential resistance data
measured in the opposite sweep directions. Such maps highlight both the
hysteresis between the sweep directions and the overall symmetry. The features
that appear doubled in these maps are hysteretic. At 5 and 5.4 GHz, the $\pm
1\rightarrow 0$ and $0\rightarrow\pm 1$ boundaries almost coincide. On the
contrary, at 6.4 GHz significant hysteresis develops, pointing to a greater
stability of the $n=0$ plateau.

##  S5 Tristable behavior at high frequency

In this section, we directly reveal the role of the $n=0$ state in the system
dynamics. We start by exploring the coexistence region between the $n=0$ and
$\pm 1$ states.

The Shapiro maps of Figures 1 and S1 are highly hysteretic. We can highlight
the hysteresis by subtracting the maps measured in the two sweep directions.
After correcting for the any delay caused by the cryogenic filters,
nonhystereic features will overlap and nearly cancel, while the hysteretic
features become doubled.

In Figure S3, we show such symmetrized maps measured in a range of frequencies
from 5 to 6.42 GHz. We focus on the boundaries between $n=0$ and $n=\pm 1$
states at the closing of the $n=0$ step, marked by a box in panels (b) and
(d). At 5.4 GHz the boundaries nearly coincide, while for 6.42 GHz they are
well separated.

As a result, at 6.42 GHz, the boundaries form three separate intersection
points. The original bifurcation point appears at the intersection of the
$-1\rightarrow 0$ and $+1\rightarrow 0$ boundaries at zero bias. Two more
intersections points appear symmetrically at the positive and negative bias,
for example one of them marks the intersection of the $-1\rightarrow 0$,
$0\rightarrow+1$, and $-1\rightarrow+1$ boundaries.

Since the sweep rate and noise are constant throughout the measurement, we
take the appearance of the hysteresis as an indication that the quasipotential
profile has grown substantially at 6.42 GHz. At 5 and 5.4 GHz, as the
stability of $n=+1,-1$ is increased with applied RF power, the $n=0$ becomes
unstable on the measurement scales. On the contrary, Figure S3d indicates that
at 6.42 GHz all three states have comparable stability. We next measure the
voltage traces at this drive frequency in the power range between these three
intersection points, Figure S4a. We indeed find that all three voltages states
can be simultaneously stable, as is shown in Figure S4b.

We note that this regime is quite distinct from the non-Markovian regime
explored in Figure 4 of the main paper and modeled in Section S5. Here, all
transitions appear to be Markovian – careful inspection of the full version
(600 sec.) of the trace in Figure S4a indicates that the system is equally
likely to switch from $n=0$ to either $n=1$ or $n=-1$ regardless of the
history. This is to be expected, because our assumption is that the memory
provided by the RC circuitry dissipates on a scale shorter than our
measurement time scale; therefore if we can directly observe a robust $n=0$
state, the system cannot demonstrate non-Markovian switching.

Despite this difference between the 6.42 GHz and lower frequencies, there
should exist a continuous transition between them. The direct observation of
switching via the $n=0$ state lends support to the idea that at lower
frequencies the system should also evolve via the $n=0$ state, albeit on the
time scales which are too short relative to the measurement bandwidth.

Figure S5: Resolvable tristable data measured with 6.42 GHz RF drive. a) A
sample time trace, showing the presence of all 3 stable voltages. There are
essentially no direct transitions between $n=-1$ and $n=+1$ all transitions
instead pass through $n=0$. b) The distribution of observed voltages, plotted
for a range of different applied RF powers. At lowest power, $n=0$ is observed
almost exclusively (blue curve); at the intermediate power all three states
have the same probability (red); and at the highest power, $n=0$ is
significantly suppressed (yellow). Note that at 5 GHz it was not possible to
achieve relative balance between all three steps (as in the red curve).
Presumably, in that case the switching rate greatly exceeded the measurement
bandwidth.

##  S6 Full Numerical Model

We have previously explored the Shapiro maps directly simulating the dynamics
of the circuit model in Ref. Larson  _et al._ (2020). Here we extend this
simulation with an additional RC stage attached, Figure S4a. This leads to a
straightforward extension of the standard Shapiro step equations:

$\displaystyle I$ $\displaystyle=I_{bias}+I_{RF}\sin{\omega t}+I_{N}(t)$ (1)
$\displaystyle=C_{1}\frac{dV_{1}}{dt}+C_{0}\frac{dV_{0}}{dt}+I_{C}\sin{\phi}+\frac{\hbar}{2eR_{j}}\frac{d\phi}{dt}+\frac{\hbar
C_{j}}{2e}\frac{d^{2}\phi}{dt^{2}}$ $\displaystyle V_{0}$
$\displaystyle=\frac{\hbar}{2e}\frac{d\phi}{dt}+R_{L}\left(I_{C}\sin{\phi}+\frac{\hbar}{2eR_{j}}\frac{d\phi}{dt}+\frac{\hbar
C_{j}}{2e}\frac{d^{2}\phi}{dt^{2}}\right)$ $\displaystyle V_{1}$
$\displaystyle=\frac{\hbar}{2e}\frac{d\phi}{dt}+(R_{L}+R_{1})\left(I_{C}\sin{\phi}+\frac{\hbar}{2eR_{j}}\frac{d\phi}{dt}+\frac{\hbar
C_{j}}{2e}\frac{d^{2}\phi}{dt^{2}}\right)$
$\displaystyle+R_{1}C_{0}\frac{dV_{0}}{dt}$

where $V_{0}$ is the voltage on capacitor $C_{0}$ and $V_{1}$ is the voltage
on capacitor $C_{1}$.

The large separation of time scales makes it overly time consuming to simulate
the non-monotonic behavior of $\tau_{0}(T)$. However, we find it fairly
straightforward to recreate the correlated switching behavior of Figure 4 of
the main text. A portion of one such time trace is shown in Figure S6. Here,
we make use of physically reasonable parameters in Table 1. The voltage
plotted is the voltage of the junction, and thus is not expected to be non-
Markovian.

Parameter | Value
---|---
$I_{C}$ | 600 nA
$R_{J}$ | 300 Ohm
$R_{0}$ | 50 Ohm
$C_{0}$ | 1.5 pF
$R_{1}$ | 1500 Ohm
$C_{1}$ | 100 pF
$\sigma$ | 0.6
$f_{AC}$ | 5 GHz
$I_{AC}$ | 1.85 $\mu$A
Table 1: Values used for the full numerical simulation in Figure S6. Note that
noise is essentially in arbitrary units, owing to the fact that simulations
are performed with relatively large noise to allow for timely computation.
Figure S6: a) A schematic of the full model used in the simulation. It is
analogous to the model we previously studied in Ref. Larson  _et al._ (2020),
with an added RC stage. b) A sample time trace. Notice that for large periods
of time, the system rapidly bounces between $n=0$ and one of the Shapiro steps
(e.g. $n=-1$), only occasionally making short excursions to the opposite step
(in this case, $n=+1$). This clearly indicates that memory is at play: from
$n=0$, the system is more likely to go back to the state where it has spent
time just beforehand.

##  S7 Toy Model

In order to determine the minimal essential elements for the observations of
non-monotonic $\tau_{0}(T)$, we consider the following toy model. Inspired by
the tristable data (section S4) and the full numerical simulations (section
S5), wherein any switches between $n=+1$ and $n=-1$ go through $n=0$, we
represent the system by a particle in a three well potential. Without loss of
generality, we can assume the particle starts in the left well (corresponding
to $n=-1$), and solve for the average time to reach the right well
(corresponding to $n=1$), $\tau_{-1\rightarrow 1}$, which will be taken to
represent $\tau_{0}$. In this section, we will use the notation
$\tau_{i\rightarrow j}$ to define the average time to transition from well $i$
to well $j$.

In a symmetric tristable potential, we have
$\tau_{0\rightarrow-1}=\tau_{0\rightarrow 1}$ and the particle has a 50$\%$
change to go to the right well after reaching the middle well. Therefore, the
average time to reach the right well will be $\tau_{-1\rightarrow
1}=2\tau_{-1\rightarrow 0}$. (Here, we neglect the escape times from $n=0$,
the metastable middle well, assuming $\tau_{0\rightarrow\pm
1}\ll\tau_{-1\rightarrow 0}$.)

Even if the potential is not symmetric, and the system is more likely to
escape from the middle well to the left well, we can simply write the expected
time to reach the right well as a geometric series made of multiple attempts.
The series sums to $\tau_{-1\rightarrow 1}=\tau_{-1\rightarrow
0}/P_{0\rightarrow 1}$ where $P_{0\rightarrow 1}$ is the probability to go
from the 0 well to +1. Introducing $\Gamma_{+/-}\equiv 1/\tau_{0\rightarrow\pm
1}$, the escape rates from $n=0$ to $n=\pm 1$, we find that $P_{0\rightarrow
1}=\frac{\Gamma_{+}}{\Gamma_{+}+\Gamma_{-}}$, which results in
$\tau_{-1\rightarrow 1}=\tau_{-1\rightarrow
0}(\Gamma_{+}+\Gamma_{-})/\Gamma_{+}$. As expected, this result does not
naively produce non-monotonicity if $\tau_{-1\rightarrow 0}$ and
$\Gamma_{+/-}$ all follow a simple activational dependence on temperature.

For a slightly more complicated model, we consider the effect of an external
RC on the escape time. As mentioned in the main text, such external RCs
naturally arise in the cryogenic filters and cables. In full, these filters
raise our second order differential equation to 3rd order (or higher,
depending on the construction of the filter). However, the $RC$ time of the
filter should be much longer the the time scale of junction dynamics, so we
treat it adiabatically as a memory effect on our tristable well picture.

Physically, the RC circuit should be charged by the DC voltage developed in
the junction. As the junction switches from the $n=-1$ voltage to the $n=0$
voltage, the RC will start to discharge, current-biasing the junction. This
will increase the probability to return to $n=-1$ and decrease the probability
to reach $n=+1$. We can describe these corresponding switching from the middle
well with time-dependent rates $\Gamma_{+}(t)$ and $\Gamma_{-}(t)$. To make
the problem most tractable, we will approximate them as time-independent
constants $\Gamma_{+}<\Gamma_{-1}$ for $t<t_{0}$, where $t_{0}$ is taken to
represent the $RC$ time of the circuit. After that time, we will assume that
the capacitors have completely discharged and the particle will have a 50$\%$
chance of going in either direction.

Figure S7: Results from the basic toy model for $\tau_{0}=\tau_{-1\rightarrow
1}$, eq. 2. All the individual transition rates, including
$\tau_{-1\rightarrow 0}$ (red dash), are assumed to follow the standard
activational behavior. However, the total $n=-1$ to $n=1$ transition (blue
line) shows a pronounced non-monotonicity. At lower temperatures, it closely
follows $\tau_{-1\rightarrow 0}$. However, it significantly deviates at higher
temperatures, owing to the increased rate of return to $n=-1$.

Under these assumptions, it is straightforward to solve for the mean switching
time from $n=-1$ to $n=1$:

$\tau_{-1\rightarrow 1}=\frac{\tau_{-1\rightarrow
0}(\Gamma_{+}+\Gamma_{-})}{\Gamma_{+}-\frac{1}{2}(\Gamma_{+}-\Gamma_{-})\exp[-t_{0}(\Gamma_{+}+\Gamma_{-})]}$
(2)

The two limiting cases discussed earlier can be clearly reproduced. If the
mean escape rate from $n=0$ is low, $\Gamma_{+}+\Gamma_{-}\ll 1/t_{0}$ the RC
memory has little effect on the system, as the capacitor has fully discharged
long before the system switches from $n=0$, and we are back to the symmetric
tristable potential picture that we have previously considered. In the
opposite limit, $\Gamma_{+}+\Gamma_{-}\gg 1/t_{0}$ the system almost always
switches from $n=0$ before the capacitor has discharged, effectively
experiencing an asymmetric tristable behavior considered earlier. While the
first limit results in $\tau_{-1\rightarrow 1}=2\tau_{-1\rightarrow 0}$, in
the second case $\tau_{-1\rightarrow 1}=\tau_{-1\rightarrow
0}(\Gamma_{+}+\Gamma_{-})/\Gamma_{+}$, just like above. If
$\Gamma_{-}\gg\Gamma_{+}$ the second limit will greatly exceed the first, and
a region of non-monotonic $\tau_{-1\rightarrow 1}$ may appear stitching the
two limits together.

Figure S6 illustrates this nonmonotonic behavior for the set of parameters
intended to visually reproduce the experimental behavior. The dashed line
represents $\tau_{-1\rightarrow 0}$, while the solid line represents
$\tau_{-1\rightarrow+1}$. We assume $\tau_{-1\rightarrow
0}=10^{-2}t_{0}e^{E_{A}/T}$, $1/\Gamma_{-}=10^{-2}t_{0}e^{E_{-}/T}$ and
$\Gamma_{+}=10^{-2}t_{0}e^{E_{+}/T}$. Their temperature-independent energy
barriers are taken to be $E_{A}=3.5$, $E_{-}=1$, and $E_{+}=5$. (All energies
can be assumed to measured in Kelvin.)

The intuition for why non-monotonicity arises is as follows. At low
temperature, we start with $\Gamma_{+/-}t_{0}\ll 0$ and the memory effects are
negligible. As the temperature increases and the system escapes from $n=0$
faster and faster, it is more likely to escape before the external RC has
discharged, predominantly returning to $n=-1$. The system has to jump back and
forth between $n=-1$ and $n=0$ many times before finally transitioning to
$n=1$. Thus, even though every individual lifetime and rate
($\tau_{-1\rightarrow 0}$, $\Gamma_{+/-}$) follows the Arrhenius law, the
resulting time to reach $n=+1$ is non-monotonic.

An additional feature of this explanation is that it naturally accounts for
the observed reduction in the voltage steps at higher temperatures (see Figure
3 of the main text), as the system averages between $n=-1$ and $n=0$, or
between $n=1$ and $n=0$ states.

While the step function discharging model is not realistic, we believe it
should still capture all of the desired qualitative features while greatly
simplifying calculations. We have also assumed that $\tau_{-1\rightarrow 0}\gg
t_{0}$ for all temperatures, so that the RC circuit has enough time to reset
to $n=-1$ after switching from $n=0$. In other words, we can neglect any
memory of the $n=0$ state on $\tau_{-1\rightarrow 0}$. Finally, we neglected
the memory effect on switching back from $n=1$ once that state is reached. We
know that such memory is present (Figure 4 of the main text and Figure S5),
resulting in preferential switching back to $n=0$ and $n=-1$. All these
effects are important, but we do not think they are necessary to make a
functioning model that describes the nonmonotonic $\tau_{0}(T)$.

There are several interesting ways to extend the toy model. We can model the
effects of small finite bias by taking
$\Gamma_{0\rightarrow-1}\neq\Gamma_{0\rightarrow 1}$ even for $t>t_{0}$ when
the RC discharges. We have verified that one can qualitatively capture the
bias dependence shown in Figure 2d of the main text. Another direction is to
explore the necessity of the $n=0$ well, by simply removing it and introducing
the dependence of $\tau_{-1\rightarrow 1}$ and $\tau_{1\rightarrow-1}$ on the
state of the RC filter. This modification of the model may be forced to
produce the non-monotonic temperature dependence, indicating that the $n=0$
state may not be strictly essential to generate non-Markovian switching.
However, both the experimental data and the full numerical simulations of
section S5 demonstrate that the $n=0$ state is essential in our case.


code context neurons in Pythia-1B. Figure 17: Monosemantic distribution
identification context neurons in Pythia-6.9B. Figure 18: The single neuron
with highest F1 classification performance for each feature by model size.
Figure 19: Top neurons in Pythia-6.9B for each category of factual neuron.
Figure 20: Top neuron for each layer in Pythia-6.9B for the sex_or_gender and
is_alive feature categories.


# Breaking the attention bottleneck

Kalle Johannes Hilsenbek
<EMAIL_ADDRESS>

###### Abstract

Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the
attention mechanism with its quadratic complexity is a significant bottleneck
in the transformer architecture. This algorithm is only uni-directional in the
decoder and converges to a static pattern in over-parametrized decoder-only
models. I address this issue by developing a generative function as attention
or activation replacement. It still has the auto-regressive character by
comparing each token with the previous one. In my test setting with nanoGPT
this yields a smaller loss while having a smaller model. The loss further
drops by incorporating an average context vector. This concept of attention
replacement is distributed under the GNU AGPL v3 license at
https://gitlab.com/Bachstelze/causal_generation.

_Keywords_ linear attention, transformer, decoder, generative language
modeling, GPT

##  1 Introduction

The advent of attention-based transformer models has catalyzed transformative
advancements in the fields of natural language processing, video, and audio
processing. Since their introduction, these models have consistently pushed
the boundaries of what is achievable in machine learning and artificial
intelligence. The work on the Transformer architecture by Vaswani et al.
Vaswani et al. (2023) laid the foundation for a new era in which attention
mechanisms enable models to process and understand complex sequential data
with an improved accuracy.

Figure 1: This attention pattern is repeated in all heads of instruction-tuned
GPTs. Each token has a uniform distribution over all previous tokens as a
consistent auto-regressive activation.

One-directional attention models, such as GPTBrown et al. (2020), are commonly
employed for instruction-following tasks, highlighting the significance of
model scaling in the development of decoder-only architectures. However, this
increase of the decoder size leads to computational demands, a "lost-in-the-
middle"-effect of accessible input informations Liu et al. (2023) and an
informationless redundant attention pattern (see figure 1 on the right)
Hilsenbek et al. (2024). Hence, instruction-tuned GPTs only store information
in the fully connected linear feed-forward layers. They model information
mostly linear Hernandez et al. (2024)Razzhigaev et al. (2024) and represent
abstract concepts linearly Park et al. (2023). Therefore, instruction GPTs can
be seen as advanced multi-layer perceptrons and generative key-value memories
Geva et al. (2021). The paper "Transformers are Multi-State RNNs" proposes
that transformers can be viewed as a type of recurrent neural network (RNN)
with multiple states Oren et al. (2024). Traditionally, transformers and RNNs
are seen as distinct architectures. Transformers rely on self-attention
mechanisms to handle dependencies between input tokens, while RNNs auto-
regressive process sequences step-by-step. However, transformers can be
interpreted as multi-state RNNs, where each layer in a transformer manages a
set of hidden states, much like an RNN but with multiple-state
representations.

While the quadratic complexity of attention is clobbered by large tech
companies with a destructive amount of resources, it poses significant
challenges for the broader research community. Therefore, the resource-
intensive nature of these models has prompted the exploration of more
efficient transformer variants. For example, "Your Transformer is Secretly
Linear" reveals a novel linear characteristic in transformer decoders,
suggesting that their operation may be more linear than previously assumed.
This study introduces a cosine-similarity-based regularization to reduce layer
linearity, improving performance metrics on benchmarks like Tiny Stories and
SuperGLUE Razzhigaev et al. (2024). In contrast, "Not All Language Model
Features Are Linear" argues that certain features of language models resist
linear simplification, highlighting the need for a nuanced approach to
optimization Engels et al. (2024).

In this paper, I try to explore the limitations and challenges associated with
current transformer models, particularly focusing on the "attention
bottleneck". This bottleneck arises from the quadratic complexity of the
attention mechanism, which can hinder the scalability and efficiency of
transformer models when processing long sequences. I will examine simple
techniques aimed at breaking this bottleneck, paving the way for more
efficient and scalable transformer-based solutions. By addressing these
challenges, I aim to contribute to the ongoing evolution of transformer
models, ensuring their accessibility and efficacy within the open-source
community.

##  2 Related work

In the field of language modeling, linear attention mechanisms have been
developed to address quadratic complexity and keep the dynamic parameterized
behaviorWu et al. (2021)Katharopoulos et al. (2020)Schlag et al. (2021)Zhai et
al. (2021)Nguyen et al. (2022)Arora et al. (2023)Lee et al. (2024) Peng et al.
(2023)Peng et al. (2024). These methods aim to make attention computation more
efficient in terms of both memory and computational resources, allowing
transformers to handle longer sequences effectively Buckman and Gelada (2024).
E.g. by representing keys and queries with kernel feature maps, it enables the
use of associative properties to simplify the computation Shen et al. (2024)
Katharopoulos et al. (2020). The Performer uses kernel methods to approximate
the softmax function Choromanski et al. (2022). New approaches have further
optimized the computation by employing kernel tricks for different memory
components within GPUs to enhance execution speed and efficiency Qin et al.
(2024)Sun et al. (2024). Furthermore, a linear transformation by an
unparameterized Fourier Transform can encode language with a reduction in
accuracy Lee-Thorp et al. (2022). There are many more attempts to optimize
attention besides its linearization Tay et al. (2022)Hosseini and Hosseini
(2024)Munkhdalai et al. (2024). E.g. FlashAttention is introduced as a new
standard method to compute attention exactly and efficiently. The method is
designed to address the inefficiencies in memory and operations from the input
to the output that are common in traditional attention mechanisms Dao et al.
(2022).

There are also parameter-free attention approaches with convolutional neural
nets (CNN) in vision processing Körber (2022) Shi et al. (2023) Yang et al.
(2021) Guo et al. (2022) Wan et al. (2023). While they don’t explicitly delve
into language modeling, the concepts of parameter-free attention mechanisms
have relevant implications for this domain. Similar to how they aim to
simplify attention in CNNs, parameter-free attention mechanisms in language
models can reduce the computational load and memory requirements. This makes
training and inference more efficient, particularly beneficial for large-scale
and resource-intensive language models.

##  3 Method

In this section, I describe the cross-entropy loss divergence as a measurement
between the standard attention and my concept of recurrence and global context
as static attention replacement in nanoGPT. A small setting is used with a
block size of 64, a batch size of 12, 4 layers, an embedding size of 128, no
dropout, and 4 heads for attention. The only difference to the hyperparameters
in the ReadMe is the longer iteration and learning rate decay iteration number
of 5000 instead of 2000 to see that the results also hold for a longer
trainingKarpathy (2023). The small nanoGPT configuration has a number of 0.8
million parameters. Those parameters accumulate from 802,944 decayed parameter
tensors and 1,152 non-decayed parameter tensors. This amount is reduced to 0.6
million without the attention parameters of the key, value, and query
matrices. The quarter smaller size comes from the total 606,336 parameters of
decayed parameter tensors in my static attention approach. A comparison of the
different concepts is made by distinguishing the loss curve of the training on
the tiny Shakespeare dataset Karpathy (2015).

This Shakespeare dataset is so small that the middle model size of nanoGPT
overfits with 10.65 million number of parameters. The middle-sized setting has
6 layers, 6 attention heads, an embedding size of 384, batch size of 64 and
also a learning-rate of 1e-3. My concept is also applied in this middle-sized
setting to evaluate its performance in an over-parameterized setting.

##  4 Results

The auto-regressive attention can be replaced by static functions without
parameters by mimicking its equal distribution over the previous tokens (see
figure 1 in the introduction). This can achieved by returning the maximum
value of all dimensions of the current token and its precursor. The first
token remains unchanged by this vector manipulation. Such a simple concept
already achieves a smaller loss value with a smaller model size at once, while
reducing the computational cost. This finding is significant at the start of
the training and aligns with the further process (see figure 2). Causal
attention has a validation loss of 1.692, whereas my approach has a difference
of 0.054 with a loss of 1.638 in the end. Minimizing with the previous token
yields a nearly identical loss of 1.635. Training fluctuation could be a
plausible explanation for the tiny loss difference between maximization and
minimization with the previous token.

Figure 2: The training and validation loss curve of the standard attention and
recurrent maximization as well as minimization.

The integration of an average context vector as reference yields an even
further improvement of the auto-regressive maximization loss down to 1.557
respectively to 1.555 for the minimization. This drop of around 0.135 from the
standard attention loss is attained by the additional averaging of all input
vectors and an additional comparison of all tokens to this reference vector.
Calculating the mean between vectors seems to signal a complementary signal to
the maximization or minimization. Only calculating the mean with the previous
token performs between the standard attention and the discrete auto-regressive
optimization (see figure 3).

Figure 3: The validation loss curve of the standard attention and recurrent
methods with an average context.

In the over-parameterized setting the validation loss curve of the standard
attention increases after a fast drop of the validation the loss to a local
optimum. This can be mitigated by reducing the batch size from 64 to 16, which
yields a slightly better validation loss for attention (see figure 4) that is
continuously decreasing. Replacing the attention module with my recursive
function shows a similar over-fitting behaviour (see the curve of "causal_min"
in figure 4). This over-fitting can be avoided by leveraging my generative
function as activation in the fully-connected feedforward network. This
complementary approach stabilized and improves the training even with the
bigger batch size of 64 (see the "causal_context_attention"curve in figure 4).
Though the activation replacement increases the computational cost from 500ms
to around 575ms per iteration and the 2.5GB GPU-RAM to 3.6GB. This increase is
caused by the 4 times up-scaling of the inner dimension of the perceptron. The
replaced GELU function is activated before the down-scaling the normal
embedding size. Leaving the inner dimension unchanged reduces the cost of the
generative activation to 2.2GB GPU-RAM and an iteration time of 325ms while
having similar results to the upscaled version (see the
"causal_context_simple_inner_attention"-curve in figure 4). My generative
activation can also be placed before the first linear computation to keep the
larger inner dimensions with the GELU-activation.

Figure 4: The validation loss curve of the standard attention and recurrent
methods with an average context in the middle Shakespeare setting of nanoGPT.

##  5 Disussion

In "Multilayer feedforward networks are universal approximators" it is
demonstrated that perceptrons can approximate any continuous functionHornik et
al. (1989)Cybenko (1989). This result is often referred to as the Universal
Approximation Theorem and it could also describe the generative pretrained
transformers. The feedforward neural network seems to be the crucial component
within the GPT architecture by considering the static help function of
attention in decoders.111There is a discussion in BertViz about this redundant
pattern: https://github.com/jessevig/bertviz/issues/128 Therefore it should be
possible to apply all of the Universal Approximation Theorem to decoder-only
models. Attention can be seen as an activation function for the perceptron Neo
et al. (2024). Non-linear activation functions (like static attention) enable
the network to approximate complex functions. The generative power comes with
such non-linearity and model size.

An open question is the combination of the different algorithm variants at
many possible positions and furthermore the integration of my concept into
other architectures like encoder-decoder models or plain MLPs Bengio et al.
(2003). Moreover, the context vector could be gated with external resources,
e.g. for nearest neighbor speculative decodingLi et al. (2024). This could
enhance the model’s ability to incorporate external knowledge dynamically,
improving its overall contextual understanding and decision-making
capabilities while keeping a low computational complexity.

##  6 Conclusion

In this paper, I presented an efficient attention replacement for the decoder
module that addresses some of the key challenges faced by traditional
attention mechanisms. My proposed method can significantly reduce the
computational complexity and resource requirements while maintaining or even
improving the language modeling. The amount of operations is linear with the
sequence length and increases by a logarithmic factor with the self context
vector. The complexity can be brought down to O(1) and O(log(n)) for the
averaging over all tokens in parallel. This makes it particularly suitable for
deployment in environments with limited computational resources and hopefully
helps to reduce the destructive impact on our nature.

By leveraging a generative help function, my approach not only simplifies the
model architecture but also enhances interpretability and transparency. This
aligns with the growing demand for explainable AI systems that can provide
insights into their decision-making processes. The empirical results
demonstrate so far that my method achieves competitive modeling, which still
has to be proven in standard settings on downstream tasks.

I am pleased to distribute this work under the GNU affero general public
lisence version 3 ("GNU AGPL v3 license") at
https://gitlab.com/Bachstelze/causal_generation to ensure that the research
community can freely access, combine and build upon my findings. This open-
access approach fosters collaboration and innovation, encouraging a
development, which is aligned with humanistic values.

## References

  * Vaswani et al. [2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Liu et al. [2023] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.  Lost in the middle: How language models use long contexts, 2023.
  * Hilsenbek et al. [2024] Kalle Hilsenbek, Hyunjun Park, Gerti Sana, and Phillip Schmitz.  Explainable attention in instruction-tuned models, 2024.
  * Hernandez et al. [2024] Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau.  Linearity of relation decoding in transformer language models, 2024.
  * Razzhigaev et al. [2024] Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov.  Your transformer is secretly linear, 2024.
  * Park et al. [2023] Kiho Park, Yo Joong Choe, and Victor Veitch.  The linear representation hypothesis and the geometry of large language models, 2023.
  * Geva et al. [2021] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.  Transformer feed-forward layers are key-value memories, 2021.
  * Oren et al. [2024] Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz.  Transformers are multi-state rnns, 2024.
  * Engels et al. [2024] Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark.  Not all language model features are linear, 2024.
  * Wu et al. [2021] Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie.  Fastformer: Additive attention can be all you need, 2021.
  * Katharopoulos et al. [2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention, 2020.
  * Schlag et al. [2021] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.  Linear transformers are secretly fast weight programmers, 2021.
  * Zhai et al. [2021] Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind.  An attention free transformer, 2021.
  * Nguyen et al. [2022] Tan Nguyen, Richard G. Baraniuk, Robert M. Kirby, Stanley J. Osher, and Bao Wang.  Momentum transformer: Closing the performance gap between self-attention and its linearization, 2022.
  * Arora et al. [2023] Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré.  Zoology: Measuring and improving recall in efficient language models, 2023.
  * Lee et al. [2024] Heejun Lee, Jina Kim, Jeffrey Willette, and Sung Ju Hwang.  SEA: Sparse linear attention with estimated attention mask.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=JbcwfmYrob.
  * Peng et al. [2023] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu.  Rwkv: Reinventing rnns for the transformer era, 2023.
  * Peng et al. [2024] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr. au2, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu.  Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence, 2024.
  * Buckman and Gelada [2024] Jacob Buckman and Carles Gelada.  Linear Transformers Are Faster After All, 2024.
  * Shen et al. [2024] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li.  Efficient attention: Attention with linear complexities, 2024.
  * Choromanski et al. [2022] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller.  Rethinking attention with performers, 2022.
  * Qin et al. [2024] Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong.  Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models, 2024.
  * Sun et al. [2024] Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong.  Linear attention sequence parallelism, 2024.
  * Lee-Thorp et al. [2022] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon.  Fnet: Mixing tokens with fourier transforms, 2022.
  * Tay et al. [2022] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey, 2022.
  * Hosseini and Hosseini [2024] Mehran Hosseini and Peyman Hosseini.  You need to pay better attention: Rethinking the mathematics of attention mechanism, 2024.
  * Munkhdalai et al. [2024] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal.  Leave no context behind: Efficient infinite context transformers with infini-attention, 2024.
  * Dao et al. [2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022.
  * Körber [2022] Nils Körber.  Parameter-free average attention improves convolutional neural network performance (almost) free of charge, 2022.
  * Shi et al. [2023] Yuxuan Shi, Lingxiao Yang, Wangpeng An, Xiantong Zhen, and Liuqing Wang.  Parameter-free channel attention for image classification and super-resolution, 2023.
  * Yang et al. [2021] Lingxiao Yang, Ru-Yuan Zhang, Lida Li, and Xiaohua Xie.  Simam: A simple, parameter-free attention module for convolutional neural networks.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 11863–11874. PMLR, 18–24 Jul 2021.  URL http://proceedings.mlr.press/v139/yang21o.html.
  * Guo et al. [2022] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui.  Calip: Zero-shot enhancement of clip with parameter-free attention, 2022.
  * Wan et al. [2023] Cheng Wan, Hongyuan Yu, Zhiqi Li, Yihang Chen, Yajun Zou, Yuqing Liu, Xuanwu Yin, and Kunlong Zuo.  Swift parameter-free attention network for efficient super-resolution, 2023.
  * Karpathy [2023] Andrej Karpathy.  nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts.  https://github.com/karpathy/nanoGPT, 2023.  Commit on Feb 27, 2024.
  * Karpathy [2015] Andrej Karpathy.  char-rnn.  https://github.com/karpathy/char-rnn, 2015.
  * Hornik et al. [1989] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  _Neural Networks_ , 2(5):359–366, 1989.  ISSN 0893-6080.  doi:https://doi.org/10.1016/0893-6080(89)90020-8.  URL https://www.sciencedirect.com/science/article/pii/0893608089900208.
  * Cybenko [1989] George Cybenko.  Approximation by superpositions of a sigmoidal function. math cont sig syst (mcss) 2:303-314.  _Mathematics of Control, Signals, and Systems_ , 2:303–314, 12 1989.  doi:10.1007/BF02551274.
  * Neo et al. [2024] Clement Neo, Shay B. Cohen, and Fazl Barez.  Interpreting context look-ups in transformers: Investigating attention-mlp interactions, 2024.
  * Bengio et al. [2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin.  A neural probabilistic language model.  _J. Mach. Learn. Res._ , 3(null):1137–1155, mar 2003.  ISSN 1532-4435.
  * Li et al. [2024] Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen tau Yih, and Xi Victoria Lin.  Nearest neighbor speculative decoding for llm generation and attribution, 2024.