when comparing the experimental setup of the previously described papers, it
can be observed that there is not a consensus on a common testbed. Models are
evaluated on heterogeneous tasks and datasets (even when using the same
dataset, different works often preprocess it in a different manner), making
the comparison of their performances very difficult. Moreover, many of the
benchmarks are not directly targeting long-range modeling ability. Finally,
recent works fall short in decoupling the effect of large scale pretraining
from the inductive bias emerging from a certain data distribution. In an
effort to standardize long-range models evaluation, Tay et al. [282] proposed
the Long Range Arena (LRA) benchmark suite. LRA includes synthetic and real-
world tasks to assess the ability of sequential architectures to reason in
such scenarios, considering different types of data and conditions. It focuses
on understanding how well architectures can handle long sequences having
certain hierarchical or spatial structures, and aims to compare their
performance across various settings. In details, it consists of $6$ tasks:
($i$) Long ListOps, ($ii$) Text ($iii$) Retrieval ($iv$) Image ($v$)
Pathfinder and its extreme long version, ($vi$) Path-X. Sequences constituting
these task have a length $L\in[1k,16k]$, encompassing modalities and
objectives that require similarity, structural, and visuospatial reasoning.
The LRA suite is becoming increasingly popular in the scientific literature
and has been exploited by several recent works mentioned in the context of
this survey [88, 94, 87, 271, 100, 171, 235, 233, 170, 230, 224, 320, 213, 95,
159].

Long ListOps.  Long ListOps is a longer version of the ListOps task proposed
in [202], aimed at assessing the model ability to process hierarchically
structured data in a long-context scenario. In each sequence, some
mathematical operators (e.g., max, mean, median and sum_mod) and integer
operands (range 0 to 9) are enclosed by delimiters (brackets) in prefix
notation. The goal is to compute the integer result of the mathematical
expression, as exemplified by the following (short) sequence,

INPUT: [MAX 4 3 [MIN 2 3 ] 1 0 [MEDIAN 1 5 8 9, 2]] OUTPUT: 5

Input symbols are encoded via one-hot vectors, having $17$ unique possible
configurations (opening brackets and operators are grouped into a single
token). Long ListOps tests the ability to reason hierarchically while handling
long contexts with varying length, up to $L=2$k, with a reserved end-of-
sequence token. The final goal is ten-way classification (the integer result
of the expression). There are $96$k training sequences, $2$k validation
sequences, and $2$k test sequences.

Text.  Text is a byte-level text classification task, based upon the IMDb
binary sentiment classification previously described (i.e., classify whether a
movie review is positive/negative). Text sequences are processed at
byte/character-level (characters are encoded as one-hot vectors, with $129$
unique values possible), with a fixed max length of $L=4$k, and truncated or
padded when necessary. The evaluation metric is the accuracy. Compared to
ListOps, the model needs to reason with compositional unsegmented real-world
data, with less defined boundaries, that must be learnt from data. Differently
from character-level language modeling, where the local context is sufficient
to infer the next character, here the model is required to grasp the overall
sentiment of the processed text, hence it needs to compose characters into
words and then higher-level concepts. There are $25$k training examples and
$25$k test examples, while no validation set is provided.

Retrival.  Retrieval can be used to assess the model capability of encoding
long sequences into representations suitable for similarity-based matching.
Couples of texts are compressed and concatenated, then provided to a linear
classifier. The text is sampled from the ACL Anthology Network [236] dataset.
Given two textual citations, where characters are encoded as a sequence of
integer tokens (one-hot vector with $97$ unique values), the task is to
classify whether the two citations are equivalent. Byte/character level
processing is performed, resulting in sequence length of $L=4$k for each
document (i.e., total text length $8$k for this task). The performances of
this binary classification task are measured with accuracy. There are
$147,086$ training pairs, $18,090$ validation pairs, and $17,437$ test pairs.

Image.  The Image task corresponds to the already introduced sCIFAR, an image
classification task on sequences of pixels, described in Section 7.2. A
$32\times 32$ pixels image is flattened in raster scan into a sequence of
$1024$ pixels. The goal is to classify ($10$-way classification) the image by
processing the 1D pixel sequence. Models are required to capture 2D spatial
hierarchical structure between pixels even if they are processed sequentially.
Images belong to the CIFAR-10 [147] dataset, and are processed in a single
gray-scale channel (each pixel represented with 8-bit pixel intensity, hence
vocabulary size of $256$).303030Some recent works process the coloured images
instead [213]. There are $45$k training examples, $5$k validation examples,
and $10$k test examples.

Pathfinder.  Pathfinder is a synthetic visual task originally introduced in
[168, 143] for learning long-range spatial dependencies.

Figure 19: Negative (left) and positive (right) samples from Pathfinder.

A $32\times 32$ grayscale image contains two points (a start and an end point,
represented as small circles—see Figure 19). There are dashed lines over the
image. The task is to make a binary decision whether the two points are
connected by a path (Figure 19-left) or not (Figure 19-right). Models process
the image pixel sequence (raster scan), thus $L=1024$. There are $160$k
training examples, $20$k validation examples, and $20$k test examples.

Path-X.  Finally, Path-X is an harder version of Pathfinder composed by
$128\times 128$ images, resulting in sequences of $L=16$k.

Table 4: Sequence processing benchmarks. A list of the most common benchmarks (first column) and papers exploiting them (other columns). Each column report papers described in a different sections of this survey. | Benchmarks |  Sec. 3 of this paper |  Sec. 4 of this paper | Sec. 5 of this paper | Sec. 6 of this paper
---|---|---|---|---|---
Synthetic | Adding Problem [105] |  [150, 5] |  - | [157, 89, 248, 250] | [130]
Copy Task [105] |  [5, 191, 10, 30, 58] |  [86, 320, 57, 85] | [89, 140] | [191, 117, 217]
Induction Heads |  |  [57, 85] | - |
Associative Recall |  [309, 30, 223] |  [55, 85, 227] | [15] | [162]
Computer Vision | sMNIST |  [293, 86, 90, 88, 271] |  [89, 140, 248, 164, 68, 250] |  | [130, 211]
psMNIST |  |  | [248, 249, 68, 250] | [130, 217]
genMNIST |  [134] |  |  | [209]
CIFAR |  |  | [165, 89] | [130]
npCIFAR |  |  | [248, 249, 250] |
ImageNet |  [119, 170, 41] |  [159, 227] | [235] |
Natural Language | WikiText103 [192] |  [224, 118, 259, 230, 111, 51, 30, 158, 133, 180, 233, 304, 135, 235] |  [88, 171, 159, 227] | [89, 15] | [191]
PTB[181] |  [51] |  | [140, 250, 157, 25, 131] | [130]
The Pile[72] |  [277, 29, 235, 222] |  [227, 85] |  |
PG-19 [238] |  [225, 114, 304, 223] |  |  |
Enwik8 [176] |  [313, 111, 51, 30] |  | [153] |
ArXiv & GitHub [305] |  [114, 225, 158] |  |  |
LM-Eval-Harness [73] |  [309, 222, 231, 277, 223] |  [85, 57] | [15] |
WMT[24, 146] |  [224, 259, 97, 35, 133, 37] |  [171] |  |
IWSLT[33] |  [224] |  | [25] |
IMDb[172] |  [233] |  [86] | [248, 249, 25] |
Time Series |  Eigenworms [27] |  |  | [116, 249, 165] |
HAR-2 [4] |  |  [131, 248] |  |
Mackey-Glass [174] |  |  [293, 86] |  |
BDIMC[281] |  |  [90] | [249] |
SensorData |  [111] |  [90, 315] |  |
|  Speech Commands [297] |  |  [90, 88, 94, 87, 271] | [250, 142, 131] |
|  Long Range Arena [282] |  [233, 170, 230, 224] |  [320, 213, 95, 159, 87, 88, 94, 271, 100, 171] | [235, 15] |

##  8 Discussion and Future Directions

In this Section, we analyze some open problems and issues that have been
recently pointed out in the works described in this survey, as long as
highlighting possible future avenues for research.

###  8.1 The Attention-Recurrence Expressivity Gap

In the context of this survey we analyzed several architectural solutions
based on recurrence, leveraging its stateful nature. Indeed, recent models
exhibit constant-time inference and linear-time training (e.g, in SSMs [86,
90, 271], see Section 4) providing benefits when compared to stateless methods
whose complexity is quadratic in the sequence length, such as vanilla
Transformers. However, despite the computational advantages of recurrence-
based models, performances are not always on par with respect to vanilla
attention when considering specific tasks, also in synthetic settings [55,
124]. Indeed, the capability of recurrent models to effectively solve a target
task is strongly conditioned upon how efficiently the processed sequence is
compressed into a finite-size state [85], which acts as an information
bottleneck.

The role of gating mechanisms turns out to be fundamental, given their
influence on the model ability to control, in a context/input dependant
manner, how information propagates or interact along the temporal dimension.
Indeed, the relevance of an appropriate gating mechanism can be appreciated
when considered that such a mechanism is present in many high-impact works
described in this survey [126, 309, 319, 259, 118, 222, 277, 135], going well
beyond the original case of LSTMs. Indeed, the Mamba model [85] shows how a
selection mechanism (i.e., a term which refers to the mechanistic action of a
model to select or ignore inputs and facilitate data interaction along the
sequence length, inspired by the original gating in [105]) improves the
capability of State-Space Models to solve tasks such as Selective copy or
Induction Heads (see Section 7). In these task, content-aware reasoning is
fundamental to be able to memorize relevant tokens and filter out the
irrelevant ones. While the Linear Time Invariant (LTI) nature of previous SSMs
(LSSL, S4 [90, 88]) allows to achieve efficient parallelizable solutions,
their dynamics do not easily allow to select the correct information from the
current context, making it harder to encode the information stored into the
hidden state. Conversely, an appropriate selection mechanism introduces
context-awareness, allowing the model to focus on or to filter out
information. This element is also interesting in the light of recent studies
(see Section 5) suggesting that certain RNNs (equipped with linear recurrent
layers interconnected by feed-forward paths with multiplicative gating) might
be implementing attention/selection “under the hood”, when trained to solve
simple in-context learning tasks [319].

However, despite such advantages brought by gating/selection mechanisms,
performances of recurrence-based solutions are still not capable to fully
close the gap with attention-based models. Several recent works investigated
such a recurrence expressivity gap [55, 124] with respect to attention.
Indeed, inference in Transformers leverages the entire sequence processed so
far, which, in autoregressive models, is stored in the KV-cache (see Section
3), to produce the final output. By definition, the memory usage of this
approach increases over time, which becomes extremely expensive. At the cost
of inefficiency, the attention mechanism posses properties that standard
stateful models are not able to embrace: explicitly recalling earlier tokens
in the sequence (that influences their ability to copy from the input
sequence, possibly by multiplying the attention scores with $V$) and comparing
tokens across the sequence (which is achievable thanks to the quadratic
attention matrix $QK^{\prime}$)). Jelassi et al. [124] exploit the copy task
(see Section 7) to theoretically and empirically prove this property. In
short, they show that from the theoretical point of view a simple Transformer
model could be capable (when empowered by proper positional encodings and
under some other mild assumptions) to copy strings of length that is
exponential in the number of heads of the Transformer. This result relies on
the ability of the Transformer to implement a mechanism of “storage” and
retrieval of sequences of $n$ tokens ($n$-grams). Noticeably, the number of
parameters depends only logarithmically on the input sequence length. The
authors show that, conversely, any state-space model fails to solve the copy
task, unless its latent state grows linearly with the sequence length. That
is, SSMs cannot accurately copy strings with more bits than the size of the
latent state.

Such theoretical findings have been confirmed empirically. Indeed,
Transformers are both much more efficient at learning to copy (i.e., they need
$\approx 10^{4}$ training examples to learn the task, whereas Mamba [85] needs
more than $\approx 10^{6}$ samples) and also generalize better to longer
inputs (when trained on strings $L=50$, they generalize to sequences with
$L>1000$, whilst Mamba fails to generalize out of the trained context). This
finding confirm the theoretical results, i.e., to be able to copy the entire
input sequence, stateful models need to fully store it in its state space,
which requires the memory to grow linearly with the sequence length. These
results hold even when considering pre-trained models, both in copy tasks and
retrieval benchmarks (e.g., Phonebook Lookup, see Section 7).

These results are interesting in light of the fact that the selection
mechanism underlying Mamba (see Section 4) was introduced specifically to be
able to selectively propagate or forget information along the temporal
dimension, depending on the current token. Similar intuitions have previously
inspired the work of Dao et al. [55], who proposed the H3 model. As we
detailed in Section 4, their intuition on reducing the attention-recurrence
expressivity gap is to empower the copy and retrieval ability of SSMs by ($i$)
adding a layer with a shift matrix, that allows the state to copy from the
input and pass that information to the next state, ($ii$) stacking two SSMs
with multiplicative interaction, in order to compare information from previous
time steps (output of the lower shifted SSMs) with the input at the current
time steps – thus measuring similarity between tokens, which mimics local
multiplicative interactions in linear attention. Noticeably, the combination
of shift SSM and the following multiplicative interaction acts as a gate.
Empirically, these solutions allow to completely solve the Induction Heads
task and almost completely fit the Associative Recall, while other SSMs (e.g.,
S4D [87], GSS [189]) struggle with it.

Still, the work by Jelassi et al. [124] showed that these mechanisms do not
seem to be sufficient to close the recurrence expressivity gap. To counteract
this, the Griffin model includes a local attention [17], also known as sliding
window attention, into SSM-based models (i.e., LRU [213]). In practice, local
attention allows each position to attend only to a fixed number of tokens in
the past, reducing the number of FLOPs with respect to standard attention and
bounding the size of the KV-cache to the size of window, making it no longer
quadratic in the sequence length. Griffin matches the learning speed of
Transformers (in terms of necessary training steps/samples to achieve a
certain performance), and is capable to extrapolate to evaluation sequences
several orders of magnitude longer than the training sequence lengths.

These findings show that recurrent architectures working in the state-space
have still room for improvement, in terms or representational capability, when
compared to attention-based methods. Future work should focus on surpassing
the current approach to build hybrid architectures that endow state-space
models with an attention-like mechanism and investigate other solutions.
Moreover, it is of extreme interest to understand the role of model pre-
training. Indeed, Amos et al. [3] suggested that self-supervised pretraining
on the task data (referred to as self pretraining), rather than training from
scratch, allows to strongly increase the performances on long sequences (i.e.,
on LRA) both in vanilla Transformers and with simple diagonal RNNs, showing
the redundancy of sophisticated architectures and manually-designed biases
[87, 171]

###  8.2 Hardware-aware Efficiency: FLOPs vs. Memory Bandwidth

Reducing the computational and memory requirements of vanilla Transformers is
one of the main goal of the works described in this survey. In Section 3 we
showcased several attention approximations that indeed succeed in this goal,
achieving complexities linear or near-linear in sequence length [134, 259,
222] by reducing the amount of computations. However, when used in practice,
many of these solutions do not display substantial wall-clock speedup when
compared to standard attention. Recent works such as FlashAttention [54, 53]
showed how floating-point operations (FLOPs) reduction may not correlate with
fast wall-clock speed, when memory access overheads are not appropriately
taken care of.

This is a direct consequence of the way the hardware of common accelerators
(e.g., GPUs, TPUs, etc.) is structured. For instance, GPUs compute speed has
out-paced memory speed. When considering Transformers, most operations are
bottlenecked by memory accesses [54, 120]. Generally speaking, accelerators
perform reads and writes to different levels of fast and slow memory, i.e.,
between fast GPU on-chip SRAM and relatively slow GPU high bandwidth memory
(HBM). As a key example, the NVIDIA A100 is composed by 40-80 GB of HBM, with
bandwidth of 1.5-2.0 TB/s and 192KB of on-chip SRAM per each of 108 streaming
multiprocessors (SMs – that execute groups of thread, referred to as thread
blocks, in parallel) with bandwidth estimated around 19 TB/s [125, 54]. Each
thread loads input from HBM to registers and on-chip SRAM for computation
before writing back to HBM. Memory-bound operations (e.g., elementwise sum,
reductions, softmax, batch norm, layer norm) often pose significant
bottlenecks due to slower memory speeds. Vanilla Transformers store tensors
such as weights and the KV-cache in HBM for the duration of the decoding.
These tensors need to be transferred from HBM to the compute cores of the chip
once per forward pass of the model, thus giving rise to a large amount of
memory traffic.

Additionally, specialized hardware units, such as TensorCores in NVIDIA GPUs
[182] and MXUs in Google TPUs [204], are optimized for classic architectures
relying on matrix multiplications and convolutions with high FLOPs-to-byte
ratios. While RNNs benefit from this due to their dense recurrence matrices,
operations happening in diagonal RNNs (see SSMs and other diagonal models
described in Sections 4 and 5) pose computational challenges on existing
accelerators. For instance, the element-wise update equation in Griffin
results in a low FLOPs-to-byte ratio, leading to memory-bound computations.

Many recent works [90, 309, 135, 277] propose alternative computational forms
(i.e., recurrent, parallel, chunkwise) for their models, offering varying
tradeoffs which can be chosen according to specific needs, but still not
taking into account the I/O cost. Whilst basic implementations of the
recurrent forms minimize the total FLOPs, they incur in high I/O costs due to
storing 2D hidden states in HBM [180]. Katharopoulos et al. [134] reduce this
cost by avoiding state materialization and recomputing hidden states during
the backward pass (hereinafter referred to as recomputation), while Katsch
[135] leverages the parallel scan algorithm to parallelize a recurrent form
over the sequence length. However, the absence of parallelizable operations,
such as matrix multiplications, hinder the benefits from Tensorcores, thus not
translating to actual wall-time efficiency. In contrast, the quadratic-
complexity form of Transformers exhibits high FLOPs, making long-sequence
training expensive, even if Qin et al. [231] showed that it can be as
efficient as FlashAttention. The chunkwise form (see [110, 277]) provides an
interesting compromise, allowing fine-grained optimization by interpolating
between the parallel and recurrent forms with the user-selected chunk size.
Differently from the recurrent form, most operations can be done via matmuls,
enabling the use of tensor cores. However, most implementations are not
I/O-aware and thus slower than FlashAttention for moderate sequence lengths
(e.g., 2K-4K).

When considering solutions to the aforementioned issues, FlashAttention [54]
showed how even vanilla Transformers could benefit from I/O aware techniques.
The authors of [54] restructure (tiling) the attention computation to prevent
reading and writing (materializing) the large $L\times L$ attention matrix to
and from the slow HBM. Meanwhile, in large-scale training and long-sequence
modeling scenarios where the batch size tends to be small, parallelizing over
the temporal dimension enables high GPU occupancy [53].

Following similar intuitions, few recent works are starting to devise novel
I/O-aware mechanisms. The GLA model proposed by Yang et al. [309] (see Section
3) borrows concepts from Linear Transformers, gating mechanism and
FlashAttention, resulting in the proposal of FlashLinearAttention, an
hardware-efficient implementation of Linear Attention in a chunkwise form,
also generalized to a more expressive variant with data-dependent gates. The
authors propose a memory-efficient version where chunk-level states are not
materialized in HBM memory, by leveraging tiling and recomputation, and a
materialization version, which leverages sequence-level parallelism and thus
allows for higher training throughput at the cost of a slightly increasing
memory footprint. Overall, FlashLinearAttention is faster than
FlashAttention-2 [53], despite the introduction of an expressive gating
mechanism. A concurrent work by Qin et al. [232] also proposed an I/O-aware
version of linear attention, which is similar to the non-materialization
version of FlashLinearAttention.

The selection mechanism underlying Mamba [85] results in a time-varying SSM.
Time varying solutions (such as Hippo [86]) had been previously overtaken by
LTI (non-selective) models (S4, S5 and all derivatives) due to the fact that
the latter can be efficiently expressed in the form of global convolutions or
associative scans, while the former require to compute and materialize the
large latent state into HBM. However, the additional expressive power achieved
by means of “selection” procedures, calls for the need of an efficient
hardware-aware implementation. Indeed, similarly to GLA, Mamba leverages
kernel fusion (to reduce the amount of memory I/Os), parallel scan (to avoid
slowdowns due to the sequential recurrence), and recomputation (the
intermediate states are not stored but recomputed in the backward pass when
the inputs are loaded from HBM to SRAM). Overall, Mamba shows the same memory
requirements as an optimized transformer with FlashAttention.

To address the low FLOPs-to-byte ratio due to elementwise operations, the
Griffin model [57] is implemented in TPUs and leverages a custom kernel for
the computation of the state update equation exploiting linear scans. The
implemented kernel minimizes memory transfers by keeping hidden states in the
TPUs Vector Memory (VMEM) instead of HBM and performing memory transfers in
larger chunks, resulting in a $3\times$ speedup with respect to a vanilla
implementation. As an interesting remark, the gating mechanism in Griffin is
not compatible with the usual convolutional view or associative scans proper
of LTI SSMs. Even if the authors could have used associative scans similarly
to GateLoop [135], they found that these operations reduce the number of
FLOPs, but do not reduce memory overheads, which is their primary bottleneck.

Summing up, when dealing with long-sequence processing, a very relevant role
is played by the way computations are optimized, and identifying primary
performance bottlenecks on the target-hardware/modern-accelerators represent a
fundamental step for future research. Of course, novel models in this research
field could highly benefit from ad-hoc hardware and algorithmic solutions.

###  8.3 Infinite-length Sequences: Lifelong Learning

This survey describes multiple architectural and algorithmic solutions that
allow neural networks to handle sequences of increasing length, by reducing
the overall computational complexities and leveraging a stateful recurrent
representation. In the previous subsections, we showcased two very relevant
research avenues for future studies, i.e., the expressivity improvement of
recurrence-based models with respect to full-fledged attention, and the need
for hardware-aware solutions capable of effectively dealing with very long
sequences.

However, there is still a significant gap that requires specific studies to
move beyond current technologies, that shows up when thinking of transitioning
from long sequences to infinite-length sequences. Indeed, while researchers
aim to mimic human learning abilities from sequential data, the most common
existing approaches primarily deal with finite-length datasets BPTT in
stateful models, or standard Back-Propagation, whilst the alternative learning
algorithms we described in Section 6 are still not widespread in the
literature. This stands in stark contrast to human learning from continuous
perceptual stimuli, which unfold over time without being pre-segmented into
finite-length sequences for stochastic gradient descent optimization, and
without the need to explicitly store exemplars into buffers or the of need
building data collections [79]. Moreover, this misalignment deviates from the
principles of seminal works in recurrent models. Indeed, the goal of works
such as the one by Williams and Peng [302] was to devise “an online algorithm,
designed to be used to train a network while it runs; no manual state resets
or segmentations of the training stream is required”. Even the original
formulations of LSTMs, among others [105], were introduced with a learning
algorithm that unlike BPTT is “local in space and time”, where “there is no
need to store activation values observed during sequence processing in a stack
with potentially unlimited size”.

In contrast, as we described in the previous Sections, the most popular
research activities in the field of RNNs are not considering the case of
learning online from a continuous, possibly infinite, stream of data (i.e.,
infinite-length sequence), without resets or segmentations [19]. This
extremely challenging scenario involves two primary demands: ($i.$) computing
meaningful gradients at time $t$ based on processed data up to that point,
without fully unrolling the network across the entire (potentially infinite)
input sequence (in some cases, only relying on information from times $t$ and
$t-1$), and updating model parameters at each time step; ($ii.$) possessing
the ability to adapt to the characteristics of the streamed data, retaining
acquired skills over time, as emphasized in continual learning contexts [216].
In Section 6 we described the scientific efforts aimed at developing
alternatives to BPTT. We showcased online learning algorithms for RNNs,
particularly addressing requirement ($i.$). RTRL stands as the most recognized
method for online learning with recurrent models, which has been recently
refined to mitigate computational and memory constraints [117], alongside
various direct or indirect adaptations [184]. We described recent studies that
explore forward computation of gradients, enabled by state-space modeling
[320], albeit reliant on finite-length sequences processed without updating
learnable parameters at each time step, thereby not fully satisfying
requirement ($i.$). Considering demand ($ii.$) leads to a scenario akin to
Online Continual Learning [216] with substantial overlap with Streaming
Learning, as outlined in [93]. It’s important to distinguish this from
approaches focused solely on continual learning from streams of finite-length
sequences [49, 62].

Summing up, the scenario described in this section remains largely unexplored.
Despite advancements in related domains, meeting requirements ($i.$) and
($ii.$) independently poses significant challenges, and their combined
fulfillment presents an even more daunting task. Nonetheless, this scenario
reflects the reality of learning from continuous streams of visual data,
auditory stimuli, perpetual video feeds, language inputs, etc. Whilst common
benchmarks for sequential data (see Section 7) do provide valuable
opportunities to assess the fulfillment of requirement ($i.$), it is extremely
needed to operate within the typical framework of online learning problems,
where all data contributes to training and errors accumulate during
learning.313131The input at time $u_{t}$ is processed and the produced output
$y_{t}$ is compared with the available ground truth to update the accuracy
estimate; then, $y_{t}$ is used to evaluate a loss function, compute gradients
at $t$ and update the model. Certain tasks such as time-series forecasting
offer greater flexibility in this context, compared to classification tasks,
due to the fact that they provide ground truth at each $t$. Additionally, the
properties of the series might change over time, partially allowing to explore
requirement ($ii.$). However, a comprehensive evaluation of property ($ii.$)
remains challenging with existing datasets.

All these considerations suggest that there is room for future research.
Providing sources for streaming data becomes extremely important, in a context
where data privacy and storage become more and more relevant [79]. For
instance, in computer vision, a paradigm-shift from image-based processing to
videos is happening [263, 287, 69, 20, 190, 31, 26], thus requiring to rethink
data handling pipelines and data engineering. From the pure machine learning
side, research on novel models and learning methods which are better at
capturing regularities on the input stream in an online manner are needed. At
the same time, model evaluation and selection procedures must evolve [77], to
be capable of handling settings where models self-adapt their parameters over
time continuously, potentially in a truly lifelong learning manner.

##  9 Conclusions

While Transformers initially overshadowed Recurrent Neural Networks, recent
developments in large-context Transformers and (deep) State-Space Models have
highlighted the importance of recurrent computations as a viable road to go
beyond the limitations of Transformers both in terms of performances on long
sequences and scalability. This survey outlined the latest trends and
approaches for sequential data processing, offering insights about recent
architectural and novel algorithmic solutions. Interestingly, the recent
literature in the context of recurrent long-context Transformers and of State-
Space Models share the same features, leading towards a computational model
based on stateful computations, confirming the advantages brought by the idea
of working in the state space. We surveyed the most important works, and we
discussed novel trends in learning algorithms for sequential data, offering a
broad and detailed perspective on the last generation of neural architectures.
This survey also outlined novel research opportunities. In fact, we emphasized
how the research direction of lifelong online learning from a stream of data,
intended to be an infinite-length sequence, represents a challenge that is
still very open and might be faced starting from this renewed interest in
recurrent models.

## References

  * [1] Luis B Almeida.  A learning rule for asynchronous perceptrons with feedback in a combinatorial environment.  In Artificial neural networks: concept learning, pages 102–111. 1990.
  * [2] Carmen Amo Alonso, Jerome Sieber, and Melanie N Zeilinger.  State space models as foundation models: A control theoretic overview.  arXiv preprint arXiv:2403.16899, 2024.
  * [3] Ido Amos, Jonathan Berant, and Ankit Gupta.  Never train from scratch: Fair comparison of long-sequence models requires data-driven priors.  In The Twelfth International Conference on Learning Representations, 2024.
  * [4] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L Reyes-Ortiz.  Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine.  In Ambient Assisted Living and Home Care: 4th International Workshop, IWAAL 2012, Vitoria-Gasteiz, Spain, December 3-5, 2012. Proceedings 4, pages 216–223. Springer, 2012.
  * [5] Martin Arjovsky, Amar Shah, and Yoshua Bengio.  Unitary evolution recurrent neural networks.  In International conference on machine learning, pages 1120–1128. PMLR, 2016.
  * [6] Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré.  Zoology: Measuring and improving recall in efficient language models.  arXiv preprint arXiv:2312.04927, 2023.
  * [7] Armin Askari, Geoffrey Negiar, Rajiv Sambharya, and Laurent El Ghaoui.  Lifted neural networks.  arXiv preprint arXiv:1805.01532, 2018.
  * [8] Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley.  Effective gene expression prediction from sequence by integrating long-range interactions.  Nature methods, 18(10):1196–1203, 2021.
  * [9] Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu.  Using fast weights to attend to the recent past.  Advances in neural information processing systems, 29, 2016.
  * [10] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [11] Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, and Atilla Baskurt.  Sequential deep learning for human action recognition.  In Human Behavior Understanding: Second International Workshop, HBU 2011, Amsterdam, The Netherlands, November 16, 2011. Proceedings 2, pages 29–39. Springer, 2011.
  * [12] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [13] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  arXiv preprint arXiv:1803.01271, 2018.
  * [14] David Balduzzi and Muhammad Ghifary.  Strongly-typed recurrent neural networks.  In International Conference on Machine Learning, pages 1292–1300. PMLR, 2016.
  * [15] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter.  xlstm: Extended long short-term memory.  arXiv preprint arXiv:2405.04517, 2024.
  * [16] Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, C James Taylor, and Gerhard Neumann.  Recurrent kalman networks: Factorized inference in high-dimensional deep feature spaces.  In International conference on machine learning, pages 544–552. PMLR, 2019.
  * [17] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  arXiv preprint arXiv:2004.05150, 2020.
  * [18] Yoshua Bengio, Patrice Simard, and Paolo Frasconi.  Learning long-term dependencies with gradient descent is difficult.  IEEE transactions on neural networks, 5(2):157–166, 1994.
  * [19] Alessandro Betti, Lapo Faggi, Marco Gori, Matteo Tiezzi, Simone Marullo, Enrico Meloni, and Stefano Melacci.  Continual learning through hamilton equations.  In Conference on Lifelong Learning Agents, volume 199, pages 201–212. PMLR, 2022.
  * [20] Alessandro Betti, Marco Gori, and Stefano Melacci.  Learning visual features under motion invariance.  Neural Networks, 126:275–299, 2020.
  * [21] Filippo Maria Bianchi, Enrico Maiorino, Michael C. Kampffmeyer, Antonello Rizzi, and Robert Jenssen.  Recurrent Neural Networks for Short-Term Load Forecasting - An Overview and Comparative Analysis.  Springer Briefs in Computer Science. Springer, 2017.
  * [22] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.  Piqa: Reasoning about physical commonsense in natural language.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
  * [23] Guy E Blelloch.  Prefix sums and their applications.  1990\.
  * [24] Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al.  Findings of the 2014 workshop on statistical machine translation.  In Proceedings of the ninth workshop on statistical machine translation, pages 12–58, 2014.
  * [25] James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher.  Quasi-recurrent neural networks.  arXiv preprint arXiv:1611.01576, 2016.
  * [26] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Video generation models as world simulators.  2024\.
  * [27] André EX Brown, Eviatar I Yemini, Laura J Grundy, Tadas Jucikas, and William R Schafer.  A dictionary of behavioral motifs reveals clusters of genes affecting caenorhabditis elegans locomotion.  Proceedings of the National Academy of Sciences, 110(2):791–796, 2013.
  * [28] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [29] Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev.  Scaling transformer to 1m tokens and beyond with rmt.  arXiv preprint arXiv:2304.11062, 2023.
  * [30] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev.  Recurrent memory transformer.  Advances in Neural Information Processing Systems, 35:11079–11091, 2022.
  * [31] João Carreira, Michael King, Viorica Pătrăucean, Dilara Gokay, Cătălin Ionescu, Yi Yang, Daniel Zoran, Joseph Heyward, Carl Doersch, Yusuf Aytar, et al.  Learning from one continuous video stream.  arXiv preprint arXiv:2312.00598, 2023.
  * [32] Miguel Carreira-Perpinan and Weiran Wang.  Distributed optimization of deeply nested systems.  In Artificial Intelligence and Statistics, pages 10–19. PMLR, 2014.
  * [33] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico.  Report on the 11th iwslt evaluation campaign.  In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2–17, 2014.
  * [34] Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi.  Antisymmetricrnn: A dynamical system view on recurrent neural networks.  In International Conference on Learning Representations, 2018.
  * [35] Kehai Chen, Rui Wang, Masao Utiyama, and Eiichiro Sumita.  Recurrent positional embedding for neural machine translation.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1361–1367, 2019.
  * [36] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  Advances in neural information processing systems, 34:15084–15097, 2021.
  * [37] Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, et al.  The best of both worlds: Combining recent advances in neural machine translation.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76–86, 2018.
  * [38] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  Advances in neural information processing systems, 31, 2018.
  * [39] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio.  Babyai: A platform to study the sample efficiency of grounded language learning.  arXiv preprint arXiv:1810.08272, 2018.
  * [40] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller.  Masked language modeling for proteins via linearly scalable long-context transformers.  arXiv preprint arXiv:2006.03555, 2020.
  * [41] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  In International Conference on Learning Representations, 2020.
  * [42] Stephen Chung and Hava Siegelmann.  Turing completeness of bounded-precision recurrent neural networks.  Advances in Neural Information Processing Systems, 34:28431–28441, 2021.
  * [43] Philippe G Ciarlet.  Linear and nonlinear functional analysis with applications, volume 130.  Siam, 2013.
  * [44] Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons.  Theoretical foundations of deep selective state-space models.  arXiv preprint arXiv:2402.19047, 2024.
  * [45] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  arXiv preprint arXiv:1905.10044, 2019.
  * [46] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.
  * [47] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, 2015.
  * [48] Christopher M Conway and Morten H Christiansen.  Sequential learning in non-human primates.  Trends in cognitive sciences, 5(12):539–546, 2001.
  * [49] Andrea Cossu, Antonio Carta, Vincenzo Lomonaco, and Davide Bacciu.  Continual learning for recurrent neural networks: an empirical evaluation.  Neural Networks, 143:607–627, 2021.
  * [50] Francis Crick.  The recent excitement about neural networks.  Nature, 337(6203):129–132, 1989.
  * [51] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019.
  * [52] Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.  In Proceedings of ACL 2019, 2019.
  * [53] Tri Dao.  Flashattention-2: Faster attention with better parallelism and work partitioning.  arXiv preprint arXiv:2307.08691, 2023.
  * [54] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  Advances in Neural Information Processing Systems, 35:16344–16359, 2022.
  * [55] Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré.  Hungry hungry hippos: Towards language modeling with state space models.  In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  * [56] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In International conference on machine learning. PMLR, 2017.
  * [57] Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al.  Griffin: Mixing gated linear recurrences with local attention for efficient language models.  arXiv preprint arXiv:2402.19427, 2024.
  * [58] Aniket Didolkar, Kshitij Gupta, Anirudh Goyal, Nitesh Bharadwaj Gundavarapu, Alex M Lamb, Nan Rosemary Ke, and Yoshua Bengio.  Temporal latent bottleneck: Synthesis of fast and slow processing mechanisms in sequence learning.  Advances in Neural Information Processing Systems, 35:10505–10520, 2022.
  * [59] SiYu Ding, Junyuan Shang, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-doc: A retrospective long-document modeling transformer.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2914–2927, 2021.
  * [60] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Neil Houlsby, Sylvain Gelly, Xiaohua Zhang, and Jakob Uszkoreit.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [61] Philipp Dufter, Martin Schmitt, and Hinrich Schütze.  Position information in transformers: An overview.  Computational Linguistics, 48(3):733–763, September 2022.
  * [62] Benjamin Ehret, Christian Henning, Maria Cervera, Alexander Meulemans, Johannes Von Oswald, and Benjamin F Grewe.  Continual learning in recurrent neural networks.  In International Conference on Learning Representations, 2020.
  * [63] Lukas Eisenmann, Zahra Monfared, Niclas Alexander Göring, and Daniel Durstewitz.  Bifurcations and loss jumps in rnn training, 2023.
  * [64] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al.  A mathematical framework for transformer circuits.  Transformer Circuits Thread, 1:1, 2021.
  * [65] Jeffrey L Elman.  Finding structure in time.  Cognitive science, 14(2):179–211, 1990.
  * [66] Jeffrey L Elman and David Zipser.  Learning the hidden structure of speech.  The Journal of the Acoustical Society of America, 83(4):1615–1626, 1988.
  * [67] N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney.  Lipschitz recurrent neural networks.  In International Conference on Learning Representations, 2021.
  * [68] N. Benjamin Erichson, Omri Azencot, Alejandro F. Queiruga, Liam Hodgkinson, and Michael W. Mahoney.  Lipschitz recurrent neural networks.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [69] Lapo Faggi, Alessandro Betti, Dario Zanca, Stefano Melacci, and Marco Gori.  Local propagation of visual stimuli in focus of attention.  Neurocomputing, 560:126775, 2023.
  * [70] P. Frasconi, M. Gori, and A. Sperduti.  A general framework for adaptive processing of data structures.  IEEE Transactions on Neural Networks, 9(5):768–786, 1998.
  * [71] Claudio Gallicchio and Alessio Micheli.  Architectural and markovian factors of echo state networks.  Neural Networks, 24(5):440–456, 2011.
  * [72] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027, 2020.
  * [73] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.
  * [74] John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett.  Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1.  NASA STI/Recon technical report n, 93:27403, 1993.
  * [75] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  In International conference on machine learning, pages 1243–1252. PMLR, 2017.
  * [76] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins.  Learning to forget: Continual prediction with lstm.  Neural computation, 12(10):2451–2471, 2000.
  * [77] Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem.  Real-time evaluation in online continual learning: A new hope.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11888–11897, 2023.
  * [78] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré.  It’s raw! audio generation with state-space models.  In International Conference on Machine Learning, pages 7616–7633. PMLR, 2022.
  * [79] M. Gori and S. Melacci.  Collectionless artificial intelligence.  arXiv 2309.06938, 2023.
  * [80] Marco Gori, Gabriele Monfardini, and Franco Scarselli.  A new model for learning in graph domains.  In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729–734. IEEE, 2005.
  * [81] Alex Graves and Alex Graves.  Supervised sequence labelling.  Springer, 2012.
  * [82] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural turing machines.  arXiv preprint arXiv:1410.5401, 2014.
  * [83] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber.  Lstm: A search space odyssey.  IEEE transactions on neural networks and learning systems, 28(10):2222–2232, 2016.
  * [84] Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves.  Memory-efficient backpropagation through time.  Advances in neural information processing systems, 29, 2016.
  * [85] Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces, 2023.
  * [86] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.  Hippo: Recurrent memory with optimal polynomial projections.  Advances in neural information processing systems, 33:1474–1487, 2020.
  * [87] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré.  On the parameterization and initialization of diagonal state space models.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 35971–35983. Curran Associates, Inc., 2022.
  * [88] Albert Gu, Karan Goel, and Christopher Re.  Efficiently modeling long sequences with structured state spaces.  In International Conference on Learning Representations, 2021.
  * [89] Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu.  Improving the gating mechanism of recurrent neural networks.  In International Conference on Machine Learning, pages 3800–3809. PMLR, 2020.
  * [90] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  Advances in neural information processing systems, 34:572–585, 2021.
  * [91] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.  Ava: A video dataset of spatio-temporally localized atomic visual actions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6047–6056, 2018.
  * [92] Fangda Gu, Armin Askari, and Laurent El Ghaoui.  Fenchel lifted networks: A lagrange relaxation of neural network training.  In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3362–3371. PMLR, 26–28 Aug 2020.
  * [93] Nuwan Gunasekara, Bernhard Pfahringer, Heitor Murilo Gomes, and Albert Bifet.  Survey on online streaming continual learning.  In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6628–6637, 2023.
  * [94] Ankit Gupta, Albert Gu, and Jonathan Berant.  Diagonal state spaces are as effective as structured state spaces.  In Advances in Neural Information Processing Systems, 2022.
  * [95] Ankit Gupta, Harsh Mehta, and Jonathan Berant.  Simplifying and understanding state space models with diagonal linear rnns.  arXiv preprint arXiv:2212.00768, 2022.
  * [96] Stephen J Hanson and Judy Kegl.  Parsnip: A connectionist network that learns natural language grammar from exposure to natural language sentences.  In Proceedings of the Eight Annual Meeting of the Cognitive Science Society, pages 106–119. Lawrence Erlbaum Associates Hillsdale, NJ, 1987.
  * [97] Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu.  Modeling recurrence for transformer.  In Proceedings of NAACL-HLT, pages 1198–1207, 2019.
  * [98] Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, and Daniela Rus.  Closed-form continuous-time neural networks.  Nature Machine Intelligence, 4(11):992–1003, 2022.
  * [99] Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu.  Liquid time-constant networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7657–7666, 2021.
  * [100] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus.  Liquid structural state-space models.  arXiv preprint arXiv:2209.12951, 2022.
  * [101] Hongyu Hè and Marko Kabic.  A unified view of long-sequence models towards million-scale dependencies.  arXiv preprint arXiv:2302.06218, 2023.
  * [102] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  arXiv preprint arXiv:2009.03300, 2020.
  * [103] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [104] Sepp Hochreiter.  Untersuchungen zu dynamischen neuronalen netzen.  Diploma, Technische Universität München, 91(1):31, 1991.
  * [105] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [106] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  An empirical analysis of compute-optimal large language model training.  Advances in Neural Information Processing Systems, 35:30016–30030, 2022.
  * [107] Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola.  Kernel methods in machine learning.  The Annals of Statistics, 36(3):1171 – 1220, 2008.
  * [108] John J Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
  * [109] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2021.
  * [110] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le.  Transformer quality in linear time.  In International Conference on Machine Learning, pages 9099–9117. PMLR, 2022.
  * [111] Feiqing Huang, Kexin Lu, CAI Yuxi, Zhen Qin, Yanwen Fang, Guangjian Tian, and Guodong Li.  Encoding recurrence into transformers.  In The Eleventh International Conference on Learning Representations, 2022.
  * [112] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [113] J Stuart Hunter.  The exponentially weighted moving average.  Journal of quality technology, 18(4):203–210, 1986.
  * [114] DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur.  Block-recurrent transformers.  Advances in Neural Information Processing Systems, 35:33248–33261, 2022.
  * [115] Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber.  Practical computational power of linear transformers and their recurrent and self-referential extensions.  arXiv preprint arXiv:2310.16076, 2023.
  * [116] Kazuki Irie, Francesco Faccio, and Jürgen Schmidhuber.  Neural differential equations for learning to program neural nets through continuous learning rules.  Advances in Neural Information Processing Systems, 35:38614–38628, 2022.
  * [117] Kazuki Irie, Anand Gopalakrishnan, and Jürgen Schmidhuber.  Exploring the promise and limits of real-time recurrent learning.  arXiv preprint arXiv:2305.19044, 2023.
  * [118] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber.  Going beyond linear transformers with recurrent fast weight programmers.  Advances in Neural Information Processing Systems, 34:7703–7717, 2021.
  * [119] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber.  A modern self-referential weight matrix that learns to modify itself.  In International Conference on Machine Learning, pages 9660–9677. PMLR, 2022.
  * [120] Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler.  Data movement is all you need: A case study on optimizing transformers.  Proceedings of Machine Learning and Systems, 3:711–732, 2021.
  * [121] Khurram Javed, Haseeb Shah, Rich Sutton, and Martha White.  Scalable online state construction using recurrent networks.  2021\.
  * [122] Khurram Javed, Haseeb Shah, Rich Sutton, and Martha White.  Online real-time recurrent learning using sparse connections and selective learning.  arXiv preprint arXiv:2302.05326, 2023.
  * [123] Khurram Javed, Martha White, and Rich Sutton.  Scalable online recurrent learning using columnar neural networks.  arXiv preprint arXiv:2103.05787, 2021.
  * [124] Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach.  Repeat after me: Transformers are better than state space models at copying.  arXiv preprint arXiv:2402.01032, 2024.
  * [125] Zhe Jia and Peter Van Sandt.  Dissecting the ampere gpu architecture via microbenchmarking.  In GPU Technology Conference, 2021.
  * [126] Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio.  Gated orthogonal recurrent units: On learning to forget.  Neural computation, 31(4):765–783, 2019.
  * [127] Michael I Jordan.  Serial order: A parallel distributed processing approach.  In Advances in psychology, volume 121, pages 471–495. Elsevier, 1997.
  * [128] Vanessa Isabell Jurtz, Alexander Rosenberg Johansen, Morten Nielsen, Jose Juan Almagro Armenteros, Henrik Nielsen, Casper Kaae Sønderby, Ole Winther, and Søren Kaae Sønderby.  An introduction to deep learning on biological sequence data: examples and solutions.  Bioinformatics, 33(22):3685–3690, 2017.
  * [129] Anil Kag and Venkatesh Saligrama.  Time adaptive recurrent neural network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15149–15158, 2021.
  * [130] Anil Kag and Venkatesh Saligrama.  Training recurrent neural networks via forward propagation through time.  In International Conference on Machine Learning, pages 5189–5200. PMLR, 2021.
  * [131] Anil Kag, Ziming Zhang, and Venkatesh Saligrama.  Rnns incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients?  In International Conference on Learning Representations, 2019.
  * [132] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.
  * [133] Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A Smith.  Finetuning pretrained transformers into rnns.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10630–10643, 2021.
  * [134] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In International conference on machine learning, pages 5156–5165. PMLR, 2020.
  * [135] Tobias Katsch.  Gateloop: Fully data-controlled linear recurrence for sequence modeling.  arXiv preprint arXiv:2311.01927, 2023.
  * [136] Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde.  On the computational complexity of self-attention.  In International Conference on Algorithmic Learning Theory, pages 597–619. PMLR, 2023.
  * [137] T Anderson Keller, Lyle Muller, Terrence Sejnowski, and Max Welling.  Traveling waves encode the recent past and enhance sequence learning.  arXiv preprint arXiv:2309.08045, 2023.
  * [138] T Anderson Keller and Max Welling.  Neural wave machines: Learning spatiotemporally structured representations with locally coupled oscillatory recurrent neural networks.  2023\.
  * [139] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of naacL-HLT, volume 1, page 2, 2019.
  * [140] Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie.  Non-normal recurrent neural network (nnrnn): learning long time dependencies while improving expressivity with transient dynamics.  Advances in neural information processing systems, 32, 2019.
  * [141] Patrick Kidger.  On neural differential equations.  arXiv preprint arXiv:2202.02435, 2022.
  * [142] Patrick Kidger, James Morrill, James Foster, and Terry Lyons.  Neural controlled differential equations for irregular time series.  Advances in Neural Information Processing Systems, 33:6696–6707, 2020.
  * [143] Junkyung Kim*, Drew Linsley*, Kalpit Thakkar, and Thomas Serre.  Disentangling neural mechanisms for perceptual grouping.  In International Conference on Learning Representations, 2020.
  * [144] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In International Conference on Learning Representations, 2020.
  * [145] Genshiro Kitagawa.  A self-organizing state-space model.  Journal of the American Statistical Association, pages 1203–1215, 1998.
  * [146] Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, et al.  Findings of the 2022 conference on machine translation (wmt22).  In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1–45, 2022.
  * [147] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.
  * [148] Eduard Kuriscak, Petr Marsalek, Julius Stroffek, and Peter G Toth.  Biological context of hebb learning in artificial neural networks, a review.  Neurocomputing, 152:27–35, 2015.
  * [149] Samuel Lanthaler, T Konstantin Rusch, and Siddhartha Mishra.  Neural oscillators are universal.  arXiv preprint arXiv:2305.08753, 2023.
  * [150] Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton.  A simple way to initialize recurrent networks of rectified linear units.  arXiv preprint arXiv:1504.00941, 2015.
  * [151] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [152] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara Berg, and Mohit Bansal.  Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2603–2614, 2020.
  * [153] Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi.  Simple recurrent units for highly parallelizable recurrence.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470–4481, 2018.
  * [154] Mario Lezcano-Casado and David Martınez-Rubio.  Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group.  In International Conference on Machine Learning. PMLR, 2019.
  * [155] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.  Cmmlu: Measuring massive multitask language understanding in chinese.  arXiv preprint arXiv:2306.09212, 2023.
  * [156] Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen.  Pretrained language models for text generation: A survey.  arXiv preprint arXiv:2201.05273, 2022.
  * [157] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao.  Independently recurrent neural network (indrnn): Building a longer and deeper rnn.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5457–5466, 2018.
  * [158] Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, and Qing Li.  Recurrent attention networks for long-text modeling.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 3006–3019. Association for Computational Linguistics, 2023.
  * [159] Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey.  What makes convolutional models great on long sequence modeling?  arXiv preprint arXiv:2210.09298, 2022.
  * [160] Zhong Li, Jiequn Han, Weinan E, and Qianxiao Li.  Approximation and optimization theory for linear continuous-time recurrent neural networks.  The Journal of Machine Learning Research, 23(1):1997–2081, 2022.
  * [161] Zhong Li, Jiequn Han, Qianxiao Li, et al.  On the curse of memory in recurrent neural networks: Approximation and optimization analysis.  arXiv preprint arXiv:2009.07799, 2020.
  * [162] Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard Zemel.  Reviving and improving recurrent back-propagation.  In International Conference on Machine Learning, pages 3082–3091. PMLR, 2018.
  * [163] Timothy P Lillicrap and Adam Santoro.  Backpropagation through time and the brain.  Current opinion in neurobiology, 55:82–89, 2019.
  * [164] Soon Hoe Lim, N. Benjamin Erichson, Liam Hodgkinson, and Michael W Mahoney.  Noisy recurrent neural networks.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5124–5137. Curran Associates, Inc., 2021.
  * [165] Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim.  Parallelizing non-linear sequential models over the sequence length.  arXiv preprint arXiv:2309.12252, 2023.
  * [166] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.  A survey of transformers.  AI Open, 3:111–132, 2022.
  * [167] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre.  Stable and expressive recurrent vision models.  Advances in Neural Information Processing Systems, 33:10456–10467, 2020.
  * [168] Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre.  Learning long-range spatial dependencies with horizontal gated recurrent units.  Advances in neural information processing systems, 31, 2018.
  * [169] Zachary C Lipton, John Berkowitz, and Charles Elkan.  A critical review of recurrent neural networks for sequence learning.  arXiv preprint arXiv:1506.00019, 2015.
  * [170] Jing Liu, Zizheng Pan, Haoyu He, Jianfei Cai, and Bohan Zhuang.  Ecoformer: Energy-saving attention with linear complexity.  Advances in Neural Information Processing Systems, 35:10295–10308, 2022.
  * [171] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.  Mega: Moving average equipped gated attention.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  * [172] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  * [173] Matthew MacKay, Paul Vicol, Jimmy Ba, and Roger B Grosse.  Reversible recurrent neural networks.  Advances in Neural Information Processing Systems, 31, 2018.
  * [174] Michael C Mackey and Leon Glass.  Oscillation and chaos in physiological control systems.  Science, 197(4300):287–289, 1977.
  * [175] Marco Maggini, Matteo Tiezzi, and Marco Gori.  A lagrangian framework for learning in graph neural networks.  In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pages 343–365. Elsevier, 2024.
  * [176] Matt Mahoney.  Large text compression benchmark, 2011.
  * [177] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner.  Online continual learning in image classification: An empirical survey.  Neurocomputing, 469:28–51, 2022.
  * [178] Danilo P. Mandic and Jonathon A. Chambers.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability, volume 130.  John Wiley & Sons, 2001.
  * [179] Huanru Henry Mao.  Fine-tuning pre-trained transformers into decaying fast weights.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10236–10242. Association for Computational Linguistics, 2022.
  * [180] Huanru Henry Mao.  Fine-tuning pre-trained transformers into decaying fast weights.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10236–10242, 2022.
  * [181] Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.  Building a large annotated corpus of english: The penn treebank.  1993\.
  * [182] Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter.  Nvidia tensor core programmability, performance & precision.  In 2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW), pages 522–531. IEEE, 2018.
  * [183] Giuseppe Marra, Matteo Tiezzi, Stefano Melacci, Alessandro Betti, Marco Maggini, and Marco Gori.  Local propagation in constraint-based neural networks.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2020.
  * [184] Owen Marschall, Kyunghyun Cho, and Cristina Savin.  A unified framework of online learning algorithms for training recurrent neural networks.  The Journal of Machine Learning Research, 21(1):5320–5353, 2020.
  * [185] Eric Martin and Chris Cundy.  Parallelizing linear recurrent neural nets over sequence length.  In International Conference on Learning Representations, 2018.
  * [186] Stefano Massaroli, Michael Poli, Daniel Y Fu, Hermann Kumbong, Rom N Parnichkun, Aman Timalsina, David W Romero, Quinn McIntyre, Beidi Chen, Atri Rudra, et al.  Laughing hyena distillery: Extracting compact recurrences from convolutions.  arXiv preprint arXiv:2310.18780, 2023.
  * [187] Warren S McCulloch and Walter Pitts.  A logical calculus of the ideas immanent in nervous activity.  The bulletin of mathematical biophysics, 5:115–133, 1943.
  * [188] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio.  Samplernn: An unconditional end-to-end neural audio generation model.  arXiv preprint arXiv:1612.07837, 2016.
  * [189] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur.  Long range language modeling via gated state spaces.  arXiv preprint arXiv:2206.13947, 2022.
  * [190] Enrico Meloni, Alessandro Betti, Lapo Faggi, Simone Marullo, Matteo Tiezzi, and Stefano Melacci.  Evaluating continual learning algorithms by generating 3d virtual environments.  In International Workshop on Continual Semi-Supervised Learning, pages 62–74. Springer, 2021.
  * [191] Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, and Alex Graves.  A practical sparse approximation for real time recurrent learning.  arXiv preprint arXiv:2006.07232, 2020.
  * [192] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  arXiv preprint arXiv:1609.07843, 2016.
  * [193] Alessio Micheli.  Neural network for graphs: A contextual constructive approach.  IEEE Transactions on Neural Networks, 20(3):498–511, 2009.
  * [194] Alessio Micheli and Domenico Tortorella.  Discrete-time dynamic graph echo state networks.  Neurocomputing, 496:85–95, 2022.
  * [195] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  arXiv preprint arXiv:1809.02789, 2018.
  * [196] Tomas Mikolov and Geoffrey Zweig.  Context dependent recurrent neural network language model.  In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234–239. IEEE, 2012.
  * [197] Beren Millidge, Alexander Tschantz, and Christopher L. Buckley.  Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs.  Neural Computation, 34(6):1329–1368, 05 2022.
  * [198] De Mori et al.  Bps: A learning algorithm for capturing the dynamic nature of speech.  In International 1989 Joint Conference on Neural Networks, pages 417–423. IEEE, 1989.
  * [199] James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons.  Neural controlled differential equations for online prediction tasks.  arXiv preprint arXiv:2106.11028, 2021.
  * [200] Michael C Mozer.  A focused backpropagation algorithm for temporal pattern recognition.  In Backpropagation, pages 137–169. Psychology Press, 2013.
  * [201] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal.  Leave no context behind: Efficient infinite context transformers with infini-attention.  arXiv preprint arXiv:2404.07143, 2024.
  * [202] Nikita Nangia and Samuel R Bowman.  Listops: A diagnostic dataset for latent tree learning.  arXiv preprint arXiv:1804.06028, 2018.
  * [203] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al.  Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution.  Advances in neural information processing systems, 36, 2024.
  * [204] Thomas Norrie, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi, and David Patterson.  The design process for google’s training chips: Tpuv2 and tpuv3.  IEEE Micro, 41(2):56–63, 2021.
  * [205] Junier Oliva, William Neiswanger, Barnabás Póczos, Eric Xing, Hy Trac, Shirley Ho, and Jeff Schneider.  Fast function to function regression.  In Artificial Intelligence and Statistics, pages 717–725. PMLR, 2015.
  * [206] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al.  In-context learning and induction heads.  arXiv preprint arXiv:2209.11895, 2022.
  * [207] Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz.  Transformers are multi-state rnns.  arXiv preprint arXiv:2401.06104, 2024.
  * [208] AG Ororbia, A Mali, D Kifer, and C Lee Giles.  Deep credit assignment by aligning local representations.  arXiv preprint arXiv:1803.01834, 2018.
  * [209] Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer.  Continual learning of recurrent neural networks by locally aligning distributed representations.  IEEE Transactions on Neural Networks and Learning Systems, 31(10):4267–4278, 2020.
  * [210] Alexander G Ororbia.  Brain-inspired machine intelligence: A survey of neurobiologically-plausible credit assignment.  arXiv preprint arXiv:2312.09257, 2023.
  * [211] Alexander G Ororbia and Ankur Mali.  Biologically motivated algorithms for propagating local target representations.  In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4651–4658, 2019.
  * [212] Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith.  On the universality of linear recurrences followed by nonlinear projections.  arXiv preprint arXiv:2307.11888, 2023.
  * [213] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De.  Resurrecting recurrent neural networks for long sequences.  In International Conference on Machine Learning, pages 26670–26698. PMLR, 2023.
  * [214] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The lambada dataset: Word prediction requiring a broad discourse context.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, 2016.
  * [215] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [216] German I Parisi and Vincenzo Lomonaco.  Online continual learning on sequences.  In Recent Trends in Learning From Data: Tutorials from the INNS Big Data and Deep Learning Conference (INNSBDDL2019), pages 197–221. Springer, 2020.
  * [217] Il Memming Park, Ábel Ságodi, and Piotr Aleksander Sokół.  Persistent learning signals and working memory without continuous attractors.  arXiv preprint arXiv:2308.12585, 2023.
  * [218] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  arXiv preprint arXiv:1802.05751, 2018.
  * [219] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.  On the difficulty of training recurrent neural networks.  In International conference on machine learning, pages 1310–1318. Pmlr, 2013.
  * [220] Barak A. Pearlmutter.  Learning state space trajectories in recurrent neural networks.  Neural Comput., 1(2):263–269, 1989.
  * [221] Barak A. Pearlmutter.  Gradient calculations for dynamic recurrent neural networks: a survey.  IEEE Trans. Neural Networks, 6(5):1212–1228, 1995.
  * [222] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al.  Rwkv: Reinventing rnns for the transformer era.  arXiv preprint arXiv:2305.13048, 2023.
  * [223] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, et al.  Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence.  arXiv preprint arXiv:2404.05892, 2024.
  * [224] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong.  Random feature attention.  In International Conference on Learning Representations, 2021.
  * [225] Jonathan Pilault, Mahan Fathi, Orhan Firat, Christopher Pal, Pierre-Luc Bacon, and Ross Goroshin.  Block-state transformers.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [226] Fernando J Pineda.  Generalization of back-propagation to recurrent neural networks.  Physical review letters, 59(19):2229, 1987.
  * [227] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré.  Hyena hierarchy: Towards larger convolutional language models.  In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28043–28078. PMLR, 2023.
  * [228] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean.  Efficiently scaling transformer inference.  Proceedings of Machine Learning and Systems, 5, 2023.
  * [229] Subhojeet Pramanik, Esraa Elelimy, Marlos C. Machado, and Adam White.  Recurrent linear transformers, 2023.
  * [230] Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong.  The devil in linear transformer.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025–7041, 2022.
  * [231] Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, et al.  Scaling transnormer to 175 billion parameters.  arXiv preprint arXiv:2307.14995, 2023.
  * [232] Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong.  Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models.  arXiv preprint arXiv:2401.04658, 2024.
  * [233] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong.  cosformer: Rethinking softmax in attention.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [234] Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng, Dongxu Li, Xiaodong Han, Yuchao Dai, Lingpeng Kong, and Yiran Zhong.  Linearized relative positional encoding.  arXiv preprint arXiv:2307.09270, 2023.
  * [235] Zhen Qin, Songlin Yang, and Yiran Zhong.  Hierarchically gated recurrent neural network for sequence modeling.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [236] Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara.  The acl anthology network corpus.  Language Resources and Evaluation, 47(4):919–944, 2013.
  * [237] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  In International Conference on Learning Representations, 2019.
  * [238] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  arXiv preprint, 2019.
  * [239] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  Advances in neural information processing systems, 20, 2007.
  * [240] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions, 2017.
  * [241] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In International conference on learning representations, 2016.
  * [242] Ankit Singh Rawat, Jiecao Chen, Felix Xinnan X Yu, Ananda Theertha Suresh, and Sanjiv Kumar.  Sampled softmax with random fourier features.  Advances in Neural Information Processing Systems, 32, 2019.
  * [243] Matthew A Reyna, Christopher S Josef, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Shamim Nemati, Gari D Clifford, and Ashish Sharma.  Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019.  Critical care medicine, 48(2):210–217, 2020.
  * [244] David W Romero, Anna Kuzina, Erik J Bekkers, Jakub Mikolaj Tomczak, and Mark Hoogendoorn.  Ckconv: Continuous kernel convolution for sequential data.  In International Conference on Learning Representations, 2021.
  * [245] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud.  Latent ordinary differential equations for irregularly-sampled time series.  Advances in neural information processing systems, 32, 2019.
  * [246] Walter Rudin et al.  Principles of mathematical analysis, volume 3.  McGraw-hill New York, 1976.
  * [247] David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al.  Learning internal representations by error propagation, 1985.
  * [248] T Konstantin Rusch and Siddhartha Mishra.  Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies.  In International Conference on Learning Representations, 2020.
  * [249] T Konstantin Rusch and Siddhartha Mishra.  Unicornn: A recurrent model for learning very long time dependencies.  In International Conference on Machine Learning, pages 9168–9178. PMLR, 2021.
  * [250] T Konstantin Rusch, Siddhartha Mishra, N Benjamin Erichson, and Michael W Mahoney.  Long expressive memory for sequence modeling.  In International Conference on Learning Representations, 2021.
  * [251] Michael S Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arnab.  Token turing machines.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19070–19081, 2023.
  * [252] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  Communications of the ACM, 64(9):99–106, 2021.
  * [253] Hojjat Salehinejad, Sharan Sankar, Joseph Barfett, Errol Colak, and Shahrokh Valaee.  Recent advances in recurrent neural networks.  arXiv preprint arXiv:1801.01078, 2017.
  * [254] Rajiv Sambharya.  Lifted recurrent neural networks.  2018\.
  * [255] Simone Scardapane, Claudio Gallicchio, Alessio Micheli, and Miguel C. Soriano.  Guest editorial: Trends in reservoir computing.  Cognitive Computation, 15(5):1407–1408, 2023.
  * [256] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.  The graph neural network model.  IEEE transactions on neural networks, 20(1):61–80, 2008.
  * [257] Anton Maximilian Schäfer and Hans Georg Zimmermann.  Recurrent neural networks are universal approximators.  In Artificial Neural Networks–ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006. Proceedings, Part I 16, pages 632–640. Springer, 2006.
  * [258] Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph.  Modeling irregular time series with continuous recurrent units.  In International Conference on Machine Learning, pages 19388–19405. PMLR, 2022.
  * [259] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.  Linear transformers are secretly fast weight programmers.  In International Conference on Machine Learning, pages 9355–9366. PMLR, 2021.
  * [260] Juergen Schmidhuber.  26 March 1991: Neural nets learn to program neural nets with fast weights—like today’s Transformer variants. 2021: New stuff!, AI Blog, 2021.
  * [261] Jürgen Schmidhuber.  Learning to control fast-weight memories: An alternative to dynamic recurrent networks.  Neural Computation, 4(1):131–139, 1992.
  * [262] Jürgen Schmidhuber.  A ‘self-referential’weight matrix.  In ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13–16 September 1993 3, pages 446–450. Springer, 1993.
  * [263] Javier Selva, Anders S Johansen, Sergio Escalera, Kamal Nasrollahi, Thomas B Moeslund, and Albert Clapés.  Video transformers: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [264] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016.
  * [265] Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville.  Ordered neurons: Integrating tree structures into recurrent neural networks.  In International Conference on Learning Representations, 2018.
  * [266] Jerome Sieber, Carmen Amo Alonso, Alexandre Didier, Melanie N Zeilinger, and Antonio Orvieto.  Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks.  arXiv preprint arXiv:2405.15731, 2024.
  * [267] Hava T Siegelmann.  Neural networks and analog computation: beyond the Turing limit.  Springer Science & Business Media, 2012.
  * [268] Hava T Siegelmann and Eduardo D Sontag.  On the computational power of neural nets.  In Proceedings of the fifth annual workshop on Computational learning theory, pages 440–449, 1992.
  * [269] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 510–526. Springer, 2016.
  * [270] Herbert A Simon and Kenneth Kotovsky.  Human acquisition of concepts for sequential patterns.  Psychological review, 70(6):534, 1963.
  * [271] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman.  Simplified state space layers for sequence modeling.  arXiv preprint arXiv:2208.04933, 2022.
  * [272] David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le.  Searching for efficient transformers for language modeling.  Advances in neural information processing systems, 34:6010–6022, 2021.
  * [273] Shagun Sodhani, Sarath Chandar, and Yoshua Bengio.  Toward training recurrent neural networks for lifelong learning.  Neural computation, 32(1):1–35, 2020.
  * [274] Alessandro Sperduti and Antonina Starita.  Supervised neural networks for the classification of structures.  IEEE transactions on neural networks, 8(3):714–735, 1997.
  * [275] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  In International conference on machine learning, pages 843–852. PMLR, 2015.
  * [276] Felix Stahlberg.  Neural machine translation: A review.  Journal of Artificial Intelligence Research, 69:343–418, 2020.
  * [277] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei.  Retentive network: A successor to transformer for large language models.  arXiv preprint arXiv:2307.08621, 2023.
  * [278] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei.  A length-extrapolatable transformer.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14590–14604. Association for Computational Linguistics, 2023.
  * [279] Corentin Tallec and Yann Ollivier.  Unbiased online recurrent optimization.  arXiv preprint arXiv:1702.05043, 2017.
  * [280] Corentin Tallec and Yann Ollivier.  Can recurrent neural networks warp time?  In International Conference on Learning Representation 2018, 2018.
  * [281] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I Webb.  Monash university, uea, ucr time series regression archive.  arXiv preprint arXiv:2006.10996, 2020.
  * [282] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  arXiv preprint arXiv:2011.04006, 2020.
  * [283] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  ACM Comput. Surv., 55(6):109:1–109:28, 2023.
  * [284] Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein.  Training neural networks without gradients: A scalable admm approach.  In International conference on machine learning, pages 2722–2731. PMLR, 2016.
  * [285] Matteo Tiezzi, Giuseppe Marra, Stefano Melacci, and Marco Maggini.  Deep constraint-based propagation in graph neural networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(2):727–739, 2021.
  * [286] Matteo Tiezzi, Giuseppe Marra, Stefano Melacci, Marco Maggini, and Marco Gori.  A lagrangian approach to information propagation in graph neural networks.  In ECAI 2020, pages 1539–1546. IOS Press, 2020.
  * [287] Matteo Tiezzi, Simone Marullo, Lapo Faggi, Enrico Meloni, Alessandro Betti, and Stefano Melacci.  Stochastic coherence over attention trajectory for continuous learning in video streams.  In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3480–3486. International Joint Conferences on Artificial Intelligence Organization, 7 2022.  Main Track.
  * [288] Matteo Tiezzi, Stefano Melacci, Alessandro Betti, Marco Maggini, and Marco Gori.  Focus of attention improves information transfer in visual features.  Advances in Neural Information Processing Systems, 33:22194–22204, 2020.
  * [289] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  * [290] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4344–4353, 2019.
  * [291] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [292] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29, 2016.
  * [293] Aaron Voelker, Ivana Kajić, and Chris Eliasmith.  Legendre memory units: Continuous-time representation in recurrent neural networks.  Advances in neural information processing systems, 2019.
  * [294] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In International Conference on Learning Representations, 2018.
  * [295] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  arXiv preprint arXiv:2006.04768, 2020.
  * [296] Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang.  R-transformer: Recurrent neural network enhanced transformer.  arXiv preprint arXiv:1907.05572, 2019.
  * [297] Pete Warden.  Speech commands: A dataset for limited-vocabulary speech recognition.  arXiv preprint arXiv:1804.03209, 2018.
  * [298] Paul J Werbos.  Generalization of backpropagation with application to a recurrent gas market model.  Neural networks, 1(4):339–356, 1988.
  * [299] Paul J Werbos.  Backpropagation through time: what it does and how to do it.  Proceedings of the IEEE, 78(10):1550–1560, 1990.
  * [300] James CR Whittington and Rafal Bogacz.  An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity.  Neural computation, 29(5):1229–1262, 2017.
  * [301] Bernard Widrow, Marcian E Hoff, et al.  Adaptive switching circuits.  In IRE WESCON convention record, volume 4, pages 96–104. New York, 1960.
  * [302] Ronald J Williams and Jing Peng.  An efficient gradient-based algorithm for on-line training of recurrent network trajectories.  Neural computation, 2(4):490–501, 1990.
  * [303] Ronald J Williams and David Zipser.  A learning algorithm for continually running fully recurrent neural networks.  Neural computation, 1(2):270–280, 1989.
  * [304] Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu.  Memformer: A memory-augmented transformer for sequence modeling.  In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 308–318, 2022.
  * [305] Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy.  Memorizing transformers.  arXiv preprint arXiv:2203.08913, 2022.
  * [306] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [307] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * [308] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * [309] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim.  Gated linear attention transformers with hardware-efficient training, 2023.
  * [310] Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang.  A review of recurrent neural networks: Lstm cells and network architectures.  Neural computation, 31(7):1235–1270, 2019.
  * [311] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  arXiv preprint arXiv:1808.05326, 2018.
  * [312] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019.
  * [313] Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind.  An attention free transformer.  arXiv preprint arXiv:2105.14103, 2021.
  * [314] Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Re.  Effectively modeling time series with simple discrete state spaces.  In The Eleventh International Conference on Learning Representations, 2022.
  * [315] Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Ré.  Effectively modeling time series with simple discrete state spaces.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  * [316] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * [317] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015.
  * [318] Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, and Chunhua Shen.  A survey on efficient training of transformers.  arXiv preprint arXiv:2302.01107, 2023.
  * [319] Nicolas Zucchet, Seijin Kobayashi, Yassir Akram, Johannes von Oswald, Maxime Larcher, Angelika Steger, and João Sacramento.  Gated recurrent neural networks discover attention, 2023.
  * [320] Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and João Sacramento.  Online learning of long-range dependencies.  Advances in Neural Information Processing Systems, 36:10477–10493, 2023.