# Going Beyond Linear Transformers
with Recurrent Fast Weight Programmers

Kazuki Irie111footnotemark: 1 , Imanol Schlag1 , Róbert Csordás1, Jürgen
Schmidhuber1,2
1The Swiss AI Lab, IDSIA, University of Lugano (USI) & SUPSI, Lugano,
Switzerland
2King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi
Arabia
{kazuki, imanol, robert<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Transformers with linearised attention (“linear Transformers”) have
demonstrated the practical scalability and effectiveness of outer product-
based Fast Weight Programmers (FWPs) from the ’90s. However, the original FWP
formulation is more general than the one of linear Transformers: a slow neural
network (NN) continually reprograms the weights of a fast NN with arbitrary
architecture. In existing linear Transformers, both NNs are feedforward and
consist of a single layer. Here we explore new variations by adding recurrence
to the slow and fast nets. We evaluate our novel recurrent FWPs (RFWPs) on two
synthetic algorithmic tasks (code execution and sequential ListOps),
Wikitext-103 language models, and on the Atari 2600 2D game environment. Our
models exhibit properties of Transformers and RNNs. In the reinforcement
learning setting, we report large improvements over LSTM in several Atari
games. Our code is public.111https://github.com/IDSIA/recurrent-fwp

##  1 Introduction

The Transformer [1] has become one of the most popular neural networks (NNs)
for processing sequential data. Its success on neural machine translation
quickly transferred to other problems in natural language processing, such as
language modelling [2, 3] or question answering [4]. Recently, it has also
been applied in other domains, including image processing [5, 6] or
mathematical problem solving [7, 8, 9].

Conceptually, the Transformer is a deep feedforward NN that processes all
elements of a sequence in parallel: unlike in recurrent NNs (RNNs), the
computations of a layer for the entire sequence can be packed into one big
matrix multiplication. This scales well with the number of parallel
processors.

Despite the benefits of parallelisation, a major drawback of Transformers is
that their computational complexity in time and space is quadratic in sequence
length. Furthermore, in the auto-regressive version [1, 2] — the focus of our
work — the state size increases linearly with sequence length. This makes
Transformers infeasible for auto-regressive settings dealing with very long or
potentially infinite sequences, forcing practitioners to truncate temporal
contexts and ignore long-term dependencies beyond fixed-size time windows.
Although recent work tries to address this issue [10, 11], this limitation
makes some applications of Transformers challenging, e.g., reinforcement
learning (RL) in partially observable environments [12, 13], which is still
dominated by RNNs such as the Long Short-Term Memory (LSTM; [14]) trained by
policy gradients [15, 16, 17, 18].

To scale Transformers to longer sequences, recent works have proposed to
linearise the softmax in the self-attention computation and reorganise the
latter in a sequential way [19]. Such models include Katharopoulos et al.’s
Linear Transformer (LT) [19], Choromanski et al.’s Performer [20] and Peng et
al. [21]’s variant. They enjoy time and space complexities linear in sequence
length with states of constant size. While their performance on some tasks
does not fully match the one of regular Transformers [22], several
improvements have already been proposed [21, 23] (see our review in Sec. 2.2)
which makes this Transformer family a promising alternative.

Here we go one step further in advancing linear Transformer variants as
powerful auto-regressive sequence processing models, adopting the perspective
of “Fast Weight Programmers” (FWPs) [24, 25, 26]. Recent work emphasised that
linearised Transformers are essentially equivalent to outer product-based FWPs
from the ’90s ([23]; reviewed in Sec. 2). Here we explore this connection
further and describe more powerful FWPs.

The original FWP [24] is a two-NN system: a slow and a fast net, each with
arbitrary architectures. The slow net learns to generate rapid context-
dependent weight modifications for the fast net. In the case of existing
linear Transformer variants, the slow and fast nets are simple one layer
feedforward NNs. Here we augment them with recurrent connections to obtain
recurrent FWPs (RFWPs). Recurrence enhances the model’s theoretical power [27]
and can help to solve tasks that naturally require recurrence as a part of the
solution.

Our experiments on the language modelling dataset Wikitext-103 [28] show that
our RFWPs are competitive compared to regular Transformers. We then study
various properties of the proposed models on two synthetic algorithmic tasks:
code execution [29] and sequential ListOps [30]. Finally, it is
straightforward to apply our models to RL problems as a drop-in replacement
for LSTMs. Here our RFWPs obtain large improvements over LSTM baselines across
many Atari 2600 2D game environments [31]. Although LSTM still works better in
a few environments, we show that our RFWPs generally improve by scaling them
up.

The main contribution of this work is twofold: (1) from the perspective of
FWPs, we study novel powerful FWPs for sequence processing, demonstrating that
NNs can easily learn to control NNs that are more complex than a single
feedforward layer, and (2) from the perspective of Transformer models, our
RFWPs augment linear Transformers with recurrence, addressing general
limitations of existing auto-regressive Transformer models.

##  2 Background on Fast Weight Programmers (FWPs)

Here we review the general concept of FWPs, as well as two specific instances
thereof: the linear Transformer [19, 20] and the Delta Net [23].

###  2.1 General Formulation

We refresh the concept of fast weight controllers or FWPs [24, 25] using
modern notation in a sequence processing scenario. An FWP with trainable
parameters ${\bm{\theta}}_{\text{slow}}$ sequentially transforms an input
sequence $\\{{\bm{x}}^{(t)}\\}_{t=1}^{T}$ with
${\bm{x}}^{(t)}\in\mathbb{R}^{d_{\text{in}}}$ to an output sequence
$\\{{\bm{y}}^{(t)}\\}_{t=1}^{T}$ with
${\bm{y}}^{(t)}\in\mathbb{R}^{d_{\text{out}}}$ of length $T$ as

$\displaystyle{\bm{\theta}}_{\text{fast}}^{(t)},{\bm{q}}^{(t)}$
$\displaystyle=\mathtt{SlowNet}\big{(}\\{{\bm{x}}^{(j)}\\}_{j=1}^{t},\\{{\bm{y}}^{(j)}\\}_{j=0}^{t-1},\\{{\bm{\theta}}^{(j)}_{\text{fast}}\\}_{j=0}^{t-1},\\{{\bm{q}}^{(j)}\\}_{j=0}^{t-1};{\bm{\theta}}_{\text{slow}}\big{)}$
(1) $\displaystyle{\bm{y}}^{(t)}$
$\displaystyle=\mathtt{FastNet}(\\{{\bm{q}}^{(j)}\\}_{j=1}^{t},\\{{\bm{y}}^{(j)}\\}_{j=0}^{t-1};{\bm{\theta}}^{(t)}_{\text{fast}})$
(2)

where ${\bm{y}}^{(0)}$, ${\bm{\theta}}^{(0)}_{\text{fast}}$, and
${\bm{q}}^{(0)}$ are initial variables. This is a system with two NNs called
$\mathtt{FastNet}$ and $\mathtt{SlowNet}$ in which the parameters
${\bm{\theta}}_{\text{fast}}^{(t)}$ of $\mathtt{FastNet}$ are generated by
$\mathtt{SlowNet}$ at each time step $t$. The weights of the fast net are fast
in the sense that they may rapidly change at every step of the sequence while
the weights of the slow net ${\bm{\theta}}_{\text{slow}}$ are slow because
they can only change through gradient descent during training, remaining fixed
afterwards111The fast net could also contain some additional slow weights; we
omit this possibility here.. Eq. 1 expresses a slow NN in its general form.
The slow net can generate fast weights conditioned on various variables,
depending on architectural choices for the slow and fast NNs. In addition to
the fast weights ${\bm{\theta}}_{\text{fast}}^{(t)}$, the slow net also
generates or invents an input ${\bm{q}}^{(t)}$ to be fed to the fast net
(alternatively ${\bm{q}}^{(t)}$ can simply be ${\bm{x}}^{(t)}$). While the
architectures of slow and fast nets are arbitrary, they are typically chosen
to be differentiable such that the entire FWP can be trained in an end-to-end
manner using gradient descent. By interpreting the weights of an NN as a
program [32], the slow net effectively learns to control, or program, the fast
NN. Thus, the slow net is a neural programmer of fast weights, and its
parameter set ${\bm{\theta}}_{\text{slow}}$ embodies compressed information
used to produce potentially infinite variations of context-dependent fast
weights.

In many settings, it makes sense to generate the fast weights
${\bm{\theta}}_{\text{fast}}^{(t)}$ incrementally in an iterative fashion,
where the $\mathtt{SlowNet}$ is further decomposed into two sub-parts:

$\displaystyle{\bm{z}}^{(t)},{\bm{q}}^{(t)}$
$\displaystyle=\mathtt{SlowSubnet}(\\{{\bm{x}}^{(j)}\\}_{j=1}^{t},\\{{\bm{y}}^{(j)}\\}_{j=0}^{t-1},\\{{\bm{\theta}}^{(j)}_{\text{fast}}\\}_{j=0}^{t-1},\\{{\bm{q}}^{(j)}\\}_{j=0}^{t-1},\\{{\bm{z}}^{(j)}\\}_{j=0}^{t-1};{\bm{\theta}}_{\text{slow}})$
(3) $\displaystyle{\bm{\theta}}^{(t)}_{\text{fast}}$
$\displaystyle=\mathtt{UpdateRule}({\bm{\theta}}^{(t-1)}_{\text{fast}},{\bm{z}}^{(t)})$
(4)

where $\mathtt{UpdateRule}$ takes the fast weights
${\bm{\theta}}^{(t-1)}_{\text{fast}}$ from the previous iteration to produce
the new fast weights ${\bm{\theta}}^{(t)}_{\text{fast}}$ conditioned on
${\bm{z}}^{(t)}$. The update rule is essentially the differentiable elementary
programming instruction used by the FWP. In the next section we review
concrete examples of recent FWPs.

###  2.2 Linear Transformers as Fast Weight Programmers

In general, the dimension of the fast weights
${\bm{\theta}}^{(t)}_{\text{fast}}$ is too large to be conveniently
parameterised by an NN. Instead, it was proposed in 1991 [24] to perform a
rank-one update via the outer product of two vectors generated by the slow
net. Two recent models directly correspond to such outer product-based FWPs:
linear Transformers [19] and the Delta Net [23].

#### Linear Transformer.

The “linear Transformer” [19] is a class of Transformers where the softmax in
the attention is linearised. This is achieved by replacing the softmax with a
kernel function $\phi$—then the self-attention can be rewritten as a basic
outer product-based FWP [24, 23]. Previous works focused on different $\phi$
maps with properties such as increased capacity [23] or guaranteed
approximation of the softmax in the limit [20, 21]. For our purposes, the
particular choice of $\phi$ is irrelevant and we simply assume
$\phi:\mathbb{R}^{d_{\text{key}}}\rightarrow\mathbb{R}^{d_{\text{key}}}$,
simplifying our equations below by writing ${\bm{k}},{\bm{q}}$ instead of
$\phi({\bm{k}}),\phi({\bm{q}})$. Using otherwise the same notation as above,
for each new input ${\bm{x}}^{(t)}$, the output ${\bm{y}}^{(t)}$ is obtained
by:

$\displaystyle{\bm{k}}^{(t)},{\bm{v}}^{(t)},{\bm{q}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}_{k}{\bm{x}}^{(t)},{\bm{W}}_{v}{\bm{x}}^{(t)},{\bm{W}}_{q}{\bm{x}}^{(t)}$
(5) $\displaystyle{\bm{W}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}^{(t-1)}+{\bm{v}}^{(t)}\otimes{\bm{k}}^{(t)}$ (6)
$\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}^{(t)}{\bm{q}}^{(t)}$ (7)

where the slow weight matrices
${\bm{W}}_{k}\in\mathbb{R}^{d_{\text{key}}\times d_{\text{in}}}$ and
${\bm{W}}_{v}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ are used to
obtain the key ${\bm{k}}^{(t)}\in\mathbb{R}^{d_{\text{key}}}$ and the value
${\bm{v}}^{(t)}\in\mathbb{R}^{d_{\text{out}}}$. The key and value vectors are
used to generate new weights via the outer product
${\bm{v}}^{(t)}\otimes{\bm{k}}^{(t)}\in\mathbb{R}^{d_{\text{out}}\times
d_{\text{key}}}$. A further simplification in the equations above is the
omission of attention normalisation which has been experimentally shown to be
unnecessary if the $\phi$ function produces normalised key and query vectors
[23].

In Eq. 6, the previous fast weight matrix
${\bm{W}}^{(t-1)}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{key}}}$ is
updated to yield ${\bm{W}}^{(t)}$ by adding the update term
${\bm{v}}^{(t)}\otimes{\bm{k}}^{(t)}$. This corresponds to the sum update rule
or purely additive programming instruction. Here the fast NN is a simple
linear transformation as in Eq. 7 which takes as input the query vector
${\bm{q}}^{(t)}\in\mathbb{R}^{d_{\text{key}}}$ generated by the slow weights
${\bm{W}}_{q}\in\mathbb{R}^{d_{\text{key}}\times d_{\text{in}}}$. Hence, in
linear Transformers, the previous Eq. 3 simplifies to:
${\bm{z}}^{(t)},{\bm{q}}^{(t)}=\mathtt{SlowSubnet}({\bm{x}}^{(t)};{\bm{\theta}}_{\text{slow}})$
with ${\bm{z}}^{(t)}=({\bm{k}}^{(t)},{\bm{v}}^{(t)})$.

#### Delta Net.

The Delta Net [23] is obtained by replacing the purely additive programming
instruction (Eq. 6) in the linear Transformer with the one akin to the delta
rule [33]:

$\displaystyle{\bm{W}}^{(t)}$
$\displaystyle={\bm{W}}^{(t-1)}+\beta^{(t)}({\bm{v}}^{(t)}-\bar{{\bm{v}}}^{(t)})\otimes{\bm{k}}^{(t)}$
(8)

where $\beta^{(t)}\in\mathbb{R}$ is a fast parameter (learning rate) of the
update rule generated by the slow net with weights
${\bm{W}}_{\beta}\in\mathbb{R}^{1\times d_{\text{in}}}$ and the sigmoid
function $\sigma$:

$\displaystyle\beta^{(t)}$
$\displaystyle=\sigma({\bm{W}}_{\beta}{\bm{x}}^{(t)})$ (9)

and $\bar{{\bm{v}}}^{(t)}\in\mathbb{R}^{d_{\text{out}}}$ is generated as a
function of the previous fast weights ${\bm{W}}^{(t-1)}$ and the key
${\bm{k}}^{(t)}$

$\displaystyle\bar{{\bm{v}}}^{(t)}$
$\displaystyle={\bm{W}}^{(t-1)}{\bm{k}}^{(t)}.$ (10)

This update rule was introduced to address a memory capacity problem affecting
linear Transformers with the purely additive update rule [23]. The
corresponding Eq. 3 is:
${\bm{z}}^{(t)},{\bm{q}}^{(t)}=\mathtt{SlowSubnet}({\bm{x}}^{(t)},{\bm{W}}^{(t-1)};{\bm{\theta}}_{\text{slow}})$
with
${\bm{z}}^{(t)}=({\bm{k}}^{(t)},{\bm{v}}^{(t)},\beta^{(t)},\bar{{\bm{v}}}^{(t)})$.
Thus, unlike linear Transformers, the $\mathtt{SlowNet}$ in the Delta Net
takes the previous fast weights ${\bm{W}}^{(t-1)}$ into account to generate
the new fast weight updates.

We typically use the multi-head version [1] of the computations above. After
the projection (Eq. 5), the vectors ${\bm{k}}^{(t)}$, ${\bm{v}}^{(t)}$,
${\bm{q}}^{(t)}$ are split into equally sized $H$ sub-vectors, and the rest of
the operations are conducted by $H$ computational heads independently. The
resulting output vectors from each head are concatenated to form the final
output.

#### Other approaches.

While our focus here is on outer product-based weight generation, which is an
efficient method to handle high dimensional NN weights, there are also other
approaches. For example, instead of generating a new weight matrix,
Hypernetworks [34] scale the rows of a slow weight matrix with a generated
vector of appropriate size. Weight compression to control fast weights in a
low dimensional compressed space has been also studied [35]. In the broad
sense of context-dependent weights [36, 37, 38], many concepts relate to FWPs:
e.g. dynamic convolution [39, 40, 41], LambdaNetworks [42], or dynamic
plasticity [43, 44].

##  3 Fast Weight Programmers With Slow or Fast RNNs

The original formulation of FWPs reviewed in Sec. 2.1 is more general than
existing models presented in Sec. 2.2. In particular, both fast and slow
networks in existing linear Transformers consist of a single feedforward layer
(Eqs. 5 and 7). Here we present FWPs with recurrent fast nets in Sec. 3.1 and
FWPs with recurrent slow nets in Sec. 3.2.

###  3.1 Fast Network Extensions

In principle, any NN architecture can be made fast. Its fast weight version is
obtained by replacing the networks’ weights with fast weights parameterised by
an additional slow network. For example, consider a regular RNN layer with two
weight matrices ${\bm{W}}$ and ${\bm{R}}$:

$\displaystyle{\bm{h}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}{\bm{x}}^{(t)}+{\bm{R}}{\bm{h}}^{(t-1)})$ (11)

A fast weight version can be obtained by replacing ${\bm{W}}$ and ${\bm{R}}$
with ${\bm{W}}^{(t)}$ and ${\bm{R}}^{(t)}$ which are controlled as in Eq. 8
with all necessary variables generated by a separate slow net at each time
step $t$.

While this view illustrates the generality of FWPs, the angle under which we
approach these models is slightly different: we introduce recurrence as a way
of augmenting existing linear Transformers.

#### Delta RNN.

We obtain a fast weight RNN called Delta RNN by adding an additional recurrent
term to the feedforward fast net of the linear Transformer (Eq. 7):

$\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}^{(t)}f({\bm{y}}^{(t-1)})$
(12)

where ${\bm{R}}^{(t)}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{out}}}$ is
an additional fast weight matrix which introduces recurrent connections. It is
also generated by the slow net using the delta update rule, similar to
${\bm{W}}^{(t)}$ in Eq. 8 but with additional slow weights. We apply an
element-wise activation function $f$ to the previous output of the fast
network ${\bm{y}}^{(t-1)}$ to obtain the recurrent query. The choice of
activation function is crucial here because, to achieve stable model
behaviour, the elements in key and query vectors should be positive and sum up
to one when the delta update rule is used [23]. We use the softmax function
($f=\mathrm{softmax}$ in Eq. 12) to satisfy these conditions. An ablation
study on the choice of using Eq. 12 instead of the one similar to Eq. 11 can
be found in Appendix A.2.

Analogous to the Delta RNN, we also construct a Delta LSTM with six fast
weight matrices. The exact equations can be found in Appendix A.2.

#### Alternative Feedforward Fast Nets.

While the focus of this work is on RNNs, there are also interesting fast
feedforward models to be used in Eq. 7 which might result in stronger
feedforward baselines. For example, we can replace the single layer fast net
of Eq. 7 by a $K$-layer deep network:

$\displaystyle{\bm{h}}_{k}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}_{k}^{(t)}f({\bm{h}}_{k-1}^{(t)})\quad\text{for}\enspace
k\in[1..K]\enspace\text{with}\enspace{\bm{h}}_{0}^{(t)}={\bm{q}}^{(t)}$ (13)
$\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{h}}_{K}^{(t)}$ (14)

where the slow network produces all $K$ fast weights
$\\{{\bm{W}}_{k}^{(t)}\\}_{k=1}^{K}$ and query ${\bm{q}}^{(t)}$ from a single
input ${\bm{x}}^{(t)}$. In light of the capacity limitation in linear
Transformers [23], this might introduce additional capacity without the need
of larger representations, analogous to the trade-off in a multilayer
perceptron (MLP) between narrow & deep versus shallow & wide. We refer to this
class of models as Delta MLPs. Again, for stable model behaviour with the
delta rule, we apply the softmax activation $f$ to the vectors to be used as a
query.

Another interesting approach is to use a Delta Net itself as a fast net, i.e.,
make the slow weights in the Delta Net fast (thus obtaining a Delta Delta
Net). Such a model could in principle learn to adapt the way of generating
fast weights depending on the context. While we plan to investigate the
potential of such hierarchical FWPs in future work, we also include
preliminary results of such a model in our language modelling experiments
(Sec. 4.1). A discussion on the dimensionality of such a model can also be
found in Appendix A.3.

We experimentally demonstrate that (slow) NNs can learn to control the weights
of these rather complex fast networks (Sec. 4).

###  3.2 Slow Network Extensions

In linear Transformers, the slow network is purely feedforward (Eq. 5). It can
be made recurrent at two different levels: within the slow network (i.e. the
slow network computes weight updates based on its own previous outputs e.g.,
key, value, query vectors) or via the fast network by taking the fast net’s
previous output as an input. In our preliminary experiments, we found the
former to be sub-optimal (at least in language modelling experiments). So we
focus on the latter approach: we make the slow net in the Delta Net dependent
on the previous output of the fast network. We refer to this model as the
Recurrent Delta Net (RDN).

#### Recurrent Delta Net.

We obtain the RDN by modifying the generation of key, value, and query vectors
(Eq. 5) as well as the learning rate (Eq. 9) in the Delta Net. We add
additional slow weights
(${\bm{R}}_{k},{\bm{R}}_{q}\in\mathbb{R}^{d_{\text{key}}\times
d_{\text{out}}}$, ${\bm{R}}_{v}\in\mathbb{R}^{d_{\text{out}}\times
d_{\text{out}}}$, and ${\bm{R}}_{\beta}\in\mathbb{R}^{1\times
d_{\text{out}}}$) for recurrent connections which connect the previous output
of the fast net ${\bm{y}}^{(t-1)}$ (Eq. 7) to the new ${\bm{k}}^{(t)}$,
${\bm{v}}^{(t)}$, ${\bm{q}}^{(t)}$, and $\beta^{(t)}$ as follows:

$\displaystyle{\bm{k}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}_{k}{\bm{x}}^{(t)}+{\bm{R}}_{k}\tanh({\bm{y}}^{(t-1)})$
(15) $\displaystyle{\bm{v}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}_{v}{\bm{x}}^{(t)}+{\bm{R}}_{v}\tanh({\bm{y}}^{(t-1)})$
(16) $\displaystyle{\bm{q}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}_{q}{\bm{x}}^{(t)}+{\bm{R}}_{q}\tanh({\bm{y}}^{(t-1)})$
(17) $\displaystyle\beta^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{\beta}{\bm{x}}^{(t)}+{\bm{R}}_{\beta}\tanh({\bm{y}}^{(t-1)}))$
(18)

While the rest of the model remains as in the Delta Net, with these simple
extra recurrent connections the model becomes a proper RNN. The corresponding
dependencies in Eq. 3 are:
${\bm{z}}^{(t)},{\bm{q}}^{(t)}=\mathtt{SlowSubnet}({\bm{x}}^{(t)},{\bm{y}}^{(t-1)},{\bm{W}}^{(t-1)};{\bm{\theta}}_{\text{slow}})$
with
${\bm{z}}^{(t)}=({\bm{k}}^{(t)},{\bm{v}}^{(t)},\beta^{(t)},\bar{{\bm{v}}}^{(t)})$.

###  3.3 Related Models

All the RFWP models presented in Sec. 3.1 and 3.2 can be seen as a type of
memory augmented recurrent neural networks [45, 46] in the sense that they
maintain two-dimensional fast weight states as a short-term memory, in
addition to the standard one-dimensional RNN states.

There are also several previously proposed recurrent fast weight models. For
example, Schmidhuber’s recurrent FWP from 1993 [26] has been revisited by Ba
et al. [47]. There, key and value vectors are not generated within the same
time step, unlike in our models or in linear Transformers. The Fast Weight
Memory (FWM) [48] is also a recurrent FWP: the slow net is an LSTM and the
fast net is a higher-order RNN. However, the FWM is a single pair of slow and
fast nets, and a multi-layer version, as in the linear Transformer family, was
not explored. Similarly, the Metalearned Neural Memory [49] uses an LSTM as
its slow net and a 3-layer MLP as its fast net but again limited to one pair.
Others have investigated variants of RNNs with fast weights for toy synthetic
retrieval tasks [50, 51]. In particular, Keller et al. [51] augment the LSTM
with a fast weight matrix in the cell update. In contrast, we make all weights
in the LSTM fast and, importantly, our model specifications build upon the
successful deep Transformer architecture using residual connections [52, 53],
layer-norm [54], multiple attention heads and feed-forward blocks [1].
Essentially, we replace the self-attention layers in the regular Transformers
by the fast weight programmer operations described above.

##  4 Experiments

We conduct experiments in four different settings. We start by evaluating all
models on a language modelling task (Sec. 4.1) to obtain a performance
overview and to discuss computational costs. Language modelling is an
excellent task to evaluate sequence models. However, to highlight their
different capabilities, we evaluate our models also on algorithmic tasks. In
fact, it is well-known that the actual capabilities of RNNs differ from one
architecture to another [55]. We are interested in discussing such
differences. With that goal in mind, we conduct experiments on two synthetic
algorithmic tasks, code execution (Sec. 4.2) and sequential ListOps (Sec.
4.3), which are designed to compare elementary sequence processing abilities
of models. Finally, we apply our models to reinforcement learning in 2D game
environments (Sec. 4.4) as a replacement for LSTMs.

###  4.1 Language Modelling

We first evaluate all discussed models on the generic language modelling task.
This allows for obtaining a performance overview and reviewing the
computational efficiency of different models. We use the Wikitext-103 dataset
[28] and follow the small model setting similar to what’s used in recent works
by Peng et al. [21] and Schlag et al. [23]. This allows for training and
evaluating different models with a reasonable amount of compute on this
resource-demanding language modelling task.

Table 1: WikiText-103 language model perplexity results with the small setting [21, 23]. For each model, its name, corresponding slow and fast networks, and weight update rule (Update) are specified. All models are trained and evaluated on the span of 256 tokens except for the models in the last two rows (+ full context) which are trained and evaluated without context truncation. Parameter count is in millions. See Appendix A for further experimental details and results.  Name | Slow net | Update | Fast net | Valid | Test | #Prms
---|---|---|---|---|---|---
Transformer | - | - | - | 33.0 | 34.1 | 44.0
Linear Transformer | Feedforward | sum | Linear | 37.1 | 38.3 | 44.0
Delta Net |  | delta |  | 34.1 | 35.2 | 44.0
Delta MLP | Feedforward | delta | Deep MLP | 35.8 | 36.8 | 44.3
Delta Delta Net |  |  | Delta Net | 34.0 | 35.2 | 44.6
Delta RNN |  |  | RNN | 33.8 | 35.0 | 44.6
Delta LSTM |  |  | LSTM | 32.6 | 33.8 | 47.3
RDN | Recurrent |  | Linear | 34.1 | 35.2 | 44.1
Delta RNN | \+ full context |  |  | 31.8 | 32.8 | 44.6
RDN |  |  | 32.5 | 33.6 | 44.1

#### Perplexity results.

The results are shown in Table 1 which also serves as a tabular summary
recapitulating different models described in Sec. 2 and 3, with various
architectures for slow and fast nets, and two choices of update rule. The top
block of Table 1 shows the performance of the baseline Transformer,
Katharopoulos et al. [19]’s Linear Transformer, and Schlag et al. [23]’s Delta
Net. The performance of models presented in Sec. 3 can be found in the middle
block. First of all, the Delta MLP performs worse than the baseline Delta Net
despite a slight increase in parameter count (44.3 vs. 44.0 M). This supports
the intuition that it is better to make the slow network aware of the outputs
of intermediate layers to generate fast weights in a deep network, instead of
generating fast weights for all layers at a time. In all other models, the
performance never degrades with the proposed architectural augmentation. The
Delta Delta Net yields limited improvements; we plan to study this model in
depth in future work. With the same amount of parameters (44.6 M), the Delta
RNN yields greater improvements. Among the models presented here, the Delta
LSTM variant exhibits the best performance. This shows that the slow network
successfully controls the rather complex fast LSTM network, although it also
requires more parameters (47.3 M) than other models. Finally, the benefits of
recurrent connections added to the baseline Delta Net do not directly
translate into practical improvements in language modelling as demonstrated by
the performance of RDN compared to the one of the baseline Delta Net.
Importantly, given a constant memory size w.r.t. sequence length, it is
straight-forward to train and evaluate our RNNs without context truncation
(while still limiting the backpropagation span). Corresponding performances of
Delta RNN and RDN are shown in the bottom part of Table 1: they outperform the
regular Transformer with a limited context (256 tokens).

While language modelling is useful as a sanity check (here for example, except
for the Delta MLP, all models achieve reasonable performance), the task is too
generic to identify certain important aspects of the models, such as real
benefits of recurrence. Before we move on to trickier RL applications, Sec.
4.2 and 4.3 will focus on studying such aspects using synthetic algorithmic
tasks.

#### Computational efficiency.

The modifications we proposed in Sec. 3 introduce additional computational
costs to linear Transformers/FWPs. First of all, none of them affect the core
complexity of linear Transformers: they all have a constant space and linear
time complexity w.r.t. sequence length. However, the per-time-step
computational costs differ a lot from one model to another, as quantified here
in terms of training speed using our implementation. All models are
implemented using a custom CUDA kernel except the baseline Transformer for
which we use regular PyTorch code [56]. Training speeds of LT and Delta Net in
Table 1 are 66 K and 63 K words per second respectively (vs. 33 K for the
baseline Transformer). The most expensive model is the Delta LSTM. This fast
weight LSTM with tied input-forget gates has 6 weight matrices, and each of
these are manipulated by separate delta rules. The corresponding speed is 14 K
words per second, too slow for scaling to more experiments. In contrast, the
speeds of Delta RNN and RDN remain reasonable: 41 K and 35 K words per second
respectively. Therefore, the remaining experiments will focus on these two
recurrent architectures which are promising and practical in terms of both
performance and computational costs.

###  4.2 Code Execution Task: Learning to Maintain and Update Variable States

In code execution tasks [29], models are trained to sequentially read the
input code provided as word-level text, and to predict the results of the
corresponding code execution. We adopt the task setting from Fan et al. [57]
with one conditional and three basic statements. We refer the readers to
Appendix B.1 for a precise description of the task. This code execution task
requires models to maintain the values of multiple variables, which has been
shown to be difficult for relatively shallow Transformers with only
feedforward connections [57].

The left block of Table 2 shows the results. Following again Fan et al. [57],
we control the task difficulty by modifying the number of variables (3 or 5).
The model architectures are fixed: the LSTM has only one layer with 256 nodes
and all Transformer variants have the same architecture with 4 layers with a
hidden size of 256 using 16 heads and an inner feedforward layer size of 1024.

We first note that the LSTM is the best performer for both difficulty levels,
with the smallest performance drops through increasing the number of
variables. In contrast to prior claims [57], the LSTM is clearly capable of
storing the values of multiple variables in its hidden and cell state vectors.
With three variables, the regular Transformer already largely underperforms
other models with a mutable memory: Delta Net, Delta RNN, and RDN. Linear
Transformers completely fail at this task, likely due to the memory capacity
problem pointed out by Schlag et al. [23] (see Appendix B.2 for further
discussion). By increasing the number of variables to five, the baseline
Transformers, Delta Net, and RDN become unstable as shown by high standard
deviations w.r.t. the seed. The benefits of recurrent connections introduced
in our RDN compared to the baseline Delta Net become more apparent (76.3 vs.
61.4%). In contrast, the Delta RNN remains stable and gives the best
performance (85.1%) among Transformer variants, which shows the benefits of
recurrence and in particular the regular RNN architecture in the fast net. To
match the performance of LSTM on this task, however, these models need more
layers (see Appendix B.2 for more results).

Table 2: Test accuracies (%) with standard deviations on code execution (Code Exec) and sequential ListOps (Seq ListOps). The difficulty of the task is controlled by the maximum number of possible variables (# variables) for code execution, and the list depth (10 or 15) for ListOps. For code execution with 5 variables, we report means over six seeds. In all other cases, the results are computed with three seeds. For more results, see Appendix B.2 (Code Exec) and B.4 (Seq ListOps). |  Code Exec (# variables) |  Seq ListOps (depth)
---|---|---
| 3 | 5 | 10 | 15
LSTM |  99.0 $\pm$ 0.1 |  93.2 $\pm\,\;$ 6.1 |  88.5 $\pm$ 2.9 | 24.4 $\pm$ 1.1
Transformer | 71.8 $\pm$ 2.6 | 35.4 $\pm$ 28.2 | 79.1 $\pm$ 0.9 | 75.3 $\pm$ 0.4
Linear Transformer | 0.0 $\pm$ 0.0 | 0.0 $\pm\,\;$ 0.0 | 64.0 $\pm$ 0.3 | 64.4 $\pm$ 0.4
Delta Net | 90.7 $\pm$ 2.7 | 61.4 $\pm$ 20.0 |  85.7 $\pm$ 1.8 | 77.6 $\pm$ 1.4
Delta RNN | 90.8 $\pm$ 1.7 |  85.1 $\pm\,\;$ 1.9 | 83.6 $\pm$ 1.2 | 78.0 $\pm$ 1.0
RDN |  92.6 $\pm$ 2.2 | 76.3 $\pm$ 17.6 | 83.2 $\pm$ 0.9 |  79.2 $\pm$ 1.4

###  4.3 Sequential ListOps: Learning Hierarchical Structure and Computation

The ListOps task [30] is a typical test for hierarchical structure learning,
which requires list operation executions. We use a simple variant of ListOps
whose detailed descriptions can be found in Appendix B.4. For example, the
list [MAX 6 1 [FIRST 2 3 ] 0 [MIN 4 7 1] ] is of depth two and the expected
output is 6. While early research comparing self-attention to RNNs [58] has
shown some advantages of recurrence in hierarchical structure learning, more
recent work [59] reports Transformers outperforming LSTMs on ListOps.
According to Tay et al. [22], linear Transformer variants (LT and Performers)
underperform other Transformer variants by a large margin on ListOps.

The right block of Table 2 shows results for two different depths: 10 and 15.
The model architectures are identical to those used in the code execution task
(Sec. 4.2). At depth 10, we find LSTM to perform best, while mutable memory
Transformer variants (Delta Net, Delta RNN, and RDN) outperform the regular
and linear Transformers. At depth 15, the LSTM’s performance drops drastically
(to 24.4%), while the differences between Transformer variants remain almost
the same. We note that sequences are longer for the depth 15 problem (mean
length of 185 tokens) than for the depth 10 version (mean length of 98
tokens). This turns out to be difficult for the small 256-dimensional LSTM;
see Appendix B.4 for the corresponding ablation study. The performance
differences between the baseline Delta Net and the proposed Delta RNN and RDN
are rather small for this task. Importantly, our models outperform both
regular and linear Transformers on this task requiring hierarchical structure
learning.

###  4.4 Reinforcement Learning in 2D Game Environments

We finally evaluate the performance of our models as a direct replacement for
the LSTM in reinforcement learning settings. In fact, only a limited number of
prior works have investigated Transformers for RL. Parisotto et al. [12] and
Rae et al. [11] evaluate them on the DMLab-30 [60, 61]. Parisotto et al. [12]
also evaluate them on Atari but in a multi-task setting [62]. Others [57, 13]
use toy maze environments. In contrast to Parisotto et al. [12]’s work, which
presents multi-task Atari as a side experiment, we study the Transformer
family of models on the standard Atari 2600 setting [31, 63, 64] by training
game-specific agents.

Figure 1: Relative improvements in test scores obtained by the Recurrent Delta
Net (RDN) compared to the Linear Transformer (LT) after 50 M env. steps.

Figure 2: Relative improvements in test scores obtained by the Recurrent Delta
Net (RDN) compared to the Linear Transformer (LT) after 200 M env. steps.

Figure 3: Relative improvements in test scores obtained by 2-layer RDN
compared to LSTM after 50 M env. steps.

Figure 4: Relative improvements in test scores obtained by 2-layer RDN
compared to LSTM after 200 M env. steps.

#### Settings.

We train an expert agent on each game separately with the Importance Weighted
Actor-Learner Training Architecture (IMPALA) using the V-trace actor-critic
setup [65] and entropy regularization [66] implemented in Torchbeast [67]. Our
model follows the large architecture of Espeholt et al. [65] which consists of
a 15-layer residual convolutional NN with one 256-node LSTM layer which we
replace by either the RDN (Sec. 3.2) or the Delta RNN (Sec. 3.1). In line with
the small LSTM used for Atari (only 1 layer with 256 hidden nodes) we also
configure a small RDN: 2 layers with a hidden size of 128 using 4 heads, and a
feedforward dimension of 512. We find this small model to perform already
surprisingly well. For the rest, we use the same hyperparameters as Espeholt
et al. [65] which can be found in Appendix C.

#### Main experiments.

We evaluate our models in 20 environments. According to Mott et al. [68], in
about half of them, the LSTM outperforms the feedforward baselines—which we
confirm in our setting with 50 M steps (see Appendix C). We report results at
50 M and 200 M environmental steps of training. Like Nair et al. [69], we run
the trained agent for 30 test episodes. Here we repeat this evaluation five
times to report the average score with a standard deviation. The following
analysis focuses on the RDN (Sec. 3.2) compared to the regular linear
Transformer and the LSTM. A similar study of the Delta RNN, as well as
comparisons to more baselines, and the exact scores achieved by each model on
each game can be found in Appendix C.

In all our experiments above, we have shown that the Linear Transformer, i.e.,
a Fast Weight Programmer with a purely additive update rule, consistently
underperforms other models based on the delta rule. Here we confirm this trend
once more. Figures 2 and 2 show the relative improvements of scores obtained
by Recurrent Delta Net over those achieved by the linear Transformer on each
game, respectively after 50 and 200 M interaction steps. The RDN matches or
outperforms the Linear Transformer on all games except for two out of 20 games
at both stages of training.

Figure 4 shows relative improvements of RDN over LSTM after 50 M interactions.
In 12 games, the RDN yields improvements over LSTM, whereas in 3 games, the
LSTM performs better. In the remaining 5 games, both reach similar scores.
Interestingly, this trend does not directly extrapolate to the 200 M case,
which is presented in Figure 4. With longer training, the LSTM surpasses the
performance of the RDN in Battlezone, Gopher, Seaquest and Zaxxon, while the
RDN catches up in Up’N Down and Kung-Fu Master. Overall, there are 6 games in
which LSTM clearly outperforms RDN at 200 M steps, whereas in 9 games the
result is the opposite.

On a side note, some of the scores achieved by the RDN at 200 M step are
excellent: a score of over 170 K and 980 K in Space Invader and Q*Bert
respectively beats the state-of-the-art set by MuZero [70] and Agent57 [62].
However, a direct comparison is not fair as we train game-specific agents.

#### Experiments with larger models.

Given the results above, a natural question to ask is whether a larger model
size improves the RDN in games where the LSTM dominates. We focus on four such
games: Battlezone, Berzerk, Gopher, and Seaquest (See Fig. 4). We double the
model size to 3.4 M parameters by increasing the number of layers to 4 and the
hidden size to 256, with 8 heads. As shown in Table 3, larger RDN models
reduce the gap to the LSTM (except in Berzerk). This indicates that further
scaling RDN might be as promising as scaling regular Transformers in other
domains.

Table 3: Performance of a larger RDN in games where the LSTM dominates (200 M steps).  | Battlezone | Berzerk | Gopher | Seaquest
---|---|---|---|---
LSTM |  24,873 $\pm$ 1,240 |  1,150 $\pm$ 92 |  124,914 $\pm$ 22,422 |  12,643 $\pm$ 1,627
RDN |  10,980 $\pm$ 1,104 |  348 $\pm$ 17 |  86,008 $\pm$ 11,815 |  4,373 $\pm\,\;\;$ 504
RDN larger |  28,273 $\pm$ 5,333 |  346 $\pm\,\;$ 9 |  118,273 $\pm$ 14,872 |  14,601 $\pm\,\;\;$ 712

##  5 Conclusion

Inspired by the formal equivalence of linear Transformers and certain
traditional Fast Weight Programmers (FWPs) from the early ’90s, we propose
various new linear Transformer variants with recurrent connections. Our novel
Recurrent FWPs (RFWPs) outperform previous linear and regular Transformers on
a code execution task and significantly improve over Transformers in a
sequential ListOps task. On Wikitext-103 in the “small” model setting, RFWPs
compete well with the previous best linear Transformer variants for truncated
contexts, and with full contexts, beat regular Transformers. Our RFWPs can
also be used as drop-in replacements for problems where RNNs are still
dominant. In particular, we evaluate them in reinforcement learning settings
on 20 Atari 2600 environments. They clearly outperform the regular Linear
Transformer on almost all environments. They also outperform the LSTM across
many environments with a small model size and demonstrate promising scaling
properties for larger models. Given the increasing interest in deploying
Transformers in RL [71, 72], in particular in the framework of Upside-Down RL
[73, 74], our RFWP models are particularly relevant: as RNNs, they
conveniently handle long contexts with a constant memory size, while being
powerful Transformer variants at the same time. Our work highlights the
usefulness of the FWP framework from the ’90s and its connection to modern
architectures, opening promising avenues for further research into new classes
of recurrent Transformers.

## Acknowledgments and Disclosure of Funding

We thank Aleksandar Stanić and Sjoerd van Steenkiste for valuable comments on
the first version of this paper. This research was partially funded by ERC
Advanced grant no: 742870, project AlgoRNN, and by Swiss National Science
Foundation grant no: 200021_192356, project NEUSYM. This work was partially
supported by computational resources at the CSCS Swiss National Supercomputing
Centre, project d115. We thank NVIDIA Corporation for donating several DGX
machines, and IBM for donating a Minsky machine.

## References

  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proc. Advances in Neural Information Processing Systems (NIPS)_ , pages 5998–6008, Long Beach, CA, USA, December 2017.
  * Al-Rfou et al. [2019] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones.  Character-level language modeling with deeper self-attention.  In _Proc. Conference on Artificial Intelligence (AAAI)_ , pages 3159–3166, Honolulu, HI, USA, January 2019.
  * Brown et al. [2020] Tom B Brown et al.  Language models are few-shot learners.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , Virtual only, December 2020.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLT)_ , pages 4171–4186, Minneapolis, MN, USA, June 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.
  * Zhu et al. [2021] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: Deformable transformers for end-to-end object detection.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.
  * Saxton et al. [2019] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli.  Analysing mathematical reasoning abilities of neural models.  In _Int. Conf. on Learning Representations (ICLR)_ , New Orleans, LA, USA, May 2019.
  * Schlag et al. [2019] Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao.  Enhancing the transformer with explicit relational encoding for math problem solving.  _Preprint arXiv:1910.06611_ , 2019.
  * Charton et al. [2021] Francois Charton, Amaury Hayat, and Guillaume Lample.  Learning advanced mathematical computations from examples.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.
  * Dai et al. [2019] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proc. Association for Computational Linguistics (ACL)_ , pages 2978–2988, Florence, Italy, July 2019.
  * Rae et al. [2020] Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.  Compressive transformers for long-range sequence modelling.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, April 2020.
  * Parisotto et al. [2020] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell.  Stabilizing Transformers for reinforcement learning.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , pages 7487–7498, Virtual only, July 2020.
  * Parisotto and Salakhutdinov [2021] Emilio Parisotto and Ruslan Salakhutdinov.  Efficient transformers in reinforcement learning using actor-learner distillation.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Wierstra et al. [2010] Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber.  Recurrent policy gradients.  _Logic Journal of IGPL_ , 18(2):620–634, 2010\.
  * Wierstra et al. [2007] Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber.  Solving deep memory POMDPs with recurrent policy gradients.  In _Proc. Int. Conf. on Artificial Neural Networks (ICANN)_ , pages 697–706, Porto, Portugal, September 2007.
  * OpenAI et al. [2019] OpenAI et al.  Dota 2 with large scale deep reinforcement learning.  _Preprint arXiv:1912.06680_ , 2019.
  * Vinyals et al. [2019] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Katharopoulos et al. [2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are RNNs: Fast autoregressive transformers with linear attention.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , Virtual only, July 2020.
  * Choromanski et al. [2021] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, 2021.
  * Peng et al. [2021] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong.  Random feature attention.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, 2021.
  * Tay et al. [2021] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.
  * Schlag et al. [2021a] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.  Linear Transformers are secretly fast weight programmers.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , Virtual only, July 2021a.
  * Schmidhuber [1991] Jürgen Schmidhuber.  Learning to control fast-weight memories: An alternative to recurrent nets.  Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, March 1991.
  * Schmidhuber [1992] Jürgen Schmidhuber.  Learning to control fast-weight memories: An alternative to dynamic recurrent networks.  _Neural Computation_ , 4(1):131–139, 1992.
  * Schmidhuber [1993] Jürgen Schmidhuber.  Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets.  In _International Conference on Artificial Neural Networks (ICANN)_ , pages 460–463, Amsterdam, Netherlands, September 1993.
  * Hahn [2020] Michael Hahn.  Theoretical limitations of self-attention in neural sequence models.  _Transactions of the Association for Computational Linguistics_ , 8:156–171, 2020.
  * Merity et al. [2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _Int. Conf. on Learning Representations (ICLR)_ , Toulon, France, April 2017.
  * Zaremba and Sutskever [2014] Wojciech Zaremba and Ilya Sutskever.  Learning to execute.  _Preprint arXiv:1410.4615_ , 2014.
  * Nangia and Bowman [2018] Nikita Nangia and Samuel Bowman.  ListOps: A diagnostic dataset for latent tree learning.  In _Proc. North American Chapter of the Association for Computational Linguistics (NAACL): Student Research Workshop_ , pages 92–99, New Orleans, LA, USA, June 2018.
  * Bellemare et al. [2016] Marc G. Bellemare, Georg Ostrovski, Arthur Guez, Philip S. Thomas, and Rémi Munos.  Increasing the action gap: New operators for reinforcement learning.  In _Proc. AAAI Conf. on Artificial Intelligence_ , pages 1476–1483, Phoenix, AZ, USA, February 2016. AAAI Press.
  * Schmidhuber [1990] Jürgen Schmidhuber.  Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments.  Technical Report FKI-126-90, Institut für Informatik, Technische Universität München, 1990.
  * Widrow and Hoff [1960] Bernard Widrow and Marcian E Hoff.  Adaptive switching circuits.  In _Proc. IRE WESCON Convention Record_ , pages 96–104, Los Angeles, CA, USA, August 1960.
  * Ha et al. [2017] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  In _Int. Conf. on Learning Representations (ICLR)_ , Toulon, France, April 2017.
  * Irie and Schmidhuber [2021] Kazuki Irie and Jürgen Schmidhuber.  Training and generating neural networks in compressed weight space.  In _Neural Compression: From Information Theory to Applications – Workshop, ICLR 2021_ , Virtual only, May 2021.
  * von der Malsburg [1981] Christoph von der Malsburg.  The correlation theory of brain function.  Internal Report 81-2, Goettingen: Department of Neurobiology, Max Planck Intitute for Biophysical Chemistry, 1981.
  * Feldman [1982] Jerome A Feldman.  Dynamic connections in neural networks.  _Biological cybernetics_ , 46(1):27–39, 1982\.
  * McClelland [1985] James L McClelland.  Putting knowledge in its place: A scheme for programming parallel processing structures on the fly.  _Cognitive Science_ , 9(1):113–146, 1985.
  * Klein et al. [2015] Benjamin Klein, Lior Wolf, and Yehuda Afek.  A dynamic convolutional layer for short rangeweather prediction.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 4840–4848, Boston, MA, USA, June 2015.
  * Noh et al. [2016] Hyeonwoo Noh, Paul Hongsuck Seo, and Bohyung Han.  Image question answering using convolutional neural network with dynamic parameter prediction.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 30–38, Las Vegas, NV, USA, 2016.
  * Jia et al. [2016] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool.  Dynamic filter networks.  In _Proc. Advances in Neural Information Processing Systems (NIPS)_ , pages 667–675, Barcelona, Spain, 2016.
  * Bello [2021] Irwan Bello.  LambdaNetworks: Modeling long-range interactions without attention.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.
  * Miconi et al. [2018] Thomas Miconi, Kenneth Stanley, and Jeff Clune.  Differentiable plasticity: training plastic neural networks with backpropagation.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , pages 3559–3568, Stockholm, Sweden, July 2018.
  * Miconi et al. [2019] Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley.  Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity.  In _Int. Conf. on Learning Representations (ICLR)_ , New Orleans, LA, USA, May 2019.
  * Graves et al. [2014] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural Turing Machines.  _Preprint arXiv:1410.5401_ , 2014.
  * Graves et al. [2016] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al.  Hybrid computing using a neural network with dynamic external memory.  _Nature_ , 538(7626):471–476, 2016.
  * Ba et al. [2016a] Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu.  Using fast weights to attend to the recent past.  In _Proc. Advances in Neural Information Processing Systems (NIPS)_ , pages 4331–4339, Barcelona, Spain, December 2016a.
  * Schlag et al. [2021b] Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber.  Learning associative inference using fast weight memory.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021b.
  * Munkhdalai et al. [2019] Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler.  Metalearned neural memory.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , pages 13310–13321, Vancouver, Canada, December 2019.
  * Schlag and Schmidhuber [2017] Imanol Schlag and Jürgen Schmidhuber.  Gated fast weights for on-the-fly neural program generation.  In _NIPS Metalearning Workshop_ , Long Beach, CA, USA, December 2017\.
  * Keller et al. [2018] T Anderson Keller, Sharath Nittur Sridhar, and Xin Wang.  Fast weight long short-term memory.  _Preprint arXiv:1804.06511_ , 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In _Proc. European Conf. on Computer Vision (ECCV)_ , pages 630–645, Amsterdam, Netherlands, October 2016.
  * Srivastava et al. [2015] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber.  Highway networks.  In _the Deep Learning workshop at Int. Conf. on Machine Learning (ICML)_ , Lille, France, July 2015.
  * Ba et al. [2016b] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _Preprint arXiv:1607.06450_ , 2016b.
  * Weiss et al. [2018] Gail Weiss, Yoav Goldberg, and Eran Yahav.  On the practical computational power of finite precision rnns for language recognition.  In _Proc. Association for Computational Linguistics (ACL)_ , pages 740–745, Melbourne, Australia, July 2018.
  * Paszke et al. [2019] Adam Paszke et al.  PyTorch: An imperative style, high-performance deep learning library.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , pages 8026–8037, Vancouver, Canada, December 2019.
  * Fan et al. [2020] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar.  Addressing some limitations of Transformers with feedback memory.  _Preprint arXiv:2002.09402_ , 2020.
  * Tran et al. [2018] Ke Tran, Arianna Bisazza, and Christof Monz.  The importance of being recurrent for modeling hierarchical structure.  In _Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4731–4736, Brussels, Belgium, October 2018.
  * Lu et al. [2021] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch.  Pretrained transformers as universal computation engines.  _Preprint arXiv:2103.05247_ , 2021.
  * Beattie et al. [2016] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al.  Deepmind lab.  _Preprint arXiv:1612.03801_ , 2016.
  * Leibo et al. [2018] Joel Z Leibo, Cyprien de Masson d’Autume, Daniel Zoran, David Amos, Charles Beattie, Keith Anderson, Antonio García Castañeda, Manuel Sanchez, Simon Green, Audrunas Gruslys, et al.  Psychlab: a psychology laboratory for deep reinforcement learning agents.  _Preprint arXiv:1801.08116_ , 2018.
  * Badia et al. [2020] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell.  Agent57: Outperforming the Atari human benchmark.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , pages 507–517, Virtual only, July 2020.
  * Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing Atari with deep reinforcement learning.  In _NIPS Deep Learning Workshop_ , Lake Tahoe, NV, USA, December 2013.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Espeholt et al. [2018] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu.  IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , pages 1406–1415, Stockholm, Sweden, July 2018.
  * Mnih et al. [2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , pages 1928–1937, New York City, NY, USA, June 2016.
  * Küttler et al. [2019] Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette.  Torchbeast: A PyTorch platform for distributed RL.  _Preprint arXiv:1910.03552_ , 2019.
  * Mott et al. [2019] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende.  Towards interpretable reinforcement learning using attention augmented agents.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , pages 12329–12338, Vancouver, Canada, December 2019.
  * Nair et al. [2015] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al.  Massively parallel methods for deep reinforcement learning.  In _Deep Learning Workshop, International Conference on Machine Learning (ICML)_ , Lille, France, July 2015.
  * Schrittwieser et al. [2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering Atari, Go, Chess and Shogi by planning with a learned model.  _Nature_ , 588(7839):604–609, 2020.
  * Chen et al. [2021] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision Transformer: Reinforcement learning via sequence modeling.  _Preprint arXiv:2106.01345_ , 2021.
  * Janner et al. [2021] Michael Janner, Qiyang Li, and Sergey Levine.  Reinforcement learning as one big sequence modeling problem.  _Preprint arXiv:2106.02039_ , 2021.
  * Schmidhuber [2019] Juergen Schmidhuber.  Reinforcement learning upside down: Don’t predict rewards–just map them to actions.  _Preprint arXiv:1912.02875_ , 2019.
  * Srivastava et al. [2019] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber.  Training agents using upside-down reinforcement learning.  _Preprint arXiv:1912.02877_ , 2019.
  * Hanson [1990] Stephen José Hanson.  A stochastic version of the delta rule.  _Physica D: Nonlinear Phenomena_ , 42(1-3):265–272, 1990.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Frazier-Logue and Hanson [2018] Noah Frazier-Logue and Stephen José Hanson.  Dropout is a special case of the stochastic delta rule: Faster and more accurate deep learning.  _Preprint arXiv:1808.03578_ , 2018.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  Preprint arXiv:1412.6980, 2014.
  * Gers et al. [2000] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins.  Learning to forget: Continual prediction with LSTM.  _Neural computation_ , 12(10):2451–2471, 2000\.
  * Irie et al. [2019] Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney.  Language modeling with deep Transformers.  In _Proc. Interspeech_ , pages 3905–3909, Graz, Austria, September 2019.
  * Csordás et al. [2021] Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber.  The devil is in the detail: Simple tricks improve systematic generalization of transformers.  In _Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP)_ , Punta Cana, Dominican Republic, November 2021.
  * Csordás et al. [2021] Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber.  The Neural Data Router: Adaptive control flow in Transformers improves systematic generalization.  _Preprint arXiv:2110.07732_ , 2021.
  * Tieleman and Hinton [2012] Tijmen Tieleman and Geoffrey Hinton.  Lecture 6.5- RMSProp: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural Networks for Machine Learning, 4_ , 2012.
  * Machado et al. [2018] Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling.  Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.  _Journal of Artificial Intelligence Research_ , 61:523–562, 2018.
  * Andrychowicz et al. [2021] Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al.  What matters for on-policy deep actor-critic methods? A large scale study.  In _Int. Conf. on Learning Representations (ICLR)_ , Virtual only, May 2021.

##  Appendix A Experimental Details and Ablation Studies for Language
Modelling

###  A.1 Experimental Settings

All language models in Table 1 have the same Transformer configuration: a
16-layer model with a hidden size of 128 with 8 heads, and a feed-forward
dimension of 2048. We use a dropout [75, 76, 77] rate of 0.1. The batch size
is 96 and we train for about 120 epochs with Adam optimiser [78] with an
initial learning rate of 0.00025 and 2000 learning rate warm-up steps. All
models are trained with a back-propagation span of 256 tokens. During
training, these segments are treated independently, except for the \+ full
context cases in Table 1 where the states (both recurrent states and fast
weight states) from a segment are used as initialisation for the subsequent
segment. The models in \+ full context cases are also evaluated in the same
way by carrying over the context throughout the evaluation text with a batch
size of one. For all other cases, the evaluation is done by going through the
text with a sliding window of size 256 with a batch size of one. Transformer
states are computed for all positions in each window, but only the last
position is used to compute perplexity (except in the first segment where all
positions are used for evaluation) [2]. We trained all models using two GPUs
(32 GB V100), and the longest training takes up to 10 days (see Sec. 4.1 in
the main text for speed comparison between models).

For readers interested in any further details, we refer to our code which is
publicly available.

###  A.2 Ablation Studies

In this section, we specify the exact Delta LSTM and Delta MLP models used in
Table 1, and provide a few ablation studies for Delta RNN, Delta LSTM222The
numbers reported in Table 4 for the Delta LSTM models are better than those we
presented in an earlier version. In fact, we found that in our previous code,
the slow weights for key and value generation were shared by mistake between
the forward and recurrent fast weight matrices (while the reported parameter
count was that of the correct model with separate slow weight matrices).
Fixing this resulted in the corresponding improvements., and Delta MLP models.

Table 4: Ablation studies for Delta LSTM, Delta RNN, and Delta MLP models. Language model perplexities are shown and the setting is the same as in Table 1.  | Version | Valid | Test | #Prms
---|---|---|---|---
Delta RNN | A | 35.6 | 36.7 | 44.6
| B | 33.8 | 35.0 |
Delta LSTM | A | 38.5 | 39.9 | 47.3
| B | 34.2 | 35.2 |
| C | 33.5 | 34.7 |
| D | 32.6 | 33.8 |
Delta MLP | A | 36.8 | 37.9 | 44.3
| B | 35.8 | 36.8 | 44.3

#### Delta RNN.

In Sec. 3.1, we argue for a version of fast RNN given by Eq. 12 as a natural
augmentation of the linear Transformer with recurrent connections. Here we
empirically support this choice by comparing to another variant of Delta RNN
given by:

$\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$ $\displaystyle
f({\bm{W}}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}^{(t)}{\bm{y}}^{(t-1)})$ (19)

where $f$ is again the softmax activation which makes ${\bm{y}}^{(t)}$ a valid
query vector (positive components which sum up to one) for fast weights
maintained by the delta update rule. We refer to this version as Version A in
this ablation study and the one given by Eq. 12 as Version B. As Table 4
shows, Version B performs better, and this is the one we report in Table 1 in
the main text.

#### Delta LSTM.

We evaluate four versions of Delta LSTM for ablation. In all cases, we tie the
input and forget gates to reduce the total number of fast weights to be
controlled by the slow net. All models contain six fast weights and each of
them is updated according to the delta update rule (Eq. 8). The different
versions differ in the location of the activation function and residual
connections in the LSTM architecture [14, 79], inspired by the Transformer.

Version A (analogous to Version A of Delta RNN above) is the one which is the
closest to the original LSTM with tied input and forget gate. The only
architectural difference is the usual $\tanh$ on the cell output
${\bm{c}}^{(t)}$ which is replaced by a softmax $f$ placed after the final
output of the layer $f({\bm{c}}^{(t)}\odot{\bm{o}}^{(t)})$, such that it can
directly be used as a delta rule compatible query for the next time step (we
also use a sigmoid instead of $\tanh$ for the main transformation
${\bm{u}}^{(t)}$, but this is not crucial for any models here).

$\displaystyle{\bm{u}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}^{(t)}{\bm{y}}^{(t-1)})$
(20) $\displaystyle{\bm{f}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{f}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{f}^{(t)}{\bm{y}}^{(t-1)})$
(21) $\displaystyle{\bm{o}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{o}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{o}^{(t)}{\bm{y}}^{(t-1)})$
(22) $\displaystyle{\bm{c}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{f}}^{(t)}\odot{\bm{c}}^{(t-1)}+(1-{\bm{f}}^{(t)})\odot{\bm{u}}^{(t)}$
(23) $\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$ $\displaystyle
f({\bm{c}}^{(t)}\odot{\bm{o}}^{(t)})$ (24)

Version B is obtained by delaying the application of the softmax activation
$f$ in Version A.

$\displaystyle{\bm{u}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}^{(t)}f({\bm{y}}^{(t-1)}))$
(25) $\displaystyle{\bm{f}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{f}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{f}^{(t)}f({\bm{y}}^{(t-1)}))$
(26) $\displaystyle{\bm{o}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{o}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{o}^{(t)}f({\bm{y}}^{(t-1)}))$
(27) $\displaystyle{\bm{c}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{f}}^{(t)}\odot{\bm{c}}^{(t-1)}+(1-{\bm{f}}^{(t)})\odot{\bm{u}}^{(t)}$
(28) $\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{c}}^{(t)}\odot{\bm{o}}^{(t)}$ (29)

Version C is obtained from Version B by adding a residual connection from the
feed-forward part ${\bm{z}}_{u}^{(t)}$ of the main transformation
${\bm{u}}^{(t)}$ to the output.

$\displaystyle{\bm{z}}_{u}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}^{(t)}{\bm{q}}^{(t)}$ (30) $\displaystyle{\bm{u}}^{(t)}$
$\displaystyle=$
$\displaystyle\sigma({\bm{z}}_{u}^{(t)}+{\bm{R}}^{(t)}f({\bm{y}}^{(t-1)}))$
(31) $\displaystyle{\bm{f}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{f}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{f}^{(t)}f({\bm{y}}^{(t-1)}))$
(32) $\displaystyle{\bm{o}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{o}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{o}^{(t)}f({\bm{y}}^{(t-1)}))$
(33) $\displaystyle{\bm{c}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{f}}^{(t)}\odot{\bm{c}}^{(t-1)}+(1-{\bm{f}}^{(t)})\odot{\bm{u}}^{(t)}$
(34) $\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{c}}^{(t)}\odot{\bm{o}}^{(t)}+{\bm{z}}_{u}^{(t)}$ (35)

Finally, Version D is obtained from Version B by removing the sigmoid on the
main transformation ${\bm{u}}^{(t)}$ which results in a highway net-like skip
connection [53] from ${\bm{u}}^{(t)}$ to the output. This version is then
analogous to Version B of the Delta RNN as a natural augmentation of the
linear Transformer: a recurrent term is added to the main transformation
${\bm{u}}^{(t)}$ and gating components are added to make it an LSTM
architecture:

$\displaystyle{\bm{u}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{W}}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}^{(t)}f({\bm{y}}^{(t-1)})$
(36) $\displaystyle{\bm{f}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{f}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{f}^{(t)}f({\bm{y}}^{(t-1)}))$
(37) $\displaystyle{\bm{o}}^{(t)}$ $\displaystyle=$
$\displaystyle\sigma({\bm{W}}_{o}^{(t)}{\bm{q}}^{(t)}+{\bm{R}}_{o}^{(t)}f({\bm{y}}^{(t-1)}))$
(38) $\displaystyle{\bm{c}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{f}}^{(t)}\odot{\bm{c}}^{(t-1)}+(1-{\bm{f}}^{(t)})\odot{\bm{u}}^{(t)}$
(39) $\displaystyle{\bm{y}}^{(t)}$ $\displaystyle=$
$\displaystyle{\bm{c}}^{(t)}\odot{\bm{o}}^{(t)}$ (40)

Corresponding performances can be found in Table 4. The best model, Version D,
is the one we report in Table 1 in the main text.

#### Delta MLP.

We also conduct a few ablation studies for the Delta MLP (Sec. 3.1). As MLP
architecture we used the feedforward block of the regular Transformer which
consists of two feedforward layers: one with the size of the inner feedforward
layer (2048 here) and another one with the size of hidden dimension (128
here). We test two variants which result in a similar number of parameters:
Version A with 8 overall Transformer layers where each self-attention layer
contains 4 fast MLP layers (i.e. a total of 48 feedforward layers with 32 fast
ones), and Version B with 11 overall Transformer layers where each self-
attention layer contains 2 fast MLP layers (i.e. a total of 44 feedforward
layers with 22 fast ones). As shown in Table 4, Version B which has fewer fast
layers controlled by the same slow net performs better, and, as already
mentioned in Sec. 4, they do not outperform the baseline Delta Net which has
only one fast feedforward layer (Table 1).

###  A.3 Dimensionality of Delta-Delta Net vs. Delta Net

Here we describe how the dimensionality of Delta-Delta Net scales with the
size of the Delta Net. We assume a Delta Net with a dimension $d$ for all
query, key, value and input vectors. Then its slow weight matrix (the
projection matrix) is of size $d\times(3d+1)$ as it projects a $d$-dimensional
input to query, key, value vectors ($3d$) and a scalar beta ($+1$) which are
needed to maintain a $d\times d$ fast weight matrix using the delta rule. Now
we can express the dimensionality of a Delta-Delta Net in terms of $d$, whose
fast network is a Delta Net with the dimensionality above. The size of its
fast weight matrix is thus $d\times(3d+1)$. In order to maintain a fast weight
matrix of this dimension using the delta rule, we need key and query vectors
of size $d$, a value vector of size $3d+1$, and a scalar beta ($+1$). The slow
weight matrix has to produce all these variables with a total dimension of
$(5d+2)$ from the input of size $d$. Therefore, the size of the slow weight
matrix in the Delta-Delta Net is $d\times(5d+2)$. Such a Delta-Delta Net would
have to store two fast weight matrices: one of size $d\times(3d+1)$ and
another one of size $d\times d$.

##  Appendix B Experimental Details and Additional Results for Algorithmic
Tasks

###  B.1 Task Details for Code Execution

In code execution tasks [29], models are trained to sequentially read the
input code provided as word-level text and to predict the results of the
corresponding code execution. We adopt the task setting from Fan et al. [57].
Each example is a sequence consisting of multiple statements — 100 in our
experiments. A statement can be one of the following three basic statements:
assign which assigns a value to a variable (e.g. x = 2 ;), increment which
increments or decrements an already assigned variable (e.g. x ++ ;), or print
which outputs the value of the variable (e.g. print x ;). In addition to basic
statements, there are also conditional comparisons on already defined
variables followed by a basic statement (e.g. if x < 3 : x ++ ;). The model
reads the input word-level code sequence from left to right in an auto-
regressive manner, and makes a prediction at each position: at the end of each
print statement, the model has to predict the correct variable value, and for
all other positions, the no-output token.

Here is a short example (with N denoting the no-output token):

In: | x | = | 3 | ; | y | = | 7 | ; | x | ++ | ; | if | y | < | 6 | : | print | x | ; | print | x | ;
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Out: | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | 4

In contrast to Fan et al. [57], we hard-code the last statement to be a print
statement of a randomly chosen variable such that the model always has to make
a prediction at the end of the sequence. The output vocabulary of the model is
restricted to discrete values within a pre-determined range (here between -8
and 16), and the code sequences are constructed such that the value to be
printed does not exceed this range by rejecting any statement which would
result in such values. Like Fan et al. [57], we randomly generate 10,000
sequences for training and 1,000 sequences each for validation and test. With
100 statements per sequence, we obtain sequences with lengths varying from
about 370 to 550, with an average length of about 450 tokens for both 3 and 5
variable cases, and for train, valid, and test sets. This code execution task
requires models to maintain the values of multiple variables, which has been
previously shown to be a difficult task for Transformers with only feedforward
connections [57].

###  B.2 Additional Results for Code Execution

#### Token level print accuracy.

First of all, as mentioned in the main text, the test accuracies reported in
Table 2 are on the sequence-level, i.e., an output sequence is counted as
correct only if all output tokens in the sequence match the ground truth. The
sequence level accuracy is a good evaluation measure here since for most
positions in the sequence (except at the end of print statement) the correct
target is the no-output token. This results in 0% accuracy for the Linear
Transformer, which might be shocking at first glance at Table 2. Thus, we also
provide the token accuracies following the print statements. The results can
be found in Table 5. There we can see that the accuracies for the Linear
Transformer are not zero: above 20% in both 3 and 5 variable cases.
Nevertheless, they clearly underperform other models.

Table 5: Token-level validation accuracies (%) for the print statements on code execution. Means and stds are computed with three seeds for 3-variable and six seeds for 5-variable cases. | # Variables
---|---
| 3 | 5
LSTM |  99.9 $\pm$ 0.0 |  99.6 $\pm$ 0.4
Transformer | 98.6 $\pm$ 0.2 | 75.5 $\pm$ 31.0
Linear Transformer | 24.6 $\pm$ 0.6 | 20.7 $\pm$ 1.4
Delta Net | 99.5 $\pm$ 0.1 | 97.2 $\pm$ 2.0
Delta RNN | 99.5 $\pm$ 0.0 |  99.3 $\pm$ 0.2
RDN |  99.6 $\pm$ 0.1 | 98.6 $\pm$ 1.4

#### Model configurations.

The Transformer architecture in Table 2 is adopted from Fan et al. [57]: 4
layers with a hidden dimension of 256 (where we use 16 heads instead of 4) and
a feedforward dimension of 1024, which yields 3.2 M parameters (like for Fan
et al. [57]). We use a dropout rate of 0.1. The regular Transformer makes use
of sinusoidal positional encoding (as is likely the case for Fan et al. [57])
while all other models in Table 2 don’t [80, 23]. All Transformer models use
pre-activation residual connections [52] and layer norm [54]. Our LSTM model
in Table 2 has one LSTM layer with a dimension of 256 and an input embedding
of 128 which results in 405 K parameters. We train all models with a batch
size of 64 using the Adam optimiser with a learning rate of 3e-4 for
Transformer-family models and a learning rate of 3e-3 for the LSTM. We clip
the gradients in the LSTM model at 0.1. To train the regular Transformers,
gradient accumulation was necessary to achieve the same batch size without
hitting the GPU memory limit. This was not the case for space efficient linear
Transformer variants. All models are trained for 200 epochs which takes no
more than 23 hours for any model on a single 16 GB P100 GPU.

Table 6: Test accuracies (%) on code execution with 5 variables. Mean, standard deviation (std), the lowest (min) and highest (max) accuracies are computed over six runs. The number of parameters (Prms.) is given in millions. | width | depth | mean $\pm$ std | min | max | Prms.
---|---|---|---|---|---|---
LSTM | 256 | 1 | 93.2 $\pm\,\;$ 6.1 | 84.7 | 98.5 | 0.4
| 512 |  |  97.7 $\pm\,\;$ 1.1 | 96.1 | 98.7 | 1.3
Delta Net | 256 | 4 | 61.4 $\pm$ 20.0 | 26.2 | 85.7 | 3.2
Delta RNN |  |  |  85.1 $\pm\,\;$ 1.9 | 83.1 | 88.6 | 3.7
RDN |  |  | 76.3 $\pm$ 17.6 | 40.2 | 92.5 | 3.2
Delta Net | 128 | 8 | 62.7 $\pm$ 36.3 | 0.1 | 97.3 | 1.1
Delta RNN |  |  |  94.1 $\pm\,\;$ 2.7 | 88.0 | 95.8 | 1.3
RDN |  |  | 85.0 $\pm\,\;$ 3.8 | 78.9 | 89.0 | 1.1

#### Model architecture ablation.

Here we conduct a few additional experiments to understand the models’
sensitivity to hyper-parameters. We restrict our analysis to the setting with
5 variables in which the performance gap between models is large (Table 2). We
train deeper but thinner models with 8 layers: each with a hidden size of 128
using 8 heads and a feed-forward dimension of 256. This yields a total of 1.1
M parameters for all Transformer models except for the Delta RNN which has 1.3
M parameters. These deeper but thinner models can be trained within 10 hours
using a single 16 GB V100 GPU. We present the results in the bottom part of
Table 6. We don’t report the performance of the regular Transformer since the
8-layer variant learns very slowly and does not improve over the initial 0%
sequence-level accuracy within 200 epochs of training after which we report
the performance for all models333 Extra experiments with this 8-layer regular
Transformer show that after 800 epochs with a dropout rate of 0.3, a test
accuracy of $89.1\pm 2.2$% is achieved. This is still worse than the
performance of Delta RNN trained for 200 epochs, although the comparison is
not even fair due to the longer training and extra tuning..

First, we observe that the Delta RNN with 8 layers can now match the
performance of the baseline LSTM with 256 nodes. However, increasing the LSTM
hidden size to 512 (which gives a parameter count of 1.3 M; equal to the Delta
RNN’s) further improves the LSTM. Second, the Delta Net still remains
unstable. We tried several tricks to stabilise Transformers on algorithmic
tasks [81], e.g. embedding initialisation and scaling, but with little
success. The problem seems intrinsically difficult for Transformer models,
though we note that one of six runs achieved a very good performance of 97.3%.
Finally, we observe that the Recurrent Delta Net becomes more stable and
performs better with a deep architecture.

###  B.3 Task Details for Sequential ListOps

The ListOps task [30] consists of list operation execution which is a typical
test for hierarchical structure learning. A list is constructed using
elementary list-operations written in prefix notation (typically one of six
operations: maximum, minimum, median followed by floor operation, sum modulo
10, first and last element retrieval) with a random number of random arguments
chosen to be either a single digit integer or a sub-list which itself has
random arguments. While early research comparing self-attention to RNNs [58]
has shown some advantages of recurrence in hierarchical structure learning,
more recent work [59] reports Transformers to also outperform LSTMs on
ListOps. Also relevant here, Tay et al. [22] report linear Transformer
variants (Linear Transformers and Performers) to underperform other
Transformer variants by a large margin on ListOps. It is thus natural to
evaluate our models on this task as models at the intersection of recurrent
and self-attention based models. We construct a simple variant of ListOps
which only makes use of maximum MAX, minimum MIN, and first element retrieval
FIRST operations. This turns out to be hard enough to shed light on the
differences between our models. By construction, the targets are single digit
integers. The number of arguments in each list or sub-list is random but less
than the pre-determined maximum number (here set to five, following Nangia and
Bowman [30]) and we control the difficulty of the task by changing the problem
depth. Here is a depth-two example:

In: | [MAX 6 1 [FIRST 2 3 ] 0 [MIN 4 7 1] ]
---|---
Out: | 6

In our setting, the task with depth 10 only contains sequences with depth
10444 However, here the depth is simply defined as the depth of nested
operations. Since the used operations do not always have to evaluate all
arguments to obtain the result, the effective computation may be shallower.
This problem has been addressed in a better version of ListOps in our more
recent work [82]. . Here, we refer to the task as “sequential ListOps”, as we
let the model read the sequence only once from left to right in an auto-
regressive fashion. As for the code execution experiments, we randomly
generate 10,000 sequences for training and 1,000 sequences each for validation
and test. The lengths for the depth 10 case vary from 37 to 364 with an
average length of 98 tokens. For the depth 15 case, the lengths are between 61
and 676, with an average of about 190 tokens. All experiments were conducted
using a single 16 GB P100 GPU. We use the same experimental settings as in the
code execution task, and the experiments for depth 10 and 15 take less than 4
and 16 hours, respectively.

###  B.4 Ablation Study for the LSTM on Sequential ListOps

While the main goal of Table 2 (Sec. 4.3) was to compare different fast weight
programmer variants under the same model configurations, we also pointed out
that the performance of the baseline LSTM dramatically drops for the
sequential ListOps task by increasing the list depth from 10 to 15. In Sec.
4.3, we hypothesised the reason for the performance drop of the LSTM for the
depth-15 case of sequential ListOps to be the small hidden size of the LSTM
and the increase of sequence lengths in the depth-15 case. Here we provide the
corresponding ablation study. Table 7 shows the performance of the LSTM with
different hidden layer sizes. We find that increasing the hidden size
effectively help the LSTM on this task.

Table 7: Test accuracies (%) with standard deviations over three runs for the LSTM on the depth-15 case of Sequential ListOps. Hidden size | Mean accuracy $\pm$ std
---|---
256 | 24.4 $\pm\,\;$ 1.1
1,024 | 24.4 $\pm\,\;$ 0.7
2,048 | 35.9 $\pm$ 13.0
4,096 |  72.2 $\pm\,\;$ 1.6

##  Appendix C Experimental Details and Additional Results for RL in Atari
2600

#### Settings.

We use the polybeast implementation from Torchbeast [67] with modifications
limited to model architectures. We train all our models using RMSProp [83]
with a learning rate of 0.0006, an epsilon of 0.01, and gradient clipping at
40. We use entropy regularisation with a weight of 0.01. The backpropagation
span is 50 and the batch size is 32. The model architecture and evaluation
method is described in the main text. All Transformer variants make use of
pre-activation residual connections [52, 12] and layer norm [54]. The number
of actors for IMPALA is 48. No action repeat is used. No time limit is set for
evaluation. Rewards are clipped between -1 and 1. The OpenAI Gym
implementation of the Atari learning environment [84] is used. The only source
of stochasticity is the default sticky action. We train expert models using
the game specific action spaces (models for Amidar and James Bond were trained
with an action space size of 6, which is smaller than the full action space
but enough to play these games). We train on 2 GPUs (either 16 GB P100 or 32
GB V100). An experiment for one game takes about 1.5 days. Evaluation is done
at 50 M and 200 M environmental steps, which are reported in Table 9 and 10.
For cases where performance did not improve after 50 M and 200 M, we report
the performance at 50 M again in Table 10 (we experienced this for Bank Heist
and Robotank; for Pong 50 M steps are enough to consistently achieve the
perfect score).

In what follows, we provide additional model comparisons.

#### Feedforward vs. LSTM.

On Atari, models without recurrence are also known to perform well in many
environments [68]. Since it is not easy to compare RL systems across different
settings [85], we train our own feedforward baseline. The feedforward baseline
is simply obtained by removing the LSTM layer in the LSTM model, which
corresponds to the model of Espeholt et al. [65]. At 50 M steps (Figure 6;
orange), there are 8 games in which the feedforward baseline clearly
outperforms the LSTM, and in 8 other games the trend is reversed. At 200 M
steps (Figure 6; orange), the LSTM performs clearly better in 10 games,
whereas the feedforward net clearly dominates only in 4 games.

#### Feedforward baseline with more parameters.

In the comparison above, the LSTM baseline has 1.6 M parameters, more than the
1.1 M parameters of the feedforward baseline (while we note that the RDN has
slightly fewer parameters than the LSTM, namely, 1.5 M). To verify that the
improvements obtained by the LSTM are not due to the increased parameter
count, we build a larger feedforward baseline with 1.7 M parameters by
replacing the LSTM layer by one feedforward highway-gated layer [53] (to keep
it as similar as possible to the LSTM baseline). Here the output from the
vision stem is first projected to a 320-dimensional vector which is followed
by a 256-dimensional highway-gated layer. We evaluate this model on four
environments on which the LSTM outperforms the 1.1 M-param feedforward
baseline. Table 8 shows the corresponding results. The extra parameters yield
improvements only on S. Invader, without matching LSTM’s performance. So we
can confirm that the dominance of LSTM over feedforward models in these games
is not simply due to the higher parameter count.

Table 8: Performance of feedforward baseline with more parameters.  | Params. | Berzerk | Gopher | Seaquest | S. Invader
---|---|---|---|---|---
LSTM | 1.5 M |  1,150 $\pm$ 92 |  124,914 $\pm$ 22,422 |  12,643 $\pm$ 1,627 |  137,657 $\pm\,\;$ 2,276
FF | 1.1 M |  343 $\pm$ 23 |  61,350 $\pm\,\;$ 3,891 |  667 $\pm\,\;\,\;\;\;\,$ 1 |  53,455 $\pm\,\;$ 6,694
FF gated | 1.7 M |  320 $\pm$ 29 |  42,851 $\pm\,\;$ 7,653 |  660 $\pm\,\;\,\;\;\;\,$ 0 |  95,629 $\pm$ 11,991

#### Recurrent Delta Net vs. Delta Net.

We also compare the Recurrent Delta Net to a stronger baseline, the Delta Net.
The results are shown in Figures 8 and 8 (sky blue). While the RDN performs
equally well or better than the baseline Delta Net on 13 games at 200 M steps,
there are also 7 games where the Delta Net is better. We thus can not
guarantee strict benefits of additional recurrence here. Again we note that
compared to other models, both the Delta Net and Recurrent Delta Net achieve
outstanding performance on Q*Bert.

#### Delta RNN vs. LSTM.

We also evaluate the Delta RNN (Sec. 3.1) in this RL setting. We first compare
it to the LSTM baseline. As shown in Figures 10 and 10 (green), the Delta RNN
clearly outperforms the LSTM on a few games at 50 M steps. However, the
performance gaps reduce across all games after 200 M steps. Overall, the
performance is close in 7 games, in favour of the LSTM in 8 games, and in
favour of the Delta RNN in 5 games.

#### Recurrent Delta Net vs. Delta RNN.

Finally, we also compare the Recurrent Delta Net to the Delta RNN. Figures 12
and 12 (grey) present our results. In 16 games, the relative performance gap
is within 50%. In one game (Seaquest), the Delta RNN outperforms the RDN. In 3
games, the RDN clearly outperforms the Delta RNN at 200 M steps.

Overall, the Recurrent Delta Net tends to yield decent performance compared to
all baselines. While the performance gaps between the Recurrent Delta Net and
the Delta RNN are rather close, the Recurrent Delta Net performs particularly
well in a few games. As mentioned in the main text, trying deeper
architectures might be a straight-forward way to obtain better scores.

##  Appendix D Comments on Nomenclature

To simplify references to specific Fast Weight Programmers, we gave short
names to all of them, such as Delta RNN or Recurrent Delta Net. We did not
cover, however, many other possible combinations of slow and fast networks as
well as update rules (which are the elementary programming instructions of
FWPs). This calls for a systematic nomenclature to specify the various FWP
types. For a given FWP, one could use “slow-net/update-rule" as a prefix and
"fast-net" architecture as a suffix. For example, the Delta RNN is an FWP with
a fast RNN and a feedforward slow net using the delta rule as elementary
programming instruction. Therefore, using the convention above, the full name
of the Delta RNN would be “Feedforward/Delta fast RNN." The full name of the
Recurrent Delta Net would be “Recurrent/Delta fast Linear Net," and so on.
This is also compatible with the baseline Delta Net, whose full name would be
“Feedforward/Delta fast Linear Net."

Figure 5: Rel. improvements in test scores obtained by the feedforward
baseline compared to LSTM after 50 M env. steps.

Figure 6: Rel. improvements in test scores obtained by the feedforward
baseline compared to LSTM after 200 M env. steps.

Figure 7: Rel. improvements in test scores obtained by the Recurrent Delta Net
compared to the Delta Net after 50 M env. steps.

Figure 8: Rel. improvements in test scores obtained by the Recurrent Delta Net
compared to the Delta Net after 200 M env. steps.

Figure 9: Rel. improvements in test scores obtained by the Delta RNN compared
to the LSTM after 50 M env. steps.

Figure 10: Rel. improvements in test scores obtained by the Delta RNN compared
to the LSTM after 200 M env. steps.

Figure 11: Rel. improvements in test scores obtained by the Recurrent Delta
Net compared to the Delta RNN after 50 M env. steps.

Figure 12: Rel. improvements in test scores obtained by the Recurrent Delta
Net compared to the Delta RNN after 200 M env. steps.

Table 9: Performance after 50 M environmental steps. Reported scores are mean and std of 5 mean-scores obtained over 30 episodes (total of 150 different test episodes). We remind the reader that we denote the Linear Transformer [19] as LT, and our Recurrent Delta Network as RDN. The numbers of parameters are: 1.1 M for the feedforward model, 1.6 M for the LSTM, 1.5 M for the Linear Transformer, the Delta Net, and the Recurrent Delta Net, and finally 1.6 M for the Delta RNN.  | Feedforward | LSTM | LT | Delta Net | RDN | Delta RNN
---|---|---|---|---|---|---
Alien |  1,985 $\pm$ 90 |  846 $\pm$ 81 |  2,135 $\pm$ 184 |  4,704 $\pm$ 452 |  1,754 $\pm$ 48 |  3,420 $\pm$ 834
Amidar |  208 $\pm$ 11 |  233 $\pm$ 10 |  320 $\pm$ 16 |  339 $\pm$ 28 |  368 $\pm$ 23 |  216 $\pm$ 14
Assault |  4,658 $\pm$ 2,147 |  7,551 $\pm$ 1,774 |  2,764 $\pm$ 380 |  5,710 $\pm$ 2,643 |  8,088 $\pm$ 2,851 |  7,503 $\pm$ 2,794
Battlezone |  12,267 $\pm$ 620 |  6,327 $\pm$ 380 |  933 $\pm$ 351 |  6,780 $\pm$ 461 |  7,373 $\pm$ 431 |  7,040 $\pm$ 1,098
Berzerk |  326 $\pm$ 21 |  474 $\pm$ 17 |  323 $\pm$ 6 |  331 $\pm$ 24 |  336 $\pm$ 27 |  305 $\pm$ 8
B. Heist |  323 $\pm$ 13 |  327 $\pm$ 11 |  309 $\pm$ 11 |  321 $\pm$ 8 |  316 $\pm$ 10 |  324 $\pm$ 10
B. Rider |  9,932 $\pm$ 1,592 |  13,638 $\pm$ 1,571 |  6,695 $\pm$ 941 |  9,185 $\pm$ 630 |  18,156 $\pm$ 1,522 |  13,429 $\pm$ 884
D. Attack |  36,255 $\pm$ 3,566 |  31,447 $\pm$ 1,850 |  8,939 $\pm$ 950 |  31,359 $\pm$ 3,362 |  41,726 $\pm$ 6,308 |  36,807 $\pm$ 3,700
Gopher |  10,356 $\pm$ 378 |  13,765 $\pm$ 808 |  8,197 $\pm$ 1,720 |  8,707 $\pm$ 2,381 |  19,775 $\pm$ 1,448 |  25,445 $\pm$ 1,963
James Bond |  2,942 $\pm$ 56 |  3,020 $\pm$ 68 |  2,425 $\pm$ 174 |  3,338 $\pm$ 137 |  2,979 $\pm$ 176 |  2,929.3 $\pm$ 408
Kung-fu |  5,449 $\pm$ 82 |  15,216 $\pm$ 818 |  3,722 $\pm$ 330 |  8,095 $\pm$ 240 |  9,201 $\pm$ 384 |  7,388 $\pm$ 491
MsPacman |  1,737 $\pm$ 53 |  1,676 $\pm$ 86 |  1,647 $\pm$ 101 |  2,116 $\pm$ 30 |  2,584 $\pm$ 121 |  2,287 $\pm$ 32
Pong |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0
Q*Bert |  4,967 $\pm$ 266 |  3,905 $\pm$ 252 |  4,693 $\pm$ 195 |  6,248 $\pm$ 204 |  5,897 $\pm$ 357 |  6,626 $\pm$ 240
Robotank |  7.1 $\pm$ 0.7 |  7.6 $\pm$ 0.7 |  4.8 $\pm$ 0.3 |  7.2 $\pm$ 0.7 |  7.5 $\pm$ 0.8 |  7.0 $\pm$ 0.5
Seaquest |  469 $\pm$ 1 |  708 $\pm$ 1 |  1,812 $\pm$ 61 |  8,853 $\pm$ 937 |  686 $\pm$ 1 |  5,123 $\pm$ 335
S. Invaders |  48,150 $\pm$ 7,233 |  12,461 $\pm$ 1,624 |  2,345 $\pm$ 74 |  25,769 $\pm$ 10,156 |  27,213 $\pm$ 3,359 |  2,847 $\pm$ 10
Stargunner |  9,397 $\pm$ 2,193 |  8,337 $\pm$ 1,094 |  8,915 $\pm$ 713 |  11,599 $\pm$ 3,454 |  9,737 $\pm$ 1,396 |  9,523 $\pm$ 2,214
Up’n down |  185,632 $\pm$ 16,490 |  155,847 $\pm$ 15,318 |  57,435 $\pm$ 2,283 |  120,806 $\pm$ 16,261 |  126,140 $\pm$ 19,078 |  148,759 $\pm$ 28,492
Zaxxon |  4863 $\pm$ 872 |  2,737 $\pm$ 121 |  2,719 $\pm$ 701 |  4,265 $\pm$ 263 |  5,285 $\pm$ 504 |  3,903 $\pm$ 648
Table 10: Performance after 200 M environment steps. Reported scores are mean and std of 5 mean-scores obtained over 30 episodes (total of 150 different test episodes). We remind the reader that we denote the Linear Transformer [19] as LT, and our Recurrent Delta Network as RDN. In cases where performance did not improve after 50 M, we report the performance at 50 M.  | Feedforward | LSTM | LT | Delta Net | RDN | Delta RNN
---|---|---|---|---|---|---
Alien |  3,816 $\pm$ 139 |  6,184 $\pm$ 558 |  4,751 $\pm$ 530 |  15,133 $\pm$ 1,122 |  11,220 $\pm$ 621 |  12,177 $\pm$ 968
Amidar |  433 $\pm$ 27 |  349 $\pm$ 22 |  646 $\pm$ 32 |  432 $\pm$ 27 |  832 $\pm$ 11 |  269 $\pm$ 17
Assault |  6,407 $\pm$ 3,430 |  7,977 $\pm$ 2,611 |  6,465 $\pm$ 1,437 |  7,525 $\pm$ 1,703 |  8,647 $\pm$ 3,061 |  7,670 $\pm$ 952
Battlezone |  60,527 $\pm$ 12,345 |  24,873 $\pm$ 1,240 |  2,667 $\pm$ 386 |  19,907 $\pm$ 1,409 |  10,980 $\pm$ 1,104 |  17,180 $\pm$ 1,493
Berzerk |  343 $\pm$ 23 |  1,150 $\pm$ 92 |  480 $\pm$ 38 |  333 $\pm$ 7 |  348 $\pm$ 17 |  332 $\pm$ 17
B. Heist |  331 $\pm$ 10 |  327 $\pm$ 11 |  317 $\pm$ 8 |  321 $\pm$ 8 |  328 $\pm$ 10 |  328 $\pm$ 8
B. Rider |  21,873 $\pm$ 2,000 |  18,024 $\pm$ 933 |  22,444 $\pm$ 755 |  28,594 $\pm$ 5,508 |  23,934 $\pm$ 2,292 |  28,973 $\pm$ 3,663
D. Attack |  74,904 $\pm$ 10,941 |  69,750 $\pm$ 9,593 |  57,715 $\pm$ 5,009 |  78,601 $\pm$ 16,907 |  67,039 $\pm$ 5,714 |  92,205 $\pm$ 17,933
Gopher |  61,350 $\pm$ 3,891 |  124,914 $\pm$ 22,422 |  48,261 $\pm$ 7,727 |  86,168 $\pm$ 5,069 |  86,008 $\pm$ 11,815 |  101,974 $\pm$ 10,200
James Bond |  56,459 $\pm$ 7,292 |  25,106 $\pm$ 5,889 |  16,223 $\pm$ 1,118 |  54,336 $\pm$ 7,165 |  32,923 $\pm$ 7,968 |  53,344 $\pm$ 4,768
Kung-fu |  12,292 $\pm$ 613 |  24,447 $\pm$ 407 |  13,969 $\pm$ 803 |  15,064 $\pm$ 929 |  20,319 $\pm$ 363 |  15,068 $\pm$ 513
MsPacman |  2,499 $\pm$ 141 |  3,431 $\pm$ 197 |  3,052 $\pm$ 128 |  4,180 $\pm$ 139 |  4,168 $\pm$ 585 |  3,500 $\pm$ 205
Pong |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0 |  21 $\pm$ 0
Q*Bert |  8,655 $\pm$ 371 |  11,513 $\pm$ 910 |  8,389 $\pm$ 349 |  521,839 $\pm$ 36,192 |  987,275 $\pm$ 0 |  10,381 $\pm$ 1,259
Robotank |  7.8 $\pm$ 0.8 |  7.6 $\pm$ 0.7 |  7.7 $\pm$ 0.9 |  7.5 $\pm$ 0.4 |  7.9 $\pm$ 0.6 |  7.5 $\pm$ 0.5
Seaquest |  667 $\pm$ 1 |  12,643 $\pm$ 1,627 |  12,425 $\pm$ 1,910 |  12,790 $\pm$ 1,512 |  4,373 $\pm$ 504 |  13,898 $\pm$ 1,674
S. Invaders |  53,455 $\pm$ 6,694 |  137,657 $\pm$ 2,276 |  2,333 $\pm$ 110 |  86,132 $\pm$ 5,483 |  170,871 $\pm$ 80 |  58,181 $\pm$ 14,987
Stargunner |  11,564 $\pm$ 4,598 |  12,194 $\pm$ 7,038 |  12,035 $\pm$ 6,995 |  11,734 $\pm$ 6,827 |  13,026 $\pm$ 6,431 |  11,635 $\pm$ 6,065
Up’n down |  185,632 $\pm$ 16,490 |  231,157 $\pm$ 10,603 |  252,555 $\pm$ 16,331 |  208,563 $\pm$ 22,803 |  240,003 $\pm$ 26,849 |  159,296 $\pm$ 25,013
Zaxxon |  11,960 $\pm$ 538 |  11,619 $\pm$ 663 |  7,371 $\pm$ 932 |  1,0523 $\pm$ 568 |  9,126 $\pm$ 313 |  11,365 $\pm$ 678