# Tests of general relativity in the nonlinear regime:
a parametrized plunge-merger-ringdown waveform model

E. Maggio  Max Planck Institute for Gravitational Physics (Albert Einstein
Institute), D-14476 Potsdam, Germany

###### Abstract

Gravitational waves (GWs) provide a unique opportunity to test gravity in the
dynamical and nonlinear regime. We present a parametrized test of general
relativity (GR) that introduces generic deviations to the plunge, merger and
ringdown stages of binary–black-hole coalescences. The novel feature of the
model is that it can capture signatures of beyond-GR physics in the plunge-
merger phase. We use the model to provide constraints on the plunge-merger
parameters from the analysis of GW150914. Alarmingly, we find that GW200129
shows a strong violation of GR. We interpret this result as a false violation
of GR either due to waveform systematics (mismodeling of spin precession) or
data-quality issues.

##  1 Parametrized plunge-merger-ringdown model

We use as our baseline model the waveform constructed in the effective-one-
body formalism for quasicircular and spin-aligned binaries, which is
calibrated to numerical relativity simulations and contains high-order modes,
i.e., SEOBNRv4HM$\textunderscore$PA [1, 2, 3, 4, 5, 6]. The waveform is
constructed by attaching the merger-ringdown waveform to the inspiral-plunge
waveform at a matching time,

$h_{\ell m}(t)=h_{\ell m}^{\rm insp-plunge}(t)\,\Theta\left(t_{\rm
match}^{\ell m}-t\right)+h_{\ell m}^{\rm merger-RD}(t)\,\Theta\left(t-t_{\rm
match}^{\ell m}\right)\,,$ (1)

where $\Theta(t)$ is the Heaviside function, and the matching time is defined
as

$t_{\rm match}^{\ell m}=\begin{cases}t^{22}_{\rm
peak}\,,&(\ell,m)=(2,2),\,(3,3),\,(2,1),(4,4)\,,\\\ t^{22}_{\rm
peak}-10M\,,&(\ell,m)=(5,5)\,,\end{cases}$ (2)

where $t^{22}_{\rm peak}$ is the time in which the amplitude of the $(2,2)$
mode peaks. The amplitude and the orbital frequency peak at different times,
therefore we introduce a time-lag parameter $\Delta t^{\rm{GR}}_{\ell
m}=t^{\Omega}_{\rm peak}-t^{22}_{\rm peak}$. We develop a parametrized
waveform model that allows for deviations from GR in the plunge-merger-
ringdown stage [7]. We introduce fractional deviations to the NR-informed
amplitude and frequency at the matching time and the time-lag parameter as,

$\displaystyle|h_{{\ell m}}^{\rm NR}|$ $\displaystyle\to$
$\displaystyle|h_{{\ell m}}^{\rm NR}|\,(1+\delta A_{{\ell m}})\,,$ (3)
$\displaystyle\omega_{{\ell m}}^{\rm NR}$ $\displaystyle\to$
$\displaystyle\omega_{{\ell m}}^{\rm NR}\,(1+\delta\omega_{{\ell m}})\,,$ (4)
$\displaystyle\Delta t^{\rm GR}_{\ell m}$ $\displaystyle\to$
$\displaystyle\Delta t^{\rm GR}_{\ell m}\,\left(1+\delta\Delta t_{\ell
m}\right)\,.$ (5)

We also introduce deformations to the quasi-normal modes following the same
strategy applied in Refs. [8, 9] for the oscillation frequency and damping
time of the fundamental modes,

$\displaystyle f_{{\ell m}0}$ $\displaystyle\to f_{{\ell m}0}\,(1+\delta
f_{{\ell m}0})\,,$ $\displaystyle\tau_{{\ell m}0}$
$\displaystyle\to\tau_{{\ell m}0}\,(1+\delta\tau_{{\ell m}0})\,.$

For simplicity, we assume that the plunge-merger parameters have the same
values across different modes (i.e., $\delta A_{{\ell m}}=\delta A$,
$\delta\omega_{\ell m}=\delta\omega$, $\delta\Delta t_{\ell m}=\delta\Delta
t$), and the ringdown parameters are nonzero only for the fundamental $(2,2)$
mode. Figure 1 shows the waveform morphology as a function of time where the
plunge-merger parameters vary in the range $[-0.5,0.5]$ for $\delta A$ (top
panel), $\delta\omega$ (middle panel) and $\delta\Delta t$ (bottom panel). The
GR waveform is shown by the black solid line.

Figure 1: Time evolution of the GW strain for non-zero values of the plunge-
merger parameters.

##  2 Bayesian parameter estimation

We analyse the events GW150914 [10] and GW200129 [11], which are among the
loudest binary–black-hole signals to date, by varying the binary parameters
and a subset of the deviation parameters, i.e., $\\{\delta A,\delta\omega\\}$,
to perform a “plunge-merger test of GR”. From the analysis of GW150914, we
provide constraints on the deviations from GR, i.e., $\delta
A=-0.01^{+0.27}_{-0.19}$ and $\delta\omega=0.00^{+0.17}_{-0.12}$ [7].
Deviations in the intermediate region of the IMRPhenom waveform were
constrained in the TIGER test [12]. From the analysis of GW200129, the
inferred value of $\delta\omega$
$\left(\delta\omega=-0.002^{+0.097}_{-0.082}\right)$ is consistent with GR,
however the inferred value of $\delta A$ $\left(\delta
A=0.44^{+0.38}_{-0.28}\right)$ exhibits a gross violation of GR. The apparent
deviation from GR could be due to systematic errors in the GW modeling or
data-quality issues as highlighted by the analyses of Refs. [13, 14]. We
explore the former possibility by performing synthetic-data studies where the
injected GW signal is generated with spin-precessing waveform models. We
conclude that the presence of spin-precession could bias us to find a false
evidence for beyond-GR effects when using a nonprecessing non-GR model.

## Acknowledgments

We thank Alessandra Buonanno and Hector O. Silva for discussions, and Gregorio
Carullo for comments. We acknowledge funding from the Deutsche
Forschungsgemeinschaft (DFG) - project number: 386119226. The material
presented in this paper is based upon work supported by National Science
Foundation’s (NSF) LIGO Laboratory, which is a major facility fully funded by
the NSF.

## References

## References

  * [1] A. Buonanno and T. Damour, Phys. Rev. D 59, 084006 (1999).
  * [2] A. Buonanno and T. Damour, Phys. Rev. D 62, 064015 (2000).
  * [3] A. Bohé et al, Phys. Rev. D 95, 044028 (2017).
  * [4] R. Cotesta et al, Phys. Rev. D 98, 084028 (2018).
  * [5] A. Nagar and P. Rettegno, Phys. Rev. D 99, 021501 (2019).
  * [6] D. Mihaylov, S. Ossokine, A. Buonanno and A. Ghosh, Phys. Rev. D 104, 124087 (2021).
  * [7] E. Maggio, H. O. Silva, A. Buonanno and A. Ghosh, arXiv:2212.09655 (2022).
  * [8] R. Brito, A. Buonanno and V. Raymond, Phys. Rev. D 98, 084038 (2018).
  * [9] A. Ghosh, R. Brito and A. Buonanno, Phys. Rev. D 103, 124041 (2021).
  * [10] B. P. Abbott et al, Phys. Rev. Lett. 116, 061102 (2016).
  * [11] B. P. Abbott et al, arXiv:2111.03606 (2021).
  * [12] B. P. Abbott et al., Phys. Rev. Lett. 116, 221101 (2016).
  * [13] M. Hannam et al, Nature 610, 7933 (2022).
  * [14] E. Payne et al, Phys. Rev. D 106, 104017 (2022).


# Elephant Neural Networks:
Born to Be a Continual Learner

Qingfeng Lan
Department of Computing Science
University of Alberta
<EMAIL_ADDRESS>
&A. Rupam Mahmood
Department of Computing Science
University of Alberta
CIFAR AI Chair, Amii
<EMAIL_ADDRESS>

###### Abstract

Catastrophic forgetting remains a significant challenge to continual learning
for decades. While recent works have proposed effective methods to mitigate
this problem, they mainly focus on the algorithmic side. Meanwhile, we do not
fully understand what architectural properties of neural networks lead to
catastrophic forgetting. This study aims to fill this gap by studying the role
of activation functions in the training dynamics of neural networks and their
impact on catastrophic forgetting. Our study reveals that, besides sparse
representations, the gradient sparsity of activation functions also plays an
important role in reducing forgetting. Based on this insight, we propose a new
class of activation functions, elephant activation functions, that can
generate both sparse representations and sparse gradients. We show that by
simply replacing classical activation functions with elephant activation
functions, we can significantly improve the resilience of neural networks to
catastrophic forgetting. Our method has broad applicability and benefits for
continual learning in regression, class incremental learning, and
reinforcement learning tasks. Specifically, we achieves excellent performance
on Split MNIST dataset in just one single pass, without using replay buffer,
task boundary information, or pre-training.

##  1 Introduction

One of the biggest challenges to achieving continual learning is the decades-
old issue of _catastrophic forgetting_ (French 1999). Catastrophic forgetting
stands for the phenomenon that artificial neural networks tend to forget prior
knowledge drastically when learned with stochastic gradient descent algorithms
on non-independent and identically distributed (non-iid) data. In recent
years, researchers have made significant progress in mitigating catastrophic
forgetting and proposed many effective methods, such as replay methods (Mendez
et al. 2022), regularization-based methods (Riemer et al. 2018), parameter-
isolation methods (Mendez et al. 2020), and optimization-based methods
(Farajtabar et al. 2020). However, instead of alleviating forgetting by
designing neural networks with specific properties, most of these methods
circumvent the problem by focusing on the algorithmic side. There is still a
lack of full understanding of what properties of neural networks lead to
catastrophic forgetting. Recently, Mirzadeh et al. (2022a) found that the
width of a neural network significantly affects forgetting and they provided
explanations from the perspectives of gradient orthogonality, gradient
sparsity, and lazy training regime. Furthermore, Mirzadeh et al. (2022b)
studied the forgetting problem on large-scale benchmarks with various neural
network architectures. They demonstrated that architectures can play a role
that is as important as algorithms in continual learning.

The interactions between continual learning and neural network architectures
remain to be under-explored. In this work, we aim to better understand and
reduce forgetting by studying the impact of various architectural choices of
neural networks on continual learning. Specifically, we focus on activation
functions, one of the most important elements in neural networks.
Experimentally, we study the effect of various activation functions (e.g.,
Tanh, ReLU, and Sigmoid) on catastrophic forgetting under the setting of
continual supervised learning. Theoretically, we investigate the role of
activation functions in the training dynamics of neural networks. Our analysis
suggests that not only sparse representations but also sparse gradients are
essential for continual learning. Based on this discovery, we design a new
class of activation functions, the elephant activation functions. Unlike
classical activation functions, elephant activation functions are able to
generate both sparse function values and sparse gradients that make neural
networks more resilient to catastrophic forgetting. Correspondingly, the
neural networks incorporated with elephant activation functions are named
elephant neural networks (ENNs). In streaming learning for regression, while
many classical activation functions fail to even approximate a simple sine
function, ENNs succeed by applying elephant activation functions. In class
incremental learning, we show that ENNs achieve excellent performance on Split
MNIST dataset in one single pass, without using any replay buffer, task
boundary information, or pre-training. In reinforcement learning, ENNs improve
the learning performance of agents under extreme memory constraints by
alleviating the forgetting issue.

##  2 Investigating catastrophic forgetting via training dynamics

Firstly, we look into the forgetting issue via the training dynamics of neural
networks. Consider a simple regression task. Let a scalar function
$f_{{\bm{w}}}({\bm{x}})$ be represented as a neural network, parameterized by
${\bm{w}}$, with input ${\bm{x}}$. $F({\bm{x}})$ is the true function and the
loss function is $L(f,F,{\bm{x}})$. For example, for squared error, we have
$L(f,F,{\bm{x}})=(f_{{\bm{w}}}({\bm{x}})-F({\bm{x}}))^{2}$. At each time step
$t$, a new sample $\\{{\bm{x}}_{t},F({\bm{x}}_{t})\\}$ arrives. Given this new
sample, to minimize the loss function $L(f,F,{\bm{x}}_{t})$, we update the
weight vector by ${\bm{w}}^{\prime}={\bm{w}}+\Delta_{\bm{w}}$ where
$\Delta_{\bm{w}}$ is the weight difference. With the stochastic gradient
descent (SGD) algorithm, we have
${\bm{w}}^{\prime}={\bm{w}}-\alpha\nabla_{\bm{w}}L(f,F,{\bm{x}}_{t})$, where
$\alpha$ is the learning rate. So
$\Delta_{\bm{w}}={\bm{w}}^{\prime}-{\bm{w}}=-\alpha\nabla_{\bm{w}}L(f,F,{\bm{x}}_{t})=-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})$.
By Taylor expansion,

$~{}f_{{\bm{w}}^{\prime}}({\bm{x}})-f_{{\bm{w}}}({\bm{x}})=-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})\,\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle+O(\Delta_{\bm{w}}^{2}),$
(1)

where $\langle\cdot,\cdot\rangle$ denotes Frobenius inner product or dot
product depending on the context. In this equation, since
$-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})$ is unrelated to ${\bm{x}}$, we only
consider the quantity
$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle$,
which is known as the neural tangent kernel (NTK) (Jacot et al. 2018). Without
loss of generality, assume that the original prediction
$f_{{\bm{w}}}({\bm{x}}_{t})$ is wrong, i.e. $f_{{\bm{w}}}({\bm{x}}_{t})\neq
F({\bm{x}}_{t})$ and $\nabla_{f}L(f,F,{\bm{x}}_{t})\neq 0$. To correct the
wrong prediction while avoiding forgetting, we expect this NTK to satisfy two
properties that are essential for continual learning:

######  Property 2.1 (error correction).

For ${\bm{x}}={\bm{x}}_{t}$,
$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle\neq
0$.

######  Property 2.2 (zero forgetting).

For ${\bm{x}}\neq{\bm{x}}_{t}$,
$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=0$.

In particular, 2.1 allows for error correction by optimizing
$f_{{\bm{w}}^{\prime}}({\bm{x}}_{t})$ towards the true value
$F({\bm{x}}_{t})$, so that we can learn new knowledge (i.e., update the
learned function). If
$\langle\nabla_{\bm{w}}f({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=0$,
we then have $f_{{\bm{w}}^{\prime}}({\bm{x}})-f_{{\bm{w}}}({\bm{x}})\approx
0$, failing to correct the wrong prediction at ${\bm{x}}={\bm{x}}_{t}$.
Essentially, Property 2.1 requires the gradient norm to be non-zero. On the
other hand, 2.2 is much harder to be satisfied, especially for non-linear
approximations. Essentially, to make this property hold, except for
${\bm{x}}={\bm{x}}_{t}$, the neural network $f$ is required to achieve zero
knowledge forgetting after one step optimization, i.e.
$\forall{\bm{x}}\neq{\bm{x}}_{t},f_{{\bm{w}}^{\prime}}({\bm{x}})=f_{{\bm{w}}}({\bm{x}})$.
_It is the violation of 2.2 that leads to the forgetting issue._ For tabular
cases (e.g., ${\bm{x}}$ is a one-hot vector and $f_{{\bm{w}}}({\bm{x}})$ is a
linear function), this property may hold by sacrificing the generalization
ability of deep neural networks. In order to benefit from generalization, we
propose 2.3 by relaxing 2.2.

######  Property 2.3 (local elasticity).

$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle\approx
0$ for ${\bm{x}}$ that is dissimilar to ${\bm{x}}_{t}$ in a certain sense.

2.3 is known as local elasticity (He & Su 2020). A function $f$ is locally
elastic if $f_{{\bm{w}}}({\bm{x}})$ is not significantly changed, after the
function is updated at ${\bm{x}}_{t}$ that is dissimilar to ${\bm{x}}$ in a
certain sense. For example, we can characterize the dissimilarity with the
$2$-norm distance. Although He & Su (2020) show that neural networks with
nonlinear activation functions are locally elastic in general, there is a lack
of theoretical understanding about the connection of neural network
architectures and the degrees of local elasticity. In our experiments, we find
that the degrees of local elasticity of classical neural networks are not
enough to address the forgetting issue, as we will show next.

##  3 Understanding the success and failure of sparse representations

The above properties can help to deepen our understanding of the success and
failure of sparse representations in continual learning. To be specific, we
argue that sparse representations are effective to reduce forgetting in linear
function approximations, but are less useful in non-linear function
approximations.

It is well-known that deep neural networks can automatically generate
effective representations (a.k.a. features) to extract key properties from
input data. The ability to learn useful features helps deep learning methods
achieve great success in many areas (LeCun et al. 2015). In particular, we
call a set of representations sparse when only a small part of representations
is non-zero for a given input. Sparse representations are shown to help reduce
the forgetting problem and the interference issues in both continual
supervised learning and reinforcement learning (Shen et al. 2021; Liu et al.
2019). Formally, let ${\bm{x}}$ be an input and $\phi$ be an encoder that
transforms the input ${\bm{x}}$ into its representation $\phi({\bm{x}})$. The
representation $\phi({\bm{x}})$ is sparse when most of its elements are zeros.

First, consider the case of linear approximations. A linear function is
defined as
$f_{{\bm{w}}}({\bm{x}})={\bm{w}}^{\top}\phi({\bm{x}}):\mathbb{R}^{n}\mapsto\mathbb{R}$,
where ${\bm{x}}\in\mathbb{R}^{n}$ is an input,
$\phi:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}$ is a fixed encoder, and
${\bm{w}}\in\mathbb{R}^{m}$ is a weight vector. Assume that the representation
$\phi({\bm{x}})$ is sparse and non-zero (i.e., $\|\phi({\bm{x}})\|_{2}>0$) for
${\bm{x}}\in\mathbb{R}^{n}$. Next, we show that both 2.1 and 2.3 are satisfied
in this case. Easy to know that
$\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}})=\phi({\bm{x}})$. Together with Equation
1, we have

$~{}f_{{\bm{w}}^{\prime}}({\bm{x}})-f_{{\bm{w}}}({\bm{x}})=-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})\,\phi({\bm{x}})^{\top}\phi({\bm{x}}_{t}).$
(2)

By assumption, $f_{{\bm{w}}}({\bm{x}}_{t})\neq F({\bm{x}}_{t})$ and
$\nabla_{f}L(f,F,{\bm{x}}_{t})\neq 0$. Then 2.1 holds since
$f_{{\bm{w}}^{\prime}}({\bm{x}}_{t})-f_{{\bm{w}}}({\bm{x}}_{t})=-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})\,\|\phi({\bm{x}}_{t})\|_{2}^{2}\neq
0$. Moreover, when ${\bm{x}}\neq{\bm{x}}_{t}$, it is very likely that
$\langle\phi({\bm{x}}),\phi({\bm{x}}_{t})\rangle\approx 0$ due to the sparsity
of $\phi({\bm{x}})$ and $\phi({\bm{x}}_{t})$. Thus,
$f_{{\bm{w}}^{\prime}}({\bm{x}})-f_{{\bm{w}}}({\bm{x}})=-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})\,\langle\phi({\bm{x}}),\phi({\bm{x}}_{t})\rangle\approx
0$ and 2.3 holds. We conclude that by satisfying 2.1 and 2.3, sparse
representations successfully help mitigate catastrophic forgetting in linear
approximations.

However, for non-linear approximations, sparse representations can no longer
guarantee 2.3. Consider an MLP with one hidden layer
$f_{{\bm{w}}}({\bm{x}})={\bm{u}}^{\top}\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}}):\mathbb{R}^{n}\mapsto\mathbb{R}$,
where $\sigma$ is a non-linear activation function,
${\bm{x}}\in\mathbb{R}^{n}$, ${\bm{u}}\in\mathbb{R}^{m}$,
${\mathbf{V}}\in\mathbb{R}^{m\times n}$, ${\bm{b}}\in\mathbb{R}^{m}$, and
${\bm{w}}=\\{{\bm{u}},{\mathbf{V}},{\bm{b}}\\}$. We compute the NTK in this
case, resulting in the following lemma.

######  Lemma 3.1 (NTK in non-linear approximations).

Given a non-linear function
$f_{{\bm{w}}}({\bm{x}})={\bm{u}}^{\top}\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}}):\mathbb{R}^{n}\mapsto\mathbb{R}$,
where $\sigma$ is a non-linear activation function,
${\bm{x}}\in\mathbb{R}^{n}$, ${\bm{u}}\in\mathbb{R}^{m}$,
${\mathbf{V}}\in\mathbb{R}^{m\times n}$, ${\bm{b}}\in\mathbb{R}^{m}$, and
${\bm{w}}=\\{{\bm{u}},{\mathbf{V}},{\bm{b}}\\}$. The NTK of this non-linear
function is

$\displaystyle\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})+{\bm{u}}^{\top}{\bm{u}}({\bm{x}}^{\top}{\bm{x}}_{t}+1)\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma^{\prime}({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}}),$

where $\langle\cdot,\cdot\rangle$ denotes Frobenius inner product or dot
product depending on the context.

The proof of this lemma can be found in Appendix B. Note that the encoder
$\phi$ is no longer fixed, and we have
$\phi_{{\bm{\theta}}}({\bm{x}})=\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}})$, where
${\bm{\theta}}=\\{{\mathbf{V}},{\bm{b}}\\}$ are learnable parameters. By Lemma
3.1,

$\displaystyle~{}\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=\phi_{{\bm{\theta}}}({\bm{x}})^{\top}\phi_{{\bm{\theta}}}({\bm{x}}_{t})+{\bm{u}}^{\top}{\bm{u}}({\bm{x}}^{\top}{\bm{x}}_{t}+1)\phi_{{\bm{\theta}}}^{\prime}({\bm{x}})^{\top}\phi_{{\bm{\theta}}}^{\prime}({\bm{x}}_{t}).$
(3)

Compared with the NTK in linear approximations (Equation 2), Equation 3 has an
additional term
${\bm{u}}^{\top}{\bm{u}}({\bm{x}}^{\top}{\bm{x}}_{t}+1)\phi_{{\bm{\theta}}}^{\prime}({\bm{x}})^{\top}\phi_{{\bm{\theta}}}^{\prime}({\bm{x}}_{t})$,
due to a learnable encoder $\phi_{{\bm{\theta}}}$. With sparse
representations, we have
$\phi_{{\bm{\theta}}}({\bm{x}})^{\top}\phi_{{\bm{\theta}}}({\bm{x}}_{t})\approx
0$. However, it is not necessary true that
${\bm{u}}^{\top}{\bm{u}}({\bm{x}}^{\top}{\bm{x}}_{t}+1)\phi_{{\bm{\theta}}}^{\prime}({\bm{x}})^{\top}\phi_{{\bm{\theta}}}^{\prime}({\bm{x}}_{t})\approx
0$ even when ${\bm{x}}$ and ${\bm{x}}_{t}$ are quite dissimilar, which
violates 2.3.

To conclude, our analysis indicates that sparse representations alone are not
effective enough to reduce forgetting in non-linear approximations.

##  4 Obtaining sparsity with elephant activation functions

Although Lemma 3.1 shows that the forgetting issue can not be fully addressed
with sparse representations solely in deep learning methods, it also points
out a possible solution: sparse gradients. With sparse gradients, we could
have
$\phi_{{\bm{\theta}}}^{\prime}({\bm{x}})^{\top}\phi_{{\bm{\theta}}}^{\prime}({\bm{x}}_{t})\approx
0$. Together with sparse representations, we may still satisfy 2.3 in non-
linear approximations, and thus reduce more forgetting. Specifically, we aim
to design new activation functions to obtain both sparse representations and
sparse gradients.

To begin with, we first define the sparsity of an function, which also applies
to activation functions.

######  Definition 4.1 (sparse function).

For a function $\sigma:\mathbb{R}\mapsto\mathbb{R}$, we define the sparsity of
function $\sigma$ on input domain $[-C,C]$ as

$S_{\epsilon,C}(\sigma)=\frac{\left|\\{x\mid\left|\sigma(x)\right|\leq\epsilon,x\in[-C,C]\\}\right|}{\left|\\{x\mid
x\in[-C,C]\\}\right|}=\frac{\left|\\{x\mid\left|\sigma(x)\right|\leq\epsilon,x\in[-C,C]\\}\right|}{2C},$

where $\epsilon$ is a small positive number and $C>0$. As a special case, when
$\epsilon\rightarrow 0^{+}$ and $C\rightarrow\infty$, define

$S(\sigma)=\lim_{\epsilon\rightarrow
0^{+}}\lim_{C\rightarrow\infty}S_{\epsilon,C}(\sigma).$

We call $\sigma$ a $S(\sigma)$-sparse function. In particular, $\sigma$ is
called a sparse function if it is a $1$-sparse function, i.e. $S(\sigma)=1$.

######  Remark 4.2.

Easy to verify that $0\leq S(\sigma)\leq 1$. The sparsity of a function shows
the fraction of nearly zero outputs given a symmetric input domain. For
example, both $\operatorname{ReLU}(x)$ and $\operatorname{ReLU}^{\prime}(x)$
are $\frac{1}{2}$-sparse functions. $\operatorname{Tanh}(x)$ is a $0$-sparse
function while $\operatorname{Tanh}^{\prime}(x)$ is a sparse function. In the
appendix, we summarize the results for common activation functions in Table 3
and visualize the activation functions with their gradients in Figure 5.

Next, we propose to use a novel class of bell-shaped activation functions,
elephant activation functions 111We name this bell-shaped activation function
as the elephant function since the bell-shape is similar to the shape of an
elephant (see Figure 1), honoring the work The Little Prince by Antoine de
Saint Exupéry. This name also hints that this activation function empowers
neural networks with continual learning ability, echoing the old saying that
“an elephant never forgets”.. Formally, an elephant function is defined as

$\operatorname{Elephant}(x)=\frac{1}{1+\left|\frac{x}{a}\right|^{d}},$ (4)

where $a$ controls the width of the function and $d$ controls the slope. We
call a neural network that uses elephant activation functions an elephant
neural network (ENN). For example, for MLPs and CNNs, we have elephant MLPs
(EMLPs) and elephant CNNs (ECNNs), respectively.

(a) A drawing of an elephant. (b) $\operatorname{Elephant}(x)$ (c)
$\operatorname{Elephant}^{\prime}(x)$

Figure 1: (a) “My drawing was not a picture of a hat. It was a picture of a
boa constrictor digesting an elephant.” The Little Prince, by Antoine de Saint
Exupéry. (b) Elephant functions with $a=2$ and various $d$. (c) The gradient
of elephant functions with $a=2$ and various $d$.

As shown in Figure 1, elephant activation functions have both sparse function
values and sparse gradient values, which can also be formally proved.

######  Lemma 4.3.

$\operatorname{Elephant}(x)$ and $\operatorname{Elephant}^{\prime}(x)$ are
sparse functions.

Specifically, $d$ controls the sparsity of gradients for elephant functions.
The larger the value of $d$, the sharper the slope of an elephant function and
the sparser the gradient. On the other hand, $a$ stands for the width of the
function, controlling the sparsity of the function itself.

Next, we show that 2.3 holds under certain conditions for elephant functions.

######  Theorem 4.4.

Define $f_{{\bm{w}}}({\bm{x}})$ as in Lemma 3.1. Let $\sigma$ be the elephant
activation function with $d\rightarrow\infty$. When
$\left|{\mathbf{V}}({\bm{x}}-{\bm{x}}_{t})\right|\succ 2a\mathbf{1}_{m}$, we
have
$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=0$,
where $\succ$ denotes an element-wise inequality symbol and
$\mathbf{1}_{m}=[1,\cdots,1]^{\top}\in\mathbb{R}^{m}$.

######  Remark 4.5.

Theorem 4.4 mainly proves that when $d\rightarrow\infty$, 2.3 holds for an
EMLP with one hidden layer. However, even when $d$ is a small integer (e.g.,
8), EMLPs still exhibit local elasticity, as we will show in the experiment
section. The proof is in Appendix B.

##  5 Experiments

In this section, we first perform experiments in a simple regression task in
the streaming learning setting. We will show that (1) sparse representations
alone are not enough to address the forgetting issue, (2) ENNs can continually
learn to solve regression tasks by reducing forgetting, and (3) ENNs are
locally elastic even when $d$ is a small integer. Next, we show that by
incorporating elephant functions, ENNs are able to achieve better performance
than several baselines in class incremental learning, without utilizing pre-
training, replay buffer, or task boundaries. Finally, we apply ENNs in
reinforcement learning (RL) and demonstrate ENNs’ advantage in reducing
catastrophic forgetting.

###  5.1 Streaming learning for regression

In the real world, regression tasks are everywhere, from house price
estimations (Madhuri et al. 2019) to stock predictions (Dase & Pawar 2010),
weather predictions (Ren et al. 2021), and power consumption forecasts (Dmitri
et al. 2016). However, most prior continual learning methods are designed for
classification tasks rather than regression tasks, although catastrophic
forgetting frequently arises in regression tasks as well (He & Sick 2021). In
this section, we conduct experiments on a simple regression task in the
streaming learning setting. In this setting, a learning agent is presented
with one sample only at each time step and then performs learning updates
given this new sample. Moreover, the learning process happens in a single pass
of the whole dataset, that is, each sample only occurs once. Furthermore, the
data stream is assumed to be non-iid. Finally, the evaluation happens after
each new sample arrives, which requires the agent to learn quickly while
avoiding catastrophic forgetting. Streaming learning methods enable real-time
adaptation; thus they are more suitable in real-world scenarios where data is
received in a continuous flow.

We consider streaming learning for approximating a sine function. In this
task, there is a stream of data
$(x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{n},y_{n})$, where $0\leq
x_{1}<x_{2}<\cdots<x_{n}\leq 2$ and $y_{i}=\sin(\pi x_{i})$. We set $n=200$ in
our experiment. The learning agent $f$ is an MLP with one hidden layer of size
$1,000$. At each time step $t$, the learning agent only receives one sample
$(x_{t},y_{t})$. We minimize the square error loss
$l_{t}=(f(x_{t})-y_{t})^{2}$ with Adam optimizer (Kingma & Ba 2015), where
$f(x_{t})$ is the agent’s prediction. During an evaluation, the agent
performance is measured by the mean square error (MSE) on a test dataset with
$1,000$ samples, where the inputs are evenly spaced over the interval $[0,2]$.

We compare our method (EMLP) with two kinds of baselines. One is an MLP with
classical activation functions, including $\operatorname{ReLU}$,
$\operatorname{Sigmoid}$, $\operatorname{ELU}$, and $\operatorname{Tanh}$. The
other is the sparse representation neural network (SR-NN) (Liu et al. 2019)
which is designed to generate sparse representations. Specifically, we apply
various classical activation functions ($\operatorname{ReLU}$,
$\operatorname{Sigmoid}$, $\operatorname{ELU}$, and $\operatorname{Tanh}$) in
SR-NNs. We set $d=8$ for all elephant functions applied in EMLP.

We summarize test MSEs in Table 1. A lower MSE indicates better performance.
Additional training details are included in the appendix. For the SR-NN, we
present the best result among the combinations of SR-NNs and classical
activation functions. Clearly, our method EMLP achieves the best performance,
reaching a very low test MSE compared with baselines. Generally, we find that
the SR-NN achieves slightly better performance than MLPs with classical
activation functions, showing the benefits of sparse representations. However,
compared with EMLP, the test MSE of the SR-NN is still large, indicating that
the SR-NN fails to approximate $\sin(\pi x)$.

Table 1: The test MSE of various networks in streaming learning for a simple regression task. Lower is better. All results are averaged over $5$ runs, reported with standard errors. Method | Test Performance (MSE)
---|---
MLP ($\operatorname{ReLU}$) | $0.4729\pm 0.0110$
MLP ($\operatorname{Sigmoid}$) | $0.4583\pm 0.0008$
MLP ($\operatorname{Tanh}$) | $0.4461\pm 0.0013$
MLP ($\operatorname{ELU}$) | $0.4521\pm 0.0019$
SR-NN | $0.4061\pm 0.0036$
EMLP ($\operatorname{Elephant}$) |  0.0081 $\pm$ 0.0009

Next, we plot the true function $\sin(\pi x)$, the learned function $f(x)$,
and the NTK function
$\operatorname{NTK}(x)=\langle\nabla_{\bm{w}}f_{{\bm{w}}}(x),\nabla_{\bm{w}}f_{{\bm{w}}}(x_{t})\rangle$
at different training stages for EMLP and SR-NN in Figure 2. The plots of MLP
($\operatorname{ReLU}$), MLP ($\operatorname{Sigmoid}$), MLP
($\operatorname{Tanh}$), and MLP ($\operatorname{ELU}$) are not presented,
since they are similar to the plots of the SR-NN. The NTK function
$\operatorname{NTK}({\bm{x}})$ is normalized such that the function value is
in $[-1,1]$. In Figure 2, the plots in the first row show that for EMLP with
$d=8$, $\operatorname{NTK}(x)$ quickly decreases to 0 as $x$ moving away from
$x_{t}$, demonstrating the local elasticity (2.3) of EMLP with a small $d$.
However, SR-NNs (and MLPs with classical activation functions) are not locally
elastic; the learned function basically evolves as a linear function, a
phenomenon often appears in over-parameterized neural networks (Jacot et al.
2018; Chizat et al. 2019).

(a) EMLP at $t=100$ (b) EMLP at $t=150$ (c) EMLP at $t=200$ (d) SR-NN at
$t=100$

(e) SR-NN at $t=150$ (f) SR-NN at $t=200$

Figure 2: Plots of the true function $\sin(\pi x)$, the learned function
$f(x)$, and the NTK function $\operatorname{NTK}(x)$ at different training
stages for EMLP and SR-NN. The NTK function $\operatorname{NTK}(x)$ of the
EMLP quickly reduces to 0 as $x$ moves away from $x_{t}$, demonstrating the
local elasticity (2.3) of EMLP.

By injecting local elasticity to a neural network, we can break the inherent
global generalization ability of the neural networks (Ghiassian et al. 2020),
constraining the output changes of the neural network to small local areas.
Utilizing this phenomenon, we can update a wrong prediction by “editing”
outputs of a neural network nearly point-wisely. To verify, we first train a
neural network to approximate $\sin(\pi x)$ well in the traditional supervised
learning style. We call the function the old learned function at this stage.
Now assume that the original $y$ value of an input $x$ is changed to
$y^{\prime}$, while the true values of other inputs remain the same. The goal
is to update the prediction of the neural network for input $x$ to this new
value $y^{\prime}$, while keeping the predictions of other inputs close to the
original predictions, without expensive re-training on the whole dataset. Note
that this requirement is common, especially in RL; and we will illustrate it
with more details later. For now, we focus on supervised learning for a
clearer demonstration. Specifically, we would like to update the output value
at $x=1.5$ of the learned function from $y=-1.0$ to $y^{\prime}=-1.5$. We
perform experiments on both EMLP and MLP, showing in Figure 3. Clearly, both
neural works successfully update the prediction at $x=1.5$ to the new value.
However, besides the prediction at $x=1.5$, the learned function of MLP is
changed globally while the changes of EMLP are mainly confined in a small
local area around $x=1.5$. That is, we can successfully correct the wrong
prediction nearly point-wisely by “editing” the output value for ENNs, but not
for classical neural networks. To conclude, we summarize our findings in the
following:

  * •

Sparse representations are useful but are not effective enough to solve
streaming regression tasks.

  * •

EMLP is locally elastic even when $d$ is small (e.g., $8$) and it allows
“editing” the output value nearly point-wisely.

  * •

EMLP can continually learn to solve regression tasks by reducing forgetting.

(a) EMLP ($\operatorname{Elephant}$) (b) MLP ($\operatorname{ReLU}$)

Figure 3: A demonstration of local elasticity of the EMLP. EMLP allows
updating the old prediction nearly point-wisely by “editing” the output value
of the neural network. However, we cannot achieve this for MLP since it is not
locally elastic.

###  5.2 Class incremental learning

In addition to regression tasks, our method can be applied to classification
tasks as well, by simply replacing classical activation functions with
elephant activation functions in a classification model. Though our model is
agnostic to data distributions, we test it in class incremental learning in
order to compare it with previous methods. Moreover, we adopt a stricter
variation of the continual learning setting by adding the following
restrictions: (1) same as streaming learning, each sample only occurs once
during training, (2) task boundaries are not provided or inferred (Aljundi et
al. 2019a), (3) neither pre-training nor a fixed feature encoder is allowed
(Wolfe & Kyrillidis 2022), and (4) no buffer is allowed to store old task
information in any form, such as training samples and gradients.

Surprisingly, we find no methods are designed for or have been tested in the
above setting. As a variant of EWC (Kirkpatrick et al. 2017), Online EWC
(Schwarz et al. 2018) almost meets these requirements, although it still
requires task boundaries. To overcome this issue, we propose Streaming EWC as
one of the baselines, which updates the fisher information matrix after every
training sample. Streaming EWC can be viewed as a special case of Online EWC,
treating each training sample as a new task. Besides Streaming EWC, we
consider SDMLP (Bricken et al. 2023) and FlyModel (Shen et al. 2021) as two
strong baselines, although they require either task boundary information or
multiple data passes. Finally, two naive baselines are included, which train
MLPs and CNNs without any techniques to reduce forgetting.

We test various methods on several standard datasets — Split MNIST (Deng
2012), Split CIFAR10 (Krizhevsky 2009), Split CIFAR100 (Krizhevsky 2009), and
Split Tiny ImageNet (Le & Yang 2015). We only present results on Split MNIST
here due to page limit and put all other results in Section C.2. For SDMLP and
FlyModel, we take results from Bricken et al. (2023) directly. For MLP and
CNN, we use $\operatorname{ReLU}$ by default. For our methods, we apply
elephant functions with $d=4$ in an MLP with one hidden layer and a simple
CNN. The resulting neural networks are named EMLP and ECNN, respectively.

Table 2: The test accuracy of various methods in class incremental learning on Split MNIST. Higher is better. All accuracies are averaged over $5$ runs, reported with standard errors. The number of neurons refers to the size of the last hidden layer in a neural network, i.e. the feature dimension. Method | Neurons | Dataset Passes | Task Boundary | Test Accuracy
---|---|---|---|---
MLP | 1K | 1 | ✗ | 0.665$\pm$0.014
MLP+Streaming EWC | 1K | 1 | ✗ | 0.708$\pm$0.008
SDMLP | 1K | 500 | ✗ | 0.69
FlyModel | 1K | 1 | ✓ | 0.77
EMLP (ours) | 1K | 1 | ✗ | 0.723$\pm$0.006
CNN | 1K | 1 | ✗ | 0.659$\pm$0.016
CNN+Streaming EWC | 1K | 1 | ✗ | 0.716$\pm$0.024
ECNN (ours) | 1K | 1 | ✗ | 0.732$\pm$0.007
MLP | 10K | 1 | ✗ | 0.621$\pm$0.010
MLP+Streaming EWC | 10K | 1 | ✗ | 0.609$\pm$0.013
SDMLP | 10K | 500 | ✗ | 0.53
FlyModel | 10K | 1 | ✓ | 0.91
EMLP (ours) | 10K | 1 | ✗ | 0.802$\pm$0.002
CNN | 10K | 1 | ✗ | 0.769$\pm$0.011
CNN+Streaming EWC | 10K | 1 | ✗ | 0.780$\pm$0.010
ECNN (ours) | 10K | 1 | ✗ | 0.850$\pm$0.004

The test accuracy is used as the performance metric. A result summary of
different methods on Split MNIST is shown in Table 2. The number of neurons
refers to the size of the last hidden layer in a neural network, also known as
the feature dimension. Overall, FlyModel performs the best, although it
requires additional task boundary information. Our methods (EMLP and ECNN) are
the second best without utilizing task boundary information by training for a
single pass. Other findings are summarized in the following, reaffirming the
findings in Mirzadeh et al. (2022a; b):

  * •

Wider neural networks forget less by using more neurons.

  * •

Neural network architectures can significantly impact learning performance:
CNN (ECNN) is better than MLP (EMLP), especially with many neurons.

  * •

Architectural improvement is larger than algorithmic improvement: using a
better architecture (e.g., EMLP and ECNN) is more beneficial than
incorporating Streaming EWC.

Overall, we conclude that applying elephant activation functions significantly
reduces forgetting and boosts performance in class incremental learning under
strict constraints.

###  5.3 Reinforcement Learning

Recently, Lan et al. (2023) showed that the forgetting issue exists even in
single RL tasks and it is largely masked by a large replay buffer. Without a
replay buffer, a single RL task can be viewed as a series of related but
different tasks without clear task boundaries (Dabney et al. 2021). For
example, in temporal difference (TD) learning, the true value function is
usually approximated by $v$ with bootstrapping: $v(S_{t})\leftarrow
R_{t+1}+\gamma v(S_{t+1})$, where $S_{t}$ and $S_{t+1}$ are two successive
states and $R_{t+1}+\gamma v(S_{t+1})$ is named the TD target. During
training, the TD target constantly changes due to bootstrapping, non-
stationary state distribution, and changing policy. To speed up learning while
reducing forgetting, it is crucial to update $v(S_{t})$ to the new TD target
without changing other state values too much (Lan et al. 2023), where local
elasticity can help.

By its very nature, solving RL tasks requires continual learning ability for
both classification and regression. Concretely, estimating value functions
using TD learning is a non-stationary regression task; given a specific state,
selecting the optimal action from a discrete action space is very similar to a
classification task. In this section, we demonstrate that incorporating
elephant activation functions helps reduce forgetting in RL under memory
constraints. Specifically, we use deep Q-network (DQN) (Mnih et al. 2013;
2015) as an exemplar algorithm following Lan et al. (2023). Four classical RL
tasks from Gym (Brockman et al. 2016) and PyGame Learning Environment (Tasfi
2016) are chosen: MountainCar-v0, Acrobot-v1, Catcher, and Pixelcopter. An
MLP/EMLP with one hidden layer of size $1,000$ is used to parameterize the
action-value function in DQN for all tasks. For all elephant functions used in
EMLPs, $d=4$. The standard buffer size is 1e4. For EMLP, we use a small replay
buffer with size 32; for MLP, we consider two buffer sizes — 1e4 and 32. More
details are included in the appendix.

The return curves of various methods are shown in Figure 4, averaged over $10$
runs, where shaded areas represent standard errors and $m$ denotes the size of
a replay buffer. Clearly, using a tiny replay buffer, EMLP ($m=32$)
outperforms MLP ($m=32$) in all tasks. Moreover, except Pixelcopter, EMLP
($m=32$) achieves similar performance compared with MLP ($m=1e4$), although it
uses a much smaller buffer. In summary, the experimental results further
confirm the effectiveness of elephant activation functions in reducing
catastrophic forgetting.

(a) MountainCar-v0 (b) Acrobot-v1 (c) Catcher (d) Pixelcopter

Figure 4: The return curves of DQN in four tasks with different neural
networks and buffer sizes. All results are averaged over $10$ runs, where the
shaded areas represent standard errors. Using a much smaller buffer, EMLP
($m=32$) outperforms MLP ($m=32$) and matches MLP ($m=1e4$).

##  6 Related works

##### Architecture-based continual learning

Continual learning methods can be divided into several categories, such as
regularization-based methods (Kirkpatrick et al. 2017; Schwarz et al. 2018;
Zenke et al. 2017; Aljundi et al. 2019b), replay-based methods (Kemker et al.
2018; Farquhar & Gal 2018; Van de Ven & Tolias 2019; Delange et al. 2021), and
optimization-based methods (Lopez-Paz & Ranzato 2017; Zeng et al. 2019;
Farajtabar et al. 2020). We encourage readers to check recent surveys
(Khetarpal et al. 2022; Delange et al. 2021; Wang et al. 2023) for more
details. Here, we focus on architecture-based methods that are more related to
our work. Among them, our work is inspired by Mirzadeh et al. (2022a; b),
which study and analyze the effect of different neural architectures on
continual learning. Several approaches involve careful selection and
allocation of a subset of weights in a large network for each task (Mallya &
Lazebnik 2018; Sokar et al. 2021; Fernando et al. 2017; Serra et al. 2018;
Masana et al. 2021; Li et al. 2019; Yoon et al. 2018), or the allocation of a
network that is specific to each task (Rusu et al. 2016; Aljundi et al. 2017).
Some other methods expand a neural network dynamically (Yoon et al. 2018; Hung
et al. 2019; Ostapenko et al. 2019). Finally, Shen et al. (2021) and Bricken
et al. (2023) proposed novel neural network structures inspired by biological
neural circuits, acting as two strong baseline methods in our experiments.

##### Sparse representations

Sparse representations are known to help reduce forgetting for decades (French
1992). In supervised learning, sparse training (Dettmers & Zettlemoyer 2019;
Liu et al. 2020; Sokar et al. 2021), dropout variants (Srivastava et al. 2013;
Goodfellow et al. 2013; Mirzadeh et al. 2020; Abbasi et al. 2022; Sarfraz et
al. 2023), and weight pruning methods (Guo et al. 2016; Frankle & Carbin 2019;
Blalock et al. 2020; Zhou et al. 2020) are shown to speed up training and
improve generalization. In continual learning, both SDMLP (Shen et al. 2021)
and FlyModel (Bricken et al. 2023) are designed to generate sparse
representations. In RL, Le et al. (2017), Liu et al. (2019), and Pan et al.
(2022) showed that sparse representations help stabilize training and improve
overall performance.

##### Local elasticity and memorization

He & Su (2020) proposed the concept of local elasticity. Chen et al. (2020)
introduced label-aware neural tangent kernels, showing that models trained
with these kernels are more locally elastic. Mehta et al. (2021) proved a
theoretical connection between the scale of neural network initialization and
local elasticity, demonstrating extreme memorization using large
initialization scales. Incorporating Fourier features in the input of a neural
network also induces local elasticity, which is greatly affected by the
initial variance of the Fourier basis (Li & Pathak 2021).

##  7 Conclusion

In this work, we proposed elephant activation functions that can make neural
networks more resilient to catastrophic forgetting. Specifically, we showed
that incorporating elephant activation functions in classical neural networks
helps improve learning performance in streaming learning for regression, class
incremental learning, and reinforcement learning. Theoretically, our work
provided a deeper understanding of the role of activation functions in
catastrophic forgetting. We hope our work will inspire more researchers to
study and design better neural network architectures for continual learning.

## References

  * Abbasi et al. (2022) Ali Abbasi, Parsa Nooralinejad, Vladimir Braverman, Hamed Pirsiavash, and Soheil Kolouri.  Sparsity and heterogeneous dropout for continual learning in the null space of neural activations.  In _Conference on Lifelong Learning Agents_. PMLR, 2022.
  * Aljundi et al. (2017) Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * Aljundi et al. (2019a) Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.  Task-free continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019a.
  * Aljundi et al. (2019b) Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars.  Selfless sequential learning.  In _International Conference on Learning Representations_ , 2019b.
  * Blalock et al. (2020) Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag.  What is the state of neural network pruning?  _Proceedings of machine learning and systems_ , 2020.
  * Bricken et al. (2023) Trenton Bricken, Xander Davies, Deepak Singh, Dmitry Krotov, and Gabriel Kreiman.  Sparse distributed memory is a continual learner.  In _International Conference on Learning Representations_ , 2023.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  OpenAI Gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Chen et al. (2020) Shuxiao Chen, Hangfeng He, and Weijie Su.  Label-aware neural tangent kernel: Toward better generalization and local elasticity.  _Advances in Neural Information Processing Systems_ , 33:15847–15858, 2020.
  * Chizat et al. (2019) Lenaic Chizat, Edouard Oyallon, and Francis Bach.  On lazy training in differentiable programming.  _Advances in neural information processing systems_ , 2019.
  * Chrabaszcz et al. (2017) Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter.  A downsampled variant of imagenet as an alternative to the cifar datasets.  _arXiv preprint arXiv:1707.08819_ , 2017.
  * Dabney et al. (2021) Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G Bellemare, and David Silver.  The value-improvement path: Towards better representations for reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021.
  * Dase & Pawar (2010) RK Dase and DD Pawar.  Application of artificial neural network for stock market predictions: A review of literature.  _International Journal of Machine Intelligence_ , 2010.
  * Delange et al. (2021) Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Deng (2012) Li Deng.  The MNIST database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 2012.
  * Dettmers & Zettlemoyer (2019) Tim Dettmers and Luke Zettlemoyer.  Sparse networks from scratch: Faster training without losing performance.  _arXiv preprint arXiv:1907.04840_ , 2019.
  * Dmitri et al. (2016) Kryukov Dmitri, Agafonova Maria, and Arestova Anna.  Comparison of regression and neural network approaches to forecast daily power consumption.  In _International Forum on Strategic Technology_ , 2016.
  * Farajtabar et al. (2020) Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li.  Orthogonal gradient descent for continual learning.  In _International Conference on Artificial Intelligence and Statistics_ , 2020.
  * Farquhar & Gal (2018) Sebastian Farquhar and Yarin Gal.  Towards robust evaluations of continual learning.  _arXiv preprint arXiv:1805.09733_ , 2018.
  * Fernando et al. (2017) Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra.  PathNet: Evolution channels gradient descent in super neural networks.  _arXiv preprint arXiv:1701.08734_ , 2017.
  * Frankle & Carbin (2019) Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _International Conference on Learning Representations_ , 2019.
  * French (1992) Robert M French.  Semi-distributed representations and catastrophic forgetting in connectionist networks.  _Connection Science_ , 1992.
  * French (1999) Robert M French.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ , 1999.
  * Ghiassian et al. (2020) Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White.  Improving performance in reinforcement learning by breaking generalization in neural networks.  In _Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems_ , 2020.
  * Goodfellow et al. (2013) Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout networks.  In _International conference on machine learning_ , 2013.
  * Guo et al. (2016) Yiwen Guo, Anbang Yao, and Yurong Chen.  Dynamic network surgery for efficient DNNs.  _Advances in neural information processing systems_ , 2016.
  * He & Su (2020) Hangfeng He and Weijie Su.  The local elasticity of neural networks.  In _International Conference on Learning Representations_ , 2020.
  * He & Sick (2021) Yujiang He and Bernhard Sick.  Clear: An adaptive continual learning framework for regression tasks.  _AI Perspectives_ , 2021.
  * Hung et al. (2019) Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen.  Compacting, picking and growing for unforgetting continual learning.  _Advances in Neural Information Processing Systems_ , 2019.
  * Jacot et al. (2018) Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 2018.
  * Kemker et al. (2018) Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan.  Measuring catastrophic forgetting in neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Khetarpal et al. (2022) Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup.  Towards continual reinforcement learning: A review and perspectives.  _Journal of Artificial Intelligence Research_ , 2022.
  * Kingma & Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 2017.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  _Master’s thesis, University of Toronto_ , 2009.
  * Lan et al. (2020) Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White.  Maxmin Q-learning: Controlling the estimation bias of q-learning.  In _International Conference on Learning Representations_ , 2020.
  * Lan et al. (2023) Qingfeng Lan, Yangchen Pan, Jun Luo, and A Rupam Mahmood.  Memory-efficient reinforcement learning with value-based knowledge consolidation.  _Transactions on Machine Learning Research_ , 2023.
  * Le et al. (2017) Lei Le, Raksha Kumaraswamy, and Martha White.  Learning sparse representations in reinforcement learning with sparse coding.  _International Joint Conferences on Artificial Intelligence_ , 2017\.
  * Le & Yang (2015) Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 2015.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _Nature_ , 2015.
  * Li & Pathak (2021) Alexander Li and Deepak Pathak.  Functional regularization for reinforcement learning via learned Fourier features.  _Advances in Neural Information Processing Systems_ , 2021.
  * Li et al. (2019) Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In _International Conference on Machine Learning_ , 2019.
  * Liu et al. (2020) Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K.H. So.  Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers.  In _International Conference on Learning Representations_ , 2020.
  * Liu et al. (2019) Vincent Liu, Raksha Kumaraswamy, Lei Le, and Martha White.  The utility of sparse representations for control in reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019.
  * Lopez-Paz & Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Madhuri et al. (2019) CH Raga Madhuri, G Anuradha, and M Vani Pujitha.  House price prediction using regression techniques: A comparative study.  In _International conference on smart structures and systems_ , 2019\.
  * Mallya & Lazebnik (2018) Arun Mallya and Svetlana Lazebnik.  PackNet: Adding multiple tasks to a single network by iterative pruning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018.
  * Masana et al. (2021) Marc Masana, Tinne Tuytelaars, and Joost Van de Weijer.  Ternary feature masks: zero-forgetting for task-incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Mehta et al. (2021) Harsh Mehta, Ashok Cutkosky, and Behnam Neyshabur.  Extreme memorization via scale of initialization.  In _International Conference on Learning Representations_ , 2021.
  * Mendez et al. (2020) Jorge Mendez, Boyu Wang, and Eric Eaton.  Lifelong policy gradient learning of factored policies for faster training without forgetting.  _Advances in Neural Information Processing Systems_ , 2020.
  * Mendez et al. (2022) Jorge A Mendez, Harm van Seijen, and Eric Eaton.  Modular lifelong reinforcement learning via neural composition.  In _International Conference on Learning Representations_ , 2022.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, and Hassan Ghasemzadeh.  Dropout as an implicit gating mechanism for continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020.
  * Mirzadeh et al. (2022a) Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Wide neural networks forget less catastrophically.  In _International Conference on Machine Learning_ , 2022a.
  * Mirzadeh et al. (2022b) Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Architecture matters in continual learning.  _arXiv preprint arXiv:2202.00275_ , 2022b.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, and Ioannis Antonoglou.  Playing Atari with deep reinforcement learning.  In _NIPS Deep Learning Workshop_ , 2013.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 2015.
  * Ostapenko et al. (2019) Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi.  Learning to remember: A synaptic plasticity driven framework for continual learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019.
  * Pan et al. (2022) Yangchen Pan, Kirby Banman, and Martha White.  Fuzzy tiling activations: A simple approach to learning sparse representations online.  In _International Conference on Learning Representations_ , 2022.
  * Ren et al. (2021) Xiaoli Ren, Xiaoyong Li, Kaijun Ren, Junqiang Song, Zichen Xu, Kefeng Deng, and Xiang Wang.  Deep learning-based weather prediction: a survey.  _Big Data Research_ , 2021.
  * Riemer et al. (2018) Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _International Conference on Learning Representations_ , 2018.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision_ , 2015.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Sarfraz et al. (2023) Fahad Sarfraz, Elahe Arani, and Bahram Zonooz.  Sparse coding in a dual memory system for lifelong learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * Schwarz et al. (2018) Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In _International Conference on Machine Learning_ , 2018.
  * Serra et al. (2018) Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In _International Conference on Machine Learning_ , 2018.
  * Shen et al. (2021) Yang Shen, Sanjoy Dasgupta, and Saket Navlakha.  Algorithmic insights on continual learning from fruit flies.  _arXiv preprint arXiv:2107.07617_ , 2021.
  * Sokar et al. (2021) Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy.  SpaceNet: Make free space for continual learning.  _Neurocomputing_ , 2021.
  * Srivastava et al. (2013) Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmidhuber.  Compete to compute.  _Advances in neural information processing systems_ , 2013.
  * Tasfi (2016) Norman Tasfi.  PyGame learning environment.  https://github.com/ntasfi/PyGame-Learning-Environment, 2016.
  * Trockman & Kolter (2022) Asher Trockman and J Zico Kolter.  Patches are all you need?  _arXiv preprint arXiv:2201.09792_ , 2022.
  * Van de Ven & Tolias (2019) Gido M Van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  _arXiv preprint arXiv:1904.07734_ , 2019.
  * Wang et al. (2023) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.  A comprehensive survey of continual learning: Theory, method and application.  _arXiv preprint arXiv:2302.00487_ , 2023.
  * Wolfe & Kyrillidis (2022) Cameron R Wolfe and Anastasios Kyrillidis.  Cold start streaming learning for deep networks.  _arXiv preprint arXiv:2211.04624_ , 2022.
  * Yoon et al. (2018) Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In _International Conference on Learning Representations_ , 2018.
  * Zeng et al. (2019) Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu.  Continual learning of context-dependent processing in neural networks.  _Nature Machine Intelligence_ , 2019.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _International Conference on Machine Learning_ , 2017.
  * Zhou et al. (2020) Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Song, Quoc Le, Qiang Liu, and Dale Schuurmans.  Go wide, then narrow: Efficient training of deep thin networks.  In _International Conference on Machine Learning_ , 2020.

##  Appendix A Function sparsity and gradient sparsity of various activation
functions

(a) $\operatorname{ReLU}$ (b) $\operatorname{ELU}$ (c)
$\operatorname{Sigmoid}$ (d) $\operatorname{Tanh}$

Figure 5: Visualizations of common activation functions and their gradients. Table 3: The function sparsity and gradient sparsity of various activation functions. Among them, only $\operatorname{Elephant}$ is sparse in terms of both function values and gradient values. activation | function sparsity | gradient sparsity
---|---|---
$\operatorname{ReLU}$ | 1/2 | 1/2
$\operatorname{Sigmoid}$ | 1/2 | 1
$\operatorname{Tanh}$ | 0 | 1
$\operatorname{ELU}$ | 0 | 1/2
$\operatorname{Elephant}$ | 1 | 1

##  Appendix B Proofs

##### Derivation of Equation 1

By Taylor expansion, we have

$\displaystyle f_{{\bm{w}}^{\prime}}({\bm{x}})-f_{{\bm{w}}}({\bm{x}})$
$\displaystyle=$ $\displaystyle
f_{{\bm{w}}+\Delta_{\bm{w}}}({\bm{x}})-f_{{\bm{w}}}({\bm{x}})$
$\displaystyle=$ $\displaystyle
f_{{\bm{w}}}({\bm{x}})+\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\Delta_{\bm{w}}\rangle+O(\Delta_{\bm{w}}^{2})-f_{{\bm{w}}}({\bm{x}})$
$\displaystyle=$
$\displaystyle\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\Delta_{\bm{w}}\rangle+O(\Delta_{\bm{w}}^{2})$
$\displaystyle=$
$\displaystyle-\alpha\nabla_{f}L(f,F,{\bm{x}}_{t})\,\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle+O(\Delta_{\bm{w}}^{2}).$

######  Lemma 3.1.

Given a non-linear function
$f_{{\bm{w}}}({\bm{x}})={\bm{u}}^{\top}\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}}):\mathbb{R}^{n}\mapsto\mathbb{R}$,
where $\sigma$ is a non-linear activation function,
${\bm{x}}\in\mathbb{R}^{n}$, ${\bm{u}}\in\mathbb{R}^{m}$,
${\mathbf{V}}\in\mathbb{R}^{m\times n}$, ${\bm{b}}\in\mathbb{R}^{m}$, and
${\bm{w}}=\\{{\bm{u}},{\mathbf{V}},{\bm{b}}\\}$. The NTK of this non-linear
function is

$\displaystyle\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})+{\bm{u}}^{\top}{\bm{u}}({\bm{x}}^{\top}{\bm{x}}_{t}+1)\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma^{\prime}({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}}),$

where $\langle\cdot,\cdot\rangle$ denotes Frobenius inner product or dot
product given the context.

###### Proof.

Let $\circ$ denote Hadamard product. By definition, we have

$\displaystyle\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle$
$\displaystyle=\langle\nabla_{u}f_{{\bm{w}}}({\bm{x}}),\nabla_{u}f_{{\bm{w}}}({\bm{x}}_{t})\rangle+\langle\nabla_{V}f_{{\bm{w}}}({\bm{x}}),\nabla_{V}f_{{\bm{w}}}({\bm{x}}_{t})\rangle+\langle\nabla_{b}f_{{\bm{w}}}({\bm{x}}),\nabla_{b}f_{{\bm{w}}}({\bm{x}}_{t})\rangle$
$\displaystyle=\langle\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}}),\sigma({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})\rangle+\langle{\bm{u}}\circ\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}}){\bm{x}}^{\top},{\bm{u}}\circ\sigma^{\prime}({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}}){\bm{x}}_{t}^{\top}\rangle$
$\displaystyle+\langle{\bm{u}}\circ\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}}),{\bm{u}}\circ\sigma^{\prime}({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})\rangle$
$\displaystyle=\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})+({\bm{x}}^{\top}{\bm{x}}_{t}+1)\left({\bm{u}}\circ\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}})\right)^{\top}\left({\bm{u}}\circ\sigma^{\prime}(\mathbf{{\mathbf{V}}}{\bm{x}}_{t}+{\bm{b}})\right),$
$\displaystyle=\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})+{\bm{u}}^{\top}{\bm{u}}({\bm{x}}^{\top}{\bm{x}}_{t}+1)\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma^{\prime}({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}}).$

∎

######  Lemma 4.3.

$\operatorname{Elephant}(x)$ and $\operatorname{Elephant}^{\prime}(x)$ are
sparse functions.

###### Proof.

$\operatorname{Elephant}(x)=\frac{1}{1+|\frac{x}{a}|^{d}}$ and
$\left|\operatorname{Elephant}^{\prime}(x)\right|=\frac{d}{a}\left|\frac{x}{a}\right|^{d-1}(\frac{1}{1+\left|\frac{x}{a}\right|^{d}})^{2}$.
For $0<\epsilon<1$, easy to verify that

$\left|x\right|\geq
a(\frac{1}{\epsilon}-1)^{1/d}\Longrightarrow\operatorname{Elephant}(x)\leq\epsilon\quad\text{and}\quad\left|x\right|\geq\frac{d}{2\epsilon}\Longrightarrow\left|\operatorname{Elephant}^{\prime}(x)\right|\leq\epsilon.$

For $C>a(\frac{1}{\epsilon}-1)^{1/d}$, we have
$S_{\epsilon,C}(\operatorname{Elephant})\geq\frac{C-a(\frac{1}{\epsilon}-1)^{1/d}}{C}$,
thus

$\displaystyle S(\operatorname{Elephant})=\lim_{\epsilon\rightarrow
0^{+}}\lim_{C\rightarrow\infty}S_{\epsilon,C}(\operatorname{Elephant})\geq\lim_{\epsilon\rightarrow
0^{+}}\lim_{C\rightarrow\infty}\frac{C-a(\frac{1}{\epsilon}-1)^{1/d}}{C}=\lim_{\epsilon\rightarrow
0^{+}}1=1.$

Similarly, for $C>\frac{d}{2\epsilon}$, we have
$S_{\epsilon,C}(\operatorname{Elephant}^{\prime})\geq\frac{C-\frac{d}{2\epsilon}}{C}$,
thus

$\displaystyle S(\operatorname{Elephant}^{\prime})=\lim_{\epsilon\rightarrow
0^{+}}\lim_{C\rightarrow\infty}S_{\epsilon,C}(\operatorname{Elephant}^{\prime})\geq\lim_{\epsilon\rightarrow
0^{+}}\lim_{C\rightarrow\infty}\frac{C-\frac{d}{2\epsilon}}{C}=\lim_{\epsilon\rightarrow
0^{+}}1=1.$

Note that $S(\operatorname{Elephant})\leq 1$ and
$S(\operatorname{Elephant}^{\prime})\leq 1$. Together, we conclude that
$S(\operatorname{Elephant})=1$ and $S(\operatorname{Elephant}^{\prime})=1$;
$\operatorname{Elephant}(x)$ and $\operatorname{Elephant}^{\prime}(x)$ are
sparse functions. ∎

######  Theorem 4.4.

Define $f_{{\bm{w}}}({\bm{x}})$ as it in Lemma 3.1. Let $\sigma$ be the
elephant activation function with $d\rightarrow\infty$. When
$\left|{\mathbf{V}}({\bm{x}}-{\bm{x}}_{t})\right|\succ 2a\mathbf{1}_{m}$, we
have
$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=0$,
where $\succ$ denotes an element-wise inequality symbol and
$\mathbf{1}_{m}=[1,\cdots,1]\in\mathbb{R}^{m}$.

###### Proof.

When $d\rightarrow\infty$, the elephant function is a rectangular function,
i.e.

$\displaystyle\sigma(x)=\operatorname{rect}(x)=\begin{cases}1,&\left|x\right|<a,\\\
\frac{1}{2},&\left|x\right|=a,\\\ 0,&\left|x\right|>a.\\\ \end{cases}$

In this case, it is easy to verify that $\forall\,x,y\in\mathbb{R}$,
$\left|x-y\right|>2w$, we have $\sigma(x)\sigma(y)=0$ and
$\sigma^{\prime}(x)\sigma^{\prime}(y)=0$. Denote
$\Delta_{\bm{x}}={\bm{x}}-{\bm{x}}_{t}$. Then when
$\left|{\mathbf{V}}\Delta_{\bm{x}}\right|\succ 2a\mathbf{1}_{m}$, we have
$\sigma({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})=0$
and
$\sigma^{\prime}({\mathbf{V}}{\bm{x}}+{\bm{b}})^{\top}\sigma^{\prime}({\mathbf{V}}{\bm{x}}_{t}+{\bm{b}})=0$.
In other words, when ${\bm{x}}$ and ${\bm{x}}_{t}$ are dissimilar in the sense
that $\left|{\mathbf{V}}({\bm{x}}-{\bm{x}}_{t})\right|\succ 2a\mathbf{1}_{m}$,
we have
$\langle\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}),\nabla_{\bm{w}}f_{{\bm{w}}}({\bm{x}}_{t})\rangle=0$.

∎

##  Appendix C Experimental details and additional results

All our experiments are conducted on CPUs or V100 GPUs. The computation to
repeat all experiments in this work is not high (much less $0.1$ GPU years).
However, the exact number of used computation is hard to estimate.

For all ENNs, we only replace the activation functions of the last hidden
layer with elephant activation functions. 222The initial experiments with
EMLPs on Split MNIST show that replacing all activation functions with
elephant functions hurts performance, probably due to loss of plasticity. We
leave this for future work. Before training, the weight values in all layers
are initialized from $U(-\sqrt{k},\sqrt{k})$, where
$k=1/\text{in\\_features}$. For the bias values in the layer where elephant
functions are used, we initialize them with evenly spaced numbers over the
interval $[-\sqrt{3}\sigma_{bias},\sqrt{3}\sigma_{bias}]$, where
$\sigma_{bias}$ is the standard deviation of the bias values. The bias values
are initialized in such way so that ENNs can generate diverse features. All
other bias values are initialized with $0$s.

As a limitation, there is a lack of a theoretical way to set $\sigma_{bias}$
or $a$ in elephant activation functions appropriately. The best values seem to
be depend on the input data distribution as well as the number of the input
features (i.e., in_features). In practice, we choose $\sigma_{bias}$ and $a$
from a small set.

To improve sample efficiency, we update a model for multiple times given each
new sample/mini-batch data. We call this number the update epoch $E$.

###  C.1 Streaming learning for regression

We list the (swept) hyper-parameters in streaming learning for regression in
Table 4. Among them, $d$, $a$, and $\sigma_{bias}$ are specific hyper-
parameters for ENNs, where $\lambda$ and $\beta$ are two hyper-parameters used
in SR-NN Liu et al. (2019).

Table 4: The (swept) hyper-parameters in streaming learning for regression. Hyper-parameter | Value
---|---
Optimizer | Adam
learning rate | $\\{3e-3,1e-3,3e-4,1e-4,3e-5,1e-5\\}$
$E$ | $10$
$d$ | $8$
$a$ | $\\{0.02,0.04,0.08,0.16,0.32\\}$
$\sigma_{bias}$ | $\\{0.08,0.16,0.32,0.64,1.28\\}$
Set KL loss weight $\lambda$ | $\\{0,0.1,0.01,0.001\\}$
SR-NN $\beta$ | $\\{0.05,0.1,0.2\\}$

###  C.2 Class incremental learning

####  C.2.1 Experimental results on Split MNIST and Split CIFAR10

For both MNIST and CIFAR10, we split them into 5 sub-datasets, and each of
them contains two of the classes. The CNN model consists of a convolution
layer, a max pooling operator, and a linear layer. We also try CNNs with more
convolution layers but find the performance is worse and worse as we increase
the number of convolution layers. Besides simple CNNs, we test ConvMixer
(Trockman & Kolter 2022) which can achieve around $92.5\%$ accuracy in just
$25$ epochs in classical supervised learning
333https://github.com/locuslab/convmixer-cifar10. However, we find that
ConvMixer completely fails in class incremental learning setting, as shown in
Table 5 and Table 6. Same as Mirzadeh et al. (2022b), the results show that
using more advanced models does not necessarily lead to better performance in
class incremental learning.

Table 5: The test accuracy of various methods in class incremental learning on Split MNIST. Higher is better. All accuracies are averaged over $5$ runs, reported with standard errors. Method | Neurons | Dataset Passes | Task Boundary | Test Accuracy
---|---|---|---|---
MLP | 1K | 1 | ✗ | 0.665$\pm$0.014
MLP+Streaming EWC | 1K | 1 | ✗ | 0.708$\pm$0.008
SDMLP | 1K | 500 | ✗ | 0.69
FlyModel | 1K | 1 | ✓ | 0.77
EMLP (ours) | 1K | 1 | ✗ | 0.723$\pm$0.006
CNN | 1K | 1 | ✗ | 0.659$\pm$0.016
CNN+Streaming EWC | 1K | 1 | ✗ | 0.716$\pm$0.024
ECNN (ours) | 1K | 1 | ✗ | 0.732$\pm$0.007
ConvMixer | 1K | 1 | ✗ | 0.110$\pm$0.003
EConvMixer (ours) | 1K | 1 | ✗ | 0.105$\pm$0.003
MLP | 10K | 1 | ✗ | 0.621$\pm$0.010
MLP+Streaming EWC | 10K | 1 | ✗ | 0.609$\pm$0.013
SDMLP | 10K | 500 | ✗ | 0.53
FlyModel | 10K | 1 | ✓ | 0.91
EMLP (ours) | 10K | 1 | ✗ | 0.802$\pm$0.002
CNN | 10K | 1 | ✗ | 0.769$\pm$0.011
CNN+Streaming EWC | 10K | 1 | ✗ | 0.780$\pm$0.010
ECNN (ours) | 10K | 1 | ✗ | 0.850$\pm$0.004
ConvMixer | 10K | 1 | ✗ | 0.107$\pm$0.003
EConvMixer (ours) | 10K | 1 | ✗ | 0.104$\pm$0.003
Table 6: The test accuracy of various methods in class incremental learning on Split CIFAR10, averaged over $5$ runs. Higher is better. Standard errors are also reported. Method | Neurons | Test Accuracy
---|---|---
MLP | 1K | 0.151$\pm$0.008
MLP+Streaming EWC | 1K | 0.158$\pm$0.005
EMLP (ours) | 1K | 0.197$\pm$0.003
CNN | 1K | 0.151$\pm$0.001
CNN+Streaming EWC | 1K | 0.147$\pm$0.010
ECNN (ours) | 1K | 0.192$\pm$0.004
ConvMixer | 1K | 0.100$\pm$0.0027
EConvMixer (ours) | 1K | 0.100$\pm$0.0001
MLP | 10K | 0.173$\pm$0.004
MLP+Streaming EWC | 10K | 0.169$\pm$0.003
EMLP (ours) | 10K | 0.239$\pm$0.002
CNN | 10K | 0.151$\pm$0.001
CNN+Streaming EWC | 10K | 0.179$\pm$0.007
ECNN (ours) | 10K | 0.243$\pm$0.002
ConvMixer | 10K | 0.102$\pm$0.0015
EConvMixer (ours) | 10K | 0.100$\pm$0.0001

In Figure 6, we plot the test accuracy curves during training on Split MNIST
and Split CIFAR10 when the number of neurons is $10K$. All results are
averaged over $5$ runs and the shaded regions represent standard errors.

(a) Split MNIST (b) Split CIFAR10

Figure 6: The test accuracy curves during training. The x-axis shows the number of training mini-batches. All results are averaged over $5$ runs and the shaded regions represent standard errors. Table 7: The test accuracy of various methods in class incremental learning on Split Embedding CIFAR10, averaged over $5$ runs. Higher is better. Standard errors are also reported. Method | Neurons | Dataset Passes | Task Boundary | Test Accuracy
---|---|---|---|---
MLP | 1K | 1 | ✗ | 0.662$\pm$0.006
SDMLP | 1K | 2000 | ✗ | 0.56
FlyModel | 1K | 1 | ✓ | 0.69
EMLP (ours) | 1K | 1 | ✗ | 0.726$\pm$0.008
MLP | 10K | 1 | ✗ | 0.697$\pm$0.002
SDMLP | 10K | 2000 | ✗ | 0.77
FlyModel | 10K | 1 | ✓ | 0.82
EMLP (ours) | 10K | 1 | ✗ | 0.755$\pm$0.002
Table 8: The test accuracy of various methods in class incremental learning on Split Embedding CIFAR100, averaged over $5$ runs. Higher is better. Standard errors are also reported. Method | Neurons | Dataset Passes | Task Boundary | Test Accuracy
---|---|---|---|---
MLP | 1K | 1 | ✗ | 0.157$\pm$0.001
SDMLP | 1K | 500 | ✗ | 0.39
FlyModel | 1K | 1 | ✓ | 0.36
EMLP (ours) | 1K | 1 | ✗ | 0.391$\pm$0.003
MLP | 10K | 1 | ✗ | 0.425$\pm$0.001
SDMLP | 10K | 500 | ✗ | 0.43
FlyModel | 10K | 1 | ✓ | 0.58
EMLP (ours) | 10K | 1 | ✗ | 0.449$\pm$0.002
Table 9: The test accuracy of various methods in class incremental learning on Split Embedding Tiny ImageNet, averaged over $5$ runs. Higher is better. Standard errors are also reported. Method | Neurons | Dataset Passes | Task Boundary | Test Accuracy
---|---|---|---|---
MLP | 10K | 1 | ✗ | 0.249$\pm$0.003
EMLP (ours) | 10K | 1 | ✗ | 0.241$\pm$0.002
MLP | 100K | 1 | ✗ | 0.264$\pm$0.002
EMLP (ours) | 100K | 1 | ✗ | 0.350$\pm$0.002

####  C.2.2 Combining with pre-training

To show that our method can be combined with pre-training, we relax our
experimental assumptions by adding the pre-training technique.

To be specific, for CIFAR10 and CIFAR100, we use 256 dimensional latent
embeddings which are provided by Bricken et al. (2023), taken from the last
layer of a frozen ConvMixer (Trockman & Kolter 2022) that is pre-trained on
ImageNet32 (Chrabaszcz et al. 2017). The corresponding embedding datasets are
called Embedding CIFAR10 and Embedding CIFAR100, respectively. We split
Embedding CIFAR10 into 5 sub-datasets while Embedding CIFAR100 is split into
50 sub-datasets; each sub-dataset contains two of the classes. We test
different methods on the two datasets and present results in Table 7 and Table
8. Note that the the results of SDMLP and FlyModel are from Bricken et al.
(2023).

For Tiny ImageNet, we use 768 dimensional latent embeddings which are taken
from the last layer of a frozen ConvMixer (Trockman & Kolter 2022) that is
pre-trained on ImageNet-1k (Russakovsky et al. 2015), provided by Hugging Face
444https://huggingface.co/timm/convmixer_768_32.in1k. The corresponding
embedding dataset is called Embedding Tiny ImageNet. We then split Embedding
Tiny ImageNet into 100 sub-datasets; each of them contains two of the classes.
The experimental results are presented in Table 9.

Overall, the performance of EMLP is greatly improved with the help of pre-
training. For example, the test accuracy on CIFAR10 is boosted from 25% to
75%. Moreover, EMLP still outperforms MLP significantly, showing that our
method can be combined with common practices in class incremental learning,
such as pre-training.

####  C.2.3 Hyper-parameters

We list the (swept) hyper-parameters for Split MNIST and Split CIFAR10 in
Table 10. Among them, $d$, $a$, and $\sigma_{bias}$ are specific hyper-
parameters for ENNs, where $\gamma$ and $\lambda$ are two hyper-parameters
used in (streaming) EWC (Kirkpatrick et al. 2017).

Table 10: The (swept) hyper-parameters for Split MNIST and Split CIFAR10. Hyper-parameter | Value
---|---
Optimizer | RMSProp with decay=$0.999$
learning rate | $\\{3e-6,1e-6,3e-7,1e-7,3e-8,1e-8\\}$
mini-batch size | $125$
$E$ | $\\{1,2\\}$
$d$ | $4$
$a$ | $\\{0.02,0.04,0.08,0.16,0.32\\}$
$\sigma_{bias}$ | $\\{0.04,0.08,0.16,0.32,0.64\\}$
EWC $\gamma$ | $\\{0.5,0.8,0.9,0.95,0.99,0.999\\}$
EWC $\lambda$ | $\\{1e1,1e2,1e3,1e4,1e5\\}$

Similarly, the (swept) hyper-parameters for Split Embedding CIFAR10, Split
Embedding CIFAR100, and Split Embedding Tiny ImageNet are listed in Table 11
and Table 12.

Table 11: The (swept) hyper-parameters for Split Embedding CIFAR10 and Split Embedding CIFAR100. Hyper-parameter | Value
---|---
Optimizer | RMSProp with decay=$0.999$
learning rate | $\\{1e-4,3e-5,1e-5,3e-6,1e-6,3e-7,1e-7\\}$
mini-batch size | $125$
$E$ | $\\{1,2,4\\}$
$d$ | $4$
$a$ | $\\{0.02,0.04,0.08,0.16,0.32\\}$
$\sigma_{bias}$ | $\\{0.04,0.08,0.16,0.32,0.64\\}$
Table 12: The (swept) hyper-parameters for Split Embedding Tiny ImageNet. Hyper-parameter | Value
---|---
Optimizer | RMSProp with decay=$0.999$
learning rate | $\\{1e-5,3e-6,1e-6,3e-7,1e-7\\}$
mini-batch size | $250$
$E$ | $\\{2,4,8\\}$
$d$ | $4$
$a$ | $\\{0.08,0.16,0.32,0.64\\}$
$\sigma_{bias}$ | $\\{0.32,0.64,1.28,2.56\\}$

###  C.3 Reinforcement learning

We list the (swept) hyper-parameters for reinforcement learning in Table 13.
Among them, $d$, $a$, and $\sigma_{bias}$ are specific hyper-parameters for
ENNs. Other hyper-parameters and training settings can be found in Lan et al.
(2020).

Table 13: The (swept) hyper-parameters in reinforcement learning. Hyper-parameter | Value
---|---
Optimizer | RMSProp with decay=$0.999$
learning rate | $\\{1e-2,3e-3,1e-3,3e-4,1e-4,3e-5,1e-5,3e-6\\}$
mini-batch size | $32$
$E$ | $\\{1,2\\}$
$d$ | $4$
$a$ | $\\{0.08,0.16,0.32,0.64\\}$
$\sigma_{bias}$ | $\\{0.08,0.16,0.32,0.64,1.28\\}$
discount factor $\gamma$ | $0.99$