# RanDumb: A Simple Approach that Questions the Efficacy of Continual
Representation Learning

Ameya Prabhu1 Shiven Sinha2∗ Ponnurangam Kumaraguru2 Philip H.S. Torr1
Ozan Sener3+ Puneet K. Dokania1+
1University of Oxford 2IIIT-Hyderabad 3Apple authors contributed equally, +
equal advising

###### Abstract

We propose RanDumb to examine the efficacy of continual representation
learning. RanDumb embeds raw pixels using a fixed random transform which
approximates an RBF-Kernel, initialized before seeing any data, and learns a
simple linear classifier on top. We present a surprising and consistent
finding: RanDumb significantly outperforms the continually learned
representations using deep networks across numerous continual learning
benchmarks, demonstrating the poor performance of representation learning in
these scenarios. RanDumb stores no exemplars and performs a single pass over
the data, processing one sample at a time. It complements GDumb [39],
operating in a low-exemplar regime where GDumb has especially poor
performance. We reach the same consistent conclusions when RanDumb is extended
to scenarios with pretrained models replacing the random transform with
pretrained feature extractor. Our investigation is both surprising and
alarming as it questions our understanding of how to effectively design and
train models that require efficient continual representation learning, and
necessitates a principled reinvestigation of the widely explored problem
formulation itself. Our code is available here.

##  1 Introduction

Continual learning is a specialized form of supervised learning, characterized
by sequentially arriving tasks, coupled with additional computational and
memory constraints [24, 27, 44, 49, 39] (see Verwimp et al. [51] for a survey
of applications). A notable implication of this setup is the inherent
performance gap between continual learning and traditional supervised
learning. To elucidate this further, let us consider the achievable risk under
continual learning given a dataset $\mathcal{D}$:

$\mathcal{R}_{\texttt{CL}}(\mathcal{D})\geq
R_{\texttt{Joint}}(\mathcal{D})+E_{\texttt{CL}}(\mathcal{D})$ (1)

Here, $R_{\texttt{Joint}}$ denotes the risk achievable with supervised
learning without any constraints, while $E_{\texttt{CL}}$ represents the gap
arising from transitioning from fully supervised learning to continual
learning which imposes additional constraints such as: resource-constraint
(e.g., single-epoch), distribution shift, and predefined task ordering with
limited access to data. Because of these additional constraints imposed by
continual learning, the risk is non-decreasing and $E_{\texttt{CL}}$ is non-
negative. Minimizing $E_{\texttt{CL}}$ has been the primary focus of the
continual learning literature [16, 29, 37, 41].

Figure 1: Mechanism of RanDumb. Project raw pixels to a high dimensional space
using random fourier projections ($\varphi$), then decorrelate the features
using Mahalanobis distance [31] and classify with the nearest class mean. The
online update only involves updating a single covariance matrix and class-
means.

In this work, we discover that the fundamental aspect of _continual
representation learning (CRL)_ is often overlooked in the continual learning
literature, as most methods merge continual classifier and representation
learning. We dissect these two aspects and study the effect of representation
learning on $E_{\texttt{CL}}$. The central question of our work is trying to
answer:

_Are current continual learning setups overly constrained for effective
continual representation learning?_

Specifically, consider a continual representation learning algorithm
$\mathtt{CRL}$ which learns a representation from the continual dataset
$\mathcal{D}$. Is this representation better than a random (ie. non-learned)
representation RandRep? More formally, is the following quantity positive or
negative?

$RepGain(\mathcal{D})=E_{\texttt{CL}}(\mathcal{D},\mathtt{RandRep})-E_{\texttt{CL}}(\mathcal{D},\mathtt{CRL}(\mathcal{D}))$

Here, the random representation functions are defined at initialization
without seeing any data or using domain-specific priors. Note that one expects
the impact of continual representation learning ($RepGain(\mathcal{D})$) to be
positive, as continual representation learning approaches should improve
performance over even the best random representation.

Our primary contribution is empirically demonstrating that currently
$RepGain(\mathcal{D})<0$, i.e. there exists random representations which
outperform state-of-the-art representations learned by continual learning
algorithms using deep networks. We show this by introducing a straightforward
baseline, which we name RanDumb, standing for Random representation function
and a Dumb linear classifier. Despite replacing the deep embedder with a
random function, we see from Table 1 (top) that RanDumb outperforms the
current state-of-the-art methods being exemplar-free over challenging large
task scenarios given in [59]. We extend the large task scenario to methods
with pretrained feature extractors [64]. In RanDumb, we replace our random
projection with the pretrained feature extractor here, simply learning a
linear classifier. We show results in Table 2 (top), showing that RanDumb
outperforms the best methods in this scenario as well.

Table 1: Online Continual Learning. Performance comparison of RanDumb on the
PEC setup [59] and VAE-GC [50]. Setup and numbers borrowed from PEC [59].
RanDumb outperforms the best exemplar-free OCL method. Comparing
representation learning with no representation learning, a fixed, random
function outperforms training a deep network continually, i.e.
$RepGain(\mathcal{D})<0$. Furthermore, RanDumb nearly matches performance of
one-pass joint, demonstrating the inefficacy of current benchmarks.

Method | MNIST | CIFAR10 | CIFAR100 | m-ImNet
---|---|---|---|---
Comparison with Best Method
Best (PEC) | 92.3 | 58.9 | 26.5 | 14.9
RanDumb (Ours) | 98.3 | 55.6 | 28.6 | 17.7
Improvement | +6.0 | -3.3 | +2.1 | +2.8
Only Ablating Representation Learning: Random v/s Deep Embedding
VAE-GC | 84.0 | 42.7 | 19.7 | 12.1
RanDumb (Ours) | 98.3 | 55.6 | 28.6 | 17.7
Error ($E_{\texttt{CRL}}$) | +14.3 | +12.9 | +8.9 | +5.6
Scope of Improvement
Joint (One Pass) | 98.3 | 74.2 | 33.0 | 25.3
RanDumb (Ours) | 98.3 | 55.6 | 28.6 | 17.7
Gap Covered. (%) | 100% | 75% | 87% | 70%

Table 2: Offline Continual Learning. Performance comparison with ImageNet21K
ViT-B16 model using 2 init classes and 1 new class per task. RanDumb here
assumes the features as input pixels. SLCA [62] and RanPAC [30] results
reproduced from original codebase. Comparing representation learning with no
representation learning, a fixed, pretrained feature extractor outperforms
further finetuning a deep network continually, i.e. $RepGain(\mathcal{D})<0$.
RanDumb nearly matches performance of joint, demonstrating the inefficacy of
current benchmarks. *here representation learning collapses, accuracy reported
without PETL. +here representation learning and random projection both
collapse, reporting last stable accuracy.

Method | CIFAR | IN-A | IN-R | CUB | OB | VTAB | Cars
---|---|---|---|---|---|---|---
Comparison with Best Method
Best (RanPAC) | 89.6 | 26.8+ | 67.3 | 87.2+ | 77.9* | 88.2+ | 53.7∗
RanDumb (Ours) | 86.8 | 42.2 | 64.9 | 88.5 | 75.3 | 92.3 | 69.1
Improvement | -2.9 | +15.4 | -2.4 | +1.3 | -2.6 | +3.7 | +15.4
Only Ablating Representation Learning: No Finetuning v/s Full Finetuning
SLCA | 86.8 | - | 54.2 | 82.1 | - | - | 18.2
RanDumb (Ours) | 86.8 | 42.2 | 64.9 | 88.5 | 75.3 | 92.3 | 69.1
Improvement | +0.0 | - | +10.7 | +6.4 | - | - | +50.9
Scope of Improvement
Joint | 93.8 | 70.8 | 86.6 | 91.1 | 83.8 | 95.5 | 86.9
RanDumb (Ours) | 86.8 | 42.2 | 64.9 | 88.5 | 75.3 | 92.3 | 69.1
Gap Covered. (%) | 93% | 60% | 75% | 97% | 92% | 97% | 80%

Figure 2: Mechanism of RanDumb. RanDumb projects the input datapoints to a
high-dimensional space to create a clearer separation between classes.
Subsequently, it corrects the anisotropy across different feature dimensions,
scaling them to be unit variance each. This allows a nearest class mean
classifier to accurately rely on cosine similarity. The figure is adapted from
Pilario et al. [38].

###  1.1 RanDumb: Construction and Performance

Mechanism. Our mechanism is illustrated in Figure 1. RanDumb first projects
input pixels into a high-dimensional space using a fixed kernel based on
random Fourier basis, which is a low-rank data-independent approximation of
the RBF Kernel [42]. Then, we use a simple linear classifier which first
normalizes distances across different feature dimensions (anisotropy) with
Mahalanobis distance [31] and then uses nearest class means for classification
[33]. In scenarios with pretrained feature extractors, we use the fixed
pretrained model as embedder and learn a linear classifier over it, similar to
Hayes and Kanan [20].

Properties. RanDumb needs no storage of exemplars and requires only one pass
over the data in a one-sample-per-timestep fashion. Furthermore, it only
requires online estimation of the covariance matrix and nearest class mean,
hence is invariant to data order, representing the accuracy in the worst-case
ordering.

Poor Representation Learning. We first compare RanDumb with VAE-GC [50] in
Table 1 (middle) and SLCA [62] in Table 2 (middle). The only distinction
between them is their embedding functions: RanDumb uses a fixed function
(random/pretrained network) for the embedder, whereas VAE-GC and SLCA further
continually trained deep networks, using them as the embedder. Astonishingly,
RanDumb consistently surpasses VAE-GC and SLCA baselines by wide margins of
5-15%. This shows that state-of-the-art online continual learning algorithms
fail to learn effective representations in a single pass across standard
online continual learning datasets.

Benchmarks Over-Constrained. The reader might now ask: With better
representation learning, can one outperform RanDumb by large margins in these
settings? To study this, we compare the performance of RanDumb with Joint in
both online and offline settings respectively in Table 1 (bottom) and Table 2
(bottom). We observe that our simple baseline RanDumb recovers 70-90% of the
gap to the respective joint classifiers across both settings. Improving
forgetting in continual representation learning [21, 36] offers limited scope
for improvement, as current benchmarks might be too constrained to have
effective representation learning at all.

In the next sections, we shall first describe our proposed method and then
walk through our findings in detail.

##  2 RanDumb: Mechanism & Intuitions

In this section, we thoroughly examine RanDumb, focusing on its two main
elements: random projection and the dumb learner. We illustrate the mechanism
of RanDumb using three toy examples in Figure 2.

We first describe the dumb classifier. To classify a test sample $\bf x_{\rm
test}$, we start with a simple classifier, the nearest class mean (NCM)
classifier. It predicts the class among $C$ classes by highest value of the
similarity function $f$ among class means $\mathbf{\mu}_{i}$:

$\displaystyle
y_{\textrm{pred}}=\operatorname*{arg\,max}_{{i}\in\\{1,\ldots,|C|\\}}f({\bf
x_{\textrm{test}}},\mathbf{\mu}_{i}),\text{where}$ $\displaystyle f({\bf
x_{\textrm{test}}},\mathbf{\mu}_{i}):={\bf
x_{\textrm{test}}}^{\top}{\mathbf{\mu}_{i}}$ (2)

and $\mathbf{\mu}_{i}$ are the class-means in the pixel space:

$\mathbf{\mu}_{i}=\frac{1}{|C_{i}|}\sum_{\mathbf{x}}\mathbf{x}\qquad\forall{\bf
x}\in C_{i}$

RanDumb adds two additional components to this classifier: 1) Kernelization
and 2) Decorrelation.

1

Kernelization: Why & How?  Classes are typically not linearly separable in the
pixel space, unlike in the feature space of deep models. Hence, we apply the
kernel trick to embed the pixels in a better representation space, computing
all distances between the data and class-means in this embedding space. This
phenomena is illustrated on three toy examples to build intuitions in Figure 2
(Embed). We use an RBF-Kernel, which for two points $\mathbf{x}$ and
$\mathbf{y}$ is defined as:
$K_{\text{RBF}}(\mathbf{x},\mathbf{y})=\exp\left(-\gamma\|\mathbf{x}-\mathbf{y}\|^{2}\right)$
where $\gamma$ is a scaling parameter. However, calculating the RBF kernel is
not possible due to the online continual learning constraints preventing
computation of pairwise-distance between all points. Hence, we use a data-
independent approximation, random Fourier projection $\phi(\mathbf{x})$, as
given in [42]:

$K_{\text{RBF}}(\mathbf{x},\mathbf{y})\approx\phi(\mathbf{x})^{T}\phi(\mathbf{y})$

where the random Fourier features $\phi(\mathbf{x})$ are defined by first
sampling $D$ vectors $\\{\mathbf{\omega}_{1},\ldots,\mathbf{\omega}_{D}\\}$
from a Gaussian distribution with mean zero and covariance matrix
$2\gamma\mathbf{I}$, where $\mathbf{I}$ is the identity matrix. Then
$\phi(\mathbf{x})$ is a 2$D$-dimensional feature, defined as:

$\phi(\mathbf{x})=\frac{1}{\sqrt{D}}\left[\cos(\mathbf{\omega}_{1}^{T}\mathbf{x}),\sin(\mathbf{\omega}_{1}^{T}\mathbf{x}),..,\cos(\mathbf{\omega}_{D}^{T}\mathbf{x}),\sin(\mathbf{\omega}_{D}^{T}\mathbf{x})\right]$

We keep these $\omega$ bases fixed throughout online learning. Thus, we obtain
our modified similarity function from Equation 2 as:

$\displaystyle f({\bf x_{\rm test}},\mathbf{\mu}_{i}):={\bf\phi(x_{\rm
test})}^{\top}{\bar{\mathbf{\mu}}_{i}}$ (3)

where $\bar{\mathbf{\mu}}_{i}$ are the class-means in the kernel space:

$\bar{\mathbf{\mu}}_{i}=\frac{1}{|C_{i}|}\sum_{\mathbf{x}}\phi(\mathbf{x})\qquad\forall{\bf
x}\in C_{i}$ 2

Decorrelation: Why & How?  Projected raw pixels, similar to features of deep
models, have feature dimensions with different variances (anisotropic). Hence,
instead of naively computing ${\bf\phi(x_{\rm
test})}^{\top}{\bar{\mathbf{\mu}}_{i}}$, we further decorrelate the feature
dimensions using a Mahalonobis distance with the empirical covariance matrix
$\mathbf{S}$. We illustrate this phenomena as well on three toy examples in
Figure 2 (Decorrelate) to build intuitions. Our similarity function finally
is:

$\displaystyle f({\bf x_{\rm test}},\mathbf{\mu}_{i}):=(\phi(\mathbf{x}_{\rm
test})-\bar{\mathbf{\mu}}_{i})^{T}\mathbf{S}^{-1}(\phi(\mathbf{x}_{\rm
test})-\bar{\mathbf{\mu}}_{i})$ (4)

Online Computation. Our random projection is computed and fixed before seeing
any real data. In the continual learning process, we only update the running
class mean and empirical covariance matrix, which is accurately estimable
online111Online prediction is possible using the Sherman–Morrison formula to
update the inverse of the covariance matrix. However, we only needed to
compute inverse at the last timestep for all our experiments, hence we do not
experiment with this..

Note on Equivalences. For the curious reader we draw equivalences between the
linear classifier used in this work and related works. These equivalences rely
on the assumption that the classes are equiprobable, which is the case for
most datasets here. In these cases, nearest class mean classifier with the
Mahalanobis distance metric is equivalent to linear discriminant analysis
classifier [32]. Hence, one could say RanDumb is equivalent to a Streaming LDA
classifier with an approximate RBF Kernel. Alternatively, one could think of
the decorrelation operation as explicitly decorrelating the features with ZCA
whitening [7]. We hope these alternative perspectives allow for a better
understanding of the RanDumb classifier.

Table 3: Benchmark (Overview). We evaluate RanDumb across a diverse set of
benchmarks that progressively relax constraints compared to RanDumb. Benchmark
A closely matches RanDumb with one class per timestep and no stored exemplars.
Benchmark B, D, E progressively relax the constraints on exemplars and classes
per timestep. Benchmark C and E remove the online constraint by allowing
unrestricted training and sample access within a task without exemplar-storage
of past tasks. Benchmark F allows using large pretrained models, modified by
us with one class per task, inspired by challenges detailed in Benchmark A.

Setup | Num | #Classes | #Samples | #Stored | Contrastive
---|---|---|---|---|---
| Passes | Per Task | Per Step | Exemplars | Augment
RanDumb (Ours) | 1 | 1 | 1 | 0 | No
A (Zając et al. [59]) | 1 | 1 | 10 | 0 | No
B1 (Guo et al. [18]) | 1 | 2 | 10 | 100-2000 | No
B2 (Guo et al. [18]) | 1 | 2 | 10 | 100-1000 | Yes
C (Smith et al. [48]) | Many | 10 | All | 0 | No
D (Anonymous [4]) | 1 | 2-10 | 10 | 1000 | No
E (Ye and Bors [58]) | 1 | 2-5 | 10 | 1000-5000 | No
F (Wang et al. [54]) | Many | 1 | All | 0 | No

Table 4: Benchmark A (Ref: Table 1 from PEC [59]). We perform comprehensive
comparisons of RanDumb with popular online continual learning approaches in a
1-class per task setting referred as ‘Dataset (num_tasks/1)’. This setting is
a challenging setting for OCL methods except GDumb. We observe that RanDumb
outperforms all approaches across all datasets by 2-6% margins, with an
exception of latest work PEC [59] on CIFAR10.

| Method | Memory |  MNIST (10/1) |  CIFAR-10 (10/1) |  CIFAR-100 (100/1) |  miniImageNet (100/1)
---|---|---|---|---|---|---
| Fine-tuning | all |  10.1$\pm$ 0.0 |  10.0$\pm$ 0.0 |  1.0$\pm$ 0.0 |  1.0$\pm$ 0.0
| Joint, 1 epoch | all |  98.3$\pm$ 0.0 |  74.2$\pm$ 0.1 |  33.0$\pm$ 0.2 |  25.3$\pm$ 0.2
Rehearsal-Based |  ER [13] (ICML-W ’19) | 500 |  84.4$\pm$ 0.3 |  40.6$\pm$ 1.1 |  12.5$\pm$ 0.3 |  5.7$\pm$ 0.2
A-GEM [12] (ICLR ’19) | 500 |  59.8$\pm$ 0.8 |  10.2$\pm$ 0.1 |  1.0$\pm$ 0.0 |  1.1$\pm$ 0.1
iCaRL [44] (CVPR ’17) | 500 |  83.1$\pm$ 0.3 |  37.8$\pm$ 0.4 |  5.7$\pm$ 0.1 |  7.5$\pm$ 0.1
BiC [56] (CVPR ’19) | 500 |  86.0$\pm$ 0.4 |  35.9$\pm$ 0.4 |  6.4$\pm$ 0.3 |  1.5$\pm$ 0.1
ER-ACE [10] (ICLR ’22) | 500 |  87.8$\pm$0.2 |  39.9$\pm$0.5 |  8.2$\pm$0.2 |  5.7$\pm$0.2
DER [9] (NeurIPS ’20) | 500 |  91.7$\pm$ 0.1 |  40.0$\pm$ 1.5 |  1.0$\pm$ 0.1 |  1.0$\pm$ 0.0
DER++ [9] (NeurIPS ’20) | 500 |  91.9$\pm$ 0.2 |  35.6$\pm$ 2.4 |  6.2$\pm$ 0.4 |  1.4$\pm$ 0.1
X-DER [8] (TPAMI ’22) | 500 |  83.0$\pm$ 0.1 |  43.2$\pm$ 0.5 |  15.6$\pm$ 0.1 |  8.2$\pm$ 0.4
GDumb [39] (ECCV ’20) | 500 |  91.0$\pm$0.2 |  50.7$\pm$0.7 |  8.2$\pm$0.2 | -
Rehearsal-Free |  EWC [24] (PNAS ’17) | 0 |  10.1$\pm$ 0.0 |  10.6$\pm$ 0.4 |  1.0$\pm$ 0.0 |  1.0$\pm$ 0.0
SI [60] (ICML ’17) | 0 |  12.7$\pm$ 1.0 |  10.1$\pm$ 0.1 |  1.1$\pm$0.0 |  1.0$\pm$0.1
LwF [26] (TPAMI ’17) | 0 |  11.8 $\pm$ 0.6 |  10.1$\pm$ 0.1 |  0.9$\pm$0.0 |  1.0$\pm$ 0.0
LT [61] (Arxiv ’18) | 0 |  10.9$\pm$ 0.9 |  10.0$\pm$ 0.2 |  1.1$\pm$ 0.1 |  1.0$\pm$ 0.0
Gen-NCM [22] (NeurIPS-W ’22) | 0 |  82.0$\pm$ 0.0 |  27.7$\pm$ 0.0 |  10.0$\pm$ 0.0 |  7.5$\pm$ 0.0
Gen-SLDA [20] (CVPR-W ’20) | 0 |  88.0$\pm$ 0.0 |  41.4$\pm$ 0.0 |  18.8$\pm$ 0.0 |  12.9$\pm$ 0.0
VAE-GC [50] (CVPR-W ’21) | 0 |  84.0$\pm$ 0.5 |  42.7$\pm$ 1.3 |  19.7$\pm$ 0.1 |  12.1$\pm$ 0.1
PEC [59] (ICLR ’24) | 0 |  92.3$\pm$ 0.1 | 58.9$\pm$ 0.1 |  26.5$\pm$ 0.1 |  14.9$\pm$ 0.1
RanDumb (Ours) | 0 | 98.3 (+5.9) |  55.6 (-3.3) | 28.6 (+2.1) | 17.7 (+2.8)

##  3 Experiments

We compare RanDumb with algorithms across online continual learning benchmarks
with an emphasis on exemplar-free and low-exemplar storage regime
complementing GDumb [39] in regimes where it has poor performance.

Benchmarks. We illustrate the benchmarks used in our experiments along with
key differences to training run for RanDumb in Table 3. We aim for a truly
comprehensive coverage and show results on four different benchmarks (A, B, D,
E) which reflect the latest trends in online continual learning (’22-’24)
across exemplar-free, contrastive-training222Benchmark B is split into two
distinct sections: (i) methods that do not rely on contrastive learning and
heavy augmentation, named B1, and (ii) approaches that incorporate contrastive
learning and extra augmentations, named as Scenario B2., meta-continual
learning and network-expansion based approaches respectively and a rehearsal-
free offline continual learning benchmark C. These benchmarks are ordered by
increasingly relaxed constraints, moving further away from the training
scenario of RanDumb. These benchmarks train models from scratch. Hence,
RanDumb utilizes a random Fourier transform for embedding.

We further test on exemplar-free scenarios in offline continual learning using
Benchmark F [54] with the challenging one-class per task constraint borrowed
from [59]. This benchmark allows using pretrained models along with
unrestricted training time and access to all class samples at each timestep.
However, RanDumb is restricted to learning from a single pass seeing only one
sample at a time. RanDumb only learns a linear classifier over a given
pretrained model in Benchmark F. We use LAMDA-PILOT
codebase333https://github.com/sun-hailong/LAMDA-PILOT for all methods, except
RanPAC and SLDA which use their codebase. All methods use the original
hyperparameters. We only change initial classes to 2 and number of classes per
task to 1 in the scripts and test using both ImageNet21K and ImageNet1K
ViT-B/16 models. For detailed assumptions, task orderings, implementation
details and other specific information of these benchmarks and approaches,
readers are encouraged to consult the LAMDA-PILOT codebase.

Implementation Details (RanDumb). We evaluate RanDumb using five datasets
commonly used for online continual learning: MNIST, CIFAR10, CIFAR100,
TinyImageNet200, and miniImageNet100. For the latter two datasets, we
downscale all images to 32x32. We augment each datapoint with flipped version,
hence two images are seen by the classifier at each timestep (except for MNIST
and Benchmark F). We normalize all images and flatten them into vectors,
obtaining 784-dim input vectors for MNIST and 3072-dim input vectors for all
the other datasets. For Benchmark F, we compare RanDumb on seven datasets used
in LAMDA-PILOT, replacing ObjectNet with Stanford Cars as ObjectNet license
prohibits training models on it. We use the 768-dimensional features from the
same pretrained ViT-B models used in this benchmark. We measure accuracy on
the test set of all past seen classes after completing the full one-pass over
the dataset. We take the average accuracy after the last task on all past
tasks [59, 18, 54]. In Benchmark A and F, since we have one class per task,
the average accuracy across past tasks is the same regardless of the task
ordering. In Benchmarks A-E, all datasets have the same number of samples,
hence similarly the average accuracy across past tasks is the same regardless
of the task ordering. We used the Scikit-Learn implementation of Random
Fourier Features [42] with 25K embedding size, $\gamma=1.0$. We use
progressively increasing ridge regression parameter ($\lambda$) with dataset
complexity, using $\lambda=10^{-6}$ for MNIST, $\lambda=10^{-5}$ for
CIFAR10/100 and $\lambda=10^{-4}$ for TinyImageNet200/miniImageNet100. All
experiments were conducted on a CPU server with a 48-core Intel Xeon Platinum
8268 CPU and 392GB of RAM. Our code is available at
https://github.com/drimpossible/RanDumb.

Table 5: Benchmark B.1 (Ref: Table adopted from OnPro [55], OCM[18]) We
perform comprehensive comparisons of RanDumb with popular online continual
learning approaches in less ideal many-classes per task setting referred as
‘Dataset (num_tasks/num_classes_per_task)’. Furthermore, results are
categorized by memory buffer sizes given in ‘M’ subcolumn for that dataset.
Comparing across rows, we observe that RanDumb outperforms the best among the
compared approaches without heavy-augmentations by 3-20% margins despite being
exemplar free. Only in one case, it is second best, but after GDumb. Comparing
the two columns of CIFAR100, we observe a large drop in performance of OCL
methods on benchmarks with longer timesteps indicating analyzing longer
timesteps is important in continual learning.

Method |  MNIST (5/2) |  CIFAR10 (5/2) |  CIFAR100 (10/10) |  CIFAR100 (50/2) |  TinyImageNet (100/2)
---|---|---|---|---|---
| $M=0.1k$ | $M=0.1k$ | $M=0.2k$ | $M=0.5k$ | $M=1k$ | $M=1k$ | $M=1k$ | $M=2k$
AGEM [12] (ICLR ’19) | 56.9±5.2 | 17.7±0.3 | 22.7±1.8 | 5.8±0.2 | 5.9±0.1 | 1.8±0.2 | 0.8±0.1 | 0.9±0.1
GSS [3] (NeurIPS ’19) | 70.4±1.5 | 18.4±0.2 | 26.9±1.2 | 8.1±0.2 | 11.1±0.2 | 4.3±0.2 | 1.1±0.1 | 3.3±0.5
ER [13] (ICML-W ’19) | 78.7±0.4 | 19.4±0.6 | 29.7±1.0 | 8.7±0.3 | 15.7±0.3 | 8.3±0.3 | 1.2±0.1 | 5.6±0.5
ASER [46] (AAAI ’21) | 61.6±2.1 | 20.0±1.0 | 27.8±1.0 | 11.0±0.3 | 16.4±0.3 | 9.6±1.3 | 2.2±0.1 | 5.3±0.3
MIR [2] (NeurIPS ’19) | 79.0±0.5 | 20.7±0.7 | 37.3±0.3 | 9.7±0.3 | 15.7±0.2 | 12.7±0.3 | 1.4±0.1 | 6.1±0.5
ER-AML [10] (ICLR ’22) | 76.5±0.1 | - | 40.5±0.7 | - | 16.1±0.4 | - | - | 5.4±0.2
iCaRL [44] (CVPR ’17) | - | 31.0±1.2 | 33.9±0.9 | 12.8±0.4 | 16.5±0.4 | - | 5.0±0.3 | 6.6±0.4
DER++ [9] (NeurIPS ’20) | 74.4±1.1 | 31.5±2.9 | 44.2±1.1 | 16.0±0.6 | 21.4±0.9 | 9.3±0.3 | 3.7±0.4 | 5.1±0.8
GDumb [39] (ECCV ’20) | 81.2±0.5 | 23.3±1.3 | 35.9±1.1 | 8.2±0.2 | 18.1±0.3 | 18.1±0.3 | 4.6±0.3 | 12.6±0.1
CoPE [15] (CVPR ’21) | - | 33.5±3.2 | 37.3±2.2 | 11.6±0.4 | 14.6±1.3 | - | 2.1±0.3 | 2.3±0.4
DVC [18] (CVPR ’22) | - | 35.2±1.7 | 41.6±2.7 | 15.4±0.3 | 20.3±1.0 | - | 4.9±0.6 | 7.5±0.5
Co²L [11] (ICCV ’21) | 83.1±0.1 | - | 42.1±1.2 | - | 17.1±0.4 | - | - | 10.1±0.2
R-RT [6] (CVPR ’21) | 89.1±0.3 | - | 45.2±0.4 | - | 15.4±0.3 | - | - | 6.6±0.3
CCIL [35] (CVPR ’21) | 86.4±0.1 | - | 50.5±0.2 | - | 18.5±0.3 | - | - | 5.6±0.9
IL2A [65] (NeurIPS ’21) | 90.2±0.1 | - | 54.7±0.5 | - | 18.2±1.2 | - | - | 5.5±0.7
BiC [56] (CVPR ’19) | 90.4±0.1 | - | 48.2±0.7 | - | 21.2±0.3 | - | - | 10.2±0.9
SSIL [1] (ICCV ’21) | 88.2±0.1 | - | 49.5±0.2 | - | 26.0±0.1 | - | - | 9.6±0.7
Rehearsal-Free
PASS [65] (NeurIPS ’21) | - | 33.7±2.2 | 33.7±2.2 | 7.5±0.7 | 7.5±0.7 | - | 0.5±0.1 | 0.5±0.1
RanDumb (Ours) | 98.3 (+7.8) | 55.6 (+20.4) | 55.6 (+5.9) | 28.6 (+12.6) | 28.6 (+2.6) | 28.6 (+10.5) | 11.6 (+6.6) | 11.6 (-1.0)

Table 6: (Left) Benchmark B.2 (Ref: Table from OnPro [55]) We now compare with
state-of-the-art contrastive representation learning based online continual
learning approaches which additionally use sophisticated augmentations. These
augmentations and additional loss function can improve any of the OCL methods
described in Setup B.1 by large margins [18]. We observe that RanDumb often
outperforms these sophisticated methods despite all of these factors on small-
exemplar settings. (Right) Benchmark C (Ref: Table 2 from [48]). We compare
the performance of RanDumb with latest efforts on improving rehearsal-free
methods. We outperform them by 4% margins.

Method |  MNIST (5/2) |  CIFAR10 (5/2) |  CIFAR100 (10/10) |  TinyImageNet (100/2)
---|---|---|---|---
| $M=0.1k$ | $M=0.1k$ | $M=0.5k$ | $M=1k$
SCR [28] (CVPR ’21) | 86.2±0.5 | 40.2±1.3 | 19.3±0.6 | 8.9±0.3
OCM [18] (ICML ’22) | 90.7±0.1 | 47.5±1.7 | 19.7±0.5 | 10.8±0.4
OnPro [55] (ICCV ’23) | - | 57.8±1.1 | 22.7±0.7 | 11.9±0.3
Rehearsal-Free
RanDumb (Ours) | 98.3 (+7.5) | 55.6 (-2.2) | 28.6 (+5.9) | 11.6 (-0.3)

Method | CIFAR100
---|---
| (10/10)
Rehearsal-Free
PredKD [26] | 24.6
PredKD + FeatKD | 12.4
PredKD + EWC | 23.3
PredKD + L2 | 21.5
RanDumb | 28.6 (+4.0)

Table 7: (Left) Benchmark D (Ref: Table 2 from VR-MCL [4]) We compare RanDumb
with meta-continual learning approaches operating in a high memory setting,
allowing buffer sizes up to 1K exemplars. Despite catering representations
specifically to combat forgetting, all methods with the exception of VR-MCL
are outperformed by RanDumb on TinyImageNet. RanDumb also surpasses all prior
work by a substantial 9.1% on CIFAR100. Allowing generous replay buffers
shifts scenarios to a high exemplar regime where GDumb performs the best on
CIFAR10. Yet RanDumb competes favorably even under these conditions not
optimized for its approach. (Middle) Benchmark E (Ref: Table 1 from SEDEM
[58]) We compare RanDumb with recent network expansion based online continual
learning approaches. Despite allowing access to much larger memory buffers,
RanDumb matches the performance of best method SEDEM on MNIST, while exceeding
it by 0.3% on CIFAR10 and 3.8% on CIFAR100. (Right) Architectures (Ref: Table
1 from Mirzadeh et al. [34]) RanDumb surpasses continual representation
learning across a wide range of architectures, achieving close to 94% of the
joint performance.

Method | CIFAR10 | CIFAR100 | TinyImageNet
---|---|---|---
| (5/2) | (10/10) | (20/10)
| $M=1k$ | $M=1k$ | $M=1k$
Finetune | 17.0 $\pm$ 0.6 | 5.3 $\pm$ 0.3 | 3.9 $\pm$ 0.2
A-GEM [12] (ICLR ’19) | 18.4 $\pm$ 0.2 | 6.0 $\pm$ 0.2 | 4.0 $\pm$ 0.2
IS [60] (ICML ’17) | 17.4 $\pm$ 0.2 | 5.2 $\pm$ 0.2 | 3.3 $\pm$ 0.3
MER [45] (ICLR ’19) | 36.9 $\pm$ 2.4 | – | –
La-MAML [19] (NeurIPS ’20) | 33.4 $\pm$ 1.2 | 11.8 $\pm$ 0.6 | 6.74 $\pm$ 0.4
GDumb [39] (ECCV ’20) | 61.2 $\pm$ 1.0 | 18.1 $\pm$ 0.3 | 4.6 $\pm$ 0.3
ER [13] (ICML-W ’19) | 43.8 $\pm$ 4.8 | 16.1 $\pm$ 0.9 | 11.1 $\pm$ 0.4
DER [9] (NeurIPS ’20) | 29.9 $\pm$ 2.9 | 6.1 $\pm$ 0.1 | 4.1 $\pm$ 0.1
DER++ [9] (NeurIPS ’20) | 52.3 $\pm$ 1.9 | 11.8 $\pm$ 0.7 | 8.3 $\pm$ 0.3
CLSER [5] (ICLR ’22) | 52.8 $\pm$ 1.7 | 17.9 $\pm$ 0.7 | 11.1 $\pm$ 0.2
OCM [18] (ICML ’22) | 53.4 $\pm$ 1.0 | 14.4 $\pm$ 0.8 | 4.5 $\pm$ 0.5
ER-OBC [14] (ICLR ’23) | 54.8 $\pm$ 2.2 | 17.2 $\pm$ 0.9 | 11.5 $\pm$ 0.2
VR-MCL [4] (ICLR ’24) | 56.5 $\pm$ 1.8 | 19.5 $\pm$ 0.7 | 13.3 $\pm$ 0.4
Rehearsal-Free
RanDumb (Ours) | 55.6 (-5.6) | 28.6 (+9.1) | 11.6 (-1.7)

Method | MNIST | CIFAR10 | CIFAR100
---|---|---|---
| (5/2) | (5/2) | (20/5)
| $M=2k$ | $M=1k$ | $M=5k$
Finetune | 19.8 $\pm$ 0.1 | 18.5 $\pm$ 0.3 | 3.5 $\pm$ 0.1
MIR [2] (NeurIPS ’19) | 93.2 $\pm$ 0.4 | 42.8 $\pm$ 2.2 | 20.0 $\pm$ 0.6
GEM [12] (ICLR ’19) | 93.2 $\pm$ 0.4 | 24.1 $\pm$ 2.5 | 11.1 $\pm$ 2.4
iCARL [44] (CVPR ’17) | 83.9 $\pm$ 0.2 | 37.3 $\pm$ 2.7 | 10.8 $\pm$ 0.4
G-MED [23] (NeurIPS ’21) | 82.2 $\pm$ 2.9 | 47.5 $\pm$ 3.2 | 19.6 $\pm$ 1.5
GSS [3] (NeurIPS ’19) | 92.5 $\pm$ 0.9 | 38.5 $\pm$ 1.4 | 13.1 $\pm$ 0.9
CoPE [15] (CVPR ’21) | 93.9 $\pm$ 0.2 | 48.9 $\pm$ 1.3 | 21.6 $\pm$ 0.7
CURL [43] (NeurIPS ’19) | 92.6 $\pm$ 0.7 | - | -
CNDPM [25] (ICLR ’20) | 95.4 $\pm$ 0.2 | 48.8 $\pm$ 0.3 | 22.5 $\pm$ 1.3
Dynamic-OCM [57] (ECCV ’22) | 94.0 $\pm$ 0.2 | 49.2 $\pm$ 1.5 | 21.8 $\pm$ 0.7
SEDEM [58] (ICCV ’23) | 98.3 $\pm$ 0.2 | 55.3 $\pm$ 1.3 | 24.8 $\pm$ 1.2
Rehearsal-Free
RanDumb (Ours) |  98.3 (0.0) | 55.6 (+0.3) | 28.6 (+3.8)

Model | CIFAR100
---|---
Joint | 79.58
CNN x1 | 62.2 $\pm$1.35
CNN x2 | 66.3 $\pm$1.12
CNN x4 | 68.1 $\pm$0.5
CNN x8 | 69.9 $\pm$0.62
CNN x16 | 76.8 $\pm$0.76
ResNet-18 | 45.0 $\pm$0.63
ResNet-34 | 44.8 $\pm$2.34
ResNet-50 | 56.2 $\pm$0.88
ResNet-101 | 56.8 $\pm$1.62
WRN-10-2 | 50.5 $\pm$2.65
WRN-10-10 | 56.8 $\pm$2.03
WRN-16-2 | 44.6 $\pm$2.81
WRN-16-10 | 51.3 $\pm$1.47
WRN-28-2 | 46.6 $\pm$2.27
WRN-28-10 | 49.3 $\pm$2.02
ViT-512/1024 | 51.7 $\pm$1.4
ViT-1024/1546 | 60.4 $\pm$1.56
RandDumb (Ours) | 74.8 (-2.0)

###  3.1 Results

We extensively evaluate RanDumb and detail our findings on each benchmark in
this section.

Benchmark A. Benchmark A assesses continual learning models in the challenging
setup of one class per timestep, closely mirroring our training assumptions.
We present our results in Table 4. Comparing across rows, and see that RanDumb
improves over prior state-of-the-art across all datasets with 2-6% margins.
The only exception is PEC on CIFAR10, where RanDumb underperforms by 3.3%.
Nonetheless, it still outperforms the second-best model, GDumb with a 500
memory size, by 4.9%.

Benchmark B1. We present our results comparing with non-contrastive methods in
Table 5. We notice that scenario allows two classes per task and relaxes the
memory constraints for online continual learning methods, allowing for higher
accuracies compared to Benchmark A. Despite that, RanDumb outperforms latest
OCL algorithms on MNIST, CIFAR10 and CIFAR100—often by margins exceeding
$>$10%. The lone exception is GDumb achieving a higher performance with 2K
memory samples on TinyImageNet, indicating that this already is in the high-
memory regime. Overall, RanDumb is the best algorithm across the benchmark.

Benchmark B2. We additionally compare our performance with the latest online
CL approaches using contrastive losses with sophisticated data augmentations.
As shown in in Table 6 (Left), these advancements provide large performance
improvements over methods from Benchmark B.1. To compensate, we compare on
lower exemplar budgets. The best approach, OnPro [55], which outpeforms
RanDumb on CIFAR10 by 2.2% and TinyImageNet by 0.3%, but falls significantly
short on CIFAR100 by 5.9%. Overall, RanDumb achieves strong results compared
to highly performant representation learning using state-of-the-art
contrastive learning approaches customized to continual learning, despite
storing no exemplars.

Benchmark C. We now compare against offline rehearsal-free continual learning
approaches in Table 6 (Right) on CIFAR100 dataset. Despite online training,
RanDumb outperforms PredKD by over 4% margins.

Benchmark D. We now compare performance of RanDumb against meta-continual
learning methods, which require large exemplars with buffer sizes of 1K in
Table 7 (left). RanDumb achieves strong performance under these conditions,
exceeding all prior work by a large margin of 9.1% on CIFAR100 and outperforms
all but VR-MCL approach on the TinyImageNet dataset. GDumb performs the best
on CIFAR10, indicating this is already in a large-exemplar regime uniquely
unsuited for RanDumb.

Benchmark E. We compare RanDumb against recent network expansion-based online
continual learning methods in Table 7 (middle). These approaches grow model
capacity to mitigate forgetting while dealing with shifts in the data
distribution, and are allow larger memory buffers. RanDumb matches the
performance of the state-of-the-art method SEDEM [58] on MNIST, while
exceeding it by 0.3% on CIFAR10 and 3.8% on CIFAR100. The results indicate
current benchmarks remain too restrictive for effective continual
representation learning.

Benchmark F. We compare performance of approaches which do not further train
the deep network like RanDumb (ours)444Recall that RanDumb in Benchmark F is
quite similar to SLDA [20]. and NCM [22] against popular prompt-tuning
approaches in Table 8. and discover that prompt-tuning approaches completely
collapse under large timesteps and approaches which do not finetune their
pretrained model achieve strong performance, even under challenging one class
per timestep constraint. Note that there are variants of RanDumb (_e.g_.
RanPAC - RP - PETL [30]) that achieve higher accuracies due to minor
modifications which are designed for this benchmark.

Conclusion. Overall, despite RanDumb being exemplar-free and modeling worst-
case ordering, it outperforms nearly all online continual learning methods
across various tasks when exemplar storage is limited. We specifically
benchmark on lower exemplar sizes to complement settings in which GDumb does
not perform well.

Table 8: Benchmark F We compare RanDumb with other prompt-tuning based
continual learning approaches using ViT-B/16 ImageNet-21K/1K pretrained models
using 2 init classes and 1 class per task setting. Most prompt-tuning based
methods collapse and simple baselines which do not pretrain the architecture
like NCM [22] or RanDumb achieve state-of-the-art performance. ’-’ indicates
that despite reasonable efforts, we could not run the method. +here
representation learning and random projection both collapse, reporting last
stable accuracy.

Method | CIFAR | IN-A | IN-R | CUB | VTAB
---|---|---|---|---|---
ViT-B/16 (IN-1K Pretrained)
Finetune | 1.0 | 1.2 | 1.1 | 1.0 | 2.1
L2P [54] (CVPR ’22) | 2.4 | 0.3 | 0.8 | 1.4 | 1.3
DualPrompt [53] (ECCV ’22) | 2.3 | 0.3 | 0.8 | 0.9 | 4.2
CODA-Prompt [47] (CVPR ’23) | 2.6 | 0.3 | 0.8 | 1.9 | 6.3
Adam-Adapt [64] (Arxiv ’23) | 76.7 | 49.3 | 62.0 | 85.2 | 83.6
Adam-SSF [64] (Arxiv ’23) | 76.0 | 47.3 | 64.2 | 85.6 | 84.2
Adam-VPT [64] (Arxiv ’23) | 79.3 | 35.8 | 61.2 | 83.8 | 86.9
Adam-FT [64] (Arxiv ’23) | 72.6 | 49.3 | 61.0 | 85.2 | 83.8
Memo [63] (ICLR ’23) | 69.8 | - | - | 81.4 | -
iCARL [44] (CVPR ’17) | 72.4 | - | 35.2 | 72.4 | -
Foster [52] (ECCV ’22) | 52.2 | - | 76.8 | 86.6 | -
NCM [22] (NeurIPS-W ’22) | 76.2 | 49.4 | 61.2 | 85.2 | 83.6
SLCA [62] (ICCV ’23) | 86.3 | - | 52.8 | 84.7 | -
RanPAC [30] (NeurIPS ’23) | 88.2 | 39.0+ | 72.8 | 77.7+ | 93.0
RanDumb (Ours) | 84.5 | 47.8 | 66.9 | 88.0 | 93.4
ViT-B/16 (IN-21K Pretrained)
Finetune | 2.8 | 0.5 | 1.2 | 1.2 | 0.5
Adam-Adapt [64] (Arxiv ’23) | 82.4 | 48.8 | 55.4 | 86.7 | 84.4
Adam-SSF [64] (Arxiv ’23) | 82.7 | 46.0 | 59.7 | 86.2 | 84.9
Adam-VPT [64] (Arxiv ’23) | 70.8 | 34.8 | 53.9 | 84.0 | 81.1
Adam-FT [64] (Arxiv ’23) | 65.7 | 48.5 | 56.1 | 86.5 | 84.4
Foster [52] (ECCV ’22) | 87.3 | - | 5.1 | 86.9 | -
iCARL [44] (CVPR ’17) | 71.6 | - | 35.1 | 71.6 | -
NCM [22] (NeurIPS-W ’22) | 81.3 | 48.9 | 54.6 | 86.7 | 84.4
SLCA [62] (ICCV ’23) | 86.8 | - | 54.2 | 82.1 | -
RanPAC [30] (NeurIPS ’23) | 89.6 | 26.8+ | 67.3 | 87.2+ | 88.2+
RanDumb (Ours) | 86.8 | 42.2 | 64.9 | 88.5 | 92.3

Table 9: RanDumb (Analysis). We thoroughly analyse the key components of
RanDumb, and observe: (1) contributions of decorrelation and embedding,
finding them interdependent and integral to performance; (2) consistent gains
from augmentation across datasets; (3) marginal returns and saturation with
higher embedding sizes, enabling computational tradeoffs; (4) increase in the
optimal regularisation parameter for more complex datasets; (5) comparisons
with alternate embeddings (or lack thereof) favoring RanDumb’s approach.

Method | MNIST | CIFAR10 | CIFAR100 | T-ImNet | m-ImNet
---|---|---|---|---|---
| (10/1) | (10/1) | (10/1) | (200/1) | (100/1)
Ablating Components of RanDumb
RanDumb | 98.3 | 55.6 | 28.6 | 11.1 | 17.7
-Decorrelate | 83.8 (-14.5) | 30.0 (-25.6) | 12.0 (-16.6) | 4.7 (-6.4) | 8.9 (-8.8)
-Embed | 88.0 (-10.3) | 41.6 (-14.0) | 19.0 (-9.6) | 8.0 (-3.1) | 12.9 (-4.8)
-Both | 82.1 (-16.2) | 28.5 (-27.1) | 10.4 (-18.2) | 4.1 (-7.0) | 7.28 (-10.4)
Effect of Adding Flip Augmentation
With | - | 55.6 | 28.6 | 11.1 | 17.7
Without | 98.3 | 52.5 (-3.1) | 26.9 (-1.7) | 10.7 (-0.4) | 16.6 (-1.1)
Variation with Ridge Parameter $\lambda$
$\lambda=10^{-6}$ | 98.3 | 53.9 | 27.8 | 10.3 | 15.8
$\lambda=10^{-5}$ | - | 55.6 | 28.6 | 11.1 | 15.9
$\lambda=10^{-4}$ | 96.6 | 52.6 | 26.1 | 11.6 | 17.7
Variation Across Embedding Projections
No-Embed | 88.0 | 41.6 | 19.0 | 8.0 | 12.9
RP+ReLU | 95.2 | 48.8 | 23.1 | 9.7 | 15.7
RanDumb (Ours) | 98.3 (+3.1) | 55.6 (+6.8) | 28.6 (+5.5) | 11.1 (+1.4) | 17.7 (+2.0)

###  3.2 RanDumb: Analysis

We extensively analyse RanDumb for the traditional OCL benchmarks in this
subsection. First, we ablate the two components (random embedder and
decorrelation) in RanDumb to ensure both components are necessary. Then we
study effect of varying the embedding dimensionality, shrinkage and removing
flipping augmentation and compare with alternate random embeddings proposed in
literature [30] and across architectures [34].

Ablating Components of RanDumb. We ablate the contribution of only using
Random Fourier features for embedding and decorrelation to the overall
performance of RanDumb in Table 9 (top). Ablating the decorrelation and
relying solely on random Fourier features, colloquially dubbed Kernel-NCM, has
performance drops ranging from 6-25% across the datasets. Replacing random
Fourier features with raw features, colloquially the SLDA baseline, leads to
pronounced drop in performance ranging from 3-14% across the datasets.
Moreover, ablating both components, resulting in the base nearest class mean
classifier exhibits the poorest performance with an average reduction of 17%.
The poor performance when ablating both decorrelation and embedding highlights
their interdependence.

Figure 3: Impact of Embedding Dimensions. We show the variation of accuracy of
RanDumb with embedding dimensionality across datasets. We see decreasing
marginal returns with increasing dimensions, obtaining relatively minor
improvements after 15K dimensions.

Impact of Embedding Dimensions. We vary the dimensions of the random Fourier
features ranging from compressing 3K input dimensions to 1K to projecting it
to 25K dimensions and evaluate its impact on performance in Figure 3.
Surprisingly, the random projection to a 3x compressed 1K dimensional space
allows for significant performance improvement over not using embedding, given
in Table 9 (top). Furthermore, increasing the dimension from 1K to 25K results
in improvements of 3.6%, 10.4%, 7.0%, and 2.5% on MNIST, CIFAR10, CIFAR100,
and TinyImageNet respectively. Increasing the embedding sizes beyond 15K,
however, only results in modest improvements of 0.1%, 1.4%, 1.1% and 0.2% on
the same datasets, indicating 15K dimensions would be a good point for a
performance-computational cost tradeoff.

Impact of Flip Augmentation. We evaluate the impact of adding the flip
augmentation on the performance of RanDumb in Table 9 (middle). Note that
MNIST was not augmented. Augmentation provided large gains of 3.1% on CIFAR10,
1.7% on CIFAR100, and 0.4% on TinyImageNet. We did not try further augmenting
the data with RandomCrop transform as done with standard augmentations.

Impact of Varying Ridge Parameter. All prior experiments use a ridge parameter
($\lambda$) that increases with dataset complexity: $\lambda=10^{-6}$ for
MNIST, $10^{-5}$ for CIFAR10 and CIFAR100, and $10^{-4}$ for TinyImageNet and
miniImageNet. Table 9 (middle) shows the effect of varying $\lambda$ on
RanDumb’s performance. With a smaller $\lambda=10^{-6}$, CIFAR10, CIFAR100,
TinyImageNet and miniImageNet all exhibit minor drops of 0.1%-1.7%, 0.8%,
0.8%. Increasing shrinkage to a $\lambda=10^{-4}$ reduces CIFAR10 and CIFAR100
performance more substantially by 3% and 2.5% versus their optimal
$\lambda=10^{-5}$. On the other hand, this larger $\lambda$ leads to
improvements of 0.5% and 1.8% on TinyImageNet and miniImageNet. This aligns
with the trend that datasets with greater complexity benefit from more
regularisation, with the optimal $\lambda$ surfacing as a result of balancing
under- and over-regularisation effects.

Comparison with RanPAC. We compared our random Fourier features with
projection based on a random weight matrix and ReLU [30] (RP+ReLU) in Table 9
(middle) with their best embedding size. Our method performs significantly
better on each dataset, averaging a gain of 3.4%. We believe this is
attributable to using a theoretical grounded random projection compared to
RanPAC [30], which is specifically designed for the setting using pretrained
transformers.

Comparisons across Architectures. In table 7 (right), we compare whether using
random Fourier features as embeddings outperforms models across various
architectures for continual representation learning. We use experience replay
(ER) baseline in the task-incremental CIFAR100 setup (for details, see
Mirzadeh et al. [34] as it differs significantly from earlier setups). Our
comparison involved across various architectures. The findings revealed that
RanDumb surpassed the performance of nearly all considered architectures, and
achieved close to 94% of the joint multi-task performance. This suggests that
RanDumb outperforms continual representation learning across a wide range of
architectures.

Conclusion. Overall, both random embedding and decorrelation are critical
components in the performance of RanDumb. Using random fourier features is
substantially better than RanPAC. Lastly, one can substantially reduce the
embedding dimension without a large drop in performance for large gains in
computational cost and RAM usage, additional augmentation may further
significantly help performance and optimal shrinkage parameter increases with
dataset complexity. RanDumb outperforms continual representation learning
across a wide range of architectures.

##  4 Discussion and Concluding Remarks

Our investigation reveals a surprising result, empirically demonstrating that
currently the representation gain is negative, i.e. $RepGain(\mathcal{D})<0$ —
simply using random embedding (RanDumb) consistently outperforms learned
representations from methods specifically designed for continual training.
Furthermore, using fixed pretrained networks consistently outperforms
continually finetuned method. Using random/pretrained features also recovers
70-90% of the gap to joint learning, leaving limited room for improvement in
representation learning techniques. Our investigation is both surprising and
alarming as it questions our understanding of how to effectively design and
train models that require efficient continual representation learning, and
necessitates a re-investigation of the widely explored problem formulation
itself. We believe adoption of computationally bounded scenarios without
memory constraints [41, 40, 17] could be a promising way forward.

## Acknowledgements

AP is funded by Meta AI Grant No. DFR05540. PT thanks the Royal Academy of
Engineering. PT and PD thank FiveAI for their support. This work is supported
in part by a UKRI grant: Turing AI Fellowship EP/W002981/1 and an EPSRC/MURI
grant: EP/N019474/1. The authors would like to thank Arvindh Arun, Kalyan
Ramakrishnan and Shashwat Goel for helpful feedback.

## References

  * Ahn et al. [2021] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon.  Ss-il: Separated softmax for incremental learning.  In _ICCV_ , 2021.
  * Aljundi et al. [2019a] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars.  Online continual learning with maximally interfered retrieval.  In _NeurIPS_ , 2019a.
  * Aljundi et al. [2019b] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  In _NeurIPS_ , 2019b.
  * Anonymous [2024] Anonymous.  Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction.  In _ICLR_ , 2024.
  * Arani et al. [2022] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz.  Learning fast, learning slow: A general continual learning method based on complementary learning system.  In _ICLR_ , 2022.
  * Bang et al. [2021] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi.  Rainbow memory: Continual learning with a memory of diverse samples.  In _CVPR_ , 2021.
  * Bell and Sejnowski [1996] Anthony Bell and Terrence J Sejnowski.  Edges are the’independent components’ of natural scenes.  In _NeurIPS_ , 1996.
  * Boschini et al. [2022] Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara.  Class-incremental continual learning into the extended der-verse.  _TPAMI_ , 2022.
  * Buzzega et al. [2020] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In _NeurIPS_ , 2020.
  * Caccia et al. [2022] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.  New insights on reducing abrupt representation change in online continual learning.  In _ICLR_ , 2022.
  * Cha et al. [2021] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.  Co2l: Contrastive continual learning.  In _ICCV_ , 2021.
  * Chaudhry et al. [2019a] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  In _ICLR_ , 2019a.
  * Chaudhry et al. [2019b] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  Continual learning with tiny episodic memories.  In _ICML-W_ , 2019b.
  * Chrysakis and Moens [2023] Aristotelis Chrysakis and Marie-Francine Moens.  Online bias correction for task-free continual learning.  In _ICLR_ , 2023.
  * De Lange and Tuytelaars [2021] Matthias De Lange and Tinne Tuytelaars.  Continual prototype evolution: Learning online from non-stationary data streams.  In _ICCV_ , 2021.
  * De Lange et al. [2020] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars.  Continual learning: A comparative study on how to defy forgetting in classification tasks.  In _TPAMI_ , 2020.
  * Garg et al. [2023] Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri.  Tic-clip: Continual training of clip models.  _ArXiv_ , 2023.
  * Guo et al. [2022] Yiduo Guo, Bing Liu, and Dongyan Zhao.  Online continual learning through mutual information maximization.  In _ICML_ , 2022.
  * Gupta et al. [2020] Gunshi Gupta, Karmesh Yadav, and Liam Paull.  Look-ahead meta learning for continual learning.  _NeurIPS_ , 2020.
  * Hayes and Kanan [2020] Tyler L Hayes and Christopher Kanan.  Lifelong machine learning with deep streaming linear discriminant analysis.  In _CVPR-W_ , 2020.
  * Hess et al. [2023] Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars.  Knowledge accumulation in continually learned representations and the issue of feature forgetting.  _arXiv preprint arXiv:2304.00933_ , 2023.
  * Janson et al. [2022] Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny.  A simple baseline that questions the use of pretrained-models in continual learning.  In _NeurIPS-W_ , 2022.
  * Jin et al. [2021] Xisen Jin, Junyi Du, and Xiang Ren.  Gradient based memory editing for task-free continual learning.  In _NeurIPS_ , 2021.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _PNAS_ , 2017.
  * Lee et al. [2020] Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim.  A neural dirichlet process mixture model for task-free continual learning.  In _ICLR_ , 2020.
  * Li and Hoiem [2017] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _TPAMI_ , 2017.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In _NeurIPS_ , 2017.
  * Mai et al. [2021] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner.  Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning.  In _CVPR_ , 2021.
  * Mai et al. [2022] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner.  Online continual learning in image classification: An empirical survey.  _Neurocomputing_ , 2022.
  * McDonnell et al. [2023] Mark D McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel.  Ranpac: Random projections and pre-trained models for continual learning.  In _NeurIPS_ , 2023.
  * McLachlan [1999] Goeffrey J McLachlan.  Mahalanobis distance.  _Resonance_ , 4(6):20–26, 1999.
  * McLachlan [2005] Geoffrey J McLachlan.  _Discriminant analysis and statistical pattern recognition_.  John Wiley & Sons, 2005.
  * Mensink et al. [2013] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka.  Distance-based image classification: Generalizing to new classes at near-zero cost.  _TPAMI_ , 2013.
  * Mirzadeh et al. [2022] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Architecture matters in continual learning.  _arXiv preprint arXiv:2202.00275_ , 2022.
  * Mittal et al. [2021] Sudhanshu Mittal, Silvio Galesso, and Thomas Brox.  Essentials for class incremental learning.  In _CVPR_ , 2021.
  * Mukhoti et al. [2023] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania.  Fine-tuning can cripple your foundation model; preserving features may be the solution.  _arXiv preprint arXiv:2308.13320_ , 2023.
  * Mundt et al. [2023] Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh.  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  _Neural Networks_ , 2023.
  * Pilario et al. [2020] Karl Ezra Pilario, Mahmood Shafiee, Yi Cao, Liyun Lao, and Shuang-Hua Yang.  A review of kernel methods for feature extraction in nonlinear process monitoring.  _Processes_ , 8(1), 2020.
  * Prabhu et al. [2020] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In _ECCV_ , 2020.
  * Prabhu et al. [2023a] Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener.  Online continual learning without the storage constraint.  _arXiv preprint arXiv:2305.09253_ , 2023a.
  * Prabhu et al. [2023b] Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi.  Computationally budgeted continual learning: What does matter?  In _CVPR_ , 2023b.
  * Rahimi and Recht [2007] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  _NeurIPS_ , 2007.
  * Rao et al. [2019] Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell.  Continual unsupervised representation learning.  _NeurIPS_ , 32, 2019.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In _CVPR_ , 2017.
  * Riemer et al. [2019] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _ICLR_ , 2019.
  * Shim et al. [2021] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang.  Online class-incremental continual learning with adversarial shapley value.  In _AAAI_ , 2021.
  * Smith et al. [2023a] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira.  Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning.  In _CVPR_ , 2023a.
  * Smith et al. [2023b] James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira.  A closer look at rehearsal-free continual learning.  In _CVPR-W_ , 2023b.
  * van de Ven and Tolias [2018] Gido M van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  In _NeurIPS-W_ , 2018.
  * Van De Ven et al. [2021] Gido M Van De Ven, Zhe Li, and Andreas S Tolias.  Class-incremental learning with generative classifiers.  In _CVPR-W_ , 2021.
  * Verwimp et al. [2023] Eli Verwimp, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H Lampert, et al.  Continual learning: Applications and the road forward.  _arXiv preprint arXiv:2311.11908_ , 2023.
  * Wang et al. [2022a] Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan.  Foster: Feature boosting and compression for class-incremental learning.  In _ECCV_ , 2022a.
  * Wang et al. [2022b] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al.  Dualprompt: Complementary prompting for rehearsal-free continual learning.  In _European Conference on Computer Vision (ECCV)_ , 2022b.
  * Wang et al. [2022c] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022c.
  * Wei et al. [2023] Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan.  Online prototype learning for online continual learning.  In _ICCV_ , 2023.
  * Wu et al. [2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In _CVPR_ , 2019.
  * Ye and Bors [2022] Fei Ye and Adrian G Bors.  Continual variational autoencoder learning via online cooperative memorization.  In _ECCV_ , 2022.
  * Ye and Bors [2023] Fei Ye and Adrian G Bors.  Self-evolved dynamic expansion model for task-free continual learning.  In _ICCV_ , 2023.
  * Zając et al. [2024] Michał Zając, Tinne Tuytelaars, and Gido M van de Ven.  Prediction error-based classification for class-incremental learning.  _ICLR_ , 2024.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _ICML_ , 2017.
  * Zeno et al. [2018] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry.  Task agnostic continual learning using online variational bayes.  _arXiv preprint arXiv:1803.10123_ , 2018.
  * Zhang et al. [2023] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei.  Slca: Slow learner with classifier alignment for continual learning on a pre-trained model.  In _ICCV_ , 2023.
  * Zhou et al. [2022] Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan.  A model or 603 exemplars: Towards memory-efficient class-incremental learning.  _arXiv preprint arXiv:2205.13218_ , 2022.
  * Zhou et al. [2023] Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu.  Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need.  _arXiv preprint arXiv:2303.07338_ , 2023.
  * Zhu et al. [2021] Fei Zhu, Zhen Cheng, Xu-yao Zhang, and Cheng-lin Liu.  Class-incremental learning via dual augmentation.  _NeurIPS_ , 2021.


# Automated Atrial Fibrillation Classification Based on Denoising Stacked
Autoencoder and Optimized Deep Network

Prateek Singh  Dr. Ambalika Sharma  Dr. Shreesha Maiya

###### Abstract

#### Background and Objective

The incidences of atrial fibrillation (AFib) are increasing at a daunting rate
worldwide. For the early detection of the risk of AFib, we have developed an
automatic detection system based on deep neural networks. The early detection
may help survive many patients by curative measures in clinical settings.
Eventually, this may reduce the mortality rate due to AFib.

#### Methods

For achieving better classification, it is mandatory to have good pre-
processing of physiological signals. Keeping this in mind, we have proposed a
two-fold study. First, an end-to-end model is proposed to denoise the
electrocardiogram signals using denoising autoencoders (DAE). To achieve
denoising, we have used three networks including, convolutional neural network
(CNN), dense neural network (DNN), and recurrent neural networks (RNN).
Compared the three models and CNN based DAE performance is found to be better
than the other two. Therefore, the signals denoised by the CNN based DAE were
used to train the deep neural networks for classification. Three neural
networks’ performance has been evaluated using accuracy, specificity,
sensitivity, and signal to noise ratio (SNR) as the evaluation criteria.

#### Results

The proposed end-to-end deep learning model for detecting atrial fibrillation
in this study has achieved an accuracy rate of 99.20%, a specificity of
99.50%, a sensitivity of 99.50%, and a true positive rate of 99.00%. The
average accuracy of the algorithms we compared is 96.26%, and our algorithm’s
accuracy is 3.2% higher than this average of the other algorithms. The CNN
classification network performed better as compared to the other two.
Additionally, the model is computationally efficient for real-time
applications, and it takes approx 1.3 seconds to process 24 hours ECG signal.
The proposed model was also tested on unseen dataset with different
proportions of arrhythmias to examine the model’s robustness, which resulted
in 99.10% of recall and 98.50% of precision.

###### keywords:

Atrial Fibrillation , Denoising Autoencoders , Electrocardiography , Long
short-term memory , Convolutional neural network

[inst1]organization=Electrical Engineering Department,addressline=Indian
Institute of Technology, Roorkee, postcode=247667, state=Uttrakhand,
country=India

[inst2]organization=Leicester Royal Infirmary and Glenfield
Hospital,addressline=Infirmary Square , city=Leicester, postcode=LE1 5WW,
state=United Kingdom, country=England

##  1 INTRODUCTION

Atrial Fibrillation (AFib) is an arrhythmia due to the irregular and rapid
beating of the heart’s upper chamber (atria). AFib is characterized by highly
variable ventricular beat intervals and a rhythm disorder that differs from
normal sinus rhythm (NSR) in RR intervals. AFib is the most common type of
sustained cardiac arrhythmia worldwide. According to the world health
organization (WHO) data, millions of deaths are attributed to Cardiovascular
diseases (CVDs) each year [1]. CVDs are the leading cause of sudden cardiac
death worldwide. Early detection of AFib can help avoid various heart
complications such as atrial thrombosis, stroke and heart failure etc. The
most effective and feasible approaches to detecting CVD is an
electrocardiography test. The test output is an electrocardiogram (ECG) signal
that shows the heart’s electrical activity and represents the plot of voltage
against time. Medical practitioners visually analyze the ECG signal for
diagnosing AFib, which is a time-consuming and tedious process. Therefore, the
automatic detection of AFib is required. CVD can be characterized by detecting
cardiac abnormalities that open archaic avenues for researchers.

In the past few decades, the researchers proposed several computer-aided
methods to detect AFib and other arrhythmias automatically. Several methods
are proposed to enable automation. The primary task for arrhythmia detection
is to categorize each heartbeat (typically consisting of a P wave, a QRS
complex and a T wave) into a set of predefined classes. Some of them include
Feature extraction approach [2], wavelet-based techniques [3], support vector
machine-based techniques[4], hidden Markov model-based techniques [5] and
Machine learning (ML) models, and Deep Learning (DL) approaches [6] [7] [8].
ML and DL models show promising results in classifying AFib. Models available
in literature often failed to provide adequate sensitivity and specificity
because of inter-patient variability, highly skewed class-wise performance and
noise Etc. Therefore, an improved approach to segregate different heartbeats
based on arrhythmia is the need for time.

Primary potential noise sources are device power interference, baseline drift,
muscle noise, electrode contact noise. Among these motion artifacts, caused by
muscle movements, can be mistakenly registered as arrhythmia [9]. Therefore,
proper pre-processing is an essential step before analyzing the ECG signal.
Several approaches are proposed to denoise ECG signals, the majority of them
are traditional methods that are based on parameters that are highly
vulnerable to noise, such as fixed filters like finite impulse response
filters (FIR)[10] and infinite impulse response filters(IIR)[11]. Wavelet
methods[12][13][14], adaptive filtering [15][16], and empirical mode
decomposition (EMD) [17][18] are the other set of methods which requires high
computational resources. In the modern era, mobile-based ECG devices and
patches are introduced that require less complex and high yield algorithms.
One such method for filtering ECG is Denoising Autoencoders (DAE) that has
shown better filtering capabilities than other techniques because of their
powerful nonlinear mapping capabilities.[19] proposed fully convolution
network (FCN)[20] based DAE for signal denoising. FCN network used is of 13
layers $SNR_{i}mp$ achieved was 15.49db on MIT-BIH database signal, which was
corrupted by noise from MIT-BIH-NST database [21].

A variety of methods have been proposed to improve classification accuracy. ML
and DL approaches were explored in the recent past. Most popular ML techniques
are Support Vector Machines (SVM) and decision tree [22], feature engineering
[23] and spectral analysis[24], though this often limits the classification
scope. Apart from this DL based architectures are popular and shown state-of-
the-art results. Different Dense Neural Network (DNN), Convolution Neural
Network (CNN) [25] and Long Short Term Memory (LSTM) [26] based architectures
are proposed for better classification [27] [28].DL based classification shown
better accuracy for AFib detection[29][30] [31].

To the best of the author’s knowledge, no study is conducted to analyse the
ECG denoising effect on deep networks used for classification. Therefore, the
authors have proposed improved deep learning-based denoising autoencoders
using DNN, CNN, and RNN to denoise ECG signals and compared their
performances. Then, the filtered signals are given to a deep learning model
for AFib classification to study the effect of different denoising
autoencoders on classification performance, which have been shown to improve
performance without the significant increase in model complexity typically
seen in DL architectures.

In this study, a comparison between the proposed three denoising autoencoders
(DAE) architecture is made, and their performance was studied on different
noises. Then, the filtered signals obtained from DAEs are used to classify
Atrial fibrillation (Afib) and studied the effect of robust filtering on
classification using the proposed CNN network. Deep architectures are
optimized to provide efficient and robust performance, thereby optimizing
resource utilization. The methodology is shown in Figure 1.

Figure 1: Flowchart of Proposed Methodology

##  2 MATERIALS AND METHODS:

In this section, the database and pre-processing methods are explained, for
detection of AFib. All pre-processing steps and model description are
discussed, along with the computational complexity.

###  2.1 DATASET Used :

Neural Network was trained on the MIT-BIH Atrial fibrillation database (AFDB)
and MIT-BIH Normal Sinus Rhythm Database (NSRDB) [32] for AFib classification.
AFDB includes twenty-five 10 hours of ECG recordings sampled at 250Hz of which
raw ECG of two records (“00735” and “03665”) are not available, and two
records (“04936” and “05091”) include many incorrect reference annotations.
So, only 21 records are taken for this study. These recordings come with
manually prepared rhythm annotation files. In the database, if one or more
beats are showing sign of AFib then the beat sequence are classified as
AFib(mostly paroxysmal), and all other beats are classifies as normal. The
database has 1221574 beats, out of which 519687 are Afib. NSRDB includes 18
long-term ECG recordings.No arrhythmia beats are available, except sporadic
ectopy.
The MIT-BIH Noise Stress Test Database (NSTDB)[21] is used to test the
denoising performance of DAEs . The database includes noise records (‘bw’,
‘em’, and ‘ma’) that can be added to ECG records to create noise stress test
records. It also contains 12 sample noise stress test records generated by
‘nst’, by adding ‘em’ noise to MIT-BIH Arrhythmia Database records 118 and
119. For convenience, reference annotation files are included here; all are
copies of the original reference annotation files for records 118 and 119.
Here ’bw’ represents baseline wander, ’em’ represents electrode motion and
’ma’ represents muscle artifacts.

Table 1: Database Used Databases | Time,min | AF,records | Normal,records | Sampling,Hz
---|---|---|---|---
MIT-BIH AFDB | 600 | 21 | 0 | $250$
MIT-BIH NSRDB | 1500 | 0 | 18 | $128$
Total | 2100 | 21 | 18 | $-$

###  2.2 Pre-processing

The experiment was conducted on the Keras with Google Tensorflow 2.3 backend
Deep Learning Library on python 3.7 software. The computer used has Intel Core
i7-8700 3.20GHz CPU, 16GB memory and 4GB NVIDIA GeForce GTX 1070 graphics card
with Cuda libraries. The input to the proposed denoising autoencoders is ECG
signal and noise. The ECG signal is pre-processed before giving it to the
proposed model as explained:

  1. 1.

The ECG signals from the NSRDB is resampled to 250Hz to make them compatible
to AFDB database.

  2. 2.

Then the signal is normalized between 1 and -1

  3. 3.

R peaks are detected using Pan Tompkins algorithm[33].

  4. 4.

The continuous ECG signals are divided into a sequence of heartbeats and
splitted into normal and AFib based on the annotation files. Window of 1.2 sec
is taken as the normal range for RR interval is 0.6-1.2 seconds
https://emedicine.medscape.com/article/2172196-overview as AFib beats RR
intervals are always smaller than the normal beats.The segmented windows are
zero padded as Afib has less number of samples as shown in Figure 2.

  5. 5.

Data prepared is shuffled and splitted into test, train and validation data in
the ratio of 15:75:10 percentage respectively.

  6. 6.

To remove the affect of imbalanced data 30,000 samples for AFib and non-AFib
is chosen randomly, totally 60,000 samples.

Figure 2: Normal and AFib sample window

The performance of three DAE architectures was analysed by introducing
Additive white Gaussian noise (AWGN), which is an addition of random noises
that occur in nature with different decibel(db) scale. The model was also
trained on MIT-BIH AFDB database signals after introducing the real noises
from the MIT-BIH NST database to adapt to most real noises. These datasets
signals are also re-sampled to 250 Hz to match our model specifications.

##  3 Denoising Autoencoders

DAE architecture is divided into three layers, namely input, encoding and
decoding layer. In DAE, input and output should have the same dimension as we
are constructing the input. Denoising Autoencoders(DAEs) with Dense Neural
Network (DNN), Convolution Neural Network (CNN) [25] and Long Short Term
Memory (LSTM) [26] were used for testing of noise filtering capabilities in
ECG. DAE model are shown in figure 3.

Figure 3: Denoising Autoencoder

$\theta\psi=argmin(\frac{1}{n}\sum_{i=1}^{n}L(x_{i},z_{i}))$ (1)

$=argmin(\frac{1}{n}\sum_{i=1}^{n}L(x_{i},g_{\psi}(f_{\theta}(x_{i})))$ (2)

where L is a cost function, the mean squared error is used as
$(MSE):L({x_{i}},z_{i})=[\\![x_{i}-z_{i}]\\!]^{2}\\\ $

AEs are usually used for clean signals and to denoise the noisy signals,
denoising AEs (DAE) are introduced by Vincent et al.[34] where noise is added
to the input data and designed such that more robust features can be extracted
than the AE algorithm. Figure 1 shows a schematic of DAE. Input vector
$x\in{\mathbb{[}0,1]}^{d}$ is given to the system, where d is the input
vector’s dimension. The initial input x is corrupted to $\tilde{x}$ by a
stochastic mapping $\tilde{x}\sim{C(\tilde{x}|x)}$, which partially destroys
the input data, as per destruction rate. The algorithm uses the corrupted
$\tilde{x}$ as input data and then maps it to the corresponding $y_{i}$ and
ultimately to its reconstruction $z_{i}$.
The encoder converts $\tilde{x}$ to y, that represents partial information.
This is a non-linear transformation as $y=f_{\theta}(x)=\sigma(W\tilde{x}+b)$.
The decoder converts y back to reconstructed data. This is a non-linear
transform as

Minimizing the objective function as shown in equation 2 is the main objective
that can be achieved by training parameters of DAE.

$\theta\psi=argmin(\frac{1}{n}\sum_{i=1}^{n}L(x_{i},g_{\psi}(f_{\theta}(\tilde{x_{i}})))$
(3)

In this paper, noisy ECG signals are used to train DAEs before the automated
feature extraction for classification, and their denoising performance are
tested statistically using signal to noise ratio $(SNR_{i})$, mean square
error (MSE), Peak to signal noise ratio$(PSNR_{i})$,percent root mean square
difference (PRD).

###  3.1 CNN Based DAE Architecture

Figure 4: Architecture of CNN based DAE

The proposed CNN based DAE network is represented in Figure 4. The encoder of
convolutional layers and decoder of symmetrical deconvolutional layers
together form the CNN network. ECG signal added with noise is received as an
input by the network, and the network output is a clean ECG signal. The
convolutional layer is a feature extraction layer that extracts ECG features
while performing denoising of ECG. Subsequently, the deconvolutional layer
recovers the signal’s original details by decoding the ECG abstraction
performed by the convolutional layer. As ECG signals are very noisy;
therefore, a large filter patch is required for efficient results. Large patch
size will help to extract more context information from the larger region of
the signal. As the patch size increases, the receptor field of the network
will also increase. To increase the receptive field, it is often recommended
to use the deeper network, but that will increase the computational cost.
Therefore, we chose two convolutional and two deconvolutional layers for our
model to give optimal results without being too deep. Since our data is
temporal, we adopt the 1D convolution and deconvolution layers. Pooling
operation is not generally performed in denoising tasks as we need full signal
details. But in our case, the pooling operation will help increase the
receptive field, so we used max-pooling with the stride of two. The max-
pooling layer is used for reducing the convoluted feature’s spatial size,
which will lead to lessening the computational cost and control overfitting.
The 3*3 kernel size is chosen as it was large enough to include sufficient
signal information. The ”same” padding for optimizing the output size. For
non-linearity after each layer, the ReLu activation function with a slope of
0.3 is utilized. Early stopping is also utilized in model training. A batch
size of 32 is taken for training.

###  3.2 DNN Based DAE Architecture

Figure 5: Architecture of DNN based DAE

The DNN performs the encoding and decoding of DNN based DAE. A noisy ECG
signal is received as input by the network, and the network output is a
denoised ECG signal. Three dense layers are used for encoding, and three dense
layers are used for decoding. A ReLu activation layer follows each dense
layers. The last dense layer is followed by sigmoid activation, as shown in
figure 5.

###  3.3 RNN Based DAE Architecture

RNN architecture is preferred for sequential data. The most widely used RNN
architecture is LSTMs. LSTM consists of memory blocks and memory cells, along
with gate units [35]. They are designed in such a way that they can remember
values at arbitrary intervals. The proposed architecture of the LSTM based DAE
model is described in figure 6. The LSTM performs the encoding and decoding of
LSTM based DAE. Two LSTM layers are used for encoding, and a repeat vector and
an LSTM layer is used in decoding, as shown in figure 6. Here, the LSTM
encoder extracts the ECG signals’ features while the decoder converts the
feature maps to the output. Parameters of the encoders and decoders are
computed by using an unsupervised training process.

Figure 6: Architecture of RNN(LSTM) based DAE

###  3.4 Network Training

For training, the network means squared error loss was minimized, which is
defined in equation 3. For all three DAE network, Adam algorithm [36] was
selected for optimizing, and the learning rate was $1e^{-5}$ for DNN based DAE
and CNN based DAE and $1e^{-3}$ for LSTM based DAE. The comparison of results
is shown in Table 5. CNN based DAE outperformed the other two. So we took the
output of CNN based DAE for the classification model.

##  4 Classification

For classification, we have trained and compared three proposed architectures
based on CNN, DNN and RNN. The clean signal from the CNN based DAE model is
given input to the classification models.

###  4.1 DNN Model for Classification

The DNN model contains seven dense layers with a ReLu activation function and
a softmax output layer. Model is trained to minimize categorical cross-entropy
loss [37]. Adam algorithm is used to optimize the model. A Drop-out layer of
0.3, i.e. 30 %, is used before three dense layers to avoid overfitting. A
batch size of 64 is taken, and the model got trained for 50 epochs. The
confusion matrix of classification is shown in figure 11, and the
classification report is shown in Table 2.

Table 2: DNN architecture Layer | Type | Output Shape | Number of Parameters
---|---|---|---
1 | Dense(ReLu) | (None,1024) | 308224
2 | Dropout | (None,1024) | 0
3 | Dense(ReLu) | (None,1024) | 1049600
4 | Dropout | (None,1024) | 0
5 | dense(ReLu) | (None,512) | 524800
6 | Dense(ReLu) | (None,128) | 65664
7 | Dense(ReLu) | (None,64) | 8256
7 | Dense(Sigmoid) | (None,2) | 130

###  4.2 CNN Model for Classification

The CNN model contains three convolution layers, three max-pooling layers,
three batch normalization layers, one dropout layer, one fully connected layer
and three dense layers shown in Table 3. A batch normalization layer follows
each Convolution layer. Batch normalization standardizes the inputs to a layer
for each mini-batch. This has the impact of stabilizing the learning process
and drastically decreasing the number of training epochs required to train
deep neural networks. The batch Normalization layer is followed by the Max
pooling layer, which helps control the overfitting and decrease the
computational cost. Each max-pooling layer has a pool size of two, a stride of
two, and the ’same’ padding. To tackle non-linearity ReLu activation function
is utilized after every convolution layer and dense layer. For training,
networks categorical cross-entropy loss is minimized. Adam algorithm was used
to optimize the network. A batch size of 128 is chosen, and the network got
trained for 16 epochs.

Table 3: CNN architecture Layer | Type | Output Shape | Number of Parameters
---|---|---|---
1 | Conv1D(ReLu) | (None, 296, 64) | 384
2 | Batch Normalization | (None, 296, 64) | 256
3 | Max Pooling | (None, 148, 64) | 0
4 | Conv1D(ReLu) | (None, 146, 64) | 12352
5 | Batch Normalization | (None, 146, 64) | 256
6 | Max Pooling | (None, 73, 64) | 0
7 | Conv1D(ReLu) | (None, 71, 64) | 12352
8 | Batch Normalization | (None, 71, 64) | 256
9 | Max Pooling | (None, 36, 64) | 0
10 | Flatten | (None, 2304) | 0
11 | Dropout | (None, 2304) | 0
12 | Dense(ReLu) | (None, 128) | 295040
13 | Dense(ReLu) | (None, 32) | 4128
14 | Dense(Sigmoid) | (None, 2) | 66

###  4.3 RNN Model for Classification

The details of the Bidirectional LSTM network are shown in Table 4. A CNN-
BiLSTM approach for the classification of AFib is published recently [38].
Bidirectional LSTM cells are followed by a dropout layer of 0.2. The dense
layer is used with the ReLu activation function. The last dense layer gives us
the output, and as this is a binary classification, so sigmoid activation is
used for the last layer. Thirty-eight thousand three hundred eighty-eight
parameters are trained during the process, and 50 epochs are used for
training.

Table 4: RNN(Bidirectional LSTM) architecture Layer | Type | Output Shape | Number of Parameters
---|---|---|---
1 | Input | (None,300) | -
2 | Bidirectional(LSTM) | (None,300,64) | 8960
3 | Bidirectional(LSTM) | (None,64) | 25088
4 | Dropout | (None,64) | 0
5 |  $dense_{1}$(ReLu) | (None,64) | 4160
6 |  $dense_{2}$(Sigmoid) | (None,64) | 130

##  5 Experimental Results

###  5.1 Experimental Results of Proposed Denoising Autoencoders

The denoised signals and reconstruction quality of all three proposed DAE,
i.e. DNN based DAE, CNN based DAE, and LSTM based DAE, is shown in Figure 7.
In Figure 7, a random window is taken from the test set, and AWGN noise of -10
dB is added to the signal, and the signal is passed to the three trained
models, i.e. DNN based DAE, CNN based DAE, and RNN based DAE, respectively and
the denoised signal is plotted as shown.

(a)

(b)

(c)

(d)

(e)

Figure 7: Comparison of DAE output

Figure 8 represents the violin plot of all three models, which represents the
SNR improvement. Further, statistical analysis using signal to noise ratio
(SNR), peak signal to noise ratio (PSNR), mean square error (MSE), and percent
root mean square difference(PRD) metrics were used to evaluate the denoising
results obtained for the test data. Mathematically,

$SNR=\frac{\mu_{signal}}{\sigma_{denoised}}$ (4)

$MSE=\frac{1}{N}\sum{(X-X_{denoised})^{2}}$ (5)

$PSNR=10\log_{10}\frac{{X_{max}}^{2}}{MSE}$ (6)

$PRD=\sqrt{\frac{\sum{(X-X_{denoised})^{2}}}{X^{2}}}*100$ (7)

where $\mu_{signal}$ = mean of the signal and $\sigma_{denoised}$ = standard
deviation of noise $N$ is the total number of terms for which the error is to
be calculated, $X$ is the original signal.

Figure 8: Violin Plot for SNR improvement at -10db noise

Table 5 shows the SNR improvement, PSNR, PRD comparison. It is visible that
CNN based DAE outperformed the other two, whereas LSTM based DAE was the
worst. As per the performance, we have chosen CNN based DAE output signal for
the classification of AFib.

Table 5: Comparision of DAE statistics DAE | SNR improvement | PSNR | PRD(%)
---|---|---|---
DNN Based DAE | 9.82$\pm{2.69}$dB | 0.98 | 5.8
CNN Based DAE | 13.43 $\pm{1.51}$dB | 0.95 | 3.6
RNN(LSTM) Based DAE | -5.11 $\pm{2.66}$dB | -6.17 | 18.3

We have also tested our model’s performance by infusing the real noises from
the NST database, as shown in Figure 8. We have added all the three noises in
equal proportion i.e $33.3\%$ of muscles artifacts ’ma’, baseline wander ’bw’
and electrode motion ’em’. Passed the noisy signal to our CNN based DAE model
and received clean signal as shown in Figure 9, where blue colour represents a
noisy signal, red colour represents the original signal and signal in green
represents denoised reconstructed ECG signal.

Figure 9: ’bw’,’ma’,’em’ noise from NST database Figure 10: Denoised ECG using
CNN based DAE

###  5.2 Experimental Results of Proposed Classification Models

The classification results of all three models are shown with the confusion
matrix’s help, as in Figure 11. We have tested a total of 9000 beats which are
a mix of AFib and normal beats. To evaluate the performance of the model,
standard statistical measures are performed. Mathematically given as,

$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}$ (8) $Precision=\frac{TP}{TP+FP}$ (9)
$Recall=\frac{TP}{TP+FN}$ (10)
$F1-Score=2*\frac{(Recall*Precision)}{(Recall+Precision)}$ (11)

where, TP is True Positive, TN is True Negative, FP is False Positive, FN is
False Negative.

Figure 11: Confusion matrices for all the proposed methods on the MIT-BIH AFDB
database

Hyperparameter tuning of the classification model is done with the help of
Google’s Keras-tuner library [39] for optimization. This library helps to pick
the optimal set of hyperparameters in the hidden layer and dropout layers. We
have provided different values for different parameters: regularization rate
of $L2$ regularization technique, number of neurons, dropout rate, and
learning rate of Adam optimizer. We chose number of neurons in the range of
[$2:20$], dropout rate in the range of [$0.1:0.5$], learning rate of Adam
optimizer [$1e^{-2},1e^{-3},1e^{-4},1e^{-5}$].Then, each technique’s
performance is evaluated by measuring accuracy, precision, recall, and
F-Measure. The set of parameters explained in the architectures are tuned from
the Keras-tuner library.

CNN classification model performed best for our retrospective data, where we
achieved 99.20% of accuracy ith 99.50% of recall and 99.50% of precision. As
per authors knowledge, this is the best classification results achieved on
AFDB dataset. DNN classification model came second with 97.45 % of accuracy
whereas RNN classification model came last with accuracy of 91.5% of accuracy.

Table 6: Comparison of the performances of AF detection algorithms that have been validated on MIT-BIH AFDB database. Algorithm | Precision | Recall | F-1 Score | Accuracy
---|---|---|---|---
Asgari, et al. (2015)[40] | 97.10 | 97.00 | - | -
Xia, et al. (2018)[41] | 97.87 | 98.79 | - | 98.63
Wang, et al. (2019)[38] | 97.1 | 97.9 | - | 97.4
Andersen et al. (2019)[31] | 97.80 | 98.98 | - | 97.80
Mousavi et al. (2020)[30] | 98.50 | 99.80 | - | 98.80
Petmezas et al. (2021)[42] | 99.29 | 97.87 | - | -
Proposed DNN Model | 97.50 | 97.50 | 97.50 | 97.45
Proposed CNN Model | 99.50 | 99.50 | 99.0 | 99.20
Proposed RNN Model | 91.50 | 91.50 | 91.50 | 91.5

We now wanted to test the model on real patient signals those are unseen to
the model and have AFib episodes as well as the normal rhythms. We have
created a 5 such signals of varying length and with different distribution of
AFib/Normal ratio. The signals are taken from the Long term AF Database[43]
which is sampled at 128Hz and has 84 records, we have chosen random five
signals with different distribution of AFib and Normal beats using annotation.
The distribution of data is shown in Table 7.

Table 7: Validation Signal Signal | Normal beats | AFib beats | Normal beat (%) | AFib beat (%)
---|---|---|---|---
Signal 1 | 256 | 198 | 56.4 | 43.6
Signal 2 | 471 | 83 | 85 | 15
Signal 3 | 869 | 192 | 81.9 | 18.1
Signal 4 | 234 | 15 | 94 | 6
Signal 5 | 23 | 2 | 92 | 8

We got 98.8% of average accuracy for all the five signals with recall of 99.1%
and precision of 98.5%.

##  6 Discussion

The proposed model achieved state-of-the-art results, indicating that it
outperforms the traditional feature engineering techniques. The proposed CNN
based DAE model has shown good denoising capabilities, as indicated in Table
5. Previous DL models [31] struggles with a high number of False positives
(FPs) due to the noisy signals and lack of proper filtering. Therefore, we
focused on proper denoising technique before classification, and tested our
DAE model on AWGN noise and most encountered real ECG noises from the MIT-BIH
NST database, as shown in Figure 7, 9 and 10. It is visible that our best
performing CNN-based DAE achieved an average SNR improvement of 13.43 dB and
negligible change in the morphology of ECG signal shown in Figures 7 and 10.

We then compared three proposed classification models based on DNN, CNN and
RNN; architectures are shown in Table 2, 3 and 4, respectively. Table 6 shows
the performance of all three classification models on the AFDB database. Here,
the CNN classification model outperforms the other two with an accuracy of
99.20%, and the model achieved 99.50% of precision and recall with 99% of
F1-score. We further validated our CNN classification model to investigate
false positives (FPs). We have chosen the NSRDB, which contains 18 recordings
and no AFib episodes are present in the dataset. It will give us a good
intuition on FPs. AFDB database shows false positives of 0.83%, and when it
was tested on the NSRDB database, it was 1.12%, i.e. we found an increase of
0.29% in false positives, which suggests further investigation. Inspecting raw
signals suggest that our R peak detection algorithm failed to detect some of
the R peaks where the signals suffer from multiple noises. Therefore, we
suggest using signal quality indices to remove high noise segments from the
analysis and hence not increase in false-positive rates.

Again, when we have tested our end-to-end model on unseen data with different
proportion of AFib beats, we got 98.8% of average accuracy for all the five
signals with recall of 99.1% and precision of 98.5%. So, there was an increase
in FPs due to noisy segments which can be resolved by defining proper signal
quality indices (SQIs). The proposed algorithm seems promising in identifying
the risk of AFib and can be used for real time applications after validating
it on more diverse dataset and defining some SQIs.

##  7 Conclusions

The performance of DAE was compared with performance metrics such as SNR, PSNR
and PRD. Among the three networks used, CNN-based DAE performs better with
SNR, PSNR and PRD values of 13.43, 0.95, and 3.6%, respectively. This
indicates that denoising capability and reconstruction quality of the CNN
network is better among all. The classification performed with CNN shows the
highest accuracy of 99.20 % with 99.50 % recall, 99.50 % precision and 99% of
F1-score. To the best of the authors’ knowledge, this is the highest achieved
performance reported in the literature. The proposed algorithm seems promising
in identifying the risk of AFib as evaluated with retrospective data. We
suggest an evaluation of the algorithm with a more diverse dataset to develop
a robust diagnostic method. This early detection may help clinicians to
improve survival rate.

## References

  * [1] C. Mathers, D. Fat, J. Boerma, The global burden of disease: World health organization, The World Health Report (2004).
  * [2] L. B. Marinho, N. de MM Nascimento, J. W. M. Souza, M. V. Gurgel, P. P. Rebouças Filho, V. H. C. de Albuquerque, A novel electrocardiogram feature extraction approach for cardiac arrhythmia classification, Future Generation Computer Systems 97 (2019) 564–577.
  * [3] Q. Zhao, L. Zhang, Ecg feature extraction and classification using wavelet transform and support vector machines, in: 2005 International Conference on Neural Networks and Brain, Vol. 2, IEEE, 2005, pp. 1089–1092.
  * [4] E. J. d. S. Luz, W. R. Schwartz, G. Cámara-Chávez, D. Menotti, Ecg-based heartbeat classification for arrhythmia detection: A survey, Computer methods and programs in biomedicine 127 (2016) 144–164.
  * [5] D. A. Coast, R. M. Stern, G. G. Cano, S. A. Briller, An approach to cardiac arrhythmia analysis using hidden markov models, IEEE Transactions on biomedical Engineering 37 (9) (1990) 826–836.
  * [6] G. Garcia, G. Moreira, D. Menotti, E. Luz, Inter-patient ecg heartbeat classification with temporal vcg optimized by pso, Scientific Reports 7 (1) (2017) 1–11.
  * [7] P. Rajpurkar, A. Y. Hannun, M. Haghpanahi, C. Bourn, A. Y. Ng, Cardiologist-level arrhythmia detection with convolutional neural networks, arXiv preprint arXiv:1707.01836 (2017).
  * [8] H. Shi, C. Qin, D. Xiao, L. Zhao, C. Liu, Automated heartbeat classification based on deep neural network with multiple input layers, Knowledge-Based Systems 188 (2020) 105036.
  * [9] S. K. Berkaya, A. K. Uysal, E. S. Gunal, S. Ergin, S. Gunal, M. B. Gulmezoglu, A survey on ecg analysis, Biomedical Signal Processing and Control 43 (2018) 216–235.
  * [10] J. A. Van Alste, T. Schilder, Removal of base-line wander and power-line interference from the ecg by an efficient fir filter with a reduced number of taps, IEEE transactions on biomedical engineering (12) (1985) 1052–1060.
  * [11] M. S. Chavan, R. Agarwala, M. Uplane, Suppression of baseline wander and power line interference in ecg using digital iir filter, International journal of circuits, systems and signal processing 2 (2) (2008) 356–365.
  * [12] J. P. Martínez, R. Almeida, S. Olmos, A. P. Rocha, P. Laguna, A wavelet-based ecg delineator: evaluation on standard databases, IEEE Transactions on biomedical engineering 51 (4) (2004) 570–581.
  * [13] C. Li, C. Zheng, C. Tai, Detection of ecg characteristic points using wavelet transforms, IEEE Transactions on biomedical Engineering 42 (1) (1995) 21–28.
  * [14] P. S. Addison, Wavelet transforms and the ecg: a review, Physiological measurement 26 (5) (2005) R155.
  * [15] P. Singh, K. Bhole, A. Sharma, Adaptive filtration techniques for impulsive noise removal from ecg (2017).
  * [16] C. Chandrakar, M. Kowar, Denoising ecg signals using adaptive filter algorithm, International Journal of Soft Computing and Engineering (IJSCE) 2 (1) (2012) 120–123.
  * [17] M. Blanco-Velasco, B. Weng, K. E. Barner, Ecg signal denoising and baseline wander correction based on the empirical mode decomposition, Computers in biology and medicine 38 (1) (2008) 1–13.
  * [18] A. Chacko, S. Ari, Denoising of ecg signals using empirical mode decomposition based technique, in: IEEE-International Conference On Advances In Engineering, Science And Management (ICAESM-2012), IEEE, 2012, pp. 6–9.
  * [19] H.-T. Chiang, Y.-Y. Hsieh, S.-W. Fu, K.-H. Hung, Y. Tsao, S.-Y. Chien, Noise reduction in ecg signals using fully convolutional denoising autoencoders, IEEE Access 7 (2019) 60806–60813.
  * [20] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * [21] G. B. Moody, W. Muldrow, R. G. Mark, A noise stress test for arrhythmia detectors, Computers in cardiology 11 (3) (1984) 381–384.
  * [22] J. Park, K. Kang, Pchd: Personalized classification of heartbeat types using a decision tree, Computers in biology and medicine 54 (2014) 79–88.
  * [23] C. A. Ledezma, X. Zhou, B. Rodríguez, P. Tan, V. Díaz-Zuccarini, A modeling and machine learning approach to ecg feature engineering for the detection of ischemia using pseudo-ecg, PloS one 14 (8) (2019) e0220294.
  * [24] Y. Kotriwar, S. Kachhara, K. Harikrishnan, G. Ambika, Higher order spectral analysis of ecg signals, arXiv preprint arXiv:1809.08451 (2018).
  * [25] K. Fukushima, S. Miyake, Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition, 1982.
  * [26] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [27] A. Isin, S. Ozdalili, Cardiac arrhythmia detection using deep learning, Procedia computer science 120 (2017) 268–275.
  * [28] S. Nurmaini, R. Umi Partan, W. Caesarendra, T. Dewi, M. Naufal Rahmatullah, A. Darmawahyuni, V. Bhayyu, F. Firdaus, An automated ecg beat classification system using deep neural networks with an unsupervised feature extraction technique, Applied Sciences 9 (14) (2019) 2921.
  * [29] O. Faust, A. Shenfield, M. Kareem, T. R. San, H. Fujita, U. R. Acharya, Automated detection of atrial fibrillation using long short-term memory network with rr interval signals, Computers in biology and medicine 102 (2018) 327–335.
  * [30] S. Mousavi, F. Afghah, U. R. Acharya, Han-ecg: An interpretable atrial fibrillation detection model using hierarchical attention networks, Computers in Biology and Medicine 127 (2020) 104057.
  * [31] R. S. Andersen, A. Peimankar, S. Puthusserypady, A deep learning approach for real-time detection of atrial fibrillation, Expert Systems with Applications 115 (2019) 465–473.
  * [32] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, H. E. Stanley, Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals, circulation 101 (23) (2000) e215–e220.
  * [33] J. Pan, W. J. Tompkins, A real-time qrs detection algorithm, IEEE transactions on biomedical engineering (3) (1985) 230–236.
  * [34] P. Vincent, H. Larochelle, Y. Bengio, P.-A. Manzagol, Extracting and composing robust features with denoising autoencoders, in: Proceedings of the 25th international conference on Machine learning, 2008, pp. 1096–1103.
  * [35] F. A. Gers, J. Schmidhuber, F. Cummins, Learning to forget: Continual prediction with lstm (1999).
  * [36] D. Kingma, L. Ba, Adam: A method for stochastic optimization in proceedings of the 3rd international conference on learning representations (san diego, ca:).[google scholar] (2015).
  * [37] Z. Zhang, M. R. Sabuncu, Generalized cross entropy loss for training deep neural networks with noisy labels, arXiv preprint arXiv:1805.07836 (2018).
  * [38] J. Wang, W. Li, Atrial fibrillation detection and ecg classification based on cnn-bilstm, arXiv preprint arXiv:2011.06187 (2020).
  * [39] N. K. Manaswi, N. K. Manaswi, S. John, Deep learning with applications using python, Springer, 2018.
  * [40] S. Asgari, A. Mehrnia, M. Moussavi, Automatic detection of atrial fibrillation using stationary wavelet transform and support vector machine, Computers in biology and medicine 60 (2015) 132–142.
  * [41] Y. Xia, N. Wulan, K. Wang, H. Zhang, Detecting atrial fibrillation by deep convolutional neural networks, Computers in biology and medicine 93 (2018) 84–92.
  * [42] G. Petmezas, K. Haris, L. Stefanopoulos, V. Kilintzis, A. Tzavelis, J. A. Rogers, A. K. Katsaggelos, N. Maglaveras, Automated atrial fibrillation detection using a hybrid cnn-lstm network on imbalanced ecg datasets, Biomedical Signal Processing and Control 63 (2021) 102194.
  * [43] S. Petrutiu, A. V. Sahakian, S. Swiryn, Abrupt changes in fibrillatory wave characteristics at the termination of paroxysmal atrial fibrillation in humans, Europace 9 (7) (2007) 466–470.