Two Complementary Perspectives to Continual Learning:
Ask Not Only _What_ to Optimize, But Also _How_

Timm Hess Tinne Tuytelaars Gido M. van de Ven

KU Leuven KU Leuven KU Leuven

###### Abstract

Recent years have seen considerable progress in the continual training of deep
neural networks, predominantly thanks to approaches that add replay or
regularization terms to the loss function to approximate the joint loss over
all tasks so far. However, we show that even with a perfect approximation to
the joint loss, these approaches still suffer from temporary but substantial
forgetting when starting to train on a new task. Motivated by this ‘stability
gap’, we propose that continual learning strategies should focus not only on
the optimization objective, but also on the way this objective is optimized.
While there is some continual learning work that alters the optimization
trajectory (e.g., using gradient projection techniques), this line of research
is positioned as alternative to improving the optimization objective, while we
argue it should be complementary. To evaluate the merits of our proposition,
we plan to combine replay-approximated joint objectives with gradient
projection-based optimization routines to test whether the addition of the
latter provides benefits in terms of (1) alleviating the stability gap, (2)
increasing the learning efficiency and (3) improving the final learning
outcome.

##  1 INTRODUCTION

Learning continually from a stream of non-stationary data is challenging for
deep neural networks. When these networks are trained on something new, their
default behaviour is to quickly forget most of what was learned before
(McCloskey and Cohen,, 1989; Ratcliff,, 1990). Considerable progress has been
made in recent years towards overcoming such ‘catastrophic forgetting’, for a
large part thanks to methods using replay (Robins,, 1995; Rolnick et al.,,
2019) and regularization (Kirkpatrick et al.,, 2017; Li and Hoiem,, 2017).
These methods work by adding extra terms to the loss function, and they can be
interpreted as attempts to approximate the joint loss over all tasks so far.

Recently a peculiar property of replay and regularization methods was pointed
out. These approaches tend to suffer from substantial forgetting when starting
to learn a new task, although this forgetting is often temporary and followed
by a phase of performance recovery (De Lange et al.,, 2023). We postulate that
avoiding this ‘stability gap’ is important, both because the transient drops
in performance themselves can be problematic (e.g., for safety-critical
applications) and because doing so might lead to more efficient and better
performing algorithms, as constantly having to re-learn past tasks seems
wasteful. However, in preliminary experiments we find that the stability gap
cannot be overcome by merely improving replay or regularization. This
motivates us to propose that, instead, continual learning needs an additional
perspective: rather than focusing only on _what_ to optimize (i.e., the
optimization objective), the field should also think about _how_ to optimize
(i.e., the optimization trajectory).

There are existing works that explore modifying the optimization trajectory as
a mechanism for continual learning, but so far this line of research has been
positioned as alternative to improving the optimization objective. A prime
example is Gradient Episodic Memory (GEM; Lopez-Paz and Ranzato,, 2017), which
alters the optimization process by projecting gradients to encourage parameter
updates that do not strongly interfere with old tasks. Crucially, GEM applies
this optimization routine to optimizing the loss on the new task, while our
proposal is that such modified optimization routines should be used to
optimize approximations of the joint loss. As a first evaluation of the merits
of our proposition, in this pre-registered report we plan to use GEM’s
gradient projection-based optimization routine to instead optimize replay-
approximated versions of the joint loss. Using both domain- and class-
incremental learning benchmarks, we test whether this combined approach
reduces the stability gap, and whether this in turn leads to higher learning
efficiency and better final performance.

##  2 TWO PERSPECTIVES TO CONTINUAL LEARNING

In this section we use conceptual arguments and preliminary data to develop
the proposition that continual learning should focus not only on _what_ to
optimize, but also on  _how_. We first describe the current dominant approach
to continual learning (subsection 2.1), we then point out a fundamental issue
with this approach (subsection 2.2), and we finally propose a complementary
approach and explain why it could address this issue (subsection 2.3).

To help us reason about the different approaches that continual learning
methods could take, in this section we consider the following continual
learning problem. Assume a model $f_{w}$, parameterized by $w$, that has
learned a set of weights $\widehat{w}_{\text{old}}$ for an initial task111The
term ‘task’ is used here in a rather general way; it loosely refers to a
combination of a data distribution and a loss function., or a set of tasks, by
optimizing a loss function $\ell_{\text{old}}$ on training data
$D_{\text{old}}\sim\mathcal{D}_{\text{old}}$. We then wish to continue
training the same model on a new task, by optimizing a loss function
$\ell_{\text{new}}$ on training data
$D_{\text{new}}\sim\mathcal{D}_{\text{new}}$, in such a way that the model
maintains (or possibly improves) its performance on the previously learned
task(s). As has been thoroughly described in the continual learning
literature, if the model is trained on the new task in the standard way (i.e.,
optimize the new loss $\ell_{\text{new}}$ with stochastic gradient descent),
the typical result is catastrophic forgetting and a solution
$\widehat{w}_{\text{new}}$ that is good for the new task but no longer for the
old one(s).

###  2.1 The Standard Approach to Continual Learning: Improving the Loss
Function

To mitigate catastrophic forgetting, continual learning research from the past
few years has typically focused on making changes to the loss function that is
optimized. In particular, rather than optimizing the loss on the new task,
many continual learning methods can be interpreted as optimizing an
approximate version of the joint loss:

$\widetilde{\ell}_{\text{joint}}=\ell_{\text{new}}+\widetilde{\ell}_{\text{old}}$
(1)

with $\widetilde{\ell}_{\text{old}}$ the method’s proxy for the loss on the
old tasks.

A straight-forward example of this approach is ‘experience replay’, which
approximates $\ell_{\text{old}}$ by revisiting a subset of previously observed
examples that are stored in an auxiliary memory buffer. In continual learning
experiments, typically limits are imposed on the buffer’s storage capacity
and/or on the computational budget for training the model (Lesort et al.,,
2020; Wang et al.,, 2022; Prabhu et al.,, 2023). Both constraints prevent full
replay of all previously observed data, meaning that $\ell_{\text{old}}$ can
only be approximated. A wide range of studies aims to improve the quality of
this approximation, for example by modifying the way samples are selected to
be stored in the buffer (Rebuffi et al.,, 2017; Chaudhry et al., 2019b, ;
Aljundi et al., 2019b, ; Lin et al.,, 2021; Mundt et al.,, 2023), or by
adaptively selecting which samples from the buffer to replay (Riemer et al.,,
2018; Aljundi et al., 2019a, ). As an alternative to storing past samples
explicitly, generative models can be learned to approximate the input
distributions of previous tasks (Robins,, 1995; Shin et al.,, 2017; van de Ven
et al.,, 2020).

Another popular class of methods for continual learning is based on
regularization. As proxy for the loss on the old tasks, these methods add
regularizing terms to the loss that impose penalties either for changes to the
network’s weights (‘parameter regularization’; Kirkpatrick et al.,, 2017;
Zenke et al.,, 2017; Aljundi et al.,, 2018) or for changes to the network’s
input-output mapping (‘functional regularization’; Li and Hoiem,, 2017; Dhar
et al.,, 2019; Lee et al.,, 2019; Titsias et al.,, 2020). That these methods
can be interpreted as attempts to approximate the joint loss can be shown by
taking either a Bayesian perspective (Nguyen et al.,, 2018; Farquhar and Gal,,
2019; Kao et al.,, 2021; Rudner et al.,, 2022) or a geometric perspective
(Kolouri et al.,, 2020).

In summary, both replay- and regularization-based methods for continual
learning operate by changing the loss function that is optimized, often with
the aim of creating an approximate version of the joint loss. When developing
new continual learning methods of this kind, the challenge is to design better
objective functions.

###  2.2 The Stability Gap: A Challenge for the Standard Approach to Continual
Learning

It was recently pointed out that even when replay- or regularization-based
methods are considered to perform well (in the sense that they obtain good
performance on both old and new tasks after finishing training on the new
task), these methods still suffer from substantial, albeit often temporary,
forgetting during the initial phase of training on a new task (De Lange et
al.,, 2023). Until recently, this phenomenon – referred to as the _stability
gap_ – had not been observed, or been paid little attention to, due to the
common evaluation setup in continual learning that only measures performance
after training on the new data has converged. Nevertheless, the stability gap
is undesirable, especially for safety-critical applications in which sudden
drops in performance can be highly problematic. But also from the perspective
of computational efficiency, or even if only the final learning outcome is of
interest, avoiding the stability gap might be beneficial, as preventing
forgetting seems easier and more efficient than having to re-learn later on
(see Figure 4 of Van de Ven et al. (2020) for empirical support for this
intuition).

Figure 1: The stability gap occurs even with incremental joint training (or
‘full replay’). Shown is the test accuracy on the first task while the network
is incrementally trained on all five tasks of Domain CIFAR. During the $n$-th
task, the network is trained jointly on all training data from the first $n$
tasks. Even with this ideal approximation to $\ell_{\text{joint}}$,
performance severely drops upon encountering a new task. Displayed are the
means over five repetitions, shaded areas are $\pm 1$ standard error of the
mean. Vertical dashed lines indicate task switches.

Why does the stability gap happen? One possibility is that the stability gap
is due to imprecision in the approximations of the joint loss made by replay
and regularization. If this were the case, it could be interpreted as good
news for the standard approach to continual learning, as it would imply that
by continuing to improve the quality of replay or regularization the stability
gap could be overcome. However, this is not the case, as in preliminary
experiments we find that the stability gap is consistently observed even with
incremental joint training (Figure 1). This indicates that with better
approximations to the joint loss alone, the stability gap cannot be solved.

###  2.3 Proposed Complementary Approach: Improving the Optimization
Trajectory

The above observations relating to the stability gap suggest that the standard
approach to continual learning of focusing on the loss function is not
sufficient. We believe that continual learning would benefit from an
additional perspective: rather than concentrating only on improving the
optimization objective (i.e., what loss function to optimize), we argue that
continual learning should also focus on improving the optimization trajectory
(i.e., how to optimize that loss function).

Figure 2: Schematic of the stability gap, and how adjusting the optimization
trajectory could avoid it. When, starting from a solution for the old tasks
($\widehat{w}_{\text{old}}$), a proxy of the joint loss
($\widetilde{\ell}_{\text{joint}}$) is optimized with standard stochastic
gradient descent, the optimization trajectory first passes through a region in
parameter space with high loss on the old tasks before converging to a
solution that is good for all tasks ($\widehat{w}_{\text{joint}}$). Work on
mode connectivity suggests that a low-loss path between
$\widehat{w}_{\text{old}}$ and $\widehat{w}_{\text{joint}}$ exists as well
(dashed arrow), indicating that it should be possible to overcome the
stability gap with a different optimization routine. Green shading indicates
areas of low loss on the old tasks.

To help explain why we believe that focusing on the optimization trajectory
can yield benefits, we revisit the continual learning problem discussed at the
start of section 2, which is schematically illustrated in Figure 2. Starting
point is a model $f_{w}$ that has already learned a solution
$\widehat{w}_{\text{old}}$ by optimizing loss $\ell_{\text{old}}$. Continuing
to train this model by optimizing loss $\ell_{\text{new}}$ would result in
catastrophic forgetting. Instead, as discussed, the standard approach in
continual learning is to optimize
$\widetilde{\ell}_{\text{joint}}=\ell_{\text{new}}+\widetilde{\ell}_{\text{old}}$,
an approximation to the joint loss, rather than $\ell_{\text{new}}$.
Optimizing a suitably approximated version of the joint loss results in a
solution $\widehat{w}_{\text{joint}}$ that is good for both the old and the
new tasks. However, if this loss is optimized with standard stochastic
gradient descent, the trajectory that is taken from $\widehat{w}_{\text{old}}$
to $\widehat{w}_{\text{joint}}$ goes through a region in parameter space where
the loss on the old tasks is high. The corresponding transient drop in
performance on the old tasks is the stability gap.

A first possibility that must be dealt with is that the stability gap is
unavoidable, in the sense that there simply is no path from
$\widehat{w}_{\text{old}}$ to $\widehat{w}_{\text{joint}}$ that does not
traverse a region where the performance on old tasks is poor. Although this is
theoretically possible, this option seems unlikely given recent work on mode
connectivity in deep neural networks (Draxler et al.,, 2018; Garipov et al.,,
2018) showing that different local optima found by stochastic gradient descent
are often connected by simple paths of non-increasing loss. In particular,
Mirzadeh et al., (2021) showed that when optimizing a neural network using
stochastic gradient descent on the joint loss while starting from a single
task solution, the resulting joint solution is connected to the single task
solution by a linear manifold of low loss on the single task. The same holds
when, starting from a single task solution, the replay-approximated joint loss
is optimized rather than the joint loss itself (Verwimp et al.,, 2021).

The above work on mode connectivity thus suggests that by changing the
optimization routine it should be possible to avoid the stability gap. Besides
that reducing the stability gap is important for safety-critical applications,
we believe that it could bring other benefits for continual learning as well.
For example, getting rid of the repeated re-learning cycles that characterize
the stability gap could increase the learning efficiency. Moreover, if the
loss function is non-convex, as is the case with deep neural networks,
changing the way the loss is optimized could also lead to different, and
hopefully better, final learning outcomes. The possibility of improved final
performance is supported by the observation from Caccia et al., (2022) that
the abrupt forgetting after task switches is not always recovered later on.
This leads us to the following three hypotheses:

Main hypothesis Better optimization routines for continual learning can: (H1)
reduce the stability gap. Secondary hypotheses Reducing the stability gap can:
(H2) increase learning efficiency; (H3) improve the final learning outcome.

##### How to Improve Optimization for Continual Learning?

In the above we have made an argument that continual learning should focus on
improving its optimization routines, but we have not yet discussed _how_ this
could be done. To avoid the stability gap, an optimization routine is needed
that is less greedy than standard stochastic gradient descent and that favors
parameter updates that do not substantially increase the loss on old tasks.
Interestingly, as we review in section 3, optimization routines based on
gradient projection have been explored in the continual learning literature
that already have these properties. Importantly, however, currently these
gradient projection-based optimization routines are not used in the way
envisioned by us, as they are used to optimize the loss on the new task rather
than an approximated version of the joint loss.

###  2.4 Another Way to Avoid the Stability Gap?

An alternative approach that can circumvent the stability gap in continual
learning is to use certain parts of the network only for specific tasks. This
approach is employed by network expansion methods (Rusu et al.,, 2016; Yoon et
al.,, 2018; Yan et al.,, 2021) and parameter isolation methods (Serra et al.,,
2018; Masse et al.,, 2018). To see why this approach can help to avoid the
stability gap, consider the extreme case of using a separate sub-network for
each task. In this case there is no forgetting at all, and thus also no
stability gap. However, the use of task-specific components has some important
disadvantages. Firstly, if task identity is not always provided, as is the
case with domain- and class-incremental learning (van de Ven et al.,, 2022),
it might not be clear which parts of the network should be used. This issue
could be addressed by inferring task identity, for example using generative
models or other out-of-distribution detection techniques (van de Ven et al.,,
2021; Henning et al.,, 2021; Kim et al.,, 2022; Zając et al.,, 2023), but such
task inference can be challenging. Secondly, having separate parts of the
network per task limits the potential of positive transfer between tasks,
which is an important desideratum for continual learning (Hadsell et al.,,
2020).

##  3 GRADIENT PROJECTION-BASED OPTIMIZATION

A tool that has been explored in the continual learning literature for
modifying the way a given loss function $\ell(w)$ is optimized is ‘gradient
projection’. With gradient projection, rather than basing the parameter
updates on the original gradient $g=\nabla_{w}\ell(w)$, they are based on a
projected version $\bar{g}$ of that gradient. Important from the perspective
of our paper, gradient projection does not alter the loss function that is
optimized (e.g., the loss landscape and its local minima remain unchanged), it
only changes the way the loss function is optimized (Kao et al.,, 2021). In
this section we review two current lines of continual learning studies that
make use of gradient projection.

###  3.1 Orthogonal Gradient Projection

Orthogonal gradient projection methods aim to avoid interference between tasks
by confining the training of each new task to previously unused subspaces. To
restrict parameter updates to directions that do not interfere with the
performance on old tasks, the gradient of the loss on the new task is
projected to the orthogonal complement of the ‘gradient subspaces’ of old
tasks. Various ways to construct these gradient subspaces have been proposed.
Several studies use the subspaces spanned by all layer-wise inputs of old
tasks, which they characterize using conceptors (He and Jaeger,, 2018) or by
iteratively accumulating projector matrices using a recursive least squares
algorithm (Zeng et al.,, 2019; Guo et al.,, 2022). To reduce the memory and
computational costs, Saha et al., (2021) approximate the input subspaces of
each task using singular value decomposition and its $k$-rank approximation.
Farajtabar et al., (2020) instead use the span of a set of stored gradient
directions of old tasks as the gradient subspace to protect.

Constraining sequential optimization to orthogonal subspaces can mitigate
forgetting effectively, but it restricts the learning of new tasks to
successively smaller subspaces and it eliminates the potential to further
improve the model with respect to old tasks. Orthogonal gradient projection
methods especially struggle or break down when the input spaces of different
tasks substantially overlap with each other (He,, 2018).

Recent advancements in this line of work therefore focus on relaxing the
constraints of the orthogonal projection framework to enable knowledge
transfer between tasks. Deng et al., (2021) dynamically scale gradients and
search for flatter minima, and Lin et al., (2022) use the idea of ‘trust
regions’ (Schulman et al.,, 2015) to selectively relax constraints for
protected subspaces of old tasks most related to the new task. As an
alternative, Kao et al., (2021) bring a Bayesian perspective to gradient
projection. They use the inverse of a Kronecker-factored approximation to the
Fisher information matrix as their projector matrix, and additionally protect
previous knowledge by parameter-based regularization.

###  3.2 Gradient Episodic Memories

Another class of gradient projection-based optimization methods for continual
learning, which also does not enforce strict orthogonality of future updates,
is based on Gradient Episodic Memory (GEM; Lopez-Paz and Ranzato,, 2017). The
projection mechanism of this approach is motivated by a constrained
optimization problem, where the goal is to optimize $\ell_{\text{new}}$
without increasing $\ell_{\text{old}}$:

$\text{min}_{w}\ \ell_{\text{new}}(w),~{}~{}\text{such that}\
\ell_{\text{old}}(w)\leq\ell_{\text{old}}(\widehat{w}_{\text{old}})$ (2)

To determine whether a parameter update based on
$g=\nabla_{w}\ell_{\text{new}}(w)$ might increase $\ell_{\text{old}}$, the
gradient(s) for the old task(s) are estimated using examples from a replay
buffer: $g_{\text{old}}=\nabla_{w}\widetilde{\ell}_{\text{old}}(w)$. If the
directions of $g$ and $g_{\text{old}}$ align (in the sense that their angle
does not exceed $90^{\circ}$), it is conjectured that a parameter update based
on $g$ is unlikely to increase $\ell_{\text{old}}$, and $g$ is left unchanged.
If the angle between $g$ and $g_{\text{old}}$ exceeds $90^{\circ}$, $g$ is
projected to
$\bar{g}=g-\frac{g^{\text{T}}g_{\text{old}}}{g_{\text{old}}^{\text{T}}g_{\text{old}}}g_{\text{old}}$,
which is the closest gradient to $g$ (in $l_{2}$-norm) with a $90^{\circ}$
angle to $g_{\text{old}}$. Because in our continual learning example it is the
case that $\ell_{\text{old}}$ encompasses all past tasks, this description
actually corresponds to Averaged GEM (A-GEM; Chaudhry et al., 2019a, ), a
computationally more efficient version of GEM. The original formulation of GEM
enforces $\bar{g}$ to align with the gradient of each individual past task
(Lopez-Paz and Ranzato,, 2017).

Given that GEM and A-GEM explicitly aim to prevent increases of the loss on
old tasks, these methods might be able to avoid the stability gap.
Empirically, however, this is not the case, as GEM suffers from considerably
larger stability gaps than experience replay (De Lange et al.,, 2023).
Moreover, also in terms of final performance, experience replay consistently
outperforms both GEM and A-GEM (De Lange et al.,, 2022; van de Ven et al.,,
2022).

We expect that the disappointing performance of GEM is due to its choice of
objective function: GEM optimizes the loss on the new task (i.e.,
$\ell_{\text{new}}$) rather than an approximation to the joint loss (i.e.,
$\widetilde{\ell}_{\text{joint}}$). In other words, we believe that GEM under-
utilizes its replay buffer by solely delineating gradient constraints but not
actively optimizing the replay-approximated joint loss. When GEM was proposed,
it was assumed that directly optimizing a joint loss approximated with a
relatively small replay buffer could not work well due to overfitting (Lopez-
Paz and Ranzato,, 2017), but recent work indicates such overfitting is not as
detrimental as thought (Chaudhry et al., 2019b, ; Verwimp et al.,, 2021).
Nevertheless, as far as we are aware, changing GEM’s objective function has
not been explored.

##  4 PROOF-OF-CONCEPT EXPERIMENTS

As discussed in the last section, the gradient projection-based optimization
routine of GEM encourages parameter updates that do not strongly interfere
with old tasks without imposing overly strict constraints that would fully
segregate tasks. Yet, so far this optimization routine has not been used to
optimize proxies of the joint loss. This makes GEM a convenient tool for a
first set of proof-of-concept experiments to evaluate the merits of our
proposition that continual learning should consider both _what_ and _how_ to
optimize. We plan to combine GEM’s optimization routine both with a basic
version of experience replay that explicitly approximates the joint loss
(subsection 4.1) and with state-of-the-art replay-based methods (subsection
4.2).

###  4.1 Experience Replay with Gradient Projection-based Optimization

In a first set of experiments we test whether, when the optimization objective
is a standard replay-approximated version of the joint loss, using GEM’s
gradient projection-based optimization routine provides the benefits
hypothesized in subsection 2.3.

##### Approximating the Joint Loss

To approximate the joint loss we use a basic version of Experience Replay
(ER). In our implementation of ER, at the end of each task new examples are
added to the memory buffer using class-balanced sampling from the training
set, and in each training iteration uniform sampling from the buffer is used
to choose which samples to replay. To approximate the joint loss as closely as
possible, when training on the $n$-th task, we balance the loss on the current
data and the loss on the replayed data using
$\widetilde{\ell}_{\text{joint}}=\frac{1}{n}\ell_{\text{{new}}}+(1-\frac{1}{n})\widetilde{\ell}_{\text{old}}$.
In each iteration, the total number of replayed samples from all past tasks
combined is always equal to $b$, which is the size of the mini-batch from the
current task.

In addition to approximating the joint loss with ER, we also run experiments
using the joint loss itself. For this, all training data from past tasks are
stored, and in each iteration we use $b$ samples from each past task to
compute $\ell_{\text{old}}$. This can be thought of as ‘full replay’.

##### Optimization Trajectory

To improve the sub-optimal optimization trajectory that is taken by standard
ER, we use the gradient projection-based optimization routines of GEM and
A-GEM. Importantly, we only use the optimization routines of GEM and A-GEM,
not their optimization objectives. As optimization objective we instead use
the replay-approximated joint loss $\widetilde{\ell}_{\text{joint}}$. To
achieve this, in the description of GEM in the first paragraph of subsection
3.2, we only need to replace all mentions of $\ell_{\text{new}}$ with
$\widetilde{\ell}_{\text{joint}}$. To further illustrate our proposed
combination approach, pseudocode for ER + A-GEM is provided in Algorithm 1.

Algorithm 1 ER + A-GEM

parameters $w$, loss function $\ell$, learning rate $\lambda$, for for for
data stream $\\{D_{1},...,D_{T}\\}$

$M\leftarrow\\{\\}$

for $t=1,...,T$ do

for $(x,y)\in D_{t}$ do

$g\leftarrow\nabla_{w}\ell(f_{w}(x),y)$

$(\tilde{x},\tilde{y})\leftarrow\texttt{SAMPLE}(M)$

$g_{\text{old}}\leftarrow\nabla_{w}\ell(f_{w}(\tilde{x}),\tilde{y})$

$g_{\text{joint}}\leftarrow\frac{1}{t}g+(1-\frac{1}{t})g_{\text{old}}$

$\bar{g}\leftarrow\texttt{PROJECT}(g_{\text{joint}},g_{\text{old}})$

$w\leftarrow\texttt{OPTIMIZER\\_STEP}(w,\lambda,\bar{g})$

end for

$M$ ← UPDATE_BUFFER($M,D_{t}$)

end for

function PROJECT($g,g_{\text{ref}}$)

if $g^{\text{T}}g_{\text{ref}}\geq 0$ then

return $g$

else

return
$g-\frac{g^{\text{T}}g_{\text{ref}}}{g_{\text{ref}}^{\text{T}}g_{\text{ref}}}g_{\text{ref}}$

end if

end function

##### Approaches to Compare

The main experimental comparison of interest is between standard ER and our
proposed combination approach ER + GEM, as this allows testing whether, when
doing continual learning by optimizing a proxy of the joint loss, benefits can
be gained by changing the way this objective is optimized. Additionally, to
probe the individual contributions of the optimization objective and the
optimization routine, we also include GEM itself and continual finetuning in
our experimental comparison. See Table 1 for an overview of the approaches we
compare. In this table ER can be replaced by ‘full replay’, and GEM can be
replaced by A-GEM. We run experiments with all combinations of these base
methods.

Table 1:  Overview of the approaches to compare in our proof-of-concept
experiment, illustrated with ER and GEM as base methods. GP: gradient
projection.
|  Approximate |  GP-based
---|---|---
Method |  joint loss |  optimization
Finetuning |  ✗ |  ✗
ER |  $\boldsymbol{\checkmark}$ |  ✗
GEM |  ✗ |  $\boldsymbol{\checkmark}$
ER + GEM |  $\boldsymbol{\checkmark}$ |  $\boldsymbol{\checkmark}$

###  4.2 Improving State-of-the-art

Next we ask whether the use of gradient projection-based optimization could
improve the performance of state-of-the-art replay-based methods. To test
this, we run the methods Dark Experience Replay (DER; Buzzega et al.,, 2020)
and Bias Correction (BiC; Wu et al.,, 2019) both with and without using the
optimization routine of A-GEM. For these experiments we only consider A-GEM,
as it is not straight-forward to combine DER with the original version of GEM.
We implement DER and BiC according to their original papers. For completeness,
details for both methods will be provided in the Appendix. As BiC is a method
that is specialized for class-incremental learning, it is included only with
the class-incremental learning benchmarks.

##  5 EXPERIMENTAL PROTOCOL

###  5.1 Setup

We consider a task-aware supervised continual learning setting, with a task
sequence $\mathcal{T}=\\{T_{1},...,T_{T}\\}$ of $T$ disjoint classification
tasks $T_{t}$. A fixed capacity neural network model $f_{w}$ is incrementally
trained on these tasks with a cross-entropy classification loss. When training
on task $T_{t}$, the model has only access to the training data
$D_{t}=\\{X_{t},Y_{t}\\}$ of that task and the data in the memory buffer (see
below), and the goal is to learn a model with strong performance on all tasks
$T_{\leq t}$ encountered so far. The model may be evaluated after any
parameter update.

##### Benchmarks

We conduct our study on four benchmarks, covering the domain- and class-
incremental learning scenarios (van de Ven et al.,, 2022). As class-
incremental learning benchmarks we use Split CIFAR-100, which is based on the
CIFAR-100 dataset (Krizhevsky et al.,, 2009), and Split Mini-Imagenet, which
is based on Mini-Imagenet (Vinyals et al.,, 2016). Both original datasets
contain 50,000 RGB images of 100 classes; CIFAR-100 in resolution 32x32, Mini-
Imagenet in resolution 84x84. For Split CIFAR-100 the classes are divided into
ten tasks with ten classes each, for Split Mini-Imagenet the classes are
divided into twenty tasks with five classes each. In both cases, the classes
are divided over the tasks randomly, and for each random seed a different
division is used. We also use the CIFAR-100 dataset to construct Domain CIFAR,
a domain-incremental learning benchmark. For this benchmark, each of the
twenty super-classes of CIFAR-100 is split across five tasks, such that every
task contains one member of each super-class (i.e., there are twenty classes
per task, one from each super-class). The goal in each task is to predict to
which super-class a sample belongs. The other domain-incremental learning
benchmark is Rotated MNIST. Each task consists of the entire MNIST dataset
(LeCun et al.,, 1998) with a certain static rotation applied. We construct
three tasks with rotations $\\{0^{\circ},80^{\circ},160^{\circ}\\}$, as De
Lange et al., (2023) found these to provoke the largest stability gaps without
inducing ambiguity between digits $6$ and $9$ (which would happen with
rotations close to $180^{\circ}$).

##### Architectures

For the Rotated MNIST benchmark, we use a fully-connected neural network with
two hidden layers of 400 ReLUs each, followed by a softmax output layer. For
the other benchmarks, following Lopez-Paz and Ranzato, (2017), we use a
reduced ResNet-18 architecture. Compared to a standard ResNet-18 (He et al.,,
2016), this architecture has three times less channels in each layer and
replaces the $7\times 7$ kernel with stride of 2 in the initial convolutional
layer by a $3\times 3$ kernel with stride of 1. The latter prevents an early
stark information reduction for images with small resolution. All benchmarks
are trained with a single-headed final layer that is shared between all tasks.

##### Memory Buffer

The memory buffer can store up to 100 samples of each class. For the domain-
incremental learning benchmarks this means 100 samples of each class per task
(e.g., with Rotated MNIST, for each digit the buffer can store 100 examples
with rotation $0^{\circ}$, 100 examples with rotation $80^{\circ}$ and 100
examples with rotation $160^{\circ}$). Exceptions to this are the experiments
with ‘full replay’, in which all training data are stored.

##### Offline & Online

All experiments are run in both an ‘offline version’ and an ‘online version’.
In the offline version, multiple passes over the data are allowed, and the
number of training iterations is set relatively high to encourage near
convergence for each task. In the online version only a single epoch per task
is allowed (i.e., each sample is seen just once, with the exception if it is
replayed from memory).

##### Training Hyperparameters

All models are trained using an SGD optimizer with momentum 0.9 and no weight-
decay. When gradient projection is used, this optimizer acts on the projected
gradients. Except for whitening (with mean and standard deviation of the
respective full training sets), no data augmentations are used. Exceptions to
this are the experiments with DER and BiC, for which we use the data
augmentations described in the original papers that proposed these methods
(Buzzega et al.,, 2020; Wu et al.,, 2019). In the offline experiments, we
train with mini-batch size 128 for around five epochs (Rotated MNIST) or ten
epochs (Domain CIFAR, Split CIFAR-100 and Split Mini-Imagenet) per task. To be
exact, we use 2000 iterations per task for Rotated MNIST, 800 for Domain
CIFAR, 400 for Split CIFAR-100, and 200 for Split Mini-Imagenet. For each
experiment in the offline setting, we sweep a set of static learning rates
$\\{0.1,0.01,0.001\\}$. For each experiment in the online setting, we sweep
both a set of mini-batch sizes $\\{10,64,128\\}$ and a set of static learning
rates $\\{0.1,0.01,0.001\\}$. In the online setting, the number of iterations
per task is determined by the selected mini-batch size and the number of
training samples.

###  5.2 Evaluation

We track the performance of all methods throughout training using ‘continual
evaluation’ (De Lange et al.,, 2023). In particular, after every training
iteration we evaluate for each task the accuracy of the model on a hold-out
test set.

##### Testing the Hypotheses

To quantitatively compare the stability gap of different approaches (i.e., to
evaluate H1), we use the ‘average minimum accuracy’ metric defined by De Lange
et al., (2023). For completeness, details of this metric will be provided in
the Appendix. To qualitatively compare the stability gaps, we plot per-task
accuracy curves with per-iteration resolution (e.g., as in Figure 1). To
compare the learning efficiency of different approaches (i.e., to evaluate
H2), we use the final average accuracy of the online experiments. To compare
the final learning outcomes of different approaches (i.e., to evaluate H3), we
use the final average accuracy of the offline experiments.

##### Computational Complexity

To provide insight into the computational complexity of the considered
methods, we report for each method its empirical training time on the online
version of Split CIFAR-100. For this evaluation all methods are run on
identical hardware.

##### Standard Errors

Each experiment is run five times, with a different random seed and different
division of the classes over tasks for each run. For each metric, both the
mean over these runs and the standard error of the mean are reported.

##  6 OUTLOOK

Motivated by the stability gap in replay- and regularization-based methods, in
this pre-registered report we proposed that strategies for continual learning
should focus not only on _what_ to optimize, but also on _how_. To empirically
evaluate the merits of this conceptual proposition, we described a set of
proof-of-concept experiments in which gradient projection-based optimization
is used to optimize replay-approximated versions of the joint objective. We
will now perform these experiments by rigorously following the detailed
protocol laid out in this report. We expect to communicate the results of
these experiments at the latest in June 2024.

#### Acknowledgements

We thank Matthias De Lange, Eli Verwimp and anonymous reviewers for useful
comments. This project has been supported by funding from the European Union
under the Horizon 2020 research and innovation program (ERC project
KeepOnLearning, grant agreement No. 101021347) and under Horizon Europe (Marie
Skłodowska-Curie fellowship, grant agreement No. 101067759).

## References

  * Aljundi et al., (2018) Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. (2018).  Memory aware synapses: Learning what (not) to forget.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154.
  * (2) Aljundi, R., Caccia, L., Belilovsky, E., Caccia, M., Lin, M., Charlin, L., and Tuytelaars, T. (2019a).  Online continual learning with maximal interfered retrieval.  Advances in Neural Information Processing Systems, 32.
  * (3) Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. (2019b).  Gradient based sample selection for online continual learning.  Advances in Neural Information Processing Systems, 32.
  * Buzzega et al., (2020) Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. (2020).  Dark experience for general continual learning: a strong, simple baseline.  Advances in Neural Information Processing Systems, 33.
  * Caccia et al., (2022) Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., and Belilovsky, E. (2022).  New insights on reducing abrupt representation change in online continual learning.  In International Conference on Learning Representations (ICLR).
  * (6) Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. (2019a).  Efficient lifelong learning with a-GEM.  In International Conference on Learning Representations (ICLR).
  * (7) Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. (2019b).  On tiny episodic memories in continual learning.  arXiv preprint arXiv:1902.10486.
  * De Lange et al., (2022) De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. (2022).  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366–3385.
  * De Lange et al., (2023) De Lange, M., van de Ven, G. M., and Tuytelaars, T. (2023).  Continual evaluation for lifelong learning: Identifying the stability gap.  In International Conference on Learning Representations (ICLR).
  * Deng et al., (2021) Deng, D., Chen, G., Hao, J., Wang, Q., and Heng, P.-A. (2021).  Flattening sharpness for dynamic gradient projection memory benefits continual learning.  Advances in Neural Information Processing Systems, 34.
  * Dhar et al., (2019) Dhar, P., Singh, R. V., Peng, K.-C., Wu, Z., and Chellappa, R. (2019).  Learning without memorizing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5138–5146.
  * Draxler et al., (2018) Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. (2018).  Essentially no barriers in neural network energy landscape.  In International Conference on Machine Learning (ICML), pages 1309–1318.
  * Farajtabar et al., (2020) Farajtabar, M., Azizan, N., Mott, A., and Li, A. (2020).  Orthogonal gradient descent for continual learning.  In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 3762–3773.
  * Farquhar and Gal, (2019) Farquhar, S. and Gal, Y. (2019).  A unifying bayesian view of continual learning.  arXiv preprint arXiv:1902.06494.
  * Garipov et al., (2018) Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. (2018).  Loss surfaces, mode connectivity, and fast ensembling of dnns.  Advances in Neural Information Processing Systems, 31.
  * Guo et al., (2022) Guo, Y., Hu, W., Zhao, D., and Liu, B. (2022).  Adaptive orthogonal projection for batch and online continual learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6783–6791.
  * Hadsell et al., (2020) Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. (2020).  Embracing change: Continual learning in deep neural networks.  Trends in Cognitive Sciences, 24(12):1028–1040.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
  * He, (2018) He, X. (2018).  Continual learning by conceptor regularization.  In Continual Learning Workshop (NeurIPS 2018).
  * He and Jaeger, (2018) He, X. and Jaeger, H. (2018).  Overcoming catastrophic interference using conceptor-aided backpropagation.  In International Conference on Learning Representations (ICLR).
  * Henning et al., (2021) Henning, C., Cervera, M., D’Angelo, F., Von Oswald, J., Traber, R., Ehret, B., Kobayashi, S., Grewe, B. F., and Sacramento, J. (2021).  Posterior meta-replay for continual learning.  Advances in Neural Information Processing Systems, 34.
  * Kao et al., (2021) Kao, T.-C., Jensen, K., van de Ven, G., Bernacchia, A., and Hennequin, G. (2021).  Natural continual learning: success is a journey, not (just) a destination.  Advances in Neural Information Processing Systems, 34.
  * Kim et al., (2022) Kim, G., Xiao, C., Konishi, T., Ke, Z., and Liu, B. (2022).  A theoretical study on solving continual learning.  Advances in Neural Information Processing Systems, 35.
  * Kirkpatrick et al., (2017) Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017).  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526.
  * Kolouri et al., (2020) Kolouri, S., Ketz, N. A., Soltoggio, A., and Pilly, P. K. (2020).  Sliced cramer synaptic consolidation for preserving deeply learned representations.  In International Conference on Learning Representations (ICLR).
  * Krizhevsky et al., (2009) Krizhevsky, A., Hinton, G., et al. (2009).  Learning multiple layers of features from tiny images.  _Techinal Report_ , University of Toronto, Canada.
  * LeCun et al., (1998) LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al. (1998).  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324.
  * Lee et al., (2019) Lee, K., Lee, K., Shin, J., and Lee, H. (2019).  Overcoming catastrophic forgetting with unlabeled data in the wild.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 312–321.
  * Lesort et al., (2020) Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., and Díaz-Rodríguez, N. (2020).  Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.  Information Fusion, 58:52–68.
  * Li and Hoiem, (2017) Li, Z. and Hoiem, D. (2017).  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947.
  * Lin et al., (2022) Lin, S., Yang, L., Fan, D., and Zhang, J. (2022).  TRGP: Trust region gradient projection for continual learning.  In International Conference on Learning Representations (ICLR).
  * Lin et al., (2021) Lin, Z., Shi, J., Pathak, D., and Ramanan, D. (2021).  The CLEAR benchmark: Continual LEArning on real-world imagery.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  * Lopez-Paz and Ranzato, (2017) Lopez-Paz, D. and Ranzato, M. (2017).  Gradient episodic memory for continual learning.  Advances in Neural Information Processing Systems, 30.
  * Masse et al., (2018) Masse, N. Y., Grant, G. D., and Freedman, D. J. (2018).  Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization.  Proceedings of the National Academy of Sciences, 115(44):E10467–E10475.
  * McCloskey and Cohen, (1989) McCloskey, M. and Cohen, N. J. (1989).  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of Learning and Motivation, volume 24, pages 109–165. Elsevier.
  * Mirzadeh et al., (2021) Mirzadeh, S. I., Farajtabar, M., Gorur, D., Pascanu, R., and Ghasemzadeh, H. (2021).  Linear mode connectivity in multitask and continual learning.  In International Conference on Learning Representations (ICLR).
  * Mundt et al., (2023) Mundt, M., Hong, Y., Pliushch, I., and Ramesh, V. (2023).  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  Neural Networks, 160:306–336.
  * Nguyen et al., (2018) Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E. (2018).  Variational continual learning.  In International Conference on Learning Representations (ICLR).
  * Prabhu et al., (2023) Prabhu, A., Al Kader Hammoud, H. A., Dokania, P. K., Torr, P. H., Lim, S.-N., Ghanem, B., and Bibi, A. (2023).  Computationally budgeted continual learning: What does matter?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3698–3707.
  * Ratcliff, (1990) Ratcliff, R. (1990).  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  Psychological Review, 97(2):285.
  * Rebuffi et al., (2017) Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. (2017).  ICARL: Incremental classifier and representation learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2001–2010.
  * Riemer et al., (2018) Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I., Tu, Y., and Tesauro, G. (2018).  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In International Conference on Learning Representations (ICLR).
  * Robins, (1995) Robins, A. (1995).  Catastrophic forgetting, rehearsal and pseudorehearsal.  Connection Science, 7(2):123–146.
  * Rolnick et al., (2019) Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. (2019).  Experience replay for continual learning.  Advances in Neural Information Processing Systems, 32.
  * Rudner et al., (2022) Rudner, T. G., Smith, F. B., Feng, Q., Teh, Y. W., and Gal, Y. (2022).  Continual learning via sequential function-space variational inference.  In International Conference on Machine Learning, pages 18871–18887.
  * Rusu et al., (2016) Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. (2016).  Progressive neural networks.  arXiv preprint arXiv:1606.04671.
  * Saha et al., (2021) Saha, G., Garg, I., and Roy, K. (2021).  Gradient projection memory for continual learning.  In International Conference on Learning Representations (ICLR).
  * Schulman et al., (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015).  Trust region policy optimization.  In International Conference on Machine Learning (ICML), pages 1889–1897.
  * Serra et al., (2018) Serra, J., Suris, D., Miron, M., and Karatzoglou, A. (2018).  Overcoming catastrophic forgetting with hard attention to the task.  In International Conference on Machine Learning (ICML), pages 4548–4557.
  * Shin et al., (2017) Shin, H., Lee, J. K., Kim, J., and Kim, J. (2017).  Continual learning with deep generative replay.  Advances in Neural Information Processing Systems, 30.
  * Titsias et al., (2020) Titsias, M. K., Schwarz, J., de G. Matthews, A. G., Pascanu, R., and Teh, Y. W. (2020).  Functional regularisation for continual learning with gaussian processes.  In International Conference on Learning Representations (ICLR).
  * van de Ven et al., (2021) van de Ven, G. M., Li, Z., and Tolias, A. S. (2021).  Class-incremental learning with generative classifiers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W), pages 3611–3620.
  * van de Ven et al., (2020) van de Ven, G. M., Siegelmann, H. T., and Tolias, A. S. (2020).  Brain-inspired replay for continual learning with artificial neural networks.  Nature Communications, 11:4069.
  * van de Ven et al., (2022) van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. (2022).  Three types of incremental learning.  Nature Machine Intelligence, 4(12):1185–1197.
  * Verwimp et al., (2021) Verwimp, E., De Lange, M., and Tuytelaars, T. (2021).  Rehearsal revealed: The limits and merits of revisiting samples in continual learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9385–9394.
  * Vinyals et al., (2016) Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. (2016).  Matching networks for one shot learning.  Advances in Neural Information Processing Systems, 29.
  * Wang et al., (2022) Wang, L., Zhang, X., Yang, K., Yu, L., Li, C., Hong, L., Zhang, S., Li, Z., Zhong, Y., and Zhu, J. (2022).  Memory replay with data compression for continual learning.  In International Conference on Learning Representations (ICLR).
  * Wu et al., (2019) Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., and Fu, Y. (2019).  Large scale incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 374–382.
  * Yan et al., (2021) Yan, S., Xie, J., and He, X. (2021).  DER: Dynamically expandable representation for class incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3014–3023.
  * Yoon et al., (2018) Yoon, J., Yang, E., Lee, J., and Hwang, S. J. (2018).  Lifelong learning with dynamically expandable networks.  In International Conference on Learning Representations (ICLR).
  * Zając et al., (2023) Zając, M., Tuytelaars, T., and van de Ven, G. M. (2023).  Prediction error-based classification for class-incremental learning.  arXiv preprint arXiv:2305.18806.
  * Zeng et al., (2019) Zeng, G., Chen, Y., Cui, B., and Yu, S. (2019).  Continual learning of context-dependent processing in neural networks.  Nature Machine Intelligence, 1(8):364–372.
  * Zenke et al., (2017) Zenke, F., Poole, B., and Ganguli, S. (2017).  Continual learning through synaptic intelligence.  In International Conference on Machine Learning (ICML), pages 3987–3995.


# Using Convolutional Neural Networks for fault analysis and alleviation in
accelerator systems

Jashanpreet Singh Sraw  Thapar Institute of Engineering and Technology
Patiala, India  Deepak M C  PES College of Engineering
Mandya, India

###### Abstract

Today, Neural Networks are the basis of breakthroughs in virtually every
technical domain. Their application to accelerators has recently resulted in
better performance and efficiency in these systems. At the same time, the
increasing hardware failures due to the latest (shrinked) semiconductor
technology needs to be addressed. Since accelerator systems are often used to
back time-critical applications such as self-driving cars or medical diagnosis
applications, these hardware failures must be eliminated. Our research
evaluates these failures from a systemic point of view. Based on our results,
we find critical results for the system reliability enhancement and we further
put forth an efficient method to avoid these failures with minimal hardware
overhead.

###### Index Terms:

fault analysis, neural networks, accelerator systems, deep neural networks

##  I Introduction

Deep neural networks (DNNs) have been demonstrated to be successful in massive
territories like image processing, video processing, and natural language
processing over the years [1, 2, 3]. The success further provokes the flourish
of customized neural network accelerators with massive parallel processing
engines which typically offer much higher performance and energy efficiency
compared to general purposed processors (GPPs) [4, 5, 6, 7, 8]. While the
performance of neural network accelerators has been intensively optimized from
various angles like pruning and quantization, the reliability of the
accelerators especially the ones that are deployed on FPGAs remains not well-
explored.

The shrinking semiconductor technology greatly improves the transistor density
of chips, but the circuits with smaller feature size are more susceptible to
manufacturing defects and abnormal physical processes like thermal stress,
electromigration, hot carrier injection, and gate oxide wear-out. Thereby, the
persistent (hard) faults become one of the major sources of system
unreliability. Despite the fault-tolerance of neural network models, permanent
faults in neural network accelerators can cause computing errors and
considerable wrong predictions [9]. They may even lead to disastrous
consequences to some of the mission-critical applications such as self-
driving, nuclear power plants, and medical diagnosis, which are sensitive to
neural network prediction mistakes.

It is seminal to comprehend the impact of errors on neural network
accelerators to improve the model functioning. To this end, several research
works are published that contain methods to identify errors in neural network
accelerators [10, 11, 12, 13, 14]. However, they usually experimented with
simulators and focused on the influence of hardware faults on the prediction
accuracy of the neural network models. For instance, the authors in [10]
mainly investigated the data faults of weights, activities, and hidden states,
which are stored in on-chip buffers. Then they explored the neural network
model resilience with model-wise analysis and layer-wise analysis. The work in
[13] targeted at the analysis of software errors in DNN accelerators and
explored the error propagation behaviors based on the structure of the neural
networks, data types and so on.

The simulation based approaches are fast and flexible for analyzing neural
network computing and prediction accuracy, but they typically ignore critical
controlling details and interfaces of the DNN accelerators to ensure the
simulation speed. Nevertheless, hardware faults on these components may have
considerable influence to the overall acceleration system other than the
prediction accuracy loss. For instance, faults in DMA module may result in
illegal memory accesses and corrupt the system. This must be addressed to
guarantee reliable DNN acceleration especially in mission-critical
applications. In addition, many DNN accelerators are implemented on FPGAs for
more intensive customization and convenient reconfiguration. While the
functionality of the DNN accelerators is mapped to the FPGA infrastructures
instead of the primitive logic gates, the simulation based approaches that
usually inject errors to the operations used in DNN processing do not apply to
the FPGA based DNN acceleration system. Because hardware faults in FPGAs
affect the configuration of the devices instead of the accelerator components
directly while the actual parts of the accelerators that are influenced
depends on the FPGA placing and routing.

To gain further insight of faults in DNN accelerators, we conduct the fault
analysis on running ARM-FPGA system where FPGA has a representative DNN
accelerator with 2D systolic array implemented along with hardware fault
injection modules and shares the DRAM with the ARM processors. Since the fault
distribution models are implemented with software on the ARM processors, they
are convenient to change for fault analysis using different models. Aside from
that we work with four different models for detecting the others.

Li, Shunlong et. al. [15] operated one-dimensional convolutional neural
network along the time axis to capture the temporal dependency. Shi, Chong et.
al [16]’s work focuses on the internal leakage fault diagnosis caused by the
wear based on intrinsic mode functions (IMFs).Unlike prior works that focused
on prediction accuracy loss analysis, we try to analyze the behaviours of the
DNN accelerators under hardware faults and investigate the system
functionality, fault coverage, input variation. Particularly, we study the
system stall that dramatically destroys the system functionality in detail and
present a simple yet efficient approach to alleviate the problem.

The following lines sum up our contributions to the litereature:

  * •

Our system focuses on fault analysis of neural network accelerators on ARM-
FPGAs. To make the analysis easier, it provides high-level interfaces for both
fault injection and output data comparison of neural network models.

  * •

On top of the fault analysis system, we mainly classify and analyze the
resulting behaviors of the DNN accelerators from system point of view. On top
of the prediction accuracy, we also study the fault coverage, system
functionality, input variation when the accelerators are exposed to persistent
faults.

  * •

With comprehensive experiments over representative models, we observe that
system stalls that can destroy the system functionality of DNN accelerators
cannot be ignored. And we further show that errors in data movement
instructions is a key reason for the system stalls, which can be addressed
with negligible hardware overhead.

The remainder of the paper can be described in the following paragraphs.
Section II sheds light on, the introduction to deep learning neural networks
and a typical DNN accelerators with 2D computing array. In Section III, we
describe the proposed framework on ARM-FPGAs and elaborate on the major
aspects that we will investigate from system point of view. Section IV details
the results and evaluation. In Section V, we give a brief review of prior
fault tolerant analysis and design of neural network accelerators. Finally, we
conclude this work in Section VI.

##  II Background

The main causes of unreliability are hardware flaws in DNN accelerators. The
impact of errors is inextricably linked to the microarchitecture of the DNN
accelerators. In this part, we take a standard DNN accelerator with a normal
2D processing element (PE) array as an example and develop its design to
assist understand and study the fault tolerance of the accelerators.

Figure 1 depicts a sample DNN accelerator. It uses output stationary data flow
to transfer computations like convolution to a 2D computing array. Every PE
completes all of the processes necessary to produce an output activation.
While each PE just has a single multiplier and accumulator, it progressively
accumulates all input activations in a filter window. While input activations
are organized in a batch and sent to one column of PE every cycle, each PE in
the column shares the same input data through broadcasting and it takes each
PE multiple cycles to complete the accumulation. During this period, more
batched input activations can be read and sent to the next column of PEs along
with the movement of the weights. Output activations flow from right to left
in column-wise. Eventually, each row of the PEs array produces a set of
sequential output activations in the same row of one output feature map on
y-axis. The architecture along with the compact data flow achieves high data
reuse under limited on-chip buffer bandwidth provision.

Figure 1: Typical DNN accelerator architecture

Both convolution layer and full connection layer can be mapped to the array
efficiently. While pooling and other non-linear activation functions such as
sigmoid will be performed right after the computing-intensive layers like
convolution layer in a module named XPE such that the data movement between
the two layers can be reduced. All the neural network operations can be mapped
to the accelerator. To associate the models with various layer combinations
and characteristics such as stride size and kernel size, we define a set of
instructions to generate appropriate control signals for different neural
network operations. Each neural network will be compiled to a series of
instructions and executed sequentially. In addition, neural network input
features and weights are usually larger than the on-chip buffers and PE array
size, so they must be tiled and the tiles need to be scheduled to obtain
efficient execution on the accelerator. To enable fine-grained optimizations,
each instruction only handles operations of a single tile. Thus, tiling is
performed during model compilation and it is transparent to the instructions.

Table I shows the instruction set of the neural network accelerator. It adopts
64-bit fixed length encoding and consists of four types of instructions
including parameter setup, calculation, data movement and control. The
parameter setup category defines the input/output feature size, kernel size,
Q-code, and DMA parameter. Calculation category includes different operations
in neural networks such as convolution, full connection, pooling, addition,
softmax, dot-accumulation and activation function etc. Data movement category
includes three instructions which move a block of data from DRAM to buffer,
buffer to DRAM and buffer to buffer respectively. Finally, control category
includes three instructions which are Jump, Stop and Nop. Jump instruction is
mainly used for repeated execution. Stop is used to terminate the execution of
the accelerator. Nop is used to resolve the data dependency between sequential
instructions.

TABLE I: Instruction Set of the DNN Accelerator Instruction Type |  Description
---|---
Parameter setup |  Setup parameters for the computing operations such as the input/output feature size, kernel size, Q-code, and DMA options
Calculation |  Performs various neural network operations such as convolution, full connection, pooling, addition, softmax, dot-accumulation and activation function
Data movement |  Move a block of data from buffer to buffer, buffer to DRAM and DRAM to buffer.
Control |  Control the execution of the accelerator such as Jump, Nop and Stop

The neural network accelerator architecture is general enough to support
various neural network models. In addition, it typically works along with a
general purposed processor and has an AXI slave port that allows configuration
and controlling from the attached processor. It assumes the input data, weight
and output data are stored in DRAM that can be accessed directly.

##  III DNN Acceleration Fault Analysis Platform and Fault Classification

###  III-A Fault Analysis Platform

We built a fault analysis platform as represented in Figure 2. It has an
unusual neural network accelerator based on FPGA.

Figure 2: Overview of the fault analysis system

In addition to this system, we needed a module as present in Figure 2. The
fault injection data path is marked with orange arrows. It is implemented on
both the ARM processor and FPGA. The FPGA errors are prevalent in the four
different memory types.

For FPGA configuration memory, we leverage Xilinx ICAP port [17], that allows
user logic to access configuration memory, to inject errors. We select the
frames and the number of bits for each bit error injection arbitrarily. It is
possible that the error may be present at any location of the FPGA
configuration memory. After finding the error location, we read the whole
frame out of the configuration memory [18], change the victim bit in the
frame, and write it back to the configuration memory [19].

We also produce an injection mask for block RAM and distributed memory as
represented in Figure 3. It essentially consists of a collection of address
and mask registers. The addresses in the registers reflect the positions of
the faults to be injected, while the associated masks keep track of the
precise error bits.

Figure 3: Error mask for fault injection to on-chip memory

Another important part of the fault analysis platform is the result collection
and comparison. We have the output of neural networks stored in the shared
DRAM, which can be easily accessed on the ARM processor and compared to the
pre-calculated golden reference. In addition, the computing results produced
in the intermediate layers can also be compared. The result comparison is
mainly performed on the ARM processor and can be easily utilized for further
analysis by high-level application designers.

###  III-B System fault classification

Prior fault analysis works usually focus on computing errors of the neural
networks and the incurred prediction accuracy loss. We argue that the
consequences of hardware faults on a DNN acceleration system vary and should
be classified into more categories. Table II shows the proposed
classification. From the perspective of a system, the consequences caused by
hardware faults roughly include system exception and accuracy degradation.
System exception indicates that the neural network execution behaves
abnormally, which may be stalled without returning or returns too fast or too
slow. Basically, we define them as system stall and abnormal runtime. For the
accuracy degradation, we further classified into two cases including system
stall and abnormal runtime.

TABLE II: Hardware Faults system exception | system stall
---|---
abnormal runtime
Accuracy degradation | $L_{0}$
…
$L_{k}$

We chose Neural Networks in four different application scenarios and try to
analyze the differences of error tolerance in different application scenarios
and networks. The four network applications include ResNet network for image
classification, YOLO system for target detection, LSTM network for voice
classification, and DCGAN network for image generation. We will evaluate their
fault tolerance from accuracy and output consistency.

Neural Network accelerators injected with hardware errors may produce
unexpected conditions. We define the system halt situation, which refers to a
serious error in the system, or working improperly. such as unable to read and
write registers, timeout, abnormal short runtime, etc. When the system halts,
we need to reset the FPGA and restart the system. System halt situations are
considered the result of errors in the evaluation of network accuracy and are
listed separately in the output consistency.

Network accuracy refers to the overall accuracy of the network when performing
corresponding tasks, such as the accuracy of 20,000 image recognition. When
there are errors in the operation, the accuracy of the network will show a
downward trend. For classification networks including ResNet and LSTM, top-5
accuracy is used to evaluate their accuracy, and for the YOLO system, mAP is
adopted.

Output consistency is the difference between the result of running with errors
injected and the result of normal running. We ran the corresponding data set
when no errors are injected into each network at first, and defined the
results as standard output. The results of Neural Networks prediction injected
with errors are divided into two categories: result with deviation and result
match. Result with deviation refers to system works properly with output
differently from standard output. Result match means that the system works
properly and the outputs are still standard output.

For the result with deviation case, we define its deviation quantitatively and
further subdivide the result. Due to the different application functions of
each network, the evaluation criteria of deviation are also different. For
YOLO system, the result is the target detection bounding box, and when the
detection result does not match the standard output in object type, it is
defined as detection result error. When the result target type is consistent,
the error is defined as the intersection area of the error output and the
standard output divided by the area of the union. Target types not match for
one single level, the two do not overlap with each other for one level, and
then each 20% is divided into one level. For ResNet and LSTM, the outputs are
top-5 labels. When the error output is not completely consistent with the
standard output, the number of elements in the intersection of the two is
taken, and divide the levels refer to the number. For DCGAN, we used the
universal SSIM standard, and divided it into six levels according to the
actual visual effects: 0 10%, almost impossible to recognize; 10 20%, barely
visible; 20 50%, with large deformation or distortion; 50 80% with partial
deformation or distortion; 80 90%, small deformation or distortion can be
seen; 90 100%, almost no deformation or distortion is visible.

##  IV Experiment

We seek to understand the influence of persistent errors on FPGA-based neural
network acceleration system. Particularly, we try to analyze the influence
from a system point of view and figure out the underlying reasons for severe
system problems such as system stall and dramatic prediction accuracy loss.

###  IV-A Device and Environment

Xilinx Zynq-7000 SoC ZC706 Evaluation Board will be used in the experiments’
hardware implementation. It has appropriate hardware resources and is easy to
develop and use. The hardware design and Bitstream file compilation were
completed using the upper computer with Intel Core i7-6700 processor and 2x8GB
DDR4 2400MHz memory. The system environment used was Ubuntu 16.04 LTS version,
and Xilinx Vivado Design Suite and Xilinx Software Development Kits version
2017.4.

The hardware resource utilization of the error analysis platform
implementation on ZC706 is shown in Table III.

The four models cover a broad range of applications. Yolo represent a typical
neural network model for object detection [20], Resent is a widely adopted
neural network model for classification [21], LSTM is the mostly used neural
network model for audio classification tasks [22], and DCGAN stands for a
typical neural network model for generative tasks [23].

Despite the widespread adoption of deep learning neural network on various
applications, it is particularly successful in four categories of tasks
including object detection, object classification, voice recognition and style
transfer. Among a great number of neural network models, Yolo, Resnet, LSTM
and DCGAN are four typical neural networks that are comprehensively explored
to handle the four computing tasks respectively.

TABLE III: Utilization Report of Fault Analysis Platform Resource | Utilization | Available | Utilization Percentage
---|---|---|---
LUT | 122618 | 218600 | 56.09%
LUTRAM | 185 | 70400 | 0.26%
FF | 84641 | 437200 | 19.36%
BRAM | 203 | 545 | 37.25%
DSP | 297 | 900 | 33%
MMCM | 1 | 8 | 12.5%

###  IV-B Overview

We hope to explore the possible consequences of errors, the fault-tolerant
ability of different applications and the influence of different error
locations of accelerators, which can provide meaningful references for the
subsequent network optimization and the fault-tolerant design of accelerators.

We conducted five sets of experiments, which explained the influence of single
error on different networks, the difference of fault tolerance ability of
different applications, the error classification of different applications,
the influence of different accelerator units and whether the input data
affected the error representation.

###  IV-C Error Consequences and Coverage

First, we conducted single-bit random error injection experiments for
different applications, and conducted 20,000 runs for each application, to
analyze the proportion of single-bit random error shielded in the system and
the possible influence of single-bit random error on the accelerator system.
Figure 4 shows the percentage of application errors caused by a single-bit
random hardware error. In the experiment, we found that more than 90% of the
errors were masked by software or hardware. However, while most errors are
masked without impact to the operation, a single-bit error can lead to serious
exceptions, including system halt, serious deviation in results, and so on. In
addition, LSTM network has a better fault tolerance performance than other
three networks, and the proportion of the influence caused by single error of
the other three networks is about 5 7 times that of LSTM. According to our
analysis, this is because LSTM network is smaller than other networks and uses
less storage and computing resources.

Figure 4: Application error rate caused by single-bit random hardware error

###  IV-D Effect of Error Number

We completed a single run with errors injected number of powers of 2, with
20,000 runs for each network. In general, the proportion of system halt and
result with deviation increases with the number of errors, and the accuracy of
application decreases. Figure 6 show the proportion of system halt and result
deviation, and Figure 5 shows the accuracy of each network. We found that the
influence of multiple errors on the accelerator was obvious and far beyond our
expectation. The Neural Network accelerator is still vulnerable to errors. By
the time we injected 16 errors in a single run, the system halt rate was more
than 1%, means it took more time to restore the system than to run it. From
the perspective of network accuracy, take YOLO system as an example, its mAP
decreases by 8.95% when it runs with 16 errors, means the application function
of the system is also seriously affected.

The result with deviation proportion of different networks appears differently
with the increase of error number. When 16 errors are injected, LSTM network
still has a small result deviation proportion due to its small network
structure. The proportion of YOLO system is very high, with about 70% of the
results showing errors. ResNet is relatively low, with only about 35% result
deviation. DCGAN network is in between, and about 50% of the results have
numerical errors. We believe that the result deviation may be related to the
structure of the output. The output of YOLO system contains more information
such as object type, location and size of bounding box, etc., and the output
of ResNet is simple sorting, while the simple output is obviously less
susceptible to errors.

Figure 5: Network accuracy versus error number

(a) System halt (b) Result with deviation

Figure 6: Abnormal situation proportion versus error number

###  IV-E Details of Result with Deviation

(a) YOLO (b) ResNet (c) LSTM (d) DCGAN

Figure 7: Distribution of result with deviation situations

(a) YOLO (b) ResNet (c) LSTM (d) DCGAN

Figure 8: Proportion of different error location

The results with errors are further classified and analyzed. In general, most
of the errors are small, with a certain proportion of serious errors and
relatively few moderate ones. We believe that most of the errors do not belong
to the errors with global influence, but only affect one or several
calculations. For example, a single value in the convolution kernel changes.
Alternatively, a portion of the error is masked by subsequent calculations
such as the max pooling layer. Some errors may have an impact on the control
path or the reusable module, resulting in the accumulation of errors
throughout the calculation; Or errors that cause serious deviations in the
data, such as sign bit upset, can have serious consequences. Figure 7 shows
the details of result distribution of result with deviation situations.

Specific to each network, about 70% of the YOLO system’s errors belong to the
level of bounding box overlap ratio more than 80%. The total of object type
errors and non-overlapping boxes is about 20%. The remaining 10% or so is
moderate errors. We think this is caused by the implementation of YOLO. YOLO
divides the input images into a series of grid cells, and each grid cell is
only responsible for one kind of object. Moreover, there is a binary judgment
Pr(Object) whether there is a target or not in the confidence degree, which
leads to more object type error cases. The bounding boxes that are given after
object recognition based on relative wide and high are less sensitive.

More than 70% of the errors in the ResNet and LSTM classification networks
were small impact results of matching four or five items. We think this is
because the output is SoftMax layer, which is less affected by the error, and
the error is easy to be hidden when sorting, so the result only shows a small
alter. However, with the increase of the number of errors, the proportion of
serious errors of no matching item in ResNet results increased with the
increase of the number of errors and exceeded 10%. Serious errors cannot be
hidden, and the networks fault tolerance for multiple errors is relatively
limited.

The result of DCGAN is about 90% of the results are small deviation level of
more than 90% of SSIM. Only a few of them have a large deviation. Combined
with the image, in 90% of the small deviation results compared with the
standard output, basically no difference can be directly seen, and only a few
large deviation results have serious distortion of the image. Considering that
the output is picture information, these small deviations can be ignored
without affecting the visual effect, we believe that DCGAN has relatively
strong error tolerance.

###  IV-F Effect of Error Location

In this period, we conducted the experiment results of different error
locations. We injected a single-bit random error into a designated location,
and conducted multiple experiments to observe the performance of the network.
By analyzing the influence of error in different locations on the accelerator,
it can provide specific methods for the subsequent fault-tolerant design.

In general, the system is less affected by configuration memory errors,
considering that the error injection of configuration memory is global, whose
errors may not affect the system. Errors in configuration memory can cause
system halt or result with deviation. Errors in BRAMs used for instruction
buffer can cause system halt or result with deviation, while BRAMs used in
other type buffers can only cause numerical deviations.

We focus on the analysis of the system halt situations caused by error in
instruction buffer, which take about 20% part of the system halt situations.
Compared with the instruction before and after upset, the system halts caused
by instruction errors includes three situations: instruction type changes,
wrong instruction not defined, and the parameter in the operation is abnormal.
Different instruction types of the original instruction are considered. Table
IV and Table V shows the detail of instruction error. About 50% of the system
halt situations are caused by the error of DMA instructions. The abnormal
access address or boundary violation caused by the abnormal parameters of DMA
instruction will lead to the system halt. About 30% are caused by AGU
instruction errors, which result in abnormal in-chip control flow.

TABLE IV: Instruction change Situation | Percentage
---|---
Error instruction undefined | 2.48%
Instruction type change | 7.45%
Abnormal parameter | 90.06%
TABLE V: Original instruction type type | Percentage
---|---
DMA | 55.90%
AGU | 32.92%
others | 11.18%

The proportions of errors in instruction buffer led to system halt in ResNet,
YOLO and LSTM, reached 1.15%, 2.55% and 4.25% respectively, seriously
affecting the proper application of the system. For result with deviation
case, in the experiment of YOLO, about half of the result deviation situations
caused by instruction buffer errors were serious object type errors. In the
experiment of ResNet, a large number of mismatches were also caused. DCGAN
network is limited by instruction errors, because the instruction sequence
length of this network is very short, only 2% of the instruction buffer is
used, while the instruction sequence of other networks uses instruction buffer
of 50%~80%. Combined with the result with deviation and system halt case, we
propose that instruction buffer needs to be strengthened in the fault-tolerant
design.

Errors in the buffers used in data-flow, such as weights, data, and bias
buffer, do not cause system halt, only may cause result deviations. In
general, data and weights are more sensitive than bias. Taking YOLO system as
an example, the proportion of result deviation caused by single error in
weight, data and bias buffer is 48.75%, 56.95% and 6.35% respectively.
Horizontal comparison shows that each network has different sensitivity to
different buffer errors, as shown in figure .

###  IV-G Input-Related Error

In the above experiments, we used different input data for experiments, and we
verified the relationship between errors and input data in this set of
experiments. We test different input data using the same error. Whether a
hardware error causes an application error exists in two ways, depending on
the input data or not. Errors unrelated to input data, fixed to cause system
halt or result deviation, or be masked, that is, different input data will not
influence the classification of the result. The other part of the errors is
input data related, which can be shown as replacing different input data,
result match situation and result with deviation situation both appear. In
other words, different input data has different sensitivity to an input-
related error. We believe that this is caused by structures which error
affected. Input-unrelated errors may affect the control-flow of the system,
the bus, etc. These structures are used in every operation, and the errors
will not be masked by subsequent calculations. Input-related errors may affect
the relevant data in the data-flow, and some errors may be masked in the
subsequent calculation. Due to the large number of experiments, we only
observed this phenomenon without further study on its proportion and
distribution.

##  V Related work

Deep learning accelerators that ensure both high performance and energy
efficiency of neural network are increasingly adopted in various computing
devices including IoT devices, mobile phones and cloud. While the accelerators
fabricated with the latest semiconductor technology is susceptible to the
manufacturing defects and abnormal processes due to the small transistor
feature size, the number of hardware faults grows accordingly. The faults may
further lead to computing errors, dramatic prediction accuracy loss and even
system stall when the faults are not handled appropriately. For some of the
mission-critical applications, the prediction mistakes may even cause
catastrophic failures.

To gain insight of fault tolerance of neural network accelerators, the authors
in [13] investigated the influence of data types, values, data reuses, and
types of layers on the resilience of DNN accelerators through experiments on a
DNN simulator. error resilience of xxx with simulation. Different from xxx,
the works in xxx mainly focus on the difference of fault tolerance using
different data types. Some of the researchers try The reliability of the
neural network accelerators While neural networks with large amount of
redundant computing is considered to resilient, Reliability of neural network
accelerators become critical to resilient neural network execution. 1)
Reliability espeically FPGA based reliability problem 2) The prevalence of
deep learning neural networks provokes the development of convolutional neural
network (CNN) accelerators for both higher performance and energy efficiency.
existing fault-tolerant CNN accelerator works,

Data type analysis with simulation, error propagation analysis Retraining to
improve fault tolerance and tolerate the computing errors (Change the neural
network model) Computing array based fault model Relax the design constraints
and have the accelerator obtain advantageous design trade-off between
precision and performance.DAC’18 work Basically, the analysis focuses on the
computing of the neural network. Hardware structure is not discussed in
detail. Simulation is the major approach. lack of system analysis on a CPU-CNN
accelerator. FPGA based analysis is not covered. Prior FPGA based reliability
such as soft processors etc. It is not quite relevant. Neural network inherent
fault tolerance. Error analysis on a running system remains not explored.

FPGA is a widely used hardware platform, and there are many designs and
methods for error injection on FPGAs [24][25] [26][27]. FPGA-based (also known
as emulation-based) error analysis is widely used in soft processor
sensitivity analysis and other scenarios. The FPGA-based fault injection
techniques have good controllability, observability and ideal speed.

FPGA-based fault injection techniques can be divided into two categories:
reconfiguration-based techniques and instrument-based techniques. The
reconfiguration-based techniques rely on the internal mechanism of FPGAs.
These techniques use complete or partial reconfiguration to change the
configuration bit of the FPGA device to apply the target fault model to the
desired fault location. The reconfiguration process is a speed bottleneck for
reconfiguration-based techniques. In instrument-based technology, fault
injection modules called saboteur are added to each fault point, and fault is
injected by activating the saboteurs. Instrument-based techniques have higher
speed than reconfiguration-based techniques, while implementing the saboteurs
increase the area of the circuit. We shall design the saboteurs ourselves for
instrument-based techniques, and insert them into the design under test.

##  VI Conclusion

With the wide application of neural network in many scenarios with high safety
requirements like autonomous driving, the fault-tolerant performance of neural
network accelerators has received more attention. In this paper, we designed a
fault emulation and analysis system for neural network accelerator based on
SRAM-based FPGA, and completed a series of experiments on this basis. We find
that the fault tolerance of neural network accelerator is poor, and analyze
the misrepresentation of different numbers, different network applications and
different error locations. These analyses provide a reference for the
following fault-tolerant design.

## References

  * [1] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [2] R. Collobert and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in _Proceedings of the 25th International Conference on Machine Learning_ , ser. ICML ’08. New York, NY, USA: ACM, 2008, pp. 160–167. [Online]. Available: http://doi.acm.org/10.1145/1390156.1390177
  * [3] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in _The IEEE International Conference on Computer Vision (ICCV)_ , December 2015.
  * [4] T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, “Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning,” in _ACM Sigplan Notices_ , vol. 49, no. 4. ACM, 2014, pp. 269–284.
  * [5] Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun _et al._ , “Dadiannao: A machine-learning supercomputer,” in _Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture_. IEEE Computer Society, 2014, pp. 609–622.
  * [6] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” in _ACM SIGARCH Computer Architecture News_ , vol. 44, no. 3. IEEE Press, 2016, pp. 367–379.
  * [7] C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, “Optimizing fpga-based accelerator design for deep convolutional neural networks,” in _Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , ser. FPGA ’15. New York, NY, USA: ACM, 2015, pp. 161–170. [Online]. Available: http://doi.acm.org/10.1145/2684746.2689060
  * [8] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, “Eie: Efficient inference engine on compressed deep neural network,” in _2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA)_ , June 2016, pp. 243–254.
  * [9] P. W. Protzel, D. L. Palumbo, and M. K. Arras, “Performance and fault-tolerance of neural networks for optimization,” _IEEE Transactions on Neural Networks_ , vol. 4, no. 4, pp. 600–614, July 1993.
  * [10] B. Reagen, U. Gupta, L. Pentecost, P. Whatmough, S. K. Lee, N. Mulholland, D. Brooks, and G.-Y. Wei, “Ares: A framework for quantifying the resilience of deep neural networks,” pp. 1–6, 2018.
  * [11] F. Kausar and P. Aishwarya, “Artificial neural network: Framework for fault tolerance and future,” in _2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT)_ , March 2016, pp. 648–651.
  * [12] G. Li, K. Pattabiraman, and N. Debardeleben, “Tensorfi: A configurable fault injector for tensorflow applications,” in _2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)_ , 2018.
  * [13] G. Li, S. K. S. Hari, M. Sullivan, T. Tsai, K. Pattabiraman, J. Emer, and S. W. Keckler, “Understanding error propagation in deep learning neural network (DNN) accelerators and applications,” pp. 1–12, 2017.
  * [14] B. Salami, O. Unsal, and A. Cristal, “On the Resilience of RTL NN Accelerators: Fault Characterization and Mitigation,” 2018. [Online]. Available: http://arxiv.org/abs/1806.09679
  * [15] S. Li, J. Niu, and Z. Li, “Novelty detection of cable-stayed bridges based on cable force correlation exploration using spatiotemporal graph convolutional networks,” _Structural Health Monitoring_ , p. 1475921720988666, 2021.
  * [16] C. Shi, Y. Ren, H. Tang, and L. R. Mupfukirei, “A fault diagnosis method for an electro-hydraulic directional valve based on intrinsic mode functions and weighted densely connected convolutional networks,” _Measurement Science and Technology_ , vol. 32, no. 8, p. 084015, 2021.
  * [17] Xilinx Inc., “Vivado design suite 7 series fpga and zynq-7000 soc libraries guide,” http://www.xilinx.com/support/documentation/sw_manuals/xilinx2017_4/ug953-vivado-7series-libraries.pdf, Dec 2018, UG953(v2017.4).
  * [18] ——, “Axi hwicap v3.0 logicore ip product guide,” http://www.xilinx.com/support/documentation/ip_documentation/axi_hwicap/v3_0/pg134-axi-hwicap.pdf, Oct 2016, PG134.
  * [19] ——, “7 series fpgas configuration user guide,” http://www.xilinx.com/support/documentation/user_guides/ug470_7Series_Config.pdf, Aug 2018, UG470 (v1.13.1).
  * [20] J. Redmon and A. Farhadi, “Yolo9000: Better, faster, stronger,” _arXiv preprint arXiv:1612.08242_ , 2016.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [22] H. Sak, A. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in _Fifteenth annual conference of the international speech communication association_ , 2014\.
  * [23] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [24] M. Ebrahimi, A. Mohammadi, A. Ejlali, and S. G. Miremadi, “A fast, flexible, and easy-to-develop fpga-based fault injection technique,” _Microelectronics Reliability_ , vol. 54, no. 5, pp. 1000–1008, 2014.
  * [25] C. Lopez-Ongil, L. Entrena, M. Garcia-Valderas, M. Portela, and F. Munoz, “A unified environment for fault injection at any design level based on emulation,” _IEEE Transactions on Nuclear Science_ , vol. 54, no. 4, pp. 946–950, 2007.
  * [26] N. A. Harward, M. R. Gardiner, L. W. Hsiao, and M. J. Wirthlin, “Estimating soft processor soft error sensitivity through fault injection,” in _IEEE International Symposium on Field-programmable Custom Computing Machines_ , 2015.
  * [27] J. Tarrillo, J. Tonfat, L. Tambara, F. L. Kastensmidt, and R. Reis, “Multiple fault injection platform for sram-based fpga based on ground-level radiation experiments,” in _Test Symposium_ , 2015.