# TinyCL: An Efficient Hardware Architecture for Continual Learning on
Autonomous Systems

Eugenio Ressa1, Alberto Marchisio2, Maurizio Martina1, Guido Masera1, and
Muhammad Shafique2 1Department of Electrical, Electronics and
Telecommunication Engineering, Politecnico di Torino, Turin, Italy2eBrain Lab,
Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE
<EMAIL_ADDRESS>alberto.marchisio<EMAIL_ADDRESS>maurizio.martina<EMAIL_ADDRESS>
###### Abstract.

The Continuous Learning (CL) paradigm consists of continuously evolving the
parameters of the Deep Neural Network (DNN) model to progressively learn to
perform new tasks without reducing the performance on previous tasks, i.e.,
avoiding the so-called catastrophic forgetting. However, the DNN parameter
update in CL-based autonomous systems is extremely resource-hungry. The
existing DNN accelerators cannot be directly employed in CL because they only
support the execution of the forward propagation. Only a few prior
architectures execute the backpropagation and weight update, but they lack the
control and management for CL. Towards this, we design a hardware
architecture, TinyCL, to perform CL on resource-constrained autonomous
systems. It consists of a processing unit that executes both forward and
backward propagation, and a control unit that manages memory-based CL
workload. To minimize the memory accesses, the sliding window of the
convolutional layer moves in a snake-like fashion. Moreover, the Multiply-and-
Accumulate units can be reconfigured at runtime to execute different
operations. As per our knowledge, our proposed TinyCL represents the first
hardware accelerator that executes CL on autonomous systems. We synthesize the
complete TinyCL architecture in a 65 nm CMOS technology node with the
conventional ASIC design flow. It executes 1 epoch of training on a Conv +
ReLU + Dense model on the CIFAR10 dataset in 1.76 s, while 1 training epoch of
the same model using an Nvidia Tesla P100 GPU takes 103 s, thus achieving a
$58\times$ speedup, consuming 86 mW in a 4.74 mm2 die.

Continual Learning, Deep Neural Networks, Hardware Architecture, ASIC.

##  1\. Introduction

In recent years, Deep Neural Networks (DNNs) have been deployed in several
applications, like computer vision, finance, healthcare, and robotics (Chai et
al., 2021; Chandani and Dubey, 2022; An et al., 2023; Soori et al., 2023). A
common practice is to train a DNN on the desired task using the training set,
and then deploy the trained model on the target for inference (Capra et al.,
2020a). In this way, it is possible to conduct DNN training on large data
centers (e.g., using high-end GPUs), and then conduct resource-constrained
optimizations (e.g., compression) to deploy DNN inference on autonomous
systems (Dhilleswararao et al., 2022). However, this practice limits the
dynamic capabilities of DNNs that cannot adapt to new tasks or a distribution
variation of the input data within the same task. In this regard, the
Continual Learning (CL) paradigm enables the dynamic change of DNN parameters
to evolve and learn new tasks (or new classes) (Wang et al., 2023). The main
goal of CL-based algorithms is to avoid Catastrophic Forgetting (CF), i.e.,
the DNN should maintain the knowledge of how to perform the previous tasks
while learning new tasks (McCloskey and Cohen, 1989).

###  1.1. Target Research Problem

Since they require the execution of the backpropagation and parameters’
update, CL algorithms impose more demanding computation and memory resources
than traditional systems that conduct only inference (Chen et al., 2020).
Hence, it is extremely important to execute CL algorithms in an efficient
manner to be able to deploy them onto resource-constrained autonomous systems.

Most of the existing architectures and optimizations for DNNs focus only on
optimizing the inference process (Capra et al., 2020b). Therefore, they cannot
execute the backward operations required by CL algorithms. Some architectures
(Lee and Yoo, 2021) that have been proposed to accelerate the training can be
adopted to conduct backpropagation computations, but they do not support the
execution of CF-avoiding policies.

###  1.2. Novel Contributions

To overcome these limitations, we propose TinyCL, a hardware architecture that
can efficiently execute CL operations on autonomous systems. Our architecture
reuses the same processing units for computing the forward and backward
computations, and a specialized control unit dictates the data flow based on
the CL policy. In a nutshell, our contributions are (see Figure 1):

  * •

RTL Design of the complete TinyCL architecture, in which multiple Processing
Units execute the computation in parallel and the convolutional sliding window
is designed following a snake-like pattern. (Section 3)

  * •

Synthesis of the TinyCL architecture using the conventional ASIC flow for a 65
nm CMOS technology node. (Section 4)

  * •

Compared to its equivalent software-level implementation on an Nvidia Tesla
P100 GPU, the TinyCL architecture achieves $35\times$ speedup; compared to
other DNN training accelerators in the literature, the TinyCL architecture
achieves lower latency, power consumption, and area, thus making it suitable
for being adopted on resource-constrained autonomous systems. (Section 4.3)

Figure 1. Overview of our novel contributions in this work.

##  2\. Background and Related Work

###  2.1. Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are a branch of DNNs used in image
recognition tasks. Convolutional and dense layers are the basis blocks of
every CNN. In a convolutional layer, a multi-channel input feature $V$ is
convoluted with a 4D kernel $K$ to produce a multi-channel output $Z$ (see
Equation 1).

(1) $\begin{gathered}Z(i,j,k)=c(K,V,s)_{i,j,k}=\\\ \sum_{l,m,n}[V(l,(j-1)\cdot
s+m,(k-1)\cdot s+n)\cdot K(i,l,m,n)]\end{gathered}$

Applying the Stochastic Gradient Descent (SGD), the computation to propagate
the gradient across layers is a convolution between the previous kernel $K$
and the gradient $G$ propagated from the previous layer, as shown in Equation
2, while the gradient of the kernel is computed through Equation 3.

(2) $\begin{gathered}h(K,G,s)_{i,j,k}=\frac{\partial}{\partial
V_{i,j,k}}J(V,K)=\\\ \sum_{l,m\ s.t.(l-1)\cdot s+m=j}\left(\sum_{n,p\
s.t.(n-1)\cdot s+p=k}\left(\sum_{q}G_{q,l,n}\cdot
K_{q,i,m,p}\right)\right)\end{gathered}$

(3) $\begin{gathered}g(G,V,s)_{i,j,k,l}=\frac{\partial}{\partial
K_{i,j,k}}J(V,K)=\\\ \sum_{m,n}G_{i,m,n}\cdot V_{j,(m-1)\cdot s+k,(n-1)\cdot
s+l}\end{gathered}$

The Dense layer computes a matrix multiplication between a 1D row matrix $I$
with a 2D weight matrix $W$ (see Equation 4).

(4) $y_{n}=\sum_{i=0}^{m}I_{i}\cdot W_{i,n}$

Applying SGD, we can compute the gradient propagation that is a matrix
moltiplication between the input gradient $dY$ propagated from the previous
layer and the transpose of the weight (see Equation 5).

(5) $dX_{i}=\sum_{n=0}^{N}dY_{n}\cdot W^{T}_{n,i}$

Then, the gradient of the weights is computed as in Equation 6.

(6) $dW_{i,n}=I_{i}\cdot dY_{n}$

###  2.2. Continual Learning Algorithms

The goal of CL, also known as incremental learning, lifelong learning, or
sequential learning, is to gradually learn from different data streams and
extend the acquired knowledge (Thrun and Mitchell, 1995). The data streams can
be associated with different tasks that the CL system can perform.
Incrementally learning all the tasks is an NP-hard problem (Knoblauch et al.,
2020). Moreover, the major issue of CL is represented by CF, i.e., the ability
of the system to perform previously learned tasks degrades over time when new
tasks are added (McCloskey and Cohen, 1989). In the literature, several CL
methods have been proposed to mitigate CF. CL algorithms can be categorized as
regularization-based methods, memory-based methods, and dynamic approaches.

Regularization-based methods apply constraints in the weight update phase to
mitigate CF. Weight regularization methods such as Elastic Weight
Consolidation (EWC) (Kirkpatrick et al., 2016) impose a quadratic penalty to
selectively regularize the parameters based on their importance to perform the
previous tasks, calculated through the Fisher information matrix. Function
regularization methods like the Learning without Forgetting (LwF) (Li and
Hoiem, 2018) employ knowledge distillation to learn the training samples of
the new tasks while preserving the knowledge of the previous tasks.

Memory-based approaches, also known as replay-based methods, retrain or
finetune the DNN jointly using samples from previous tasks and samples from
new tasks. The Gradient Episodic Memory (GEM) method (Lopez-Paz and Ranzato,
2017) constrains the parameter update such that the training loss of each
individual previous task does not increase. Its variant, called A-GEM
(Chaudhry et al., 2019), ensures that the average training loss for all
previous tasks does not increase. The Incremental Classifier and
Representation Learning (iCaRL) (Rebuffi et al., 2017) method stores a subset
of training samples for each task and jointly minimizes the training loss for
new tasks and the distillation loss for the previously learned tasks.
Experience Replay (ER) (Hayes et al., 2019) combines training with samples of
the new tasks and old samples that are stored in a replay memory. The
Maximally Interfered Retrieval (MIR) method (Aljundi et al., 2019a) selects
the samples from the old tasks that would have the largest impact on the
forgetting property. The Gradient-based Sample Selection (GSS) strategy
(Aljundi et al., 2019b) maximizes the gradient diversity of the stored sample
subset. Greedy Sampler and Dumb Learner (GDumb) approach (Prabhu et al., 2020)
greedily stores training samples in the memory buffer to maintain a balanced
class distribution. The replay data can also be generated at runtime by using
the Deep Generative Replay (DGR) method (Shin et al., 2017), where a generator
creates synthetic samples that contain previous tasks knowledge.

Dynamic approaches dynamically increase the DNN architecture to learn features
of new tasks. The Continual Neural Dirichlet Process Mixture (CN-DPM) method
(Lee et al., 2020) retains the knowledge of the previous tasks by building a
mixture of experts where a new model is trained for a new task, while the
existing models for the previous tasks are not modified. The Progressive
Segmented Training (PST) method (Du et al., 2019) focuses on a single network
and, when training on a new task, it divides the parameters into two groups
according to their importance to perform that task. The group of important
parameters is frozen to preserve the current knowledge, while the other group
is saved and can be updated when learning future tasks.

Our architecture supports memory-based approaches due to the simplicity of
their hardware implementation, but it can be easily extended to execute other
CL algorithms.

###  2.3. DNN Training Accelerators

Various hardware architectures implemented in ASIC or FPGA for accelerating
DNN training have been proposed in recent years. The work in (Venkataramani et
al., 2017) is composed of heterogeneous processing tiles to efficiently
execute different operations with diverse computational characteristics. The
DeepTrain architecture (Kim et al., 2018) deploys heterogeneous programmable
data flows to achieve data reuse during different training operations. The
Gist architecture (Jain et al., 2018) utilizes layer-specific encoding schemes
to exploit redundancy in DNN training by storing the feature maps computed
during the forward pass and reusing them in the backward pass. The SIGMA
accelerator (Qin et al., 2020) supports irregular sparse workload. To
efficiently handle sparsity, the Procrustes accelerator (Yang et al., 2020)
employs a dense tensor dimension for performing arithmetic operations that
involve sparse tensors. The LNPU architecture (Lee et al., 2019) implements
fine-grained mixed precision to perform training. The HNPU architecture (Han
et al., 2021) supports low-precision training by dynamically configuring the
fixed-point representation. The FlexBlock architecture (Noh et al., 2023)
supports multiple block floating-point precisions. The ETA architecture (Lu et
al., 2023) performs training based on the proposed piecewise integer format.
The work in (Kim et al., 2019) trains the DNN through a two-step process that
consists of sample collection and policy update for continuous control of the
behavior. The above-mentioned architectures allow the execution of standard
training in an efficient manner. However, the execution of CL algorithms
requires additional abstraction layers to correctly manage the workload.

###  2.4. Optimizations for Continual Learning at the Edge

Recent works have proposed specialized optimization to improve the efficiency
of CL algorithms on resource-constrained devices. Huai et al. (Huai et al.,
2023) designed a method for improving the efficiency during the re-training
phase of latency-constrained high-end edge systems. The SparCL framework (Wang
et al., 2022) enables efficient continual learning through weight and gradient
sparsity. The SIESTA algorithm (Harun et al., 2023) proposed to alternate
awake cycles with sleep cycles to consolidate the memory. The Miro methodology
(Ma et al., 2023) designs the memory hierarchy to leverage the tradeoff
between energy and accuracy.

While these works demonstrated potential to improve the efficiency in some
parts of the process, they do not consider the complete system implemented on
a specialized accelerator, as we do in this paper.

##  3\. TinyCL Architecture Design

###  3.1. Data quantization

Since high-precision data is not necessary, a 16-bit fixed-point (4-bit
integer + 12-bit fractional data) format is used. Accordingly, the work in
(Gaur et al., 2020) suggests that value clipping is useful to increase
accuracy in DNNs where batch normalization is not applied.

###  3.2. Top-Level Architecture

Our system is composed of 4 groups of data memory ( GDumb memory, Gradient
memory, Kernel memory and Partial Feature memory), a control unit and a
processing unit, as depicted in Figure 2.

Figure 2. Top-level view of the TinyCL architecture.

###  3.3. Processing Unit

Our processing unit, depicted in Figure 3, includes 9 parallel multiply and
accumulate (MAC) blocks, each of them executing an 8-operand multiplication
and addition in parallel. 3 manager units (gradient, kernel, and feature) are
designed to drive the data flow. 3 address managers (gradient, kernel, and
feature) compute the addresses needed to the forward, gradient propagation,
and kernel derivative.

Figure 3. Overview of the processing unit architecture.

###  3.4. MAC

Our MAC is composed of 8 multipliers and 8 adders, as shown in Figure 4. The
multipliers are configured in a parallel fashion, while the adders can be
configured in two ways: multi-operand mode or multi-adder mode.

Figure 4. Overview of the multiply-and-accumulate (MAC) architecture.

  * •

Multi-Adder mode: During kernel gradient calculation, the adders sum the 8
multiplication results with 8 input values to obtain the partial sum outputs.
In the weight gradient calculation, as we compute in parallel 8 channels at a
time, each time we have to multiply 8 channels of one input feature with one
channel of the input gradient. The results of these 8 multiplications will be
summed with the previous multiplication done.

  * •

Multi-Operand mode: During forward and gradient propagation, we compute 8
input channels at a time. In this case, since the output is the sum of these 8
input channels (3D-convolution), we configure the 7 adders to operate as an
adder-tree.

To reduce the loss of information, the results of the 16-bit multiplications
are kept in full precision and propagated to the 32-bit adders. After the
addition, the output is reduced to 16 bit, rounded to nearest.

###  3.5. Memory

A typical DNN accelerator that performs only inference needs to save only its
parameters. However, a CL accelerator needs to save other data.

  * •

Training Data Memory: Memory-based CL methods save old training samples when
performing a new training cycle. For this reason, such samples must be saved
in a memory that can be updated by replacing some samples of old classes with
more samples of new classes. The cardinality of each training sample set must
be equal, thus we avoid class imbalance problems.

  * •

Partial Feature Memory: During inference, each layer applies a function $f(k)$
to the input feature and computes an output feature. During back propagation,
the derivative of the output with respect to the weights is a function of the
input feature used during forward. For this reason, for each layer with
weights, we have to save the input feature during inference, to be used
backward.

  * •

Kernel Memory: Obviously, kernel values must be stored for forward and
backward processes.

  * •

Gradient Memories: To temporarily save the gradients to be used between two
computations, a couple of memories must be used. The memories shall be 2
because 1 would not be enough. In a multi-channel convolution operation, the
feature we are calculating will overwrite a feature we will need in further
calculus.

To increase throughput and decrease stalls, we design the memories with a port
width of $128$ bits, to read 8 features at a time. To further increase the
throughput, the SRAM is organized according to the channel. For example, if we
compute convolution with an input feature of $32\times 32$ with $8$ channels
and we output a feature of $32\times 32$ of $8$ channels, we will have $8$
blocks for partial features of $32\times 32\times 16$ bits, $64$ blocks of
$3\times 3\times 16$ bits to save the kernels and $16$ blocks of $32\times
32\times 16$ bits memory for gradient propagation. Moreover, dedicated buffers
are present to prefetch data from memory.

###  3.6. Control Unit and Data Flow

Our control unit manages the multi-layer computation, passing the actual
matrix input and output sizes to the PU. For each layer, the Control Unit (CU)
manages the data flow from memories to MACs for executing 6 different
computations:

  * •

Convolution - forward computation

  * •

Convolution kernel - gradient computation

  * •

Convolution - gradient propagation

  * •

Dense layer - forward

  * •

Dense layer - gradient propagation

  * •

Dense layer - wight derivative

In each of these computations, data are fetched and loaded into buffers, so
that at each clock cycle, the respective features are directly processed by
the MACs of the PU. In this way, we are able to maintain a high hardware
utilization.

####  3.6.1. Forward Computation of the Convolutional Layer

In the forward operation, each multi-channel input feature must be
3D-convoluted with a 4D kernel to create a single pixel in the new feature, as
in Equation 1.

To create more channels in the output feature, more 3D kernels are used. Using
$9$ instances of the MAC, we compute in parallel a $8\times 3\times 3$ 3D
convolution. To output a single output feature, groups of $8\times 3\times 3$
input features (a $3\times 3$ submatrix for each channel) are convolved with a
$8\times 3\times 3$ matrix of weights (kernels). If the input feature has more
input channels, this operation is repeated. However, most of the input
features that are part of the convolution of $9$ output feature (for each
output channel) are used in the next output feature calculations.

When we reach the final feature of a row and move to the next row, we do not
restart from column $0$. Instead, we start decreasing the column counter,
following a snake-like movement (see Figure 5). In this way, $6$ features are
always reused. Hence, when the computations are at full throttle, for each
output feature (i.e., for each cycle), we fetch $3$ input features for $8$
channels ($16$-bit each) and write one value. In this way, each cycle, $6$ of
the $9$ input features are saved and $3$ new are loaded. These $9\times 8$
values are then sent to the $9$ MACs. Each MAC takes $8$ channels of a feature
and multiplies them for $8$ channels of the kernel. These $9$ values are then
summed together in a $9$-operand Dadda adder to generate the output. This
process is conducted for the whole set of output features. The addresses to
read the input features and the kernels, and to write the new output features
are generated by the Forward Address Manager, which uses the dynamic size
given by the CU as the bound for the column, row, and channel counters. When
the column counter reaches the dynamic size, it will not be zeroed due to
snake movement, but its address will be maintained and increased by one row.
When the end of the matrix is reached, the channel counter is increased.

Figure 5. Example of forward computation of a channel of a convolutional
layer.

####  3.6.2. Kernel Gradient Computation

To calculate the gradient of the kernels, a 2D convolution must be executed,
as in Equation Equation 3. The input gradient is convoluted with $9$ different
sub-matrices of the input feature, which are shifted according to the
respective position of the gradient kernel. Since the input feature is padded
with $2$ stripes of zeros on the right and bottom borders, the resulting
matrix has sizes increased by $2$ in both dimensions. The index of the kernels
is also used for indexing which MAC will be used to compute the gradient,
according to Equation 7.

(7) $MAC_{k,l}<=g(G,V,s)_{i,j,k,l}=\frac{\partial}{\partial
K_{i,j,k,l}}J(V,K)$

Therefore, the features are shared between convolutions (to an even larger
extent than for the forward convolution).

####  3.6.3. Gradient Propagation

To compute the gradient propagation, the input gradient is convoluted with the
4D kernel. The data flow is the same as for the forward propagation since the
input gradient is computed as the input feature in the forward computations.

####  3.6.4. Dense Layer Computation

This section illustrates how the dense layer operations are conducted using
the same $9$ MACs of the PU.

  * •

Forward propagation: It is implemented as a matrix multiplication between the
flatted features of the previous layer with a 2D matrix. The weight matrix has
a size ($m$,$n$) where $m$ is the number of input features and $n$ is the
number of output features. Note that in the last dense layer, the output
features’ value is equal to the number of classes. This number, due to the CL
setup, is not static and changes during the operation. The output of the
forward algorithm is implemented according to Equation 8, which is described
considering to have a 3D matrix with sizes ($I$, $J$, $K$) instead of the
conventional flattened values.

(8) $\begin{gathered}y_{n}=\sum_{i=0}^{m}I_{i}\cdot
W_{i,n}=\sum_{i,j,k=0}^{I,J,K}I_{i,j,k}\cdot W_{i,j,k,n}\end{gathered}$

In this way, we can reuse the same logic of the forward computation for
convolutional layers. In this case, for each clock cycle, $8$ pixels of $8$
channels of the input feature and $8$ weights of $8$ channels are read and
sent to $8$ of the $9$ MACs. All $64$ results are then summed together and
saved into the partial sum register. This is done for the whole set of input
features and it is repeated $n$ times, where $n$ is the number of output
features. In this way, we can dynamically set the number of iterations
required.

  * •

Gradient propagation: The gradient propagation $dX$ is equal to the matrix
multiplication between input gradient propagation and the transpose of the
weights, as described in Equation 5.

In this case, the optimization is more complicated. Due to the fact the size
of $dY$ is dynamic due to the CL settings and is not necessarily a power of
$2$, we cannot reach a $100\%$ utilization of the hardware resources. We
propose to execute inside a single MAC the computation for each feature of
$dX$, iteratively using the register and the partial sum logic. Using $9$
MACs, we can compute $9$ pixel in $n/8$ clock cycles, where $n$ is the number
of output features, and the whole computation in $(I/9)\cdot(n/8)$ cycles. An
example of how MACs are indexed is shown in Equation 9.

(9) $MAC_{0,0}<=dX_{0}=\sum_{n=0}^{7}dY_{n}\cdot W^{T}_{n,0}$

The values will be saved in the partial sum register and summed in the next
addition, until the end of the vector.

  * •

Weight derivative: The derivative of the weight ($dW$) is the matrix
multiplication between the input feature $I$ used during the forward
computation and the gradient propagation coming from the loss computation
$dY$, as described in Equation 6.

Using the matrix notation, since both operands are row matrix, their product
produces a matrix. This means that the whole matrix $I$ is multiplied with the
same value $dY_{n}$. This calculus can be executed in our MAC. $64$ parallel
input features are read ($8$ pixels for $8$ channels, as in forward) and
multiplied with one feature of $dY$. The 64 multiplication results are then
added together and accumulated to previous multiplication and addition. This
is repeated for each feature of $dY$.

##  4\. Experimental Results

###  4.1. Experimental Setup

The complete TinyCL architecture has been described at the RTL level using the
SystemVerilog language. We synthesize the architecture in a 65 nm CMOS
technology node with the ASIC design flow using the Synopsys Design Compiler
tool. The synthesized netlist has been functionally tested through gate-level
simulations, compared to the equivalent software-level implementation on
TensorFlow (Abadi et al., 2016) running on an NVIDIA Tesla P100 GPU. The
complete flow of our experiments is shown in Figure 6. For comparison
purposes, we test our system executing a simple DNN model, composed of 2
convolutional layers with ReLU activation, followed by a Dense layer. This
model has been trained with the CIFAR10 dataset (Krizhevsky et al., 2015) for
10 epochs, a learning rate of 1, and a batch size of 1, while the training is
composed of 5 tasks of 2 classes per task, following the GDumb approach
(Prabhu et al., 2020). The model is used side by side with a memory of 6.144
MB, which can contain 1000 $32\times 32$ RGB training samples.

Figure 6. Experimental setup and tool flow for conducting the experiments.

###  4.2. Sysnthesis Evaluation of Area, Power, and Latency

The synthesized TinyCL architecture has a clock period of 3.87 ns, consumes 86
mW, and occupies an area of 4.74 mm2. Figure 7 analyzes in detail the power
and area breakdown for each component. It is evident that the Memory block is
responsible for the majority of area (80%) and power (76%) of the complete
architecture.

Figure 7. (a) Area and (b) power breakdown of our TinyCL architecture.

Our architecture takes 8,192 clock cycles to compute either the forward
convolution, the gradient propagation, or the gradient of the weight when we
use 8 filters and the input feature has a shape of $32\times 32\times 8$.
Instead, the dense layer for a feature of shape $32\times 32\times 8$ which
outputs an array of size 10 takes 1,280 clock cycles for forward computation,
1,821 clock cycles for the computation of the gradients of the weights, and
1,280 clock cycles for the gradient propagation.

###  4.3. Comparison with Related Works

As a first analysis, we compare our TinyCL architecture with its software-
level implementation on a P100 GPU. Following the setup described in Section
4.1, the GPU implementation lasts 103 s, while the TinyCL architecture
executes the same workload in 1.76 s, thus reaching a speedup of $58\times$.

Table 1 compares latency, power, area, and performance of our proposed TinyCL
architecture with related works. Note that since there are no available
designs specialized for CL workloads, we compare it with existing designs that
execute generic DNN training operations. The results show that the proposed
TinyCL architecture is suitable for autonomous systems with limited resources
since it achieves lower area and power consumption than prior works.

Table 1. Comparison between TinyCL and related DNN Training architectures.

Architecture Latency (ns) Power (mW) Area (mm2) Performance (TOPS) TinyCL
(our) 3.87 86 4.74 0.037 HNPU (Han et al., 2021) 4 1162 12.96 3.07 LNPU (Lee
et al., 2019) 5 367 16 0.6 ISSCC19 (Kim et al., 2019) 5 196 16 0.204

##  5\. Conclusion

We have presented TinyCL, an efficient hardware architecture for CL
algorithms, which executes all the involved operations such as forward
propagation, backward propagation, and parameter update. A specialized control
unit manages the memory and data flow to support memory-based CL policies and
different operations on the same Processing Unit. The TinyCL architecture,
synthesized in a 65 nm CMOS technology node, achieves $58\times$ speedup
compared to an equivalent software-level implementation executing on an Nvidia
Tesla P100 GPU, while achieving lower area and power consumption than related
works. Our design and analyses open new avenues for developing efficient CL
algorithms on resource-constrained autonomous systems.

###### Acknowledgements.

This work was supported in parts by the NYUAD Center for Artificial
Intelligence and Robotics (CAIR), funded by Tamkeen under the NYUAD Research
Institute Award CG010, and the NYUAD’s Research Enhancement Fund (REF) Award
on “eDLAuto: An Automated Framework for Energy-Efficient Embedded Deep
Learning in Autonomous Systems”.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi et al. 2016\.  TensorFlow: A System for Large-Scale Machine Learning. In _USENIX_.
  * Aljundi et al. (2019a) Rahaf Aljundi et al. 2019a.  Online Continual Learning with Maximal Interfered Retrieval. In _NeurIPS_.
  * Aljundi et al. (2019b) Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. 2019b.  Gradient based sample selection for online continual learning. In _NeurIPS_.
  * An et al. (2023) Qi An, Saifur Rahman, Jingwen Zhou, and James Jin Kang. 2023.  A Comprehensive Review on Machine Learning in Healthcare Industry: Classification, Restrictions, Opportunities and Challenges.  _Sensors_ (2023).
  * Capra et al. (2020a) Maurizio Capra et al. 2020a.  Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead.  _IEEE Access_ (2020).
  * Capra et al. (2020b) Maurizio Capra et al. 2020b.  An Updated Survey of Efficient Hardware Architectures for Accelerating Deep Convolutional Neural Networks.  _Future Internet_ (2020).
  * Chai et al. (2021) Junyi Chai, Hao Zeng, Anming Li, and Eric W.T. Ngai. 2021.  Deep learning in computer vision: A critical review of emerging techniques and application scenarios.  _MLWA_ (2021).
  * Chandani and Dubey (2022) Arti Chandani and Rahul Dubey. 2022.  Application of Machine Learning in Banking and Finance: A Bibliometric Analysis.  _IJDATS_ (2022).
  * Chaudhry et al. (2019) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019.  Efficient Lifelong Learning with A-GEM. In _ICLR_.
  * Chen et al. (2020) Yiran Chen et al. 2020\.  A survey of accelerator architectures for deep neural networks.  _Engineering_ (2020).
  * Dhilleswararao et al. (2022) Pudi Dhilleswararao, Srinivas Boppu, M. Sabarimalai Manikandan, and Linga Reddy Cenkeramaddi. 2022.  Efficient Hardware Architectures for Accelerating Deep Neural Networks: Survey.  _IEEE Access_ (2022).
  * Du et al. (2019) Xiaocong Du, Gouranga Charan, Frank Liu, and Yu Cao. 2019.  Single-Net Continual Learning with Progressive Segmented Training. In _ICMLA_.
  * Gaur et al. (2020) Divya Gaur, Joachim Folz, and Andreas Dengel. 2020.  Training deep neural networks without batch normalization.  _CoRR abs/2008.07970_ (2020).
  * Han et al. (2021) Donghyeon Han et al. 2021\.  HNPU: An Adaptive DNN Training Processor Utilizing Stochastic Dynamic Fixed-Point and Active Bit-Precision Searching.  _IEEE JSSC_ (2021).
  * Harun et al. (2023) Md Yousuf Harun et al. 2023\.  SIESTA: Efficient Online Continual Learning with Sleep.  _CoRR_ abs/2303.10725 (2023).
  * Hayes et al. (2019) Tyler L. Hayes, Nathan D. Cahill, and Christopher Kanan. 2019.  Memory Efficient Experience Replay for Streaming Learning. In _ICRA_.
  * Huai et al. (2023) Shuo Huai et al. 2023\.  Latency-constrained DNN architecture learning for edge systems using zerorized batch normalization.  _Future Gener. Comput. Syst._ (2023).
  * Jain et al. (2018) Animesh Jain et al. 2018\.  Gist: Efficient Data Encoding for Deep Neural Network Training. In _ISCA_.
  * Kim et al. (2019) Changhyeon Kim et al. 2019\.  A 2.1TFLOPS/W Mobile Deep RL Accelerator with Transposable PE Array and Experience Compression. In _ISSCC_.
  * Kim et al. (2018) Duckhwan Kim, Taesik Na, Sudhakar Yalamanchili, and Saibal Mukhopadhyay. 2018.  DeepTrain: A Programmable Embedded Platform for Training Deep Neural Networks.  _IEEE TCAD_ (2018).
  * Kirkpatrick et al. (2016) James Kirkpatrick et al. 2016\.  Overcoming catastrophic forgetting in neural networks.  _CoRR_ abs/1612.00796 (2016).
  * Knoblauch et al. (2020) Jeremias Knoblauch, Hisham Husain, and Tom Diethe. 2020.  Optimal Continual Learning has Perfect Memory and is NP-hard. In _ICML_.
  * Krizhevsky et al. (2015) Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2015.  CIFAR-10 (Canadian Institute for Advanced Research).  (2015).
  * Lee et al. (2019) Jinsu Lee et al. 2019\.  7.7 LNPU: A 25.3 TFLOPS/W sparse deep-neural-network learning processor with fine-grained mixed precision of FP8-FP16. In _ISSCC_.
  * Lee and Yoo (2021) Jinsu Lee and Hoi-Jun Yoo. 2021.  An overview of energy-efficient hardware accelerators for on-device deep-neural-network training.  _IEEE OJ-SSCS_ (2021).
  * Lee et al. (2020) Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. 2020.  A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning. In _ICLR_.
  * Li and Hoiem (2018) Zhizhong Li and Derek Hoiem. 2018.  Learning without Forgetting.  _IEEE TPAMI_ (2018).
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient Episodic Memory for Continual Learning. In _NeurIPS_.
  * Lu et al. (2023) Jinming Lu, Chao Ni, and Zhongfeng Wang. 2023.  ETA: An Efficient Training Accelerator for DNNs Based on Hardware-Algorithm Co-Optimization.  _IEEE TNNLS_ (2023).
  * Ma et al. (2023) Xinyue Ma et al. 2023\.  Cost-effective On-device Continual Learning over Memory Hierarchy with Miro. In _MobiCom_.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_.
  * Noh et al. (2023) Seock-Hwan Noh et al. 2023\.  FlexBlock: A Flexible DNN Training Accelerator With Multi-Mode Block Floating Point Support.  _IEEE Trans. Computers_ (2023).
  * Prabhu et al. (2020) Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. 2020.  GDumb: A Simple Approach that Questions Our Progress in Continual Learning. In _ECCV_.
  * Qin et al. (2020) Eric Qin et al. 2020\.  SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training. In _HPCA_.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017.  iCaRL: Incremental Classifier and Representation Learning. In _CVPR_.
  * Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017.  Continual Learning with Deep Generative Replay. In _NeurIPS_.
  * Soori et al. (2023) Mohsen Soori, Behrooz Arezoo, and Roza Dastres. 2023.  Artificial intelligence, machine learning and deep learning in advanced robotics, a review.  _Cognitive Robotics_ (2023).
  * Thrun and Mitchell (1995) Sebastian Thrun and Tom M. Mitchell. 1995.  Lifelong robot learning.  _RAS_ (1995).
  * Venkataramani et al. (2017) Swagath Venkataramani et al. 2017\.  ScaleDeep: A Scalable Compute Architecture for Learning and Evaluating Deep Networks. In _ISCA_.
  * Wang et al. (2023) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023.  A Comprehensive Survey of Continual Learning: Theory, Method and Application.  _CoRR_ abs/2302.00487 (2023).
  * Wang et al. (2022) Zifeng Wang et al. 2022\.  SparCL: Sparse Continual Learning on the Edge. In _NeurIPS_.
  * Yang et al. (2020) Dingqing Yang et al. 2020\.  Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training. In _MICRO_.