EuclidNets]EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models

1]Xinlin<EMAIL_ADDRESS>
2]Mariana<EMAIL_ADDRESS>
2]Adam<EMAIL_ADDRESS>
1]Alireza<EMAIL_ADDRESS>
2]Masoud<EMAIL_ADDRESS>
[1,3]Vahid Partovi<EMAIL_ADDRESS>
*[1]Huawei Noah's Ark Lab, Montreal Research Centre, 7101 Park Avenue, Montreal, H3N 1X9, QC, Canada

[2]Department of Mathematics and Statistics, McGill University, 805 Sherbrooke Street West, Montreal, H3A 0B9, QC, Canada

[3]Department of Mathematics and Industrial Engineering, Polytechnique Montreal, 2500 Chem. Polytechnique, Montreal, H3T 1J4, QC, Canada

With the advent of deep learning application on edge devices, researchers actively try to optimize their deployments on low-power and restricted memory devices. There are established compression method such as quantization, pruning, and architecture search that leverage commodity hardware. Apart from conventional compression algorithms, one may redesign the operations of deep learning models that lead to more efficient implementation. To this end, we propose EuclidNet, a compression method, designed to be implemented on hardware which replaces multiplication, $xw$, with Euclidean distance $(x-w)^2$.
We show that EuclidNet is aligned with matrix multiplication and it can be used as a measure of similarity in case of convolutional layers. Furthermore, we show that under various transformations and noise scenarios, EuclidNet exhibits the same performance compared to the deep learning models designed with multiplication operations.

§ INTRODUCTION

While majority of deep neural networks are trained on GPUs, they are increasingly being deployed on edge devices, such as mobile devices. These edge devices require

to compress the architecture for a given hardware design (e.g. GPU or lower precision chips)  due to memory and power constraints <cit.>. Moreover, application specific hardware are being designed to accommodate the deployment of deep learning models. Thus,  designing efficient deep learning architectures that are efficient for the deployment (i.e. inference) has become a new challenge in the deep learning community.

The combined problem of hardware and deep learning model design is complex, and the precise measurement of efficiency is both device and model specific. This is because researchers have to  take into account various efficiency factors such as latency, memory footprint, energy consumption.  Here we deliberately oversimplify the problem in order to make it tractable, by addressing a fundamental element of hardware cost. Knowing that power consumption is directly related to the chip area in a digital circuit, we use the chip area required to implement an arithmetic operation on a hardware as a surrogate to measure the efficiency of a deep learning model. While this is very coarse, and full costs will depend on other aspects of hardware implementation, it nevertheless represents a fundamental unit of cost in hardware design <cit.>.

In a deep learning model, weights are multiplied by inputs, hence on of the fundamental operations in deep learning models is multiplication $S_{\Conv}(x,w) = wx$. In our work, we replace multiplication with the EuclidNet operator,

\begin{equation}\label{eq: euclid}
S_{\Euclid}(x,w) = -\frac{1}{2}\|x-w\|_2^2.
\end{equation}

which combines a difference with a square operator. We will refer to the family of deep learning models that use equation (<ref>) as EuclidNets.
These models compromise between standard multiplicative models and AdderNets<cit.>, which remove multiplication entirely, but at the cost of a significant loss of accuracy and difficult training procedure. Replacing multiplication with square can potentially reduce the computation cost. The feature representation of each of the architectures is illustrated in Figure <ref>. EuclidNets can be implemented on 8-bit precision without loss of accuracy as demonstrated in Table <ref>.

The square operator is cheaper than multiplication and it can also be implemented using look up tables <cit.>.  In <cit.>, authors prove that replacing look up table can replace actual float computing, while works such as LookNN in <cit.> take the first step in designing hardware for look up table use. On a low precision hardware, we can compute $S_\Euclid$ for about half the cost of computing $S_\Conv$.

Furthermore, using EuclidNets,  the deep learning model does not lose expressivity, as explained ins Section <ref>.
To summarize, we make the following contributions:

* We design a deep learning architecture based on replacing the multiplication $S_\Conv(x,w) = wx$ by the squared difference equation (<ref>).  We show that using square operator can potentialy reduce the hardwaer cost.

* These deep learning models are just as expressive as convolutional neural networks.  In practice, they have comparable accuracy (drop of less than 1 percent on ImageNet on ResNet50 going from full precision convolutional to 8-bit EuclidNets).

* We show theoretically and empirically that EuclidNets have the same behaviour compared to convolutional neural network in the case that the input is transformed (e.g. linear transformation) or affected by noise (e.g. Guassian noise).

* We provide an easy approach to train EuclidNets using homotopy.

Feature representation of traditional convolution with $S(x,w) = xw$ (left), AdderNet $S(x,w) = -\|x-w\|_1$ (middle), EuclidNet $S(x,w) = -\frac{1}{2}\|x-w\|_2^2$ (right).

Euclid-Net Accuracy with full precision and 8-bit quantization: Results on ResNet-20 with Euclidian similarity for CIFAR10 and CIFAR100, and results on ResNet-18 for ImageNet.  Euclid-Net achieves comparable or better accuracy with 8-bit precision, compared to the conventional full precision convolutional neural network.

3*Chip Efficiency

3cTop-1 accuracy

CIFAR10     CIFAR100    ImageNet

Full precision    	   	92.97
68.14     69.56

8-bit    	   	92.07     68.02    	69.59

Full precision            93.32     68.84     69.69

8-bit    	   	93.30     68.78     68.59

Full precision            91.84     67.60     67.0

8-bit    	   	91.78     67.60     68.8

1-bit            84.87     54.14     51.2

§ CONTEXT AND RELATED WORK

Compressing deep learning models comes at the costs of accuracy loss, and increasing training time (to a greater extent on quantized networks) <cit.>.
Part of the accuracy loss comes simply from decreasing model size, which is required for mobile and edge devices  <cit.>. Some of the most common deep learning compression methods include pruning <cit.>, quantization <cit.>, knowledge distillation <cit.>, and efficient design <cit.>. Between the compression methods, the most prominent approach is low bit quantization <cit.>. In this case, the inference can speed up with lowering bit size, at the cost of
accuracy drop and  longer training times. In the extreme quantization, such as binary networks, operations have negligible cost at inference but exhibits a considerable accuracy drop <cit.>.

Here we focus on a small sub-field of compression, that optimizes mathematical operations in a deep learning model.  This approach can be combined successfully with other conventional compression methods, such as quantization <cit.> and pruning <cit.>.

On the other hand, knowledge distillation <cit.> consists of transferring information from a larger teacher network to a smaller student network. The idea is easily extended by thinking of information transfer between different similarity measures, which <cit.> explores in the context of AdderNets. Knowledge distillation is an uncommon training procedure and requires extra implementation effort. However, EuclidNet preserves the accuracy without knowledge distillation. We suggest a straightforward training using a smooth transition between common convolution and Euclid operation using homotopy.

§ SIMILARITY AND DISTANCES

§.§ Inner Products versus Distances

Consider an intermediate layer of a deep learning model  with input $x\in\R^{H\times W \times c_{\In}}$ and output $y~\in~\R^{H\times W \times c_{\Out}}$ where $H,W$ are the dimensions of the input feature, and $c_{\In}, c_{\Out}$ the number of input and output channels, respectively. For a standard convolutional network, we represent the input to output transformation via weights $w~\in~\R^{d\times d\times c_{\In}\times c_{\Out}}$ as

\begin{equation}\label{eq: layer}
y_{mnl} = \sum_{i = m}^{m+d} \sum_{j=n}^{n+d} \sum_{k = 0}^{c_{\In}}
x_{ijk} w_{ijkl}
%S(x_{ijk}, w_{ijkl}).
\end{equation}

Setting $d=1$ reduces the equation (<ref>) to a fully-connected layer.
We can abstract the multiplication of the weights $w_{ijkl}$ by $x_{ijkl}$ in the equation above by using a similarity measure $S:\R\times\R\to\R$. The original convolutional layer corresponds to

S_{\Conv}(x,w) = xw.

In our work, we replace $S_\Conv$ with $S_\Euclid$, given by equation (<ref>).  A number of works have also replaced the multiplication operator in  deep learning models.
The most relevant work is the AdderNet <cit.>, which uses

\begin{equation}\label{eq: adder}
S_{\Adder}(x,w) = -\|x-w\|_1.
\end{equation}

to replace multiplication by $\ell_1$ norm, i.e. summation of the absolute value of differences.
This operation can be implemented very efficiently on a custom hardware, knowing that subtraction and absolute value of different $n$-bit integers cost $\mathcal{O}(n)$ gate operations, compared to $\mathcal{O}(n^2)$ for multiplication i.e. $S_\Conv(x,w) = xw$. However, AdderNet comes with a significant loss in accuracy, and is difficult to train.

§.§ Other Similarity Measures

The idea of replacing multiplication operations to save resources within the context of neural networks dates back to 1990s. Equally motivated by computational speed-up and hardware logic minimization, authors of <cit.> defined perceptrons that use the synapse similarity,

\begin{equation}\label{eq: comp_syn}
S_{\Synapse}(x,w) = \sign(x)\cdot \sign(w) \cdot \min(\|x\|,\|w\|),
\end{equation}

which is cheaper than multiplication in terms of hardware complexity.

Equation (<ref>) has not been experimented with modern deep learning models and datasets. Moreover, in <cit.> a slight variation is introduced which is also a multiplication-free operator,

\begin{equation}\label{eq: mf}
S_{\Mfo}(x,w) = \sign(x)\cdot\sign(w)\cdot(\|x\|+\|w\|)).
\end{equation}

Note that both equations (<ref>) and (<ref>) use $\ell_1$-norm. Also note that in <cit.>, the updated design choice  allows contributions from both operands $x$ and $w$. Furthermore, in <cit.>,  the similarity in image classification on CIFAR10 is studied. Other applications of  equation (<ref>) are studied in <cit.>.

In <cit.>, the similarity operation is further combined with a bit-shift, leading to an improved accuracy with negligible added hardware cost. However, the accuracy results for AdderNet appear to be lower than those reported in <cit.>. Another follow-up work uses knowledge distillation to further improve the accuracy of AdderNets <cit.>.

Instead of simply replacing the similarity on the summation, there is also the possibility to replace the full expression of equation (<ref>) as, for example,  proposed in  <cit.>, by approximating the activation of a given layer with an exponential term. Unfortunately, these methods only lead to speed-up in certain cases and, in particular, they do not improve CPU inference time. Moreover, the reported accuracy on the benchmark problems is also lower than the typical baseline.

In <cit.>, authors used three layer morphological neural networks for image classification. Morphological neural networks were introduced in  1990s in <cit.> and use the notion of erosion and dilation to replace equation (<ref>):

\begin{align*}
\mbox{Erosion}(x,w) &= \min_j S(x_j, w_j) = \min_j (x_j - w_j), \\
\mbox{Dilation}(x,w) &= \max_j S(x_j, w_j) = \max_j (x_j + w_j).
\end{align*}

The authors proposed two methods by stacking layers to expand networks, but they admitted the  possibility of  over-fitting and difficult training issues, casting doubt on scalability of the method.

§ THEORETICAL JUSTIFICATION

This section provides some theoretical ground for the connections among AdderNets, EuclidNets, and conventional convolution.

§.§ Equivalence with Multiplication

Euclidean distance has a close tie with multiplication and hence, it can replace the multiplications in convolution and linear layers. Here, we delve into the details of this claim a bit more.

Let us consider Euclidean distance between the two vectors $\x$ and $\w$ as $\norm{\x-\w}=(\x-\w)\t(\x-\w)$ where $\x$ and $\w$ are the vectors of inputs and weights respectively. Moreover, $\x$ and $\w$ are vectors of random variables, so it is of interest to study the expected value of the EuclidNet operation first,

\begin{equation}
-\frac 1 2 \mathbb{E} \norm{\x-\w} = -\frac 1 2 \mathbb{E}\norm \x - \frac 1 2 \mathbb{E}\norm \w + \mathbb{E}(\x\t\w) .
\label{eq:euclid_expect}
\end{equation}

In other words (<ref>), convlution similarity measure, i.e.  the inner product $\x\t\w$, is embedded in EuclidNet form. However, the result is biased with two extra terms i.e. $-\frac 1 2 \mathbb{E}\norm\x$ and $-\frac 1 2 \mathbb{E}\norm\w$. Thus we may  conclude that Euclidean distance is aligned with multiplication shifted by two bias terms. The induced bias by the EuclidNet operation remains controlled in both training or inference, most deep learning models use some sort of normalization mechanism such as batch norm,  layer norm, and weight norm.

Euclidean distance also has a close relationship with cosine similarity. Let us define $S_\Cos$ as

\begin{equation}
S_\Cos(\x,\w):= \frac{\x^\top \w}{\rootnorm \x \rootnorm\w}.
\label{eq:cosine_sim}
\end{equation}

It is easy to see that in the case of having a normalization mechanism (i.e. $\rootnorm \x=\rootnorm \w=1$) the cosine similarity and Euclid similarity become equivalent

\begin{eqnarray}
S_\Euclid(\x,\w)= S_\Cos(\x,\w)-1 &\mathrm{\quad s.t.}& \rootnorm \x = \rootnorm \w=1 .
\label{eq:cosine_distance}
\end{eqnarray}

Moreover, Euclidean norm is a transitive similarity measure since it satisfies the following inequality

\begin{equation}
\|\x-\w\|_2 \geq\lvert~\|\x\|_2-\|\w\|_2 ~\rvert .
\label{eq:inv_triangle}
\end{equation}

It is noteworthy to mention that this transitivity holds for p-norms (i.e. $\|\mathbf{a}\|_p= (\sum_i \|{a}_i\|^p)^{\frac{1}{p}}$). This means that the AdderNet <cit.> operator is also transitive. According to equation (<ref>), however, the only norm that has such a close relationship with the cosine similarity is Euclidean norm. This is the distinguishing feature of the EuclidNets that while they are distance based, and hence enjoy the transitivity property in measuring similarity, their performance is also completely aligned with those based on Cosine similarity.

§.§ Expressiveness of EuclidNets

Deep learning models that use the EuclidNet operation are just as expressive as those using multiplication. Note the polarization identity,

\[
S_\Conv(x,w) = S_\Euclid(x,w) - S_\Euclid(x,0) - S_\Euclid(0,w)
\]

which means that any multiplication operation can be expressed using only Euclid operations.

§.§ Hardware cost

Traditionally, hardware developers
use smaller multipliers to create larger multipliers <cit.>. They use various methods of multiplier tiling or divide and conquer to form larger multiplier. Karatsuba algorithm and its generalization <cit.> is among the most known algorithms to implement large multipliers. Here we show that Euclidean distance can be potentially implemented with fewer multipliers in hardware.

Karatsuba algorithm is a form of divide and conquer algorithm to perform $n-$bit multiplication using $m-$bit multipliers. Let us assume $a$ and $b$ are $n-$bit integer numbers and they can be re-written using two $m-$bits partitions

\begin{align}
\nonumber
&a = a_1 \times 2^m + a_2,\\ \nonumber
&b = b_1 \times 2^m + b_2.\\
\label{eq:karatsuba_parts}
\end{align}

In the case of multiplication, we have

\begin{align}
\nonumber
&ab = (a_1 \times 2^m + a_2) (b_1 \times 2^m + b_2)\\ \nonumber
&~~~= 2^{2m}a_1b_1+2^m a_1b_2+2^m a_2b_1+a_2b_2,\\
\label{eq:mult}
\end{align}

which is comprised of three additions and four $m-$bits multiplications. However for the squaring operation, we have

\begin{align}
\nonumber
&a^2 = (a_1 \times 2^m + a_2) (a_1 \times 2^m + a_2)\\ \nonumber
&~~~= 2^{2m}a_1^2+2^{m+1} a_1a_2+a_2^2,\\
\label{eq:square}
\end{align}

which is comprised of two additions and three $m-$bits multiplications. Thus, the squaring operation can be cheaper in hardware. Also note that such divide and conquer techniques are used commonly in designing accelerator on FPGA targets.

§ TRAINING EUCLIDNETS

Training EuclidNets are much easier compared to other similarity measures such as AdderNets. This makes EuclidNet attractive for complex tasks such as image segmentation, and object detection where training compressed networks are challenging and causes large accuracy drops. However, EuclidNets are more expensive than AdderNets when using floating-point number format, however, their quantization is easy since, unlike AdderNets, they behave  similar to traditional convolution to a great extent. In another words EuclidNets are easy to quantize.

While training a deep learning model using EuclidNets, it is more appropriate to use the identity

\begin{equation}
S_{\Euclid}(x,w) = -\frac {x^2}{2} - \frac{w^2}{2} + x w,
\end{equation}

that is more appropriate for GPUs that are optimized for inner product computations. As such,  training EuclidNets does not require additional CUDA kernel implementation unlike AdderNets <cit.>. The official implementation of AdderNet <cit.> reflects order of  $20\times$ slower training than the traditional convolution on PyTorch. This is specially problematic for large deep learning models and complex tasks since even traditional convolution training takes few days or even weeks. EuclidNet training is about $2\times$ slower in the worst case and their implementation is natural in deep learning frameworks such as PyTorch and Tensorflow.

Time (seconds) and maximum training batch-size that can fit in a signle GPU Tesla V100-SXM2-32GB, during ImageNet training. In parenthesis is the slowdown with respect to the $S_{conv}$ baseline.
We do not show times for AdderNet, which is much slower than both, because it is not implemented in CUDA

2*Model     2*Method     2l Maximum Batch-size     2lTime per step

1lc]@c@ power of 2     1linteger     Training     Testing
2*ResNet-18     $S_{\Conv}$     1024     1439     0.149     0.066

$S_{\Euclid}$     512     869 ($1.7\times$)     0.157 ($1.1\times$)     0.133 ($2\times$)
2*ResNet-50     $S_{\Conv}$     256     371     0.182     0.145

$S_{\Euclid}$     128     248 ($1.5\times$)     0.274 ($1.5\times$)     0.160 ($1.1\times$)

A common method in training neural networks is fine-tuning, that means initializing with weights trained on different data but with a similar nature. Here, we introduce the idea of using a weight initialization from a model trained on a related similarity measure.

Rather than training from scratch, we wish to fine-tune EuclidNet starting from accurate  CNN weights.  This is achieved by an “architecture homotopy" where we change hyperparameters to convert a regular convolution to an EuclidNet operation

\begin{equation}
S(x,w; \lambda_k) = xw - \lambda_k\frac{x^2 + w^2}{2},\qquad \mbox{ with }\lambda_k = \lambda_0 + \frac{1 - \lambda_0}{n} \cdot k,
\label{eq: homotopy}
\end{equation}

where $n$ is the total number of epochs and $0<\lambda_0<1$ is the initial  transition phase. Note that $S(x,w,0) = S_{\Conv}(x,w)$ and $S(x,w,1) =S_{\Euclid}(x,w)$ and equation (<ref>) is a convex combination of these two similarities. One may interpret  $\lambda_k$ as a scheduler for the homotopy, similar to the way learning rate is scheduled in training a deep learning model. We found that a linear scheduling as shown in equation (<ref>) is empirically effective.

Transformations like equation (<ref>) are commonly used in scientific computing <cit.>. The idea of using homotopy  in training neural networks can be traced back to <cit.>. Recently, homotopy was used in deep learning in the context of activation functions <cit.>, loss functions <cit.>, compression <cit.> and  transfer learning <cit.>. Here, we use homotopy in the context of transforming operations of a deep learning model.

Fine-tuning method in equation (<ref>) is inspired by continuation methods in partial differential equations. Assume $S$ is a solution to a differential equation with the initial condition  $S(x,0) = S_0(x)$. In certain situations, solving this differential equation for $S(x,t)$ and then evaluating at $t=1$ might be easier than solving directly for $S_1$. One may think of this homotopy method as an evolution for deep learning model weights. At time zero the deep learning model consists of regular convolutional layers, but they gradually transform to Euclidean layers.

The homotopy method can also be interpreted as a sort of  of knowledge distillation. Whereas knowledge distillation methods tries to match a student network to a teacher network, the homotopy can be seen as a slow transformation from the teacher network into a student network. Figure <ref>  demonstrates the idea. Interestingly, problems that have been solved with homotopy have also been tackled by knowledge distillation <cit.>.

Training schema of EuclidNet using Homotopy, i.e. transitioning from traditional convolution $S(x,w)=xw$ towards EuclidNet $S(x,w)=-\frac{1}{2} |x-w|^2$ through equation (<ref>).

§ EXPERIMENTS

To illustrate performance of the EuclidNets, We apply our proposed method on image classification tasks. We also test our trained deep learning model under different transformations on the input image and compare the accuracy to standard convolutional networks.

§.§ CIFAR10

First, we consider the CIFAR10 dataset, consisting of $32\times32$ RGB images with 10 possible classifications <cit.>. We normalize and augment the dataset with random crop and random horizontal flip. We consider two ResNet models <cit.>, ResNet-20 and ResNet-32.

We train EuclidNet using the optimizer from <cit.>, which we will refer to as AdderSGD, to evaluate EuclidNet under a similar setup. We use initial learning rate $0.1$ with cosine decay, momentum $0.9$, batch size 128 and weight decay $5\times 10^{-4}$. We follow <cit.> in setting the learning-rate scaling parameter $\eta$. For traditional convlutional network, we use the same hyper-parameters with stochastic gradient descent optimizer.

The details of classification accuracy is provided in Table <ref>. We consider two different weight initialization for EuclidNets. First, we initialize the weights randomly and second, we initialize them with pre-trained on a convolutional network. The accuracy for EuclidNets has negligible accuracy loss compared to the standard ResNets. We see that for CIFAR10 training from scratch achieves even a higher accuracy, while initializing with convolution network and using linear homotopy training improves it even further.

Results on CIFAR10. The initial learning rate is adjusted for non-random initialization.

2*Model     2*Similarity     2*Initialization     2*Homotopy     2*Epochs     2cTop-1 accuracy

CIFAR10     CIFAR100
4*ResNet-20     $S_{\Conv}$     Random     None     400     92.97     69.29

3*$S_{\Euclid}$     Random     None     450     93.00     68.84

2*Conv     None     100     90.45     64.62

Linear     100     93.32

4*ResNet-32     $S_{\Conv}$     Random     None     400      93.93     71.07

3*$S_{\Euclid}$     Random     None     450     93.28     71.22

2*Conv     None     150     91.28     66.58

Linear     100     92.62     68.42

EuclidNets can become unstable during the training, despite careful choice of the  optimizer. Figure <ref> shows a comparison of the EuclidNet training with a standard convolutional network.  As it can be seen in the Figure <ref>, fine-tuning the EuclidNets directly from convolutional networks' weights is more stable than training from scratch. Also observe that when we train EuclidNets from scrach, accuracy is lower but the convergence is faster. Finally, using homotopy in the training procedure, the accuracy is improved.  Note that the pre-trained convolution weights are commonly available in the most of neural compression tasks, so initializing EuclidNets with pre-trained convolution is a commonplace procedure in optimizing deep learning models for inference.

Evolution of testing accuracy during training of ResNet-20 on CIFAR10, initialized with random weights, or initialized from convolution pre-trained network. Initializing from a pretrained convolution network speeds up the convergence. EuclidNet is harder to train compared with convolution network when both initialized from random weights.

EuclidNets are not only faster to train compared to other norm based similarity measures, but also stand superior in terms of accuracy. AdderNet performs slightly worse in terms of accuracy and also is much slower to train. The accuracy is significantly lower for the synapse <cit.> and the multiplication-free <cit.> operators. Table <ref> demonstrates a top-1 accuracy comparison of different methods. The reported results on AdderNet are from <cit.>. Note that although for AdderNet in <cit.>, authors  used knowledge distillation to close the gap with the full precision, it still falls short compared with EuclidNet.

Full precision results on ResNet-20 for CIFAR10 for different multiplication-free similarities.

14emSimilarity      13em$S_{\Conv}$     13em$S_{\Euclid}$     $S_{\Adder}$      $S_{\Mfo}$     $S_{\Synapse}$

Accuracy     92.97
82.05     73.08

Training a quantized $S_{\Euclid}$  is very similar to convolutional neural networks. This allows a wider use of such models for lower resource devices. Quantization of the EuclidNets to 8bits keeps the accuracy drop within the range of one percent <cit.> similar to traditional convolutional neural networks.  Table <ref> shows 8-bit quantization of EuclidNet where the accuracy drop remains negligible. Furthermore, training EuclidNets on CIFAR100 dataset exhibits a negligible accuracy drop when the weights are initialized with pre-trained standard model weights.

§.§ ImageNet

Next, we consider testing EuclidNet classifier on ImageNet <cit.> which is known to be a challenging classification task comparing to CIFAR10. We trained our baseline convolutional neural network with standard augmentations of random resized crop and horizontal flip and normalization. We consider ResNet-18 and ResNet-50 models with the same hyper-parameters as those used in Section <ref>.

Table <ref> shows top-1 and top-5 classification accuracy of ImageNet dataset. As shown in Table <ref>,  the accuracy of EuclidNet when it is trained from scratch is lower than the baseline emphasizing the importance of homotopy training. We believe that the accuracy drop with no homotopy is because the hyper-parameter tuning is harder for large datasets such as ImageNet. This means that even though there exists hyper-parameters that achieve equivalent accuracy with random initialization, however it is too difficult to find them. Thus, it is much easier to use the existing hyper-parameters of traditional convolutional neural network, and use homotopy to smoothly transfer the weights to wights that are suitable for EuclidNets.

Full precision results on ImageNet. Best result for each model is in bold.

Model     Similarity     Initialization     Homotopy     Epochs     1lTop-1 Accuracy     1lTop-5 Accuracy
6*ResNet-18     $S_{\Conv}$     Random     None     90     69.56     89.09
5*$S_{\Euclid}$     Random     None     90     64.93     86.46
4*Conv     None     90     68.52     88.79
3*Linear     10     65.36     86.71

60     69.21     89.13

90      69.69      89.38
6*ResNet-50     $S_{\Conv}$     Random     None     90     75.49     92.51
5*$S_{\Euclid}$     Random     None     90     37.89     63.99
4*Conv     None     90     75.12     92.50
3*Linear     10     70.66     90.10

60     74.93     92.52

90      75.64      92.86

§.§ Transformation and blurring

Here we provide empirical evidence that Euclidean norm is aligned with the multiplication. First, we show that EuclidNets perform as well as standard convolutional neural networks in the case of pixel transform. Second, we show that when the image is blurred with Guassian noise, EuclidNets closely follow the behaviour of the convolutional neural networks.

§.§.§ Pixel transformation

We define pixel transformation of an image as

\begin{equation}
\mathbf{I_T} = a\mathbf{I}+b,
\label{eq:transform}
\end{equation}

where $\mathbf{I}$ is a tensor representing the original image, scalars $a$ and $b$ are transformation parameters, and $\mathbf{I_T}$ is the transformed image. Note that in (<ref>), $a$ controls the contrast and $b$ controls the brightness of the image. Such transformations are widely used in various stages of the imaging systems for instance in color correction, and gain-control (ISO).

Figure <ref> shows the accuracy of the standard ResNet-18 and EuclidNet ResNet-18 when the input image is affected by the pixel transformation of equation (<ref>). We can see that when changing $a$ and $b$, EuclidNet ResNet-18 closely follow the accuracy of the standard ResNet-18.

Accuracy of CIFAR10 classification affected by pixel transformation for a standard ResNet-18 (left) and EuclidNet ResNet-18 (right).

Accuracy of CIFAR10 classification affected by Guassian noise for a standard ResNet-18 (left) and EuclidNet ResNet-18 (right).

§.§.§ Gaussian Blurring

Additive noise can be injected to an image in different stages of the imaging system due to faulty equipments or environmental conditions. We tested EuclidNet when the input image is affected by a Guassian additive noise. Figure <ref> demonstrates comparison of the standard ResNet-18 and EuclidNet ResNet-18 for different noise intensities i.e. $\sigma$ and kernel sizes. This experiment is done for classification of the CIFAR10 dataset. We can see that EuclidNet ResNet-18 closely follow the behaviour of the standard ResNet-18 in the case of different kernel sizes and noise intensities.

§ CONCLUSION

EuclidNets are a class of deep learning models in which the multiplication operator is replaced  with the Euclidean distance. They are designed to be implemented on application specific hardware, with the idea that subtraction and squaring are cheaper than multiplication when designing efficient hardware for inference.
Furthermore, in contrast to other efficient architectures that are difficult to train in low precision, EuclideNets are easily trained in low precision.
EuclidNets can be initialized with pre-trained weights of the standard convolutional neural networks and hence, the training procedure of EuclidNets using homotopy is considered as a fine tuning of convolutional networks for inference. The homotopy method further improves training in such scenarios and training using this method sometimes surpass regular convolution accuracy. .


# Increasing Depth of Neural Networks for Life-long Learning

1st Jedrzej Kozal  Department of Systems and Computer Networks
Wrocław University of Science and Technology
Wrocław, Poland
<EMAIL_ADDRESS>2nd Michał Wozniak  Department of Systems and
Computer Networks
Wrocław University of Science and Technology
Wrocław, Poland
<EMAIL_ADDRESS>
###### Abstract

Increasing neural network depth is a well-known method for improving neural
network performance. Modern deep architectures contain multiple mechanisms
that allow hundreds or even thousands of layers to train. This work is trying
to answer if extending neural network depth may be beneficial in a life-long
learning setting. In particular, we propose a novel method based on adding new
layers on top of existing ones to enable the forward transfer of knowledge and
adapting previously learned representations for new tasks. We utilize a method
of determining the most similar tasks for selecting the best location in our
network to add new nodes with trainable parameters. This approach allows for
creating a tree-like model, where each node is a set of neural network
parameters dedicated to a specific task. The proposed method is inspired by
Progressive Neural Network (PNN) concept, therefore it is rehearsal-free and
benefits from dynamic change of network structure. However, it requires fewer
parameters per task than PNN. Experiments on Permuted MNIST and SplitCIFAR
show that the proposed algorithm is on par with other continual learning
methods. We also perform ablation studies to clarify the contributions of each
system part.

###### Index Terms:

life-long learning, continual learning, deep learning, machine learning

##  I Introduction

Neural networks may benefit greatly from increasing depth [13, 14, 38].
Especially in computer vision introduction of residual connection [13] allowed
for training much deeper networks, which further boosted the results obtained
on ILSVRC ImageNet challenge [33], stimulating further research based on this
architecture [43, 36, 38]. Greater depth increases models’ capacity and allows
for learning more complicated functions. Features learned by network are
hierarchical. This can be easily shown with convolutional networks, where we
can visualize filters learned by the network. Filters located at the bottom of
the network contain generic features responsible for detecting edges or color
conjunctions [44]. With increasing depth, filters in convectional layers are
responsible for recognizing more abstract concepts. Layers at the top of a
network are specific to the task solved at hand. This property is used in
transfer learning technique [1, 18], where the model pre-trained on a large
dataset is later fine-tuned on a typically smaller dataset. It helps to avoid
collecting a large dataset, which can be prohibitive for some applications.
Fine-tuning of already pretrained backbones was also utilized in life-long
learning [25], as it allows for transmitting learned knowledge among different
tasks.

Life-long learning or continual learning is a branch of machine learning where
new data arrive sequentially. Usually, in machine learning, we assume that
data is i.i.d. In life-long learning, this assumption is not met, as new
incoming data can come from a different distribution than the previous ones.
The arrival of new concepts to generalize leads to forgetting old already
learned concepts. This phenomenon is known as catastrophic forgetting [10].
For this reason, continual learning algorithms should be able to adapt to
changes in data distribution and, at the same time, contain mechanisms that
prevent forgetting already acquired knowledge [5]. In literature this notion
is known as _stability-plasticity dilemma_ [7]. The same notion was captured
as generalization-forgetting trade-off in [31]. Both humans and animals can
learn new tasks without degrading already acquired knowledge. Our brains
handle continual learning well and can refine knowledge based on new
experiences to make better decisions [2]. For this reason, life-long learning
is closer to human learning compared to standard machine learning. Also,
continual learning development is essential for building autonomous learning
agents and robots capable of operating in the open world. If no closed set of
knowledge can be defined, then the agent must learn constantly based on the
experiences provided to it. Perpetual learning from scratch is largely
ineffective in this setup as a model would have been trained too often. This
has sparked interest in recent years in continual learning [29].

This work aims to combine advancements created over the years in neural
network architecture development and life-long learning. We evaluate the
ability to increase network depth for life-long learning, stacking new layers
on top of existing ones. Using this approach, we hope to benefit from
increasing the depth and training of new parameters by utilizing previously
learned representations. This allows for direct forward knowledge transfer
that emerges as a natural consequence of the proposed method. Recently
published [28] evaluates impact of architecture on continual learning
performance. Authors suggest that it is more beneficial to use wider, not
deeper, networks in a life-long learning setup. Results presented in our work
prove that naive increasing of depth does not improve performance; however,
when using the proposed approach, one can obtain better outcomes.

##  II Related Works

Increasing the depth of neural network architectures is an inherent part of
deep learning progress. Most of breakthroughs in computer visions may be
attributed to increasing the size of network [35, 37, 13, 38, 39]. With this
progress, other areas of computer vision started to benefit from the
utilization of large pretrained backbones [11, 12]. In [14] One interesting
proposal was Stochastic Depth, which was introduced to train very deep
networks, allowing ResNets with depths exceeding 1000 layers to be trained. It
was later incorporated into newer architectures [38, 39]. Over time Neural
Architecture Search (NAS) algorithms have been used to find new neural network
architectures [45, 46]. Due to this development, some researchers focused on
determining the proper search space for NAS algorithms. In [38] principles of
expanding neural networks are studied in depth. Authors considered three
dimensions for increasing network size: depth (number of layers), width
(number of filters), and input resolution. This allowed for the construction
of a network with smaller parameters and better performance.

Many algorithms have been developed so far for life-long learning. Elastic
Weight Consolidation (EWC) [17] is a method where a new regularization term
was introduced to slow down learning for parameters that were important in
previous tasks. Li and Hoiem [25] proposed Learning without Forgetting (LwF),
that is a method where a new task-specific classification head is added
backbone common for all tasks. Firstly, incoming data for the new task is
labeled according to the outputs of classification heads for previous tasks.
These soft labels are used to prevent forgetting of earlier tasks. Gradient
Episodic Memory (GEM) [27] is a rehearsal method that utilizes samples stored
in memory to project gradients computed for the current task to ensure that
loss over all previous tasks is not increasing. A-GEM [4] is a modification of
GEM that relaxes the condition of decreasing loss for previous tasks to only
decreasing average loss for previous tasks. This modification speeds up
training with an approximate ability to solve the tasks

The idea of expanding network capacity to handle changing data distribution is
well known in the literature. Progressive Neural Networks (PNN) [34] is a
life-long learning method that introduces growing neural networks in terms of
a number of parameters with each new task. When data for a new task arrives, a
new column with randomly initialized weights is added to an existing
structure. Each newly added layer takes as an input features generated by
layers from current and previous tasks. For this reason, features learned in
each layer are a function of data from the current task and knowledge learned
from earlier tasks. In [23] extension of PNN was proposed, where expanding
neural network is combined with selective retraining. Firstly network fine-
tunes selected parameters that have already been learned. New layers are added
during training if the resulting loss is greater than the defined threshold.
L1 normalization is used to introduce sparsity into the network. In [24] NAS
framework is utilized to modify network structure for life-long learning. For
every new task, each layer is either reused with no changes, fine-tuned or
expanded with new parameters.

##  III Method

We propose a gradual increase of neural network depth when a new task is
presented by adding new layers on top of already trained ones. We may divide
layers into three categories: layers with common weights for all tasks (common
backbone), layers used by current and future tasks, and layers specific for
the current task (please refer to Fig. 1). Backbone contains pre-trained
weights from ImageNet. The first task is trained with a standard neural
network structure (Fig. 1A). When a new task appears, we create new common and
task-specific layers. We will refer to this structure as a node. A new node is
attached to the previously shared layer or common backbone (Fig. 1B). Weights
from previous tasks (including previously added common layers) and backbone
are frozen during training. Only new layers added for the current task are
trained. Due to this design, we may obtain simultaneous full knowledge
retention and forward knowledge transfer. Multiple nodes could be attached to
an existing node or backbone. The network grows a tree-like structure, with
each path from the root to the leaf containing the parameters for the specific
task. Example structure learned by our network is presented in Fig. 2.

Figure 1: Stacking layers for new tasks. (A) Standard neural network
structure. Rectangles denote convolutional layers, trapeze convolutions with
stride 2 or pooling operations, wide rectangles at the end of network denotes
classifiers. Layers are divided into 3 groups: common backbone for all tasks,
common layers that future tasks can be attached to, and task-specific layers
with separate classifier. (B) We add new layers by attaching them directly to
backbone or to previous shared weights.

Figure 2: Example structure learned by network for 10 tasks.

The critical element of the proposed method is the selection of where new
layers should be attached. Adding a new node directly to the latest learned
weights can be sub-optimal, as the representation learned for the previous
task may not be suitable for current data. For this reason, we utilize the
selection of the most similar task with entropy. Let us assume that a new task
$t$ with the dataset $D_{t}$ is given. The entropy $H$ of outputs obtained
from neural network $f$ with weights $\theta$ learned for previous tasks $i<t$
is

$H_{i}=-\frac{1}{|D_{i}|}\sum_{x\in D_{i}}f(\theta_{i},x)logf(\theta_{i},x)$
(1)

The maximum entropy value can vary depending on the number of classes for the
$i$th task, but the number of classes for each task could be different.
Therefore, we cannot directly compare entropy for different paths in our
network. To unify, we compute the maximum value of entropy for a given task:

$H_{imax}=-\sum_{j=0}^{C_{i}}\frac{1}{C_{i}}log\frac{1}{C_{i}}$ (2)

where $C_{i}$ is number of classes for task $i$. Then to compare the task
similarity of each previous task $i$ to the new one, we use the ratio
$\frac{H_{i}}{H_{imax}}$ with value in $[0,1]$ interval, regardless of the
number of classes for each task. We select the most similar task index $l$ as:

$l=argmin_{i}\\{\frac{H_{i}}{H_{imax}}:\frac{H_{i}}{H_{imax}}<t\\}$ (3)

where $t$ is a threshold for maximal entropy value. New layers are directly
attached to the backbone if all entropy ratio values exceed $t$ or otherwise
to node $l$.

The proposed algorithm for tree growing increases the time complexity of the
proposed method, as for each new task, we must compute entropy for previous
nodes. It can be optimized by utilizing tree structure and storing
intermediate activations that can be later passed to multiple nodes. Another
alternative is to use a multi GPU system, as entropy calculation can be
trivially paralleled to multiple processes. During training, time complexity
can be written as $O(f_{B}+df_{S}+f_{T})$, where $O(f_{B})$ is the time
complexity of backbone training (forward and backward pass), $d$ is the depth
of the current path, $O(f_{S})$ is the complexity of the shared part and
$O(f_{T})$ is the complexity of the task-specific part. It means that crucial
for time complexity is the depth of the current path $d$. If a tree is wider,
$d$ is smaller. The width of a tree can be controlled to some extent by
changing hyperparameter $t$.

###  III-A Very deep neural networks

Model depth cannot be increased indefinitely. For implementation reasons, let
us note that very deep models cannot be fitted into single GPU memory, and
additionally, depth increasing may cause overfitting. In general, the first
problem could be solved by using model parallelism [6] and assigning different
parts of networks to different GPUs. Here we may simply solve this problem by
introducing the depth limit. When the network grows to maximum depth, the
latest nodes are not considered when selecting where new layers should be
attached. We avoid the second problem by utilization of stochastic depth [14].
It also acts as a form of normalization. Our model is built with ResNet
stochastic depth blocks. This should allow for training networks with
thousands of layers.

###  III-B Batch normalization

Batch normalization [15] is used in many modern neural networks [13, 38, 43,
36, 39]. This layer computes activation statistics heavily dependent on data
distribution and contains learnable parameters that interact with activation
statistics. In a life-long learning scenario, data distribution varies for
each task. In the preliminary experiment, we noticed that this could impact
our model behavior in the long term. Also, the performance of neural networks
with batch normalization is dependent on batch size [41]. With increased
depth, smaller batch size can be required to fit the model into available GPU
memory. For this reason, we swap batch normalization layers in our model to
instance normalization [40].

##  IV Experiments

We conduct broad experiments with multiple datasets and baselines used in
continual learning. Pretrained models were utilized for all life-long learning
methods to ensure a fair evaluation and explore how pretraining impacts
performance. We further examine what parts of our solution are important by
performing ablation studies.

As a result of the experimental research, the following research questions
(RQ) are to be answered:

  1. 1.

Could network with gradual increasing depth be used in continual learning?

  2. 2.

What is the benefit of utilizing pretrained architectures in life-long
learning?

  3. 3.

How does the proposed method of building tree structure compare to other
approaches?

  4. 4.

What is the performance of our algorithm in a multi-dataset environment?

Code of our experiments was implemented with Pytorch [30] and Avalanche [26]
libraries, and is available online 111https://github.com/w4k2/stochastic-
depth-data-streams.

###  IV-A Dataset

We chose commonly used benchmarks for continual learning:

  * •

Permuted MNIST [17] created from original MNIST [22] dataset. New tasks are
created by applying pixel permutations to MNIST images. We used this benchmark
with 50 tasks to evaluate long-term knowledge retention.

  * •

Split CIFAR100 [19] created by splitting CIFAR100 dataset [20] into 20
different tasks, 5 classes for each task.

  * •

Tiny ImageNet dataset [21] with 20 tasks and 5 classes for each task.

For experiments with multiple datasets MNIST [22], CIFAR10 [19] and Fashion
MNIST [42] were used. Images were resized to 64x64 pixels for models with the
ResNet base model. In the case of Cifar and Tiny ImageNet datasets, horizontal
flip transform was used.

###  IV-B Baselines

We chose several commonly used baselines in our experiments.

  * •

As a higher bound of performance for convolutional models, we used separate
models trained on different tasks for Permuted MNIST and models trained with
data from all tasks seen so far for CIFAR100.

  * •

Experience Replay (ER) - a method that creates buffers with samples from
previous tasks.

  * •

A-GEM - a more advanced version of replay.

  * •

PNN is the most similar method to ours that expands the network with each
task.

  * •

EWC - the regularization-based method.

  * •

LwF - the method that uses pseudo labels generated by heads for previous
tasks.

###  IV-C Architecture

We employed ResNet18 [13] with weights pretrained on ImageNet as a base model
for each method except for PNN. In the case of PNN we utilized a simpler MLP
base model with 4 layers, size of hidden layers equal to 256, and a smaller
image size (32 for CIFAR and 28 for MNIST). We trained replay, A-GEM, baseline
methods, and our method with Adam [16] optimizer, and LwF, EWC, and PNN with
SGD [32].

###  IV-D Hyperparameters

We performed grid search hyperparameter optimization with the following
parameters: learning rate, weight decay, and a number of epochs. Other
algorithm-specific parameters were found manually before the grid search. We
used the strength of regularization in EWC equal to 1000; we allowed memory
size for 250 patterns per task for A-GEM and ER, and distillation and
temperature equal to 1 for LwF. According to [4] and [8] we allowed
hyperparameter search only for the first three tasks, and reported final
results for all tasks. We trained networks for multiple epochs of learning, as
the number of learning examples in each task was usually too small to train
for only one epoch.

###  IV-E Results

We report average task accuracy after finished training (acc) and forgetting
measure (FM) [3]. Results of our experiments are presented in Tab. I. For all
datasets, the upper bound sets limit performance for convolutional models. In
the case of permuted MNIST, PNN obtains betters results than the upper bound.
We can explain this with the simpler architecture used in PNN compared to
ResNet18 used in upper. This also explains why other methods fail at Permuted
MNIST when used with ResNet. As indicated in [9] data distribution in Permuted
MNIST is not in line with convolutional networks inductive bias. Regardless of
this fact, our method, although based on ResNet, obtains good performance on
Permuted MNIST with zero forgetting. This is further illustrated in Fig. 3.
Across the whole learning process, both PNN and the proposed method closely
match the performance of the upper bound for ResNets. EWC, A-GEM, and LwF
instantly forget accumulated knowledge as new tasks arrive. Replay gradually
lose performance due to the limited size of the memory buffer. Some samples
already stored in memory must be dropped when a new task arrives. This causes
a slow fall in rehearsal effectiveness for many tasks.

For Cifar100 dataset proposed method has accuracy close to replay. All other
methods offer lower average accuracy with larger forgetting. The simpler
architecture used in PNN no longer offers an advantage when applied to the
dataset with more realistic data distribution. On Tiny ImageNet, our method
obtains the best performance. We know that the literature contains better
results for A-GEM, EWC, and LwF. However, we used the same pretrained backbone
for all baselines in our experiment. Pretraining in many environments provides
a boost to performance. Using it only with the proposed method would result in
unfair comparison, as our method would start with additional knowledge stored
in pretrained weights. For this reason, we equalize the comparison between our
method and baselines and the same time, study the impact of larger,
pretraining on the commonly used life-long learning methods.

Figure 3: Cumulative accuracy for Permuted MNIST dataset. TABLE I: Results for pMNIST, CIFAR100, and Tiny ImageNet datasets Method | pMNIST | Cifar100 | Tiny ImageNet
---|---|---|---
Acc | FM | Acc | FM | Acc | FM
upper | 0.9653 | 0 | 0.8637 | 0.0098 | 0.6739 | 0.0155
ewc [17] | 0.2096 | 0.7424 | 0.6396 | 0.2054 | 0.1 | 0.193
replay | 0.7195 | 0.2359 | 0.7973 | 0.0526 | 0.4361 | 0.1728
a-gem [4] | 0.1175 | 0.8507 | 0.514 | 0.2992 | 0.4145 | 0.2618
pnn [34] | 0.9839 | 0 | 0.6956 | 0 | 0.4354 | 0
lwf [25] | 0.1689 | 0.7926 | 0.4591 | 0.418 | 0.2528 | 0.4539
ours | 0.9427 | 0 | 0.7935 | 0 | 0.5814 | 0

###  IV-F Ablation studies

To estimate what parts of our solution contribute most to obtained results, we
performed ablation studies on the Cifar100 dataset. First, we removed
pretraining. Next, instead of selecting node attachment place based on
entropy, we added each new node to the previously trained node (Sequential
structure). Lastly, we attached each node directly to the backbone to obtain a
parallel structure (similar to the MultiHead approach). Results are presented
in Tab. II. The drop in accuracy after removing ImageNet pretraining is
probably due low amount of training data available. Each task in the Split
Cifar benchmark has, on average only 2500 training samples. This is not enough
to train a convolutional neural network from scratch. When it comes to
selecting where a new node should be attached, our method is on par with the
parallel approach. This is surprising, as both sequential and parallel
structures could be obtained with a method based on entropy and, therefore,
they can be seen as special cases of the proposed approach. The proposed
method obtained better results than the sequential node structure.

TABLE II: Ablation study results. Variant | Accuracy
---|---
base | 0.7897
no pretraining | 0.6473
sequential node structure | 0.5707
parallel node structure | 0.6466

###  IV-G Multi-dataset benchmarks

We also tested our method in a more demanding environment where each task was
a separate dataset encompassing a different domain. We utilized Cifar10,
Fashion MNIST, and MNIST datasets. As the best performing baseline in previous
experiments, the Experience Replay was selected for comparison. We trained
both methods for 10 epochs with different task sequences:
Cifar10→MNIST→Fashion MNIST (C-M-F), MNIST→Fashion MNIST→Cifar10 (M-F-C),
Fashion MNIST→Cifar10→MNIST (C-M-F). Results are presented in Tab. III.
Interestingly replay obtains better performance when MNIST is the first task.
This is probably caused by the fact that MNIST is quite simple to learn, and
fewer samples are needed in memory to prevent forgetting. The proposed method
is insensitive to task order and performs better than the replay.

TABLE III: Results for multi-dataset setup Method | C-M-F | M-F-C | F-C-M
---|---|---|---
Acc | FM | Acc | FM | Acc | FM
replay | 0.7763 | 0.0985 | 0.8205 | 0.0192 | 0.7666 | 0.0828
ours | 0.8735 | 0 | 0.8728 | 0 | 0.875 | 0

##  V Lessons learned

The first part of our experiments proves that when correctly utilized,
increasing neural network depth could be beneficial for continual learning.
However, it is important to note that results obtained for some baselines with
Permuted MNIST, CIFAR100, and Tiny ImageNet are worse than the results present
in the literature. This is due to the utilization of larger pretrained
networks in our experiments; however, even if we consider better results from
the literature, the proposed method still performs well. This answers RQ1.
Worse results obtained for some baselines compared to literature suggest that
there is no particular gain from utilizing larger, pretrained architectures
with popular life-long learning methods. However, at the same time, our
ablation studies show that ImageNet pretraining is responsible for a
significant part of our method performance. This validates the utilization of
pretrained models in our experiments. With no special adjustments in the life-
long algorithm, knowledge stored in pretrained weights is quickly forgotten
and does not improve the performance in the long run. This answers RQ2.

Ablations also suggest that naïve sequential stacking of new layers does not
obtain good results. Utilization of task similarity is better. However, its
performance is comparable to the parallel structure, which does not
necessarily mean that our method is equivariant to simple multihead
architecture. A single node contains a classifier and several residual blocks
in our case. It calls for future research on a better method for building a
tree structure. This answers RQ3. Our algorithm performs better when the
number of learning examples per task is larger. This could be noticed in the
first experiment with Permuted MNIST and in a multi-dataset setup, where each
task contains 50000 examples for training. The proposed method can obtain
better results than Experience Reply in these cases. A larger number of
samples allows for more detailed training of our network, yet at the same
time, regardless of the number of samples, the built-in properties of our
algorithm ensure no forgetting. This answers RQ4.

##  VI Conclusion

We proposed a novel method for continual learning based on the increasing
depth of neural networks. We indicate where new layers should be attached by
computing the entropy of each head prediction for a new task. We created a
tree-like structure that can grow based on task similarity based on this
information. Our method is similar in principle to PNN; however, PNN allocates
a lot of memory for each task. The proposed method alleviates this by adding
only part of a network for a new task. At the same time, we may retain the
benefit of PNN, such as no forgetting, without the need for a memory buffer by
design.

Conducted experiments suggest that the introduced algorithm obtained good
results, especially for datasets with distribution close to real images. In a
more challenging setup with a single computer vision dataset as a separate
task, our method outperforms Experience Replay.

Future work may include adding the ability to select tasks in the class
incremental scenario, where no label of incoming task is provided. Also,
methods for tree structure pruning should be developed to limit memory usage.

## Acknowledgment

This work is supported by the CEUS-UNISONO programme, which has received
funding from the National Science Centre, Poland under grant agreement No.
2020/02/Y/ST6/00037.

## References

  * [1] Stevo Bozinovski.  Reminder of the first paper on transfer learning in neural networks, 1976\.  Informatica (Slovenia), 44, 2020.
  * [2] Andrew Bremner, David Lewkowicz, and Charles Spence.  Multisensory development, 11 2013.
  * [3] Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  CoRR, abs/1801.10112, 2018.
  * [4] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with A-GEM.  CoRR, abs/1812.00420, 2018.
  * [5] Zhiyuan Chen, Bing Liu, Ronald Brachman, Peter Stone, and Francesca Rossi.  Lifelong Machine Learning.  Morgan & Claypool Publishers, 2nd edition, 2018.
  * [6] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng.  Large scale distributed deep networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  * [7] Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar.  Learning in nonstationary environments: A survey.  IEEE Computational Intelligence Magazine, 10(4):12–25, 2015.
  * [8] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach.  Uncertainty-guided continual learning with bayesian neural networks.  CoRR, abs/1906.02425, 2019.
  * [9] Sebastian Farquhar and Yarin Gal.  Towards robust evaluations of continual learning, 2019.
  * [10] Robert French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3:128–135, 05 1999.
  * [11] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  CoRR, abs/1311.2524, 2013.
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  CoRR, abs/1703.06870, 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [14] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.  Deep networks with stochastic depth.  CoRR, abs/1603.09382, 2016.
  * [15] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  CoRR, abs/1502.03167, 2015.
  * [16] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 12 2014.
  * [17] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  CoRR, abs/1612.00796, 2016.
  * [18] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Large scale learning of general visual representations for transfer.  CoRR, abs/1912.11370, 2019.
  * [19] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * [20] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  University of Toronto, 05 2012.
  * [21] Ya Le and Xuan S. Yang.  Tiny imagenet visual recognition challenge.  2015\.
  * [22] Yann LeCun and Corinna Cortes.  The mnist database of handwritten digits.  2005\.
  * [23] Jeongtae Lee, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  CoRR, abs/1708.01547, 2017.
  * [24] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  CoRR, abs/1904.00310, 2019.
  * [25] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  CoRR, abs/1606.09282, 2016.
  * [26] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni.  Avalanche: an end-to-end library for continual learning.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Workshop, 2021.
  * [27] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continuum learning.  CoRR, abs/1706.08840, 2017.
  * [28] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Architecture matters in continual learning, 2022.
  * [29] German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  CoRR, abs/1802.07569, 2018.
  * [30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [31] Krishnan Raghavan and Prasanna Balaprakash.  Formalizing the generalization-forgetting trade-off in continual learning.  CoRR, abs/2109.14035, 2021.
  * [32] Sebastian Ruder.  An overview of gradient descent optimization algorithms.  arXiv preprint arXiv:1609.04747, 2016.
  * [33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  CoRR, abs/1409.0575, 2014.
  * [34] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  CoRR, abs/1606.04671, 2016.
  * [35] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv 1409.1556, 09 2014.
  * [36] Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke.  Inception-v4, inception-resnet and the impact of residual connections on learning.  CoRR, abs/1602.07261, 2016.
  * [37] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  CoRR, abs/1512.00567, 2015.
  * [38] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  CoRR, abs/1905.11946, 2019.
  * [39] Mingxing Tan and Quoc V. Le.  Efficientnetv2: Smaller models and faster training.  CoRR, abs/2104.00298, 2021.
  * [40] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  CoRR, abs/1607.08022, 2016.
  * [41] Yuxin Wu and Kaiming He.  Group normalization.  CoRR, abs/1803.08494, 2018.
  * [42] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  CoRR, abs/1708.07747, 2017.
  * [43] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  CoRR, abs/1611.05431, 2016.
  * [44] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  CoRR, abs/1311.2901, 2013.
  * [45] Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning.  CoRR, abs/1611.01578, 2016.
  * [46] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.  Learning transferable architectures for scalable image recognition.  CoRR, abs/1707.07012, 2017.