# ClamNet: Using contrastive learning with variable depth Unets for medical
image segmentation

Samayan Bhattacharya
Department of Computer Science and Engineering
Jadavpur University
Kolkata, India
<EMAIL_ADDRESS>
Sk Shahnawaz∗
Department of Computer Science and Engineering
Jadavpur University
Kolkata, India
<EMAIL_ADDRESS>
Avigyan Bhattacharya∗
Department of Computer Science and Engineering
Jadavpur University
Kolkata, India
<EMAIL_ADDRESS>

(June 9, 2022)

###### Abstract

Unets have become the standard method for semantic segmentation of medical
images, along with fully convolutional networks (FCN). Unet++ was introduced
as a variant of Unet, in order to solve some of the problems facing Unet and
FCNs. Unet++ provided networks with an ensemble of variable depth Unets, hence
eliminating the need for professionals estimating the best suitable depth for
a task. While Unet and all its variants, including Unet++ aimed at providing
networks that were able to train well without requiring large quantities of
annotated data, none of them attempted to eliminate the need for pixel-wise
annotated data altogether. Obtaining such data for each disease to be
diagnosed comes at a high cost. Hence such data is scarce. In this paper we
use contrastive learning to train Unet++ for semantic segmentation of medical
images using medical images from various sources including magnetic resonance
imaging (MRI) and computed tomography (CT), without the need for pixel-wise
annotations. Here we describe the architecture of the proposed model and the
training method used. This is still a work in progress and so we abstain from
including results in this paper. The results and the trained model would be
made available upon publication or in subsequent versions of this paper on
_arxiv_.

**footnotetext: These authors contributed equally to this work

_K_ eywords semantic segmentation $\cdot$ fully convolutional networks $\cdot$
contrastive learning

##  1 Introduction

Figure 1: Different stages of image transformation by the proposed method from
input chest X-ray image (on the left) to output segmentation (on the right)

Image segmentation is highly important for medical images as it allows human
professionals to deal with the more challenging patients, leaving the ordinary
patients to the automatic system. As opposed to classification, segmentation
provides some insight into how the system is analyzing the image and hence is
more reliable for applications at scale.

Encoder-decoder networks are widely used for semantic segmentation of images
[1, 2, 2, 3, 4, 5]. Unet [75] introduced skip connections between the encoder
and the decoder. This improved performance of the network by allowing the
convolutional layers in the decoder to access the fine-grained feature maps
from the shallow layers of the encoder. However, the appropriate depth of the
network for a particular application had to be determined empirically or
heuristically, leading to a waste of resources and suboptimal results. The
standard way to address this issue was to train networks of different depths
and take an ensemble of these at inference time [6, 7, 8]. However, this
practice was inefficient while deploying at scale [9, 10]. Another issue was
that the feature maps had to be of the same size to be combined. However,
there was no theoretical guarantee that this was the most efficient
combination.

Unet++ [75] was introduced with a built-in ensemble of Unets of different
depths and was able to combine feature maps of different sizes. Hence it was
able to perform better than classical Unets.It achieved these by including
additional convolutional units in between the encoder and decoder parts of the
standard Unet architecture.This also allowed the model to be pruned to improve
efficiency without adversely affecting performance. However, it still relied
heavily on pixel-wise annotated data to train. Such annotations are tediously
obtained at the cost of a lot of man-hours and financial expenditure. Hence,
large pixel-wise annotated datasets of medical images are hard to come by.
Datasets with image specific labels are much more easily available. However,
these are used for classification rather than semantic segmentation of images.
Classification networks work as a black box and the only way to gain insights
into what part of the image the network focussed on, while classifying it as
belonging to a particular class, can only be obtained by using tools like
gradcam [76]. Such tools analyze the gradient map of a particular
convolutional layer and indicates the region of interest. For medical images,
the region of interest may or may not be restricted to the abnormality we are
looking for in an image to diagnose an underlying medical condition. For
example, for diagnosing pneumonia using CT images, a patch might occur in
different parts of the lung and hence the whole lung would be the region of
interest for the network. This does not indicate if there is a patch or where
it is.

Contrastive learning [73] is commonly used for self-supervised learning. It
works by making sure that the pixels with same label produce the same output
in two identical models, while the pixels with different labels produce
different outputs. In this paper, we use contrastive learning in an
unsupervised setting.

In this paper we propose to use contrastive learning to train a pair of unet++
models using labelled images without pixel-wise annotations. For this purpose,
we use a probabilistic loss term. We use our model on different types of
medical images, including X-ray, magnetic resonance imaging (MRI) and computed
tomography (CT). We also test our model for the diagnosis of different medical
conditions, including pneumonia, carcinoma, trauma, etc. Preliminary results
are promising and we would present a comprehensive study upon publication of
our work or in subsequent versions of this paper on _arxiv_.

##  2 Related works

###  2.1 Self-supervised contrastive learning

This approach works by contrasting the outputs of two identical models for
positive pairs (pairs of inputs belonging t the same class) and negative
pairs. In recent years several approaches have used contrastive loss [73] for
learning representations of visual information [41, 42, 43, 44, 45, 46, 48].
These approaches use data augmentation techniques to generate positive pairs
and contrast them with other instances. Some approaches use hard negative
mining techniques [54, 55]. Some approaches used data banks to store
representations as performance is seen to improve with increase in number of
negative pairs [44, 53, 46]. Some approaches have extended its use to semantic
segmentation of images by classification of individual pixels[47, 49, 50, 51,
52] and by using large datasets of pixel-wise annotate images.

###  2.2 Semantic segmentation

Convolutional Neural Networks (CNNs) have been used for segmentation of images
for a long time [56, 57]. Such models have undergone incremental improvements
by using more convolutional layers in deeper networks [58, 59, 60, 61, 62, 63,
64, 65, 66]. An alternative technique is pretraining the model on a large
dataset, for example the ImageNet classification dataset or weak supervision
techniques like unlabeled images [4, 11, 12, 13, 14, 15], bounding boxes [16,
17, 18, 19], image-level labels [1, 22, 23, 18] or points [2] and scribbles
[20, 21].

Another approach learns pixel relations by using region-based loss function
[70, 71]. A region Mutual Information (MI) loss is used to maximize the MI
between patch label distributions [71]. A pairwise affinity loss based on KL
divergence between the probabilities, predicted for belonging to each class,
is used by [70]. Some works by [67, 68, 69, 74] performed instance and
semantic segmentation by using metric learning based on similarity and
dissimilarity pairs. The work by [72] trained the feature extractor of a
semantic segmentation model by maximizing log likelihood of an extracted
feature under several vMF distribution models. At the time of inference, they
used K-means clustering to segment the pixel features and used K nearest
neighbours to get labels from available segments.

###  2.3 Unet++

Since Unet++ was first introduced by [35], it has been used in multiple ways.
[36, 37, 38, 39] used it as a baseline, while [29, 30, 31, 32, 33, 34]
proposed models inspired by the Unet++ architecture. It was used for natural
image segmentation by [26], for biomedical image segmentation by [24, 25] and
for satellite image segmentation [27, 28]. It has also been used for contact
prediction by [15].

##  3 Proposed model

Figure 2: Proposed model architecture. The shape resembles a clam. The name
might be altered at a later stage of our work.

###  3.1 Unet++ architecture

We use a classical unet++ architecture with 5 levels of convolutional layers.
The input image is of size 256x256. The subsequent feature maps have double
the number of channels of the layer above it and length and breadth dimensions
are reduced by half, using stride=2 for the second convolution of the two
convolutions at each level. The kernel size is higher in lower levels allowing
more context information. The intermediate convolutions are taken according to
the architecture of the original unet++.

As an alteration of the above approach, we try having two subsequent layers
with the same dimensions of feature map. We make this change for randomly
selected levels. Initial results indicate that this approach gives better
results than a deeper unet++.

###  3.2 Preprocessing

In this study we use labelled images with image-wise annotations instead of
pixel-wise annotations. The images are then passed to an image segmentation
network that is trained to detect the organ of interest. Segmentation of
organs is much easier due to easy availability of data. This is because humans
have few organs but each organ has many diseases. After segmentation, the
image is cropped such that it contains minimum background. This process is
helpful for the probabilistic loss function we describe later in this paper.
Each image is resized to 256x256 pixels.

###  3.3 Contrastive training

We take two identical models, as indicated above.These are trained
simultaneously on positive pairs and negative pairs of slices of the image.
The target is to have both networks produce the same output for positive pairs
and different outputs for negative pairs.

Positive pair is a pair of slices belonging to the same class label. We
generate a positive pair by (i) blurring the image (such that resultant image
has a resolution between 90 to 100 percent of the original resolution) (ii)
stretch and compression distortions (such that resultant image has dimensions
between 80 to 100 percent of the original resolution), followed by resizing
the image to the original size or padding it to the original size. Apart from
these, we also generate positive pairs by taking slices from images labelled
as normal (negative for the medical condition to be diagnosed).

Negative pairs are pairs of slices that belong to different label classes.
These are generated by taking a slice from a positive image and one from a
negative image. Since the whole of the positive image does not contain the
features of the medical condition to be diagnosed, we use the probabilistic
loss function.

###  3.4 Probabilistic loss function

Since we are trying to produce pixel-wise annotated images by training the
model on image-wise annotated images, there is no way to be sure which part of
the image has markers of the medical condition to be diagnosed. But we
increase the probability of the marker being in a given region of the image by
segmenting the organ which is supposed to be affected and removing the
background. Now, we consider the probability of a marker being present in a
given region of the image while calculating the loss.

Mathematically this is given as,

$Hybrid\\_loss(Y,P)=-\frac{1}{N}.\sum_{c=1}^{C}\sum_{n=1}^{N}(y_{n,c}.logp_{n,c}+\frac{y_{n,c}.p_{n,c}}{y_{n,c}^{2}+p_{n,c}^{2}})$
(1)

where $y_{n,c}\in Y$, representing the target labels and $p_{n,c}\in P$
representing the predicted probabilities for class c and $n^{th}$ pixel in the
batch, where $n\in N$, representing the number of pixels in one batch.

The total loss of the model is the weighted sum of the individual hybrid
losses, given by $\sum_{i=1}^{d}(\eta.Hybrid\\_loss(Y,P^{i}))$, where d is the
number of slices in the batch and $\eta$ is the weight that is the probability
of the slice containing the marker for the medical condition to be diagnosed.
If this probability is not known for a dataset, it is treated as a
hyperparameter, to be tuned.

##  4 Conclusion

In this paper we proposed a novel method for using contrastive learning, in an
unsupervised setting, to train a UNET++ architecture. We are in the process of
trying out different methods and testing them on different datasets. This
paper is intended to provide an overview of the method. We do not make any
claims about its performance in this paper. We plan to provide a comprehensive
report of our final model, datasets used to train it and a comparative study
of the results obtained for our model versus other commonly used
architectures, upon publication of our work or in subsequent versions of this
paper on _arxiv_.

## References

  * [1] Shen, D., Wu, G. and Suk, H.I., 2017. Deep learning in medical image analysis. Annual review of biomedical engineering, 19, p.221.
  * [2] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B. and Sánchez, C.I., 2017. A survey on deep learning in medical image analysis. Medical image analysis, 42, pp.60-88.
  * [3] Chartrand, G., Cheng, P.M., Vorontsov, E., Drozdzal, M., Turcotte, S. and Pal, C.J., Deep Learning: A Primer for Radiologists. RadioGraphics [Internet]. 2017; 37 (7): 2113–31.
  * [4] Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K. and Dovzhenko, A., 2019. U-Net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1), pp.67-70.
  * [5] Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z. and Ding, X., 2020. Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis, 63, p.101693.
  * [6] Dietterich, T.G., 2000, June. Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Springer, Berlin, Heidelberg.
  * [7] Shin, H.C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D. and Summers, R.M., 2016. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE transactions on medical imaging, 35(5), pp.1285-1298.
  * [8] Ciompi, F., de Hoop, B., van Riel, S.J., Chung, K., Scholten, E.T., Oudkerk, M., de Jong, P.A., Prokop, M. and van Ginneken, B., 2015. Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2D views and a convolutional neural network out-of-the-box. Medical image analysis, 26(1), pp.195-202.
  * [9] Bengio, Y., 2009. Learning deep architectures for AI. Now Publishers Inc.
  * [10] Zhang, Y. and Yang, Q., 2021. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering.
  * [11] Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.J., Fei-Fei, L., Yuille, A., Huang, J. and Murphy, K., 2018. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV) (pp. 19-34).
  * [12] Long, J., Shelhamer, E. and Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440).
  * [13] Chaurasia, A. and Culurciello, E., 2017, December. Linknet: Exploiting encoder representations for efficient semantic segmentation. In 2017 IEEE Visual Communications and Image Processing (VCIP) (pp. 1-4). IEEE.
  * [14] Yu, F., Wang, D., Shelhamer, E. and Darrell, T., 2018. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2403-2412).
  * [15] Shenoy, A., 2019. Feature optimization of contact map predictions based on inter-residue distances and U-Net++ architecture. no. July.
  * [16] He, K., Gkioxari, G., Dollár, P. and Girshick, R., 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969).
  * [17] Meyer, M.G., Hayenga, J.W., Neumann, T., Katdare, R., Presley, C., Steinhauer, D.E., Bell, T.M., Lancaster, C.A. and Nelson, A.C., 2015. The Cell-CT 3-dimensional cell imaging technology platform enables the detection of lung cancer using the noninvasive LuCED sputum test. Cancer cytopathology, 123(9), pp.512-523.
  * [18] Zhao, H., Qi, X., Shen, X., Shi, J. and Jia, J., 2018. Icnet for real-time semantic segmentation on high-resolution images. In Proceedings of the European conference on computer vision (ECCV) (pp. 405-420).
  * [19] Jiang, J., Hu, Y.C., Liu, C.J., Halpenny, D., Hellmann, M.D., Deasy, J.O., Mageras, G. and Veeraraghavan, H., 2018. Multiple resolution residually connected feature streams for automatic lung tumor segmentation from CT images. IEEE transactions on medical imaging, 38(1), pp.134-144.
  * [20] Simonyan, K. and Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  * [21] Zhu, Q., Du, B., Turkbey, B., Choyke, P.L. and Yan, P., 2017, May. Deeply-supervised CNN for prostate segmentation. In 2017 international joint conference on neural networks (IJCNN) (pp. 178-184). IEEE.
  * [22] Dou, Q., Yu, L., Chen, H., Jin, Y., Yang, X., Qin, J. and Heng, P.A., 2017. 3D deeply supervised network for automated segmentation of volumetric medical images. Medical image analysis, 41, pp.40-54.
  * [23] Kistler, M., Bonaretti, S., Pfahrer, M., Niklaus, R. and Büchler, P., 2013. The virtual skeleton database: an open access repository for biomedical research and collaboration. Journal of medical Internet research, 15(11), p.e2930.
  * [24] Zyuzin, V., Sergey, P., Mukhtarov, A., Chumarnaya, T., Solovyova, O., Bobkova, A. and Myasnikov, V., 2018, May. Identification of the left ventricle endocardial border on two-dimensional ultrasound images using the convolutional neural network Unet. In 2018 Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) (pp. 76-78). IEEE.
  * [25] Cui, H., Liu, X. and Huang, N., 2019, October. Pulmonary vessel segmentation based on orthogonal fused u-net++ of chest CT images. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 293-300). Springer, Cham.
  * [26] Sun, K., Xiao, B., Liu, D. and Wang, J., 2019. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5693-5703).
  * [27] Peng, D., Zhang, Y. and Guan, H., 2019. End-to-end change detection for high resolution satellite images using improved UNet++. Remote Sensing, 11(11), p.1382.
  * [28] Zhang, Y., Gong, W., Sun, J. and Li, W., 2019. Web-Net: A novel nest networks with ultra-hierarchical sampling for building extraction from aerial imageries. Remote Sensing, 11(16), p.1897.
  * [29] Zhang, J., Jin, Y., Xu, J., Xu, X. and Zhang, Y., 2018. Mdu-net: Multi-scale densely connected u-net for biomedical image segmentation. arXiv preprint arXiv:1812.00352.
  * [30] Chen, F., Ding, Y., Wu, Z., Wu, D. and Wen, J., 2018, December. An improved framework called Du++ applied to brain tumor segmentation. In 2018 15th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) (pp. 85-88). IEEE.
  * [31] Zhou, C., Chen, S., Ding, C. and Tao, D., 2018, September. Learning contextual and attentive information for brain tumor segmentation. In International MICCAI brainlesion workshop (pp. 497-507). Springer, Cham.
  * [32] Wu, S., Wang, Z., Liu, C., Zhu, C., Wu, S. and Xiao, K., 2019, July. Automatical segmentation of pelvic organs after hysterectomy by using dilated convolution u-net++. In 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 362-367). IEEE.
  * [33] Song, T., Meng, F., Rodriguez-Paton, A., Li, P., Zheng, P. and Wang, X., 2019. U-next: A novel convolution neural network with an aggregation u-net architecture for gallstone segmentation in ct images. IEEE Access, 7, pp.166823-166832.
  * [34] Yang, C. and Gao, F., 2019, October. EDA-Net: dense aggregation of deep and shallow information achieves quantitative photoacoustic blood oxygenation imaging deep in human breast. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 246-254). Springer, Cham.
  * [35] Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N. and Liang, J., 2018. Unet++: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support (pp. 3-11). Springer, Cham.
  * [36] Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W. and Wang, J., 2019. High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.04514.
  * [37] Fang, Y., Chen, C., Yuan, Y. and Tong, K.Y., 2019, October. Selective feature aggregation network with area-boundary constraints for polyp segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 302-310). Springer, Cham.
  * [38] Fang, J., Zhang, Y., Xie, K., Yuan, S. and Chen, Q., 2019, October. An improved mpb-cnn segmentation method for edema area and neurosensory retinal detachment in sd-oct images. In International Workshop on Ophthalmic Medical Image Analysis (pp. 130-138). Springer, Cham.
  * [39] Meng, C., Sun, K., Guan, S., Wang, Q., Zong, R. and Liu, L., 2020. Multiscale dense convolutional neural network for DSA cerebrovascular segmentation. Neurocomputing, 373, pp.123-134..
  * [40] Hadsell, R., Chopra, S. and LeCun, Y., 2006, June. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) (Vol. 2, pp. 1735-1742). IEEE.
  * [41] Chen, T., Kornblith, S., Norouzi, M. and Hinton, G., 2020, November. A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.
  * [42] Chen, T., Kornblith, S., Swersky, K., Norouzi, M. and Hinton, G.E., 2020. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33, pp.22243-22255.
  * [43] Dosovitskiy, A., Springenberg, J.T., Riedmiller, M. and Brox, T., 2014. Discriminative unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27.
  * [44] He, K., Fan, H., Wu, Y., Xie, S. and Girshick, R., 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9729-9738).
  * [45] Li, J., Zhou, P., Xiong, C. and Hoi, S.C., 2020. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966.
  * [46] Wu, Z., Xiong, Y., Yu, S.X. and Lin, D., 2018. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3733-3742).
  * [47] Chaitanya, K., Erdil, E., Karani, N. and Konukoglu, E., 2020. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33, pp.12546-12558.
  * [48] Ye, M., Zhang, X., Yuen, P.C. and Chang, S.F., 2019. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6210-6219).
  * [49] O Pinheiro, P.O., Almahairi, A., Benmalek, R., Golemo, F. and Courville, A.C., 2020. Unsupervised learning of dense visual representations. Advances in Neural Information Processing Systems, 33, pp.4489-4500.
  * [50] Wang, X., Zhang, R., Shen, C., Kong, T. and Li, L., 2021. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3024-3033).
  * [51] Ding, J., Xie, E., Xu, H., Jiang, C., Li, Z., Luo, P. and Xia, G.S., 2021. Unsupervised pretraining for object detection by patch reidentification. arXiv preprint arXiv:2103.04814.
  * [52] Xie, Z., Lin, Y., Zhang, Z., Cao, Y., Lin, S. and Hu, H., 2021. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16684-16693).
  * [53] Tian, Y., Krishnan, D. and Isola, P., 2020, August. Contrastive multiview coding. In European conference on computer vision (pp. 776-794). Springer, Cham.
  * [54] Kalantidis, Y., Sariyildiz, M.B., Pion, N., Weinzaepfel, P. and Larlus, D., 2020. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33, pp.21798-21809.
  * [55] Robinson, J., Chuang, C.Y., Sra, S. and Jegelka, S., 2020. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592.
  * [56] Farabet, C., Couprie, C., Najman, L. and LeCun, Y., 2012. Learning hierarchical features for scene labeling. IEEE transactions on pattern analysis and machine intelligence, 35(8), pp.1915-1929.
  * [57] Long, J., Shelhamer, E. and Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440).
  * [58] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L., 2014. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062.
  * [59] Chen, L.C., Papandreou, G., Schroff, F. and Adam, H., 2017. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.
  * [60] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H., 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 801-818).
  * [61] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z. and Lu, H., 2019. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3146-3154).
  * [62] Vemulapalli, R., Tuzel, O., Liu, M.Y. and Chellapa, R., 2016. Gaussian conditional random field network for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3224-3233).
  * [63] Yuan, Y., Chen, X. and Wang, J., 2020, August. Object-contextual representations for semantic segmentation. In European conference on computer vision (pp. 173-190). Springer, Cham.
  * [64] Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X. and Wang, J., 2018. Ocnet: Object context network for scene parsing. arXiv preprint arXiv:1809.00916.
  * [65] Zhang, H., Zhang, H., Wang, C. and Xie, J., 2019. Co-occurrent features in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 548-557).
  * [66] Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J., 2017. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2881-2890).
  * [67] De Brabandere, B., Neven, D. and Van Gool, L., 2017. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551.
  * [68] Fathi, A., Wojna, Z., Rathod, V., Wang, P., Song, H.O., Guadarrama, S. and Murphy, K.P., 2017. Semantic instance segmentation via deep metric learning. arXiv preprint arXiv:1703.10277.
  * [69] Fathi, A., Wojna, Z., Rathod, V., Wang, P., Song, H.O., Guadarrama, S. and Murphy, K.P., 2017. Semantic instance segmentation via deep metric learning. arXiv preprint arXiv:1703.10277..
  * [70] Ke, T.W., Hwang, J.J., Liu, Z. and Yu, S.X., 2018. Adaptive affinity fields for semantic segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 587-602).
  * [71] SZhao, S., Wang, Y., Yang, Z. and Cai, D., 2019. Region mutual information loss for semantic segmentation. Advances in Neural Information Processing Systems, 32..
  * [72] Hwang, J.J., Yu, S.X., Shi, J., Collins, M.D., Yang, T.J., Zhang, X. and Chen, L.C., 2019. Segsort: Segmentation by discriminative sorting of segments. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 7334-7344).
  * [73] Hadsell, R., Chopra, S. and LeCun, Y., 2006, June. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) (Vol. 2, pp. 1735-1742). IEEE.
  * [74] Kong, S. and Fowlkes, C.C., 2018. Recurrent pixel embedding for instance grouping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9018-9028).
  * [75] Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N. and Liang, J., 2019. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE transactions on medical imaging, 39(6), pp.1856-1867.
  * [76] Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D. and Batra, D., 2016. Grad-CAM: Why did you say that?. arXiv preprint arXiv:1611.07450.


# VERSE: Virtual-Gradient Aware Streaming Lifelong Learning
with Anytime Inference

Soumya Banerjee1, Vinay K. Verma2, Avideep Mukherjee1, Deepak Gupta2, Vinay P.
Namboodiri3, Piyush Rai1 1 IIT Kanpur, India, {soumyab, avideep,
<EMAIL_ADDRESS>Amazon, India, {vinayugc, deepakgupta.cbs}@gmail
.com3 University Of Bath, UK<EMAIL_ADDRESS>
###### Abstract

Lifelong learning, also referred to as continual learning, is the problem of
training an AI agent continuously while also preventing it from forgetting its
previously acquired knowledge. Most of the existing methods primarily focus on
lifelong learning within a static environment and lack the ability to mitigate
forgetting in a quickly-changing dynamic environment. Streaming lifelong
learning is a challenging setting of lifelong learning with the goal of
continuous learning in a dynamic non-stationary environment without
forgetting. We introduce a novel approach to lifelong learning, which is
streaming, requires a single pass over the data, can learn in a class-
incremental manner, and can be evaluated on-the-fly (anytime inference). To
accomplish these, we propose virtual gradients for continual representation
learning to prevent catastrophic forgetting and leverage an exponential-
moving-average-based semantic memory to further enhance performance. Extensive
experiments on diverse datasets demonstrate our method’s efficacy and superior
performance over existing methods.

##  I INTRODUCTION

Continuous machine perception is crucial for AI agents to learn while
interacting with the environment, preventing catastrophic forgetting [1].
Lifelong Learning (LL) or Continual Learning (CL) [2, 3] methods are designed
with the goal to accomplish this. Recent CL research focuses mainly on static
environments [4, 5, 6, 7, 2], assuming large batch data and overlooking
changing data distribution and permitting multiple passes over the data to
facilitate CL. However, these approaches are not suitable for rapidly changing
dynamic environments. While there have been efforts to enable CL in online
settings [8], these methods have various limitations, such as batch data
requirements, lack of anytime-inference, and the need for large replay
buffers, limiting their applicability in Streaming Lifelong Learning (SLL) [9,
10, 11]. In SLL, the goal is to learn by observing each training example only
once without forgetting.

TABLE I: Baseline approaches are categorized based on simplifying assumptions. $\zeta(n)$ denotes the number of times the network visits the data for continual learning. It follows that $\zeta(n)\gg\zeta(2)>\zeta(1)$ with ’-’ indicating we are unable to find the exact value. Categories: B (Batch), O (Online), S (Streaming). Methods |  LWF [3] |  EWC++ [2, 12] |  MAS [4] |  SI [13] |  VCL [14] |  CVCL [14] |  GEM [15] |  AGEM [16] |  GDumb [8] |  TinyER [17] |  DER [18] |  DER++ [18] |  ExStream [9] |  REMIND [10] |  CLS-ER [19] |  VERSE  (Ours) |  CISLL Constraints
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Type | B | B | B | B | B | B | O | O | O | O | O | O | S | S | B | S |
Batch-Size $\mathbf{(N_{t})}$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}=1\;$ | $\;N_{t}=1\;$ | $\;N_{t}\gg 1\;$ | $\;N_{t}=1\;$ |
Fine-tuning | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
Single Pass | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓
Follows CIL | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Subset Replay | n/a | n/a | n/a | n/a | n/a | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓
Training Time | $\zeta(n)$ | $\zeta(n)$ | $\zeta(n)$ | $\zeta(n)$ | $\zeta(n)$ | $\zeta(n)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(n)$ | $\zeta(1)$ | $\zeta(1)$
Inference Time | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(n)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(n)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$ | $\zeta(1)$
Buffer Size | n/a | n/a | n/a | n/a | n/a | - | - | - | - | $\leq 5\%$ | $\leq 5\%$ | $\leq 5\%$ | $\leq 5\%$ | $\gg 10\%$ | $\leq 5\%$ | $\leq 5\%$ |
Doesn’t violate  CISLL | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ |

Below, we outline the key properties of SLL [9, 11]:

  * •

The AI agent observes each training example only once without storing it in
memory (desirable).

  * •

The agent is required to adapt to new sample(s) in a single pass (desirable).

  * •

The input data stream may exhibit temporal correlations, deviating from the
typical i.i.d pattern (essential).

  * •

The agent is required to be evaluated at any time (anytime inference) without
fine-tuning its parameters (essential).

  * •

The agent needs to perform class-incremental streaming lifelong learning
(CISLL), i.e., predict a class label from all the previously observed classes
(essential).

  * •

To make it practical, especially in resource-constrained environments, the
agent should minimize its memory requirements (desirable).

Existing CL approaches often make strong assumptions that violate one or more
key constraints required for SLL. Despite being desirable because of also
being closer to biological learning [10], SLL hasn’t received much attention.
The SLL setting is natural in real-world scenarios like home robots, smart
appliances, and drones, where AI agents must adapt quickly and continuously
without forgetting. Table I categorizes existing CL approaches based on their
underlying assumptions, revealing that only a few non-SLL methods can be
adapted to the SLL setting without violating the constraints. Notably,
ExStream [9], being an SLL method, violates subset replay in SLL by using all
past samples for CL. Non-SLL methods like TinyER [17] and DER/DER++ [18]
perform poorly when applied in the SLL setting.

Figure 1: SLL involves continuous learning from non-i.i.d. labeled streams
with multiple views without forgetting. This fig. shows temporally ordered cup
frames from CoRe50 [20].

We introduce Virtual GradiEnt AwaRe Streaming LEarning (VERSE), a rehearsal-
based CL model, facilitating CISLL in deep neural networks (DNNs). VERSE
_implicitly_ regularizes the network parameters by employing virtual gradient
updates, fostering robust representations that require minimal changes to
adapt to the new task/sample(s), preventing catastrophic forgetting. We also
utilize a small episodic buffer to store past samples, which are used for both
local/virtual and global parameter updates to perform a single-step virtual
gradient regularization, enabling CL. VERSE first adapts to a new example with
a _virtual_ (local) parameter update and generalizes to past samples with a
_global_ parameter update, promoting convergence between the two. This process
facilitates SLL allowing the AI agent to be evaluated on-the-fly (anytime
inference) without parameter fine-tuning with the stored samples.

Moreover, VERSE utilizes an exponential-moving-average [21, 22, 23, 24] based
semantic memory akin to long-term memory in mammalian brains [25, 26, 27].
Semantic memory is updated intermittently, consolidating new knowledge within
the agent’s parameters. It interacts with episodic memory, interleaving the
past predictions on stored buffer samples, minimizing self-distillation loss
[28, 29] to prevent forgetting and enhance the agent’s performance.

Experimental results on three temporally contiguous datasets show that VERSE
is effective in challenging SLL scenarios. It outperforms recent SOTA methods,
with ablations confirming the significance of its components.

In summary, our contributions are as follows:

  * •

We present a novel approach VERSE, a rehearsal-based virtual gradient
regularization framework, that incorporates both virtual and global parameter
updates to mitigate catastrophic forgetting in CISLL, enabling ’any-time-
inference’ without fine-tuning.

  * •

We propose a semantic memory based on an exponential-moving-average approach,
which enhances the agent’s overall performance.

  * •

Through empirical evaluations and ablation studies conducted on three
benchmark datasets with temporal correlations, we affirm the superiority of
VERSE over the existing SOTA methods.

##  II RELATED WORK

This section briefly summarizes different CL paradigms.

Task Incremental Learning (TIL). In TIL, the AI agent learns from task-
batches, observing samples related to specific tasks [30, 4, 12, 3, 2, 31, 13,
14, 32], each involving learning a few distinct classes. These methods rely on
knowing the task-identifier during inference; otherwise, it leads to severe
catastrophic forgetting [12].

Incremental Class Batch Learning (IBL). IBL, also referred to as class
incremental learning (CIL), assumes that the dataset is divided into batches,
each containing samples from different classes [12, 17, 33, 5, 34, 18, 19, 35,
36, 37, 38]. The AI agent observes and can loop over these batches in each
incremental session. During inference, the agent isn’t provided with task
labels and evaluated over all the observed classes.

Online Continual Learning (OCL). Unlike TIL and IBL, OCL involves an AI agent
sequentially observing and adapting to samples in a _single pass_ through the
entire dataset, avoiding catastrophic forgetting [8, 17, 39, 40, 16, 15].
While these methods enable continuous learning in dynamic environments, they
have various limitations: $(i)$ they require data in batches, assuming
$\forall t,|B_{t}|\gg 1$, where $B_{t}$ is a batch of samples at time $t$,
$(ii)$ they need fine-tuning before inference, lacking any-time-inference
ability (e.g., GDumb [8] being a superior OCL method, requires fine-tuning
model parameters with replay-buffer samples before each inference), $(iii)$
they require large replay buffers [10].

Streaming Lifelong Learning (SLL). SLL, a challenging variant of LL, enables
CL in a rapidly changing environment without forgetting [11, 9, 10]. It shares
similarities with OCL but has additional constraints: $(i)$ SLL limits the
batch size to one datum per incremental step, while OCL requires $|B_{t}|\gg
1$, $(ii)$ it doesn’t allow AI agent to fine-tune its parameters during
training or inference. Additionally, in SLL, the input data stream can be
temporally correlated in terms of class instance and instance ordering.
Detailed essential and desirable properties of SLL are discussed in Section I.

To our knowledge, ExStream [9], REMIND [10], and BaSiL [11] are the three
methods tackling the challenging SLL setting. However, it’s important to note
some of the key differences: ExStream [9] uses full-buffer replay, violating
the subset-replay constraint; REMIND [10] stores a large number of past
samples compared to other baselines (e.g., iCaRL [41] stores 10K past ImageNet
[42, 43] samples, whereas REMIND stores 1M samples); BaSiL [11] focuses on
Bayesian methods for SLL and relies entirely on pretrained weights for visual
features in SLL. It does not adapt convolutional layers to sequential data and
only trains linear layers, potentially posing severe challenges with
non-i.i.d. data. In contrast, our approach (VERSE) adheres to the SLL
constraints, stores a limited number of past samples, replays only a subset of
buffer samples, and trains both convolutional and fully connected (FC) layers
for SLL.

In this paper, we introduce VERSE, adhering to CISLL constraints, enabling CL
in challenging SLL settings. We compare VERSE with SLL frameworks, REMIND [10]
and ExStream [9], as well as various OCL and IBL methods.

##  III Proposed Approach

In this section, we introduce the proposed approach VERSE (Fig. 2), which
trains the CNN architecture in CISLL setup. The model, consisting of the
parameters $\Theta=\\{\xi,\theta\\}$, comprises of two components: $(i)$ a
non-plastic feature extractor $(G_{\xi})$ with parameter ${\xi}$, and $(ii)$ a
plastic neural network $(F_{\theta})$ with parameter $\theta$. $G_{\xi}$
includes the initial CNN layers, and $F_{\theta}$ encompasses the final few
layers, including the fully connected (FC) layers. The class label output for
a given input $\bm{x}$, is predicted as: $y=F_{\theta}(G_{\xi}(\bm{x}))$.

We focus on adapting the plastic network $F_{\theta}(\cdot)$ while keeping
non-plastic parameters $\xi$ frozen throughout. In each streaming incremental
step of CISLL, data arrives sequentially,
$\mathcal{D}_{t}=(\bm{x}_{t},y_{t})$, one datum at a time, and the learner
adapts without catastrophic forgetting [44, 45] by observing this datum only
once. The following section provides brief details of the proposed model.

Figure 2: In VERSE, Virtual-gradient-regularization (VGR) enables CL by
adapting to new sample(s) with a virtual model $(\theta^{v})$, which computes
the final model $(\theta)$ through rehearsal. Episodic memory (TEM) stores a
few observed samples, while Semantic memory (SEM) enforces consistency with
self-distillation loss, improving overall performance.

###  III-A Virtual Gradient as Regularizer

We denote by $\mathcal{D}_{1},\mathcal{D}_{2},\dots,\mathcal{D}_{T}$ the set
of tasks, with
${(\bm{z}_{t},y_{t})}={(G_{\xi}(\bm{x}_{t}),y_{t})}\sim\mathcal{D}_{t}$ with
$|\mathcal{D}{t}|=1$, arriving one-by-one in each incremental step. During the
training of the $t^{th}$ task only the data $\mathcal{D}_{t}$ is available,
the previous tasks’ data
$\mathcal{D}_{1},\mathcal{D}_{2}\dots\mathcal{D}_{t-1}$ are discarded, and we
only keep a few samples into a small memory buffer $\mathcal{M}$. Optimizing
network parameters with a single example is challenging; therefore, we select
a subset of samples with size $C$ from memory $\mathcal{M}$, that is,
$\mathcal{D}_{v}^{t}\sim\mathcal{M}$ with $|\mathcal{D}_{v}^{t}|=C$. We
combine the subset $\mathcal{D}_{v}^{t}$ with the new example
$\mathcal{D}_{t}$ to form a joint batch:
$\mathcal{D}_{v}^{t}\leftarrow\mathcal{D}_{t}\cup\mathcal{D}_{v}^{t}$ and
compute the cross-entropy loss, as defined below:

$\mathcal{L}^{t}_{v}=\mathbb{E}_{(\bm{z},y)\sim\mathcal{D}^{t}_{v}}\left[\mathcal{L}_{CE}(y,F_{\theta}(\bm{z}))\right]$
(1)

Let $\nabla_{\theta}\mathcal{L}^{t}_{v}(F_{\theta})$ be the gradient of the
loss (Eq. 1) w.r.t. the plastic network parameters $\theta$. With this
gradient, we compute the updated _local parameter_ as follows:

$\theta^{v}\leftarrow\theta-\alpha\nabla_{\theta}\mathcal{L}^{t}_{v}(F_{\theta})$
(2)

TABLE II: $\bm{\Omega}_{\text{all}}$ results. Best-performing CISLL method is
highlighted in Bold. The reported results are an average over 10 runs with
different permutations of data. Offline model is trained once.
$\widehat{\text{Offline}}=\frac{1}{T}\sum_{t=1}^{T}\bm{\alpha}_{\text{Offline},t}$,
with ’-’ denoting experiments we are unable to run due to compatibility
issues.

Method | iid | Class-iid | instance | Class-instance
---|---|---|---|---
iCub1.0 |  iCub28 |  CoRe50 |  iCub1.0 |  iCub28 |  CoRe50 |  iCub1.0 |  iCub28 |  CoRe50 |  iCub1.0 |  iCub28 |  CoRe50
Fine-tune | 0.9550 | 0.8432 | 0.9681 | 0.3902 | 0.4265 | 0.4360 | 0.1981 | 0.2483 | 0.2468 | 0.3508 | 0.4810 | 0.3400
EWC++ [2, 12] | - | - | - | 0.3747 | 0.4218 | 0.4307 | - | - | - | 0.3507 | 0.4805 | 0.3401
MAS [4] | - | - | - | 0.3758 | 0.4334 | 0.4333 | - | - | - | 0.3509 | 0.4807 | 0.3401
AGEM [16] | - | - | - | 0.4626 | 0.7507 | 0.5633 | - | - | - | 0.3510 | 0.4811 | 0.3399
FIFO | 0.9269 | 0.9774 | 0.9943 | 0.4971 | 0.6550 | 0.4763 | 0.3257 | 0.2807 | 0.1481 | 0.3609 | 0.4811 | 0.3399
GDumb [8] | 0.9269 | 0.7076 | 0.9502 | 0.9683 | 0.8293 | 0.9767 | 0.6240 | 0.4704 | 0.6521 | 0.7734 | 0.6481 | 0.6628
TinyER [17] | 0.9852 | 0.9752 | 1.0064 | 0.9766 | 0.8584 | 0.9723 | 0.9324 | 0.7995 | 0.9315 | 0.8825 | 0.7074 | 0.8525
DER [18] | 0.5976 | 0.8625 | 0.9807 | 0.8727 | 0.8402 | 0.9734 | 0.7972 | 0.8397 | 0.9870 | 0.8293 | 0.8286 | 0.9630
DER++ [18] | 0.9004 | 0.9020 | 0.9985 | 0.9398 | 0.8746 | 0.9786 | 0.8785 | 0.8547 | 0.9933 | 0.9125 | 0.8484 | 0.9696
CLS-ER [19] | 0.9573 | 0.1837 | 0.1107 | 0.5010 | 0.6664 | 0.3182 | 0.6854 | 0.1837 | 0.1107 | 0.5007 | 0.5858 | 0.2580
ExStream [9] | 0.9114 | 0.8053 | 0.9286 | 0.9035 | 0.8375 | 0.8884 | 0.8713 | 0.7389 | 0.8530 | 0.8806 | 0.8339 | 0.9091
REMIND [10] | 0.9666 | 0.9483 | 0.9988 | 0.9544 | 0.8197 | 0.9507 | 0.9102 | 0.7764 | 0.8993 | 0.8453 | 0.6784 | 0.8259
Ours | 1.0087 | 1.0045 | 1.0202 | 1.0069 | 0.8874 | 0.9918 | 0.9613 | 0.8555 | 0.9945 | 0.9985 | 0.8840 | 0.9851
Offline | 1.000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000
$\widehat{\text{Offline}}$ | 0.8046 | 0.8726 | 0.9038 | 0.8785 | 0.9266 | 0.9268 | 0.8046 | 0.8726 | 0.9038 | 0.8785 | 0.9266 | 0.9268

The optimization above is a virtual/local gradient update, as it doesn’t alter
the model parameters $\theta$. However, $\theta^{v}$ is optimized focusing on
the novel sample, which may not generalize well to the observed past samples
due to changes in the previously optimal network parameters, leading to
forgetting. To address this, we perform a global optimization with rehearsal.
For this, we choose two more sample subsets from memory:
$\mathcal{D}^{t}_{l},\mathcal{D}_{\mathcal{M}}^{t}\sim\mathcal{M}$, each with
size $C$. Let $H_{l}^{t}\leftarrow F_{\Phi}(\mathcal{D}_{l}^{t})$ represent
the logits obtained over replay samples $\mathcal{D}^{t}_{l}$, with
$F_{\Phi}(\cdot)$ denoting the semantic memory (see Sec. III-C). Then, we
compute the loss over virtual parameters $\theta^{v}$ using both subsets from
the replay buffer as the sum of cross-entropy and knowledge distillation loss
[28], defined as:

$\small\mathcal{L}^{t}=\mathbb{E}_{(\bm{z},y)\sim\mathcal{D}^{t}_{\mathcal{M}}}\left[\mathcal{L}_{CE}(y,F_{\theta^{v}}(\bm{z}))\right]+\lambda\mathcal{L}_{MSE}(H^{t}_{l},F_{\theta^{v}}(\mathcal{D}^{t}_{l}))$
(3)

Eq. 3 assesses the generalization loss over the rehearsal subsets using
virtual parameters optimized for the new streaming example. If the model
exhibits forgetting, then the virtual parameters will incur high loss due to
poor generalization. Otherwise, the loss will be small.

Suppose $\nabla_{\theta^{v}}\mathcal{L}^{t}(F_{\theta^{v}})$ be the gradient
of the loss in Eq. 3 w.r.t. the virtual model parameters $(\theta^{v})$. Then,
we can compute the _global parameters_ , for the plastic network $(\theta)$,
as follows:

$\theta\leftarrow\theta-\beta\nabla_{\theta^{v}}\mathcal{L}^{t}(F_{\theta^{v}})$
(4)

Eq. 4 updates $\theta$ using the gradient of the virtual parameter, which
might appear counter-intuitive. However, the alternating competitive training
between Eq. 2 and Eq. 4 is crucial. Below, we briefly discuss this training
behavior.

The updates in Eq. 2 and Eq. 4 occur alternately. Eq. 2 assesses how well
$\theta$ generalizes to new data, while Eq. 4 focuses on $\theta^{v}$’s
generalization to past data. For minimum loss (in Eq. 1), $\theta$ must adapt
to new samples, which is more challenging as compared to $\theta^{v}$, which
is optimized for new samples. Conversely, $\theta^{v}$ may struggle to
generalize to past samples due to its emphasis on new data. Hence, both
$\theta$ and $\theta^{v}$ need to generalize effectively to new and replayed
samples, minimizing losses in both Eq. 1 and Eq. 3. In this alternating
learning, convergence occurs when these losses approach zero, implying mutual
generalization and reduced forgetting.

###  III-B Tiny Episodic Memory (TEM)

The model uses a fixed-sized tiny episodic memory (TEM) to act as short-term
memory. In each incremental step, it $(i)$ replays subsets of $(C)$ samples
uniformly selected from memory for continual learning, and $(ii)$ stores the
new sample in memory. It employs Reservoir Sampling [46] and Class Balancing
Random Sampling to maintain a fixed-sized replay buffer. Reservoir Sampling
selects a random buffer sample to replace with the new example, while Class
Balancing Random Sampling picks a sample from the most populated class in the
buffer to replace the new one.

###  III-C Semantic Memory (SEM)

Semantic memory (SEM) retains long-term knowledge and combats forgetting using
self-distillation-loss [28, 47, 48, 29, 49] minimization (Eq. 3), aligning the
current model’s decision boundary with past memories. SEM, based on a DNN and
initialized with working model parameters, absorbs knowledge from the working
network $\theta$ in incremental steps. Inspired by mean-teacher [21, 22, 23,
24], SEM is updated stochastically via exponential moving average (EMA) rather
than at every iteration. Given a randomly sampled value $u$ from a uniform
distribution ($u\sim\mathcal{U}(0,1)$) and an acceptance probability $r$, the
update process for SEM denoted with $\Phi$ and the working model parameter
$\theta$, is defined as follows

$\Phi=\begin{cases}\gamma\;\Phi+(1-\gamma)\;\theta&,u<r\\\ \Phi&,Otherwise\\\
\end{cases}$ (5)

The acceptance probability is a hyper-parameter that regulates the frequency
of SEM updates. A lower (higher) acceptance probability means less (more)
frequent updates, retaining more (less) information from the remote model.
This update resembles the mammalian brain, with information initially stored
in short-term memory before transitioning to long-term memory. Algorithm 1
illustrates the different stages of the proposed model.

##  IV Experiments

###  IV-A Datasets, Data Orderings and Metrics

Datasets. We evaluate our method (VERSE) through extensive experiments on
three temporally coherent datasets: iCub1.0 [50], iCub28 [51], and CoRe50
[20]. iCub1.0 involves object recognition from video frame sequences, with
each frame containing a single object instance, while iCub28 is similar but
spans across four days. CoRe50, like iCub1.0/28, includes temporally ordered
images divided into 11 sessions with varying backgrounds and lighting.

Data Orderings. To test VERSE’s robustness in a challenging SLL setup, we
assess its streaming learning capability using four data-ordering schemes, as
in [10, 9, 11]. These schemes include (i) streaming i.i.d., (ii) streaming
class i.i.d., (iii) streaming instance, and (iv) streaming class instance
ordering.

Algorithm 1 VERSE

1:Initialize: $\Phi=\theta$

2:Hyperparameters: $\lambda,r,\alpha,\beta,\gamma$, Memory: $\mathcal{M}$

3:for $t\in 1,\dots,T,\dots$ do

4:
$\\{(\bm{z}_{t},y_{t})\\}=\\{(G_{\xi}(\bm{x}_{t}),y_{t})\\}\sim\mathcal{D}_{t}$
$\triangleright$ $|\mathcal{D}_{t}|=1$

5: Select samples from $\mathcal{M}$: $\mathcal{D}_{v}^{t}\sim\mathcal{M}$
$\triangleright$ $|\mathcal{D}_{v}^{t}|=C$

6: $\mathcal{D}_{v}^{t}\leftarrow\mathcal{D}_{t}\cup\mathcal{D}_{v}^{t}$

7: Compute $\mathcal{L}^{t}_{v}$ using Eq. 1 and evaluate
$\nabla_{\theta}\mathcal{L}^{t}_{v}(F_{\theta})$

8: $\theta^{v}=\theta-\alpha\nabla_{\theta}\mathcal{L}^{t}_{v}(F_{\theta})$
$\triangleright$ Virtual gradient update

9: Select samples from $\mathcal{M}$:
$\mathcal{D}^{t}_{l},\mathcal{D}_{\mathcal{M}}^{t}\sim\mathcal{M}$
$\triangleright$ $|\mathcal{D}_{l}^{t}|=|\mathcal{D}_{\mathcal{M}}^{t}|=C$

10: $H_{l}^{t}\leftarrow F_{\Phi}(\mathcal{D}_{l}^{t})$

11: Compute $\mathcal{L}^{t}$ using Eq. 3 and evaluate
$\nabla_{\theta^{v}}\mathcal{L}^{t}(F_{\theta^{v}})$

12: $\theta=\theta-\beta\nabla_{\theta^{v}}\mathcal{L}^{t}(F_{\theta^{v}})$
$\triangleright$ Global update

13: sample $u\sim\mathcal{U}(0,1)$

14: if $u<r$ then

15: $\Phi\leftarrow\gamma\;\Phi+(1-\gamma)\;\theta$

16: $\textit{UpdateMemory}(\mathcal{M},\mathcal{D}_{t},t)$ $\triangleright$
Add sample to $\mathcal{M}$

17:return $\theta,\Phi$

TABLE III: Buffer capacity used for various datasets.

Dataset | iCub1.0 | iCub28 | CoRe50 | ImageNet100
---|---|---|---|---
Buffer Capacity | 230 | 230 | 1000 | 1000
Training-Set Size | 6002 | 20363 | 119894 | 127778

Metrics. To assess the learner’s performance in a CISLL setup, we employ the
$\bm{\Omega}_{\text{all}}$ metric, following the approach in [7, 9, 10, 11].
This metric quantifies CL performance normalized against an _Offline_
baseline:

$\bm{\Omega}_{\text{all}}=\frac{1}{T}\sum^{T}_{t=1}\frac{\bm{\alpha}_{t}}{\bm{\alpha}_{\text{Offline},t}}$
(6)

where $(i)$ $T$ is the total number of testing events, $(ii)$
$\bm{\alpha}_{t}$ is the streaming learner’s unnormalized performance at time
$t$, and $(iii)$ $\bm{\alpha}_{\text{Offline},t}$ is the unnormalized
performance of the Offline baseline at time $t$.

Figure 3: Plots of $\bm{\alpha}_{t}$ as a function of streaming learning model
and data-orderings. VERSE outperforms other SLL models in both streaming
class-iid (top-row) and streaming class-instance (bottom-row) orderings across
datasets.

###  IV-B Baselines and Compared Methods

VERSE adheres to the challenging CISLL approach. We compare it with ExStream
[9] and REMIND [10]. We also evaluate various IBL and OCL approaches (EWC++,
MAS, AGEM, GDumb, TinyER, DER/DER++, CLS-ER, FIFO), as well as two additional
baselines: $(i)$ offline training with full dataset access (Offline/Upper
Bound), and $(ii)$ fine-tuning with one example at a time and no CL strategy
(Fine-tuning/Lower Bound). All comparisons are performed under the SLL setup,
except for GDumb, which fine-tunes with replay buffer samples, giving it an
unfair advantage.

Figure 4: Performance ($\bm{\mu}_{\text{all}}$) comparison between VERSE
(Ours) and the other baselines on ImageNet100.

###  IV-C Implementation Details

In all experiments, baselines are trained with one sample at a time using the
same network architecture. We employ ResNet-18 [52] pretrained on ImageNet-1K
[42, 43] available in PyTorch [53] TorchVision package, using its first 15
convolutional (conv) layers and 3 downsampling layers as the feature extractor
$(G)$. The remaining 2 conv layers and 1 fully connected (FC) layer constitute
the plastic network $(F)$. For ExStream [9], all 17 conv and 3 downsampling
layers are utilized for feature extraction $(G)$, and the final FC layer
serves as the plastic network $(F)$. Feature embeddings are stored in memory
for all baselines, including VERSE. Replay buffer capacity is specified in
Table III. We employ reservoir sampling for class-instance and instance
ordering and class-balancing random sampling for class-iid and iid ordering.
Experience-replay and self-distillation consistently use $C=16$ samples across
all baselines. Hyperparameters are set as follows:
$\alpha=0.005,\beta=0.01,\lambda=0.3$, and $\gamma=0.9$. $r$ values are set
as: $(i)$ $r=0.4$ for iCub1.0, $(ii)$ $r=0.1$ for iCub28, and $(iii)$ $r=0.05$
for CORe50 dataset. Each experiment is repeated 10 times with different data
permutations, and the average accuracy is reported.

Figure 5: Performance ($\bm{\mu}_{\text{all}}$) comparison between VERSE and
Lifelong MAML [54] on iCub1.0. Figure 6: Plots of $\bm{\Omega}_{\text{all}}$
as as function of $(i)$ replay buffer capacity $(|\mathcal{M}|)$, $(ii)$
knowledge-distillation hyper-parameter $(\lambda)$, and $(iii)$ acceptance-
rate hyper-parameter $(r)$ on iCub1.0.

###  IV-D Results

Table II presents VERSE’s performance in various experimental setups with
different data orderings and datasets. We conducted 10 repetitions of each
experiment, reporting average accuracy. Notably, VERSE consistently surpasses
the baselines by a significant margin. It demonstrates robustness to different
data-ordering schemes, which are known to induce catastrophic forgetting. In
contrast, IBL methods like EWC++ [2, 12] and MAS [4] experience severe
forgetting. Even GDumb [8], which fine-tunes network parameters with buffer
samples before each inference, fails to outperform VERSE.

iCub1.0/28 and CoRe50 are temporally coherent datasets, offering a more
realistic and challenging evaluation scenario. Class-instance and instance
ordering requires the agent to learn from temporally ordered video sequences
one at a time. Table II shows that VERSE achieves notable improvements: $(i)$
up to $8.6\%$ and $2.89\%$ on iCub1.0 for class-instance and instance
ordering, $(ii)$ $3.56\%$ on iCub28 for class-instance ordering, and $(iii)$
$1.55\%$ on CoRe50 for class-instance ordering. Fig. 3 plots accuracy
($\alpha_{t}$) of VERSE (Ours) and other baselines for class-iid and class-
instance ordering. Notably, VERSE better retain knowledge of old classes
compared to other baselines, particularly excelling in class-instance
ordering.

We also assess VERSE and other baselines on ImageNet100, a subset of
ImageNet-1K (ILSVRC-2012) [42, 43], comprising randomly selected 100 classes,
each with 700-1300 training samples and 50 validation samples. As ImageNet-1K
lacks labels for test samples, we used the validation set for testing,
following [10]. Fig. 4 illustrates the performance ($\bm{\mu}_{\text{all}}$)
of various baselines, including VERSE, showing VERSE consistently
outperforming all other CL methods. $\bm{\mu}_{\text{all}}$ represents the
mean-absolute accuracy with:

$\bm{\mu}_{\text{all}}=\frac{1}{T}\sum^{T}_{t=1}\bm{\alpha}_{t}$ (7)

where $(i)$ $T$ is the total number of testing events and $(ii)$
$\bm{\alpha}_{t}$ is the accuracy of the streaming learner at time $t$.

Finally, we conduct a performance comparison between VERSE and Lifelong MAML
[54], a continual learning variant of MAML [55], on the iCub1.0 dataset. Fig.
5 illustrates the performance comparison using $\bm{\mu}_{\text{all}}$ metric
with VERSE consistently outperforming LifeLong MAML across all data-orderings.

Figure 7: Plots of $\bm{\mu}_{\text{all}}$ as a function of EMA $(\Phi)$ used
to compute the semantic memory (Eq. 5) on CORe50.

##  V Ablations

We perform extensive ablations to validate the importance of the various
components of VERSE.

Choice Of Buffer Capacity. Fig. 6 (left) shows the impact of different buffer
capacities on iCub1.0. Increased buffer capacity leads to improved model
performance.

Choice Of Hyper-parameter $(\lambda)$. Fig. 6 (middle) shows the impact of
changing the self-distillation hyper-parameter $(\lambda)$ on iCub1.0. The
best performance is consistently achieved across all data orderings with
$\lambda=0.3$.

Significance Of Self-Distillation Loss. Fig. 6 (middle) depicts the model’s
performance with $\lambda=0.0$, indicating no self-distillation. While the
best performance is achieved with $\lambda=0.3$, self-distillation alone does
not significantly improve performance.

Significance Of Acceptance-Rate $(r)$. Fig. 6 (right) illustrates the impact
of changing the acceptance-rate $(r)$ on iCub1.0. The best performance is
achieved with $r=0.4$. However, increasing $r$ to $0.50$ leads to performance
degradation. For instance ordering, the model tends to perform best with
$r=0.0$.

Significance of Exponential Moving Average (EMA). Fig. 7 highlights the
importance of SEM $(\Phi)$ in the model’s performance. Without using SEM and
relying solely on the working model $(\theta)$ for computing logits in self-
distillation loss (Eq. 3), the model’s performance degrades. Additionally, for
temporally coherent orderings (instance and class instance orderings), not
using EMA to update SEM severely degrades VERSE’s performance.

Significance of Buffer Replacement Policies. Fig. 8 shows the model’s
performance with different buffer replacement policies used for TEM. For
temporally ordered data (instance and class instance ordering), reservoir
sampling yields the best performance. However, for i.i.d and class i.i.d
ordering, class balancing random sampling or balanced sampling achieves the
best results.

Figure 8: Plots of $\bm{\mu}_{\text{all}}$ as a function of buffer replacement
policies on iCub1.0.

##  VI Conclusion

We address the challenging problem of streaming lifelong learning, where the
learner is given only one sample at a time during training, the learned model
is required to have anytime inference capability. Our replay-based virtual-
gradient-regularization with global and virtual/local parameters
generalization to both previous and novel task samples. Tiny episodic memory
for rehearsal and semantic memory help align the decision boundary with past
memories through self-distillation-loss. Extensive experiments and ablations
on various datasets and data orderings demonstrate our approach’s efficacy.

## References

  * [1] S. Thrun, “Lifelong learning algorithms,” in _Learning to learn_. Springer, 1998, pp. 181–209.
  * [2] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, _et al._ , “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the national academy of sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [3] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 12, pp. 2935–2947, 2017\.
  * [4] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 139–154.
  * [5] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 374–382.
  * [6] C. Wu, L. Herranz, X. Liu, J. van de Weijer, B. Raducanu, _et al._ , “Memory replay gans: Learning to generate new categories without forgetting,” in _Advances In Neural Information Processing Systems_ , 2018, pp. 5962–5972.
  * [7] R. Kemker and C. Kanan, “Fearnet: Brain-inspired model for incremental learning,” _arXiv preprint arXiv:1711.10563_ , 2017.
  * [8] A. Prabhu, P. H. Torr, and P. K. Dokania, “Gdumb: A simple approach that questions our progress in continual learning,” in _European Conference on Computer Vision_. Springer, 2020, pp. 524–540.
  * [9] T. L. Hayes, N. D. Cahill, and C. Kanan, “Memory efficient experience replay for streaming learning,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 9769–9776.
  * [10] T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, and C. Kanan, “Remind your neural network to prevent catastrophic forgetting,” _arXiv preprint arXiv:1910.02509_ , 2019.
  * [11] S. Banerjee, V. K. Verma, and V. P. Namboodiri, “Streaming lifelong learning with any-time inference,” _arXiv preprint arXiv:2301.11892_ , 2023.
  * [12] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr, “Riemannian walk for incremental learning: Understanding forgetting and intransigence,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 532–547.
  * [13] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 3987–3995.
  * [14] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, “Variational continual learning,” _arXiv preprint arXiv:1710.10628_ , 2017.
  * [15] D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6467–6476.
  * [16] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, “Efficient lifelong learning with a-gem,” _arXiv preprint arXiv:1812.00420_ , 2018.
  * [17] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato, “On tiny episodic memories in continual learning,” _arXiv preprint arXiv:1902.10486_ , 2019.
  * [18] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,” _arXiv preprint arXiv:2004.07211_ , 2020.
  * [19] E. Arani, F. Sarfraz, and B. Zonooz, “Learning fast, learning slow: A general continual learning method based on complementary learning system,” _arXiv preprint arXiv:2201.12604_ , 2022.
  * [20] V. Lomonaco and D. Maltoni, “Core50: a new dataset and benchmark for continuous object recognition,” in _Conference on Robot Learning_. PMLR, 2017, pp. 17–26.
  * [21] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [22] Q. Cai, Y. Pan, C.-W. Ngo, X. Tian, L. Duan, and T. Yao, “Exploring object relation in mean teacher for cross-domain detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 457–11 466.
  * [23] J. Deng, W. Li, Y. Chen, and L. Duan, “Unbiased mean teacher for cross-domain object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4091–4101.
  * [24] Z. Chen, L. Zhu, L. Wan, S. Wang, W. Feng, and P.-A. Heng, “A multi-task mean teacher for semi-supervised shadow detection,” in _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition_ , 2020, pp. 5611–5620.
  * [25] R. F. Thompson and D. J. Krupa, “Organization of memory traces in the mammalian brain,” _Annual review of neuroscience_ , vol. 17, no. 1, pp. 519–549, 1994.
  * [26] D. J. Linden, “Long-term synaptic depression in the mammalian brain,” _Neuron_ , vol. 12, no. 3, pp. 457–472, 1994.
  * [27] A. Holtmaat and K. Svoboda, “Experience-dependent structural synaptic plasticity in the mammalian brain,” _Nature Reviews Neuroscience_ , vol. 10, no. 9, pp. 647–658, 2009.
  * [28] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [29] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3713–3722.
  * [30] R. Aljundi, P. Chakravarty, and T. Tuytelaars, “Expert gate: Lifelong learning with a network of experts,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3366–3375.
  * [31] H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 2990–2999.
  * [32] R. Aljundi, M. Rohrbach, and T. Tuytelaars, “Selfless sequential learning,” _arXiv preprint arXiv:1806.05421_ , 2018.
  * [33] A. Rios and L. Itti, “Closed-loop memory gan for continual learning,” _arXiv preprint arXiv:1811.01146_ , 2018.
  * [34] E. Belouadah and A. Popescu, “Il2m: Class incremental learning with dual memory,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 583–592.
  * [35] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 831–839.
  * [36] X. Tao, X. Chang, X. Hong, X. Wei, and Y. Gong, “Topology-preserving class-incremental learning,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16_. Springer, 2020, pp. 254–270.
  * [37] J. Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong learning with dynamically expandable networks,” _arXiv preprint arXiv:1708.01547_ , 2017.
  * [38] V. K. Verma, K. J. Liang, N. Mehta, P. Rai, and L. Carin, “Efficient feature transformations for discriminative and generative continual learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 13 865–13 875.
  * [39] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio, “Gradient based sample selection for online continual learning,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 11 816–11 825.
  * [40] R. Aljundi, L. Caccia, E. Belilovsky, M. Caccia, M. Lin, L. Charlin, and T. Tuytelaars, “Online continual learning with maximally interfered retrieval,” _arXiv preprint arXiv:1908.04742_ , 2019.
  * [41] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2001–2010.
  * [42] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [43] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [44] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” _arXiv preprint arXiv:1312.6211_ , 2013.
  * [45] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in _Psychology of learning and motivation_. Elsevier, 1989, vol. 24, pp. 109–165.
  * [46] J. S. Vitter, “Random sampling with a reservoir,” _ACM Transactions on Mathematical Software (TOMS)_ , vol. 11, no. 1, pp. 37–57, 1985.
  * [47] Z. Allen-Zhu and Y. Li, “Towards understanding ensemble, knowledge distillation and self-distillation in deep learning,” _arXiv preprint arXiv:2012.09816_ , 2020.
  * [48] H. Mobahi, M. Farajtabar, and P. Bartlett, “Self-distillation amplifies regularization in hilbert space,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 3351–3361, 2020.
  * [49] L. Zhang, C. Bao, and K. Ma, “Self-distillation: Towards efficient and compact neural networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 8, pp. 4388–4403, 2021.
  * [50] S. Fanello, C. Ciliberto, M. Santoro, L. Natale, G. Metta, L. Rosasco, and F. Odone, “icub world: Friendly robots help building good vision data-sets,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2013, pp. 700–705.
  * [51] G. Pasquale, C. Ciliberto, F. Odone, L. Rosasco, and L. Natale, “Teaching icub to recognize objects using deep convolutional neural networks,” in _Machine Learning for Interactive Systems_. PMLR, 2015, pp. 21–25.
  * [52] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [53] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in neural information processing systems_ , 2019, pp. 8026–8037.
  * [54] G. Gupta, K. Yadav, and L. Paull, “Look-ahead meta learning for continual learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 11 588–11 598, 2020.
  * [55] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.