# Latent Properties of Lifelong Learning Systems

Corban Rivera, Chace Ashcraft, Alexander New, James Schmidt, Gautam Vallabha
Intelligent Systems Center
Johns Hopkins Applied Physics Lab
<EMAIL_ADDRESS>

###### Abstract

Creating artificial intelligence (AI) systems capable of demonstrating
lifelong learning is a fundamental challenge, and many approaches and metrics
have been proposed to analyze algorithmic properties. However, for existing
lifelong learning metrics, algorithmic contributions are confounded by task
and scenario structure. To mitigate this issue, we introduce an algorithm-
agnostic explainable surrogate-modeling approach to estimate latent properties
of lifelong learning algorithms. We validate the approach for estimating these
properties via experiments on synthetic data. To validate the structure of the
surrogate model, we analyze real performance data from a collection of popular
lifelong learning approaches and baselines adapted for lifelong classification
and lifelong reinforcement learning.

##  1 Introduction

Inspired by the way that humans acquire new skills sequentially and improve
over time, lifelong or continual learning (Chen & Liu (2018); Silver et al.
(2013)) describes the goal of enabling AI systems to learn tasks sequentially
over time while improving performance on both previous and future tasks.
Lifelong learning has received much attention in the AI community, and many
algorithms have been proposed for both supervised (Delange et al. (2021)) and
reinforcement learning (Khetarpal et al. (2020)). We include additional review
of lifelong learning approaches in Appendix A.2. A key challenge, apart from
the learning algorithm itself, is the assessment of the learner: how well it
works, where it fails, and what factors influence its success or failure. Many
existing approaches focus on defining continual learning metrics, such as
_forgetting_ , _backward transfer_ , and _forward transfer_ (Díaz-Rodríguez et
al. (2018); Powers et al. (2021)); recently a suite of metrics for continual
learning has been proposed New et al. (2022). However, a metric such as
forward transfer confounds three factors: the relationship between the tasks
to be learned, the particular sequencing of tasks (curriculum), and the
capabilities of the learner (Farquhar & Gal (2018)). Use of systematic
benchmarks (e.g., Powers et al. (2021); Lomonaco & Maltoni (2017)) can address
part of this concern by fixing the tasks and curricula, but does not address
how the learner might do on other (non-benchmark) tasks.

In this work, we present a novel approach we refer to as Continual Learning
Analysis via a Model of Performance, or _CLAMP_ , to estimate the separate
contributions of task structure and algorithm capabilities from performance
data. We refer to these contributions as latent properties by analogy to
latent variable modeling, where the goal is to identify a lower-dimensional
space that effectively captures the probability distribution of high-
dimensional data. In our case, we treat the performance curves (the time
series of the learner’s performance over the curriculum) as the high-
dimensional data, and consider how to identify underlying properties of the
learner and tasks that can best explain the observed performance curves. Our
strategy is to (a) define a generative surrogate performance model of lifelong
learning that has a small set of explainable parameters, and (b) estimate
these parameters of the surrogate model from a set of performance curves. The
explainable parameters specify algorithm properties (transfer efficiency,
skill retention, expertise translation) and task properties (the “similarity”
between the tasks in the curriculum). Our approach works equally well with
classification learners and reinforcement learners, and relies solely on the
performance curves (i.e., it does not need access to the implementation of the
learner or the original task environment, similar to other post-hoc ML model
assessment approaches (Martin et al. (2021))). Additional review of
explainable surrogate models literature is included in Appendix A.1.

##  2 Approach

Here we introduce Continual Learning Analysis via a Model of Performance
(_CLAMP_) and define the latent properties being estimated by the approach.
This work represents what we believe is the first explainable surrogate
performance model for lifelong learning system analysis. We further believe
this is the first model-agnostic attempt to quantitatively characterize both
algorithm capabilities and inter-task relationships using only performance
data available from multiple continual learning algorithms and multiple tasks.

###  2.1 Definitions

A lifelong learning system (called the agent) experiences $n$ tasks
${t_{1},t_{2},\ldots,t_{n}}$ from an set $\mathcal{T}$. Formally, task
experiences are tuples $(x,y)\in\mathcal{X}\times\mathcal{Y}$ for an input set
$\mathcal{X}$ and output set $\mathcal{Y}$. We assume task experiences are
ordered and known. Agents experience a sequence of $m$ experience tuples
$(x_{1},y_{1}),\ldots,(x_{m},y_{m})$ drawn from a paired sequence of tasks
$\mathcal{C}=[c_{1},c_{2},\ldots,c_{m}]$, called a curriculum, where each
$c_{j}\in\mathcal{T}$. While single-task learning problems suppose that
$(x_{j},y_{j})\sim\mathbb{P}_{\mathcal{X}\times\mathcal{Y}}$, sampling with
curriculum supposes, by contrast, that
$(x_{j},y_{j})\sim\mathbb{P}^{c_{j}}_{\mathcal{X}\times\mathcal{Y}}$ for
$c_{j}\in\mathcal{T}$.111Formally, let
$S:=\\{(x_{1},y_{1}),\ldots,(x_{m},y_{m})\\}\subset\mathcal{X}\times\mathcal{Y}$
be a labeled data set, and $S^{j}=\\{(x_{i},y_{i})\in S:\,c_{i}=c_{j}\\}$;
then we suppose $(x,y)\sim\mathbb{P}^{c_{j}}_{\mathcal{X}\times\mathcal{Y}}$
for $(x,y)\in S^{j}$. As noted in Appendix A.3, task incremental learning
includes task $c_{j}$ along with labeled data $(x_{j},y_{j})$ for the learning
algorithm, while domain incremental learning does not.

We consider two categories of properties. Intrinsic-task properties
characterize a task to be encountered and are assumed to be invariant to
changes in the lifelong learning algorithm encountering that task. Likewise,
intrinsic-algorithm properties characterize properties of a given lifelong
learning algorithm and are assumed to be invariant to what tasks that
algorithm encounters. These strong assumptions enable us to ensure that
_CLAMP_ is explainable.

The task transfer matrix $\textbf{A}\in\mathbb{R}^{n\times n}$ (recall that
$|\mathcal{T}|=n$) with entries $\mathbf{A}_{i,j}\in[-1,1]$ indicating how
gaining experience from task $i$ affects performance on task $j$ independent
of the algorithm selected. We assume a linear model of task transfer.

The task difficulty score $d$ in range $[0,\infty)$ as a task associated
parameter that represents the intrinsic difficulty in increasing performance
on a task given experience.

The transfer efficiency score $\gamma$ is an intrinsic property of lifelong
learning algorithms in the range $[0,\infty)$ that indicates how efficiently
the algorithm translates experience from one task to experience on another.

The experience retention score $h$ as a property of lifelong learning
algorithms that conveys how well the algorithm retains prior experiences.
Values are in the range $[0,1]$ with $0$ indicating complete forgetting of
prior task knowledge and $1$ indicating complete retention of prior task
performance.

The expertise translation score $\lambda$ to be an algorithm property
reflecting the ability of the algorithm to translate performance from one task
into experience on another task, for any fixed task transfer matrix. Values
are in the range $[0,\infty)$.

###  2.2 Lifelong Surrogate Model Formulation

We assume a functional form for experience accumulation. For an algorithm $a$
and curriculum $\mathcal{C}$, accumulated experience for position $i$ in
curriculum $\mathcal{C}$, we define experience $\mathcal{E}^{a}_{j}(c_{l})$ of
task $t_{j}$ at all points $l$ in the curriculum as follows. For the base
case, we assume for all $t_{j}\in\mathcal{T}$ that the initial experience for
all tasks is zero. $\mathcal{E}^{a}_{j}(c_{0})=0$ For curriculum steps $j>0$
we define experience accumulation by the following expression:

$\displaystyle\mathcal{E}^{a}_{j}(c_{l})=\mathcal{E}(\mathcal{C},\textbf{A},\gamma,h,\lambda)=\mathcal{E}^{a}_{j}(c_{l-k})h_{a}+\textbf{A}_{i,j}(\gamma_{a}+\mathcal{P}_{i}(c_{l-1})\lambda_{a}),$

where $\mathbf{A},\gamma,h,\lambda$ are task and algorithm properties defined
in Section 2.1. Experience in task $t_{j}$ is mapped to performance
$\mathcal{P}(c_{j})$ in task $t_{j}$ through a sigmoid function
$S(x)=(1+e^{-x})^{-1}$, which is then shifted and scaled so that the
inflection passes through zero. If curriculum step $c_{l}$ experiences task
$t_{j}$, then its performance $\mathcal{P}_{i}(c_{l})$ is given by:

$\displaystyle\mathcal{P}_{j}(c_{l})=S(\mathcal{E}_{j}(c_{l})/d_{i})=2/(1+e^{-((\mathcal{E}_{j}(c_{l})/d_{i}))})-1$

###  2.3 Estimating parameters from data

Suppose we have a curriculum $\mathcal{C}$ containing $m$ encounters of $n$
different tasks, and, for a set of algorithms $a=1,\ldots,p$, a time-series of
performance values $P^{a}\in\mathbb{R}^{n\times m}$, where the entries
$P^{a}_{jl}$ are the performance of the $j$th task at the $l$th curriculum
entry. We desire to estimate the task transfer matrix $\mathbf{A}$ and task
difficulty score $d$, as well as, for each algorithm $a$, the transfer
efficiency $\gamma_{a}$, experience retention $h_{a}$, and expertise
translation $\lambda_{a}$. We can group these parameters together as
$\Theta=\\{\mathbf{A},d\\}\cup\bigcup_{a}\\{\gamma_{a},h_{a},\lambda_{a}\\}$.
Then _CLAMP_ uses $\mathbf{A},d,\gamma_{a},h_{a}$, and $\lambda_{a}$ to
predict a time-series of performance values $\hat{P}^{a}\in\mathbb{R}^{n\times
m}$, where $\hat{P}^{a}_{jl}$ is the predicted performance of the $j$th task
at the $l$th curriculum entry. Our parameters $\Theta$ can be estimated by
solving the following minimization problem:

$\displaystyle\min_{\Theta}$
$\displaystyle\sum_{a}||\hat{P}^{a}-P^{a}||_{F}^{2}$ s.t.
$\displaystyle-1\leq\mathbf{A}_{ij}\leq 1$ $\displaystyle
d,\gamma_{a},\lambda_{a}\geq 0$ $\displaystyle 0\leq h_{a}\leq 1,$

This minimization problem is differentiable in its parameters and has linear
constraints, so it may be solved with techniques like projected subgradient
descent. In practice, we implement our approach in PyTorch Paszke et al.
(2019) and use 1,000 steps of gradient descent with an Adam optimizer Kingma &
Ba (2014) with default parameters and learning rate. After each step, the
parameters are projected back onto their feasible set.

##  3 Results

We designed experiments to (i) validate the approach used to estimate latent
properties of lifelong learning, (ii) validate the functional form of the
surrogate performance model with data coming from multiple baseline algorithms
and multiple baseline datasets; including MNIST, CIFAR100 and Atari. Due to
space constraints, several sections are included in the appendices including:
quantitative validation of _CLAMP_ based on synthetic lifelong learning
performance data in Appendix B, analysis of domain adaptation with MNIST in
Appendix C.1, and task adaptation with CIFAR100 in Appendix C.2.

To illustrate the adaptability of _CLAMP_ , we analysed the performance of
various lifelong learning reinforcement learning approaches from an experiment
consisting of a set of Atari tasks from AGI-Labs222https://github.com/AGI-
Labs/continual_rl/blob/develop/docs/ATARI_RESULTS.md. In particular, the
algorithms we consider include Continual Learning with Experience And Replay
(CLEAR, Rolnick et al. (2019)), online EWC, Progress & Compress (P&C, Schwarz
et al. (2018)), and Importance Weighted Actor-Learner Architecture (IMPALA,
Espeholt et al. (2018)).

Figure 1: Performance of several common lifelong reinforcement learning
approaches and baselines applied to a curriculum of 6 Atari tasks. Each phase
of the curriculum was trained for 5 million steps. Green vertical bars
indicate the regions where the current task is being trained. Dashed lines
indicate the predicted performance curves estimates by _CLAMP_.

Before running _CLAMP_ , we downsampled the performance data for each
continual learning algorithm to capture task performance at task transition
boundaries. We then normalized reward results across tasks using the procedure
detailed in New et al. (2022). The resulting data is shown in Figure 1. We fit
a _CLAMP_ model to the data as described in Section 2.3 resulting in the
estimated algorithm performance data shown in 1 ($0.007$ MSE).

The property estimates from _CLAMP_ shown in Tables 1a, 1b, and 7
qualitatively match well with our observations about the data shown in Figure
1. Of the approaches considered in this experiment, CLEAR was estimated to
have the best transfer efficiency $0.12$ and experience retention $0.9$ due to
the experience replay. The data in Figure 1 show that Clear frequently had the
highest task performance and the highest task performance for tasks not
actively being trained. We observed that EWC and online EWC had relatively
lower task performance, but they retained task performance while other tasks
were training. The CLAMP analysis explains the relatively poor performance of
EWC and online EWC as being a result of relatively low transfer efficiency.
This makes sense as the mechanism of EWC and online EWC are based on
regularization which adds additional loss terms that are simultaneously being
optimized to slow parameter changes to parts of the network identified as
useful for other tasks. The algorithm parameter estimates from from CLAMP
qualitatively match our expectations from the given performance data and a
mechanistic understanding of the lifelong learning approaches.

Approach | $\gamma$ | $h$ | $\lambda$
---|---|---|---
Clear | 0.12 | 0.90 | 0.03
Progress & Compress | 0.06 | 0.35 | 1.16
EWC_online | 0.03 | 0.60 | 0.82
EWC | 0.01 | 0.74 | 0.72
Impala | 0.10 | 0.33 | 1.26

(a) Estimated latent properties of different algorithms for based on Atari
tasks

Task |  | Space
---
Invaders
Krull |  | Beam
---
Rider
Hero |  | Star
---
Gunner
| Ms.
---
Packman
| Space
---
Invaders
1.00 | 0.15 | -0.13 | 0.01 | -0.16 | 0.03
Krull | -0.09 | 1.00 | 0.21 | -0.02 | -0.05 | 0.02
| Beam
---
Rider
0.04 | -0.15 | 1.00 | 0.01 | 0.04 | 0.08
Hero | 0.13 | 0.05 | -0.33 | 1.00 | 0.10 | 0.11
| Star
---
Gunner
0.01 | 0.07 | -0.16 | -0.12 | 1.00 | -0.03
| Ms.
---
Packman
-0.03 | -0.16 | -0.12 | -0.01 | -0.38 | 1.00

(b) Estimated task transfer between Atari environments

Task |  | Space
---
Invaders
Krull |  | Beam
---
Rider
Hero |  | Star
---
Gunner
| Ms.
---
Pacman
Difficulty | 0.09 | 0.08 | 0.15 | 0.07 | 0.10 | 0.08

(c) Estimated task difficulty for Atari tasks

Table 1: CLAMP estimates for the curriculum of Atari tasks show in Figure 1
including (a) latent properties for lifelong learning algorithms including
transfer efficiency $\gamma$, experience retention $h$, and expertise
translation $\lambda$, (b) task transfer matrix indicating how experience on
one task effects the others, and (c) intrinsic task difficulty.

##  4 Discussion

The analysis of synthetic data in Appendix B demonstrates that the
optimization approach in _CLAMP_ was able to recover underlying parameters of
the model from synthetic lifelong learning performance data with low error.
From Section 3 and Appendix C.1 and C.2, we found that _CLAMP_ had low mean
squared error to real performance data from several benchmarks. Qualitative
assessment of the latent property estimates from analysis of the Atari
experiment indicated broad agreement with expectations given the performance
data and the underlying mechanisms of the lifelong learning approaches.

Existing metrics for evaluating lifelong learning conflate task structure with
algorithm performance. We have introduced _CLAMP_ \- Continual Learning
Analysis via a Model of Performance - as the first attempt to estimate the
separate contributions of task structure from lifelong learning algorithm
performance. _CLAMP_ is model and task agnostic, and it can estimate several
important properties of lifelong learning systems, including how well they
learn from new experiences and how well they retain prior experience. We have
also demonstrated the applicability of this approach to continual learning in
both the classification and reinforcement learning settings.

## References

  * Adadi & Berrada (2018) Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: a survey on explainable artificial intelligence (xai).  _IEEE access_ , 6:52138–52160, 2018.
  * Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 139–154, 2018.
  * Angelov et al. (2021) Plamen P Angelov, Eduardo A Soares, Richard Jiang, Nicholas I Arnold, and Peter M Atkinson.  Explainable artificial intelligence: an analytical review.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 11(5):e1424, 2021.
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  _Information fusion_ , 58:82–115, 2020.
  * Chen & Liu (2018) Zhiyuan Chen and Bing Liu.  _Lifelong machine learning_.  Morgan & Claypool Publishers, 2018.
  * Delange et al. (2021) Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2021.  doi: 10.1109/tpami.2021.3057446.  URL https://doi.org/10.1109%2Ftpami.2021.3057446.
  * Doran et al. (2017) Derek Doran, Sarah Schulz, and Tarek R. Besold.  What does explainable ai really mean? a new conceptualization of perspectives, 2017.  URL https://arxiv.org/abs/1710.00794.
  * Díaz-Rodríguez et al. (2018) Natalia Díaz-Rodríguez, Vincenzo Lomonaco, David Filliat, and Davide Maltoni.  Don’t forget, there is more than forgetting: new metrics for continual learning, 2018.  URL https://arxiv.org/abs/1810.13166.
  * Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu.  IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 1407–1416. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/espeholt18a.html.
  * Farquhar & Gal (2018) Sebastian Farquhar and Yarin Gal.  Towards robust evaluations of continual learning, 2018.  URL https://arxiv.org/abs/1805.09733.
  * French (1999) Robert M. French.  Catastrophic forgetting in connectionist networks.  _Trends in Cognitive Sciences_ , 3(4):128–135, 1999.  ISSN 1364-6613.  doi: https://doi.org/10.1016/S1364-6613(99)01294-2.  URL https://www.sciencedirect.com/science/article/pii/S1364661399012942.
  * Gunning & Aha (2019) David Gunning and David Aha.  Darpa’s explainable artificial intelligence (xai) program.  _AI magazine_ , 40(2):44–58, 2019.
  * Holzinger (2018) Andreas Holzinger.  From machine learning to explainable ai.  In _2018 World Symposium on Digital Intelligence for Systems and Machines (DISA)_ , pp. 55–66, 2018.  doi: 10.1109/DISA.2018.8490530.
  * Hsu et al. (2018) Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines, 2018.  URL https://arxiv.org/abs/1810.12488.
  * Khetarpal et al. (2020) Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup.  Towards continual reinforcement learning: A review and perspectives, 2020\.  URL https://arxiv.org/abs/2012.13490.
  * Kingma & Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.  URL https://arxiv.org/abs/1412.6980.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Lindley (1972) Dennis Victor Lindley.  _Bayesian statistics: A review_.  SIAM, 1972.
  * Lomonaco & Maltoni (2017) Vincenzo Lomonaco and Davide Maltoni.  CORe50: a New Dataset and Benchmark for Continuous Object Recognition.  In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pp. 17–26. PMLR, November 2017.  URL https://proceedings.mlr.press/v78/lomonaco17a.html.
  * Lopez-Paz & Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, pp. 6470–6479, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Martin et al. (2021) Charles H. Martin, Tongsu (Serena) Peng, and Michael W. Mahoney.  Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.  _Nature Communications_ , 12(1):4122, Jul 2021\.  ISSN 2041-1723.  doi: 10.1038/s41467-021-24025-8.  URL https://doi.org/10.1038/s41467-021-24025-8.
  * McCloskey & Cohen (1989) Michael McCloskey and Neal J. Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  volume 24 of _Psychology of Learning and Motivation_ , pp. 109–165. Academic Press, 1989.  doi: https://doi.org/10.1016/S0079-7421(08)60536-8.  URL https://www.sciencedirect.com/science/article/pii/S0079742108605368.
  * New et al. (2022) Alexander New, Megan Baker, Eric Nguyen, and Gautam Vallabha.  Lifelong learning metrics, 2022.  URL https://arxiv.org/abs/2201.08278.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems 32_ , pp. 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Powers et al. (2021) Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta.  Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents, 2021.  URL https://arxiv.org/abs/2110.10067.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”why should i trust you?”: Explaining the predictions of any classifier, 2016.  URL https://arxiv.org/abs/1602.04938.
  * Rolnick et al. (2019) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne.  Experience replay for continual learning, 2019.
  * Schwarz et al. (2018) Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 4528–4537. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/schwarz18a.html.
  * Silver et al. (2013) Daniel Silver, Qiang Yang, and Lianghao Li.  Lifelong machine learning systems: Beyond learning algorithms, 2013.  URL https://www.aaai.org/ocs/index.php/SSS/SSS13/paper/view/5802.
  * Tritscher et al. (2020) Julian Tritscher, Markus Ring, Daniel Schlr, Lena Hettinger, and Andreas Hotho.  Evaluation of post-hoc xai approaches through synthetic tabular data.  In _International symposium on methodologies for intelligent systems_ , pp. 422–430. Springer, 2020.
  * van de Ven & Tolias (2018) Gido M. van de Ven and Andreas S. Tolias.  Generative replay with feedback connections as a general strategy for continual learning, 2018.  URL https://arxiv.org/abs/1809.10635.
  * von Toussaint (2011) Udo von Toussaint.  Bayesian inference in physics.  _Rev. Mod. Phys._ , 83:943–999, Sep 2011.  doi: 10.1103/RevModPhys.83.943.  URL https://link.aps.org/doi/10.1103/RevModPhys.83.943.
  * Zagoruyko & Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks, 2016.  URL https://arxiv.org/abs/1605.07146.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _International Conference on Machine Learning_ , pp. 3987–3995. PMLR, 2017.
  * Zou & Adams (2012) James Y. Zou and Ryan P. Adams.  Priors for diversity in generative latent variable models.  In _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’12, pp. 2996–3004, Red Hook, NY, USA, 2012. Curran Associates Inc.

##  Appendix A Related Work

###  A.1 Explainable surrogate models

Explainable surrogate models are a common approach in the explainable AI (XAI,
Angelov et al. (2021); Adadi & Berrada (2018); Arrieta et al. (2020); Gunning
& Aha (2019)) literature. These are a form of explanation by simplification
(Tritscher et al. (2020)). By fitting a simplified model to a more complex
model, by comparison, it may be possible to better interpret salient factors
for prediction. For example, a linear model or decision tree might be fit to
the original model to reduce the model complexity compared to the original
model. Doran et al. (2017) refer to this category of models as interpretable
in that it is possible to mathematically analyze the algorithmic properties.

Another characteristic of explainable models are the assumptions that they
make about knowledge of the original model structure. Some approaches assume
knowledge of the original model composition (white box assumption) while other
approaches only assume that access to inputs and output are available from the
full model (black box assumption) (Adadi & Berrada (2018)). Approaches that
make black-box assumptions are also referred to as model-agnostic, in the
sense that they can be used without detailed knowledge of the original models
structure. For example, Local Interpretable Model-agnostic Explanations
(Ribeiro et al. (2016)) (LIME) is a recent example of a model agnostic
approach that creates locally-optimized explanations by training a surrogate
model.

Furthermore, there is a rich history of latent parameter inference in the
Bayesian (Lindley (1972); von Toussaint (2011)) and generative model (Zou &
Adams (2012)) literature. Bayesian belief networks (Holzinger (2018)) have
been explored for use in XAI.

###  A.2 Algorithms to mitigate catastrophic forgetting

While lifelong learning attempts to address various shortcomings of modern AI,
numerous lifelong learning algorithms pay special attention to _catastrophic
forgetting_ French (1999); McCloskey & Cohen (1989)). Catastrophic forgetting
is the behavior of an algorithm or model where, after training the model on
one task, training on a new task adversely effects model’s performance on the
first task; often to the point that the model performs worse than a model
behaving randomly. Nuance between how to calculate relevance and minimization
of adverse effects differentiates algorithms such as elastic weight
consolidation (EWC, Kirkpatrick et al. (2017)), continual learning through
synaptic intelligence (SI, Zenke et al. (2017)), and memory aware synapses
(MAS, Aljundi et al. (2018)). SI, for example, holds weights relatively fixed
according to the sensitivity of loss function with respect to them, whereas
MAS considers sensitivity of weights in the predictor itself. EWC, on the
other hand, uses Fisher information for determining how to update weights.
Gradient episodic memory (GEM, Lopez-Paz & Ranzato (2017)) like SI stores a
subset of previous task data and seeks to minimize cost function on current
labeled data point while constraining that loss on the episodic memory does
not increase.

###  A.3 Continual learning experimental categories

We make use of two different continual learning experimental categories in
this study which we briefly describe below. They are referred to as domain
incremental learning and task incremental learning.

_Domain incremental learning_ as defined by Hsu et al. (2018) and van de Ven &
Tolias (2018) implies that task identifiers are not given to the lifelong
learning algorithm. The algorithm must infer from the given data which task is
being performed.

_Task incremental learning_ is defined by Hsu et al. (2018) as prediction
where the task label is given to the algorithm. Lifelong learning approaches
for task adaptation often have an output prediction head for each task. During
inference, only the predictions from the current task head are considered.

##  Appendix B Analysis of CLAMP with Synthetic Algorithm Performance Data

We aimed to verify that our surrogate model functional form yields problems
that are identifiable – i.e., that a performance curve can be mapped back to a
unique set of parameters. We designed an experiment using synthetic data where
we could be certain of the underlying parameters governing the model. We
sampled ground truth parameters ($\mathbf{A},d,\gamma,h,$ and $\lambda$)
uniformly from $[-1,1]$ for $\mathbf{A}$ and $[0,1]$ for the other parameters,
and we generated a random curriculum of length $9$ from a set of five tasks.
Then we used ${E}^{a}_{j}(c_{l})$ and $\mathcal{P}_{j}(c_{l})$ to generate
synthetic lifelong learning performance data for 3 algorithms over a
curriculum composed of $5$ tasks.

Figure 2: Synthetic performance data and the fit of the surrogate model to the
data. Green bars on the $x$-axis for a given task indicate periods in the
curriculum where that task was trained. Our curriculum was randomly generated,
so Task 5 was never trained on. In all cases, the estimated performance curves
(dashed lines) overlap very closely to the synthetic performance data (solid
lines). Table 2: Parameter distributions and estimation errors for synthetic
scenario generation experiment

| Symbol | Ranges | MSE
---|---|---|---
Task Parameters |  |  |
Transfer Matrix | $\mathbf{A}$ | $[-1,1]$ | 0.12
Difficulty | $d$ | $[0,\infty)$ | 0.04
|  |  |
Algorithm Parameters |  |  |
Transfer Efficiency | $\gamma$ | $[0,\infty)$ | 0.02
Experience Retention | $h$ | $[0,1]$ | 0.0
Expertise Translation | $\lambda$ | $[0,\infty)$ | 0.01

To validate the model fitting approach to recover underlying model parameters,
we fit a randomly initialized surrogate performance model to the synthetic
performance data for 1000 epochs using an Adam optimizer (Kingma & Ba (2014))
with mean squared error as the loss function. The estimated curves match the
synthetic data with low error as expected. We then computed the mean squared
error between the ground truth parameters and the parameters in the fitted
model. The results are shown in Table 2. We can conclude that the approach
used to fit the explainable performance surrogate model can be used to recover
accurate estimates of the true underlying parameter values with small error.

##  Appendix C Analysis of MNIST and CIFAR100 Lifelong Learning Performance

Here we evaluate whether the strong assumptions we used to develop our model
of learning can describe real lifelong learning performance data for
classification algorithms. We designed experiments including the MNIST and
CIFAR100 datasets to quantitatively evaluate how well our model fits real
lifelong performance data. To generate performance data, we followed the
experimental protocol described by Hsu et al. (2018).

Performance curves from four classification algorithms were considered. The
first (NormalNN) makes use of a neural network where weights are transferred
from one task to the next. A second (L2) included $\ell^{2}$ regularization in
its loss function to minimize the change in weights from those that were
previously learned. Naive rehearsal (Naive_Rehearsal_[k]) was a third
algorithm that made use of an experience replay memory to store and retrain on
a subset of size $k$ of previously experienced data at random. Each training
batch was composed of equal parts from the current task and the experience
replay memory. The number of stored experiences over all tasks is a parameter
of the approach.

We also included curves from four state-of-the-art lifelong learning
algorithms: elastic weight consolidation (EWC, Kirkpatrick et al. (2017)),
online elastic weight consolidation (EWC_online, Schwarz et al. (2018)),
synaptic intelligence (SI, Zenke et al. (2017)), memory-aware synapses (MAS,
Aljundi et al. (2018)), and gradient episodic memory (GEM, Lopez-Paz & Ranzato
(2017)).

###  C.1 Domain adaptation with Split-MNIST

For MNIST, we considered the problem of domain-incremental learning. We
followed the experimental protocol described by Hsu et al. (2018), which we
briefly describe below.

Figure 3: Continual learning performance for incremental domain learning on
Split-MNIST along with performance curves estimated with _CLAMP_ . Naive
rehearsal was considered with experience storage limits of $1100$ and $4400$.

We used the dataset splitting procedure to create multiple binary
classification tasks (e.g., distinguish between images of 0 and images of 1)
from the MNIST dataset with 60k training images ($\sim$6,000 per digit) and
10,000 test images. The 32x32 greyscale images were normalized to have a mean
of zero and variance of one.

Table 3: Estimated latent properties of different algorithms for the MNIST
task-incremental learning experiment

Approach | $\gamma$ | $h$ | $\lambda$
---|---|---|---
GEM_1100 | 0.79 | 0.99 | 0.30
GEM_4400 | 0.88 | 1.00 | 0.11
EWC_online | 0.96 | 0.44 | 0.58
EWC | 0.86 | 0.44 | 0.56
MAS | 0.32 | 0.91 | 0.09
L2 | 0.74 | 0.53 | 0.65
Naive_Rehearsal_1100 | 1.02 | 0.89 | 0.35
Naive_Rehearsal_4400 | 0.92 | 0.97 | 0.63
NormalNN | 0.99 | 0.45 | 0.53

Table 4: Estimated task transfer scores for MNIST task-incremental learning
experiment

Task | 0,1 | 2,3 | 4,5 | 6,7 | 8,9
---|---|---|---|---|---
0,1 | 1.0 | 0.0 | 0.03 | 0.05 | -0.1
2,3 | -0.4 | 0.88 | 0.07 | 0.07 | 0.12
4,5 | -0.2 | 0.38 | 0.99 | -0.1 | -0.4
6,7 | 0.27 | -0.2 | -0.7 | 1.0 | 0.4
8,9 | -0.2 | 0.0 | -0.2 | -0.2 | 0.75

The results of our MNIST analysis are given in Fig. 3 (estimated and true
performance curves) and Tables 3 (estimated properties) and 4 (estimated
transfer scores). Averaged over each classification algorithm $a$, the
surrogate models achieved a squared error
$\text{mean}_{a}||\hat{P}^{a}-P^{a}||_{F}^{2}$ of $0.005$. The ability of the
surrogate model to fit the performance data with low error is a partial
validation of the formulation of the surrogate performance model. Table 4 is
the estimated task transfer matrix. The estimated task transfer diagonal was
strongly positive. The highest off-diagonal transfer estimate was that
training with the $6,7$ task is beneficial for the task $8,9$ task. We believe
this result follows intuition because of the character similarity between $6$
and $9$.

The estimated algorithm properties are shown in Table 3. At first glance it
might seem surprising that a normal neural network (Normal NN) had one of the
highest estimated transfer efficiencies $\gamma$ for the MNIST set of tasks,
given that we know that normal neural networks are ill-suited for continual
learning. One intuition for the result is that normal neural networks are able
to transfer well because their weight updates are unconstrained, while the
regularization in other methods prevents such rapid transfer. However, because
the normal neural network updates are unconstrained, performance on past tasks
is not maintained, while the regularized methods are able to maintain
performance on various tasks throughout the curricula.

###  C.2 CIFAR100 task-incremental learning

For analysis on CIFAR100, we used results from _task-incremental learning_. We
used the protocol by Hsu et al. (2018) for preparing the CIFAR100 dataset and
running the continual learning approaches.

Like the MNIST-task set, we used the dataset splitting procedure to create 5
binary classification tasks based on the CIFAR100 dataset. The 100 CIFAR
classes were subdivided at random into 10 classes. Each binary classification
task consisted of two of the 10 distinct classes. Data preprocessing resulted
in 32x32 images that were normalized with a mean of zero and variance of one.
To ensure a fair comparison among lifelong learning approaches, all lifelong
learning approaches used the same WideResNet (Zagoruyko & Komodakis (2016))
architecture.

The solid lines in Figure 4 show the performance (i.e. 1-error) for each of
the tasks across the curriculum. The highlighted green regions in the figure
illustrate when the task is actively being trained. As expected, we broadly
observe that performance of a task increases sharply when actively being
trained.

The fit of the surrogate performance model is shown as dashed lines in Figure
4, and illustrates broad general agreement between predicted performance and
the underlying data. We observed a MSE of $0.01$ between the predicted and
observed performance curves.

Figure 4: Continual learning performance for incremental task learning on
Split-CIFAR100 tasks of several lifelong learning algorithms along with the
performance estimated with _CLAMP_ . Naive rehearsal was considered with
experience storage limits of $1400$ and $5600$. Table 5: Estimated latent
properties of different algorithms for the CIFAR100 task-incremental learning
experiment

Approach | $\gamma$ | $h$ | $\lambda$
---|---|---|---
EWC_online | 0.35 | 1.0 | 2.11
EWC | 0.60 | 0.61 | 0.00
MAS | 0.33 | 0.94 | 1.04
L2 | 0.12 | 1.00 | 0.00
Naive_Rehearsal_1400 | 0.57 | 0.84 | 1.03
Naive_Rehearsal_5600 | 0.58 | 0.91 | 1.37
NormalNN | 0.52 | 0.62 | 0.00
SI | 0.39 | 0.93 | 1.21

Table 6: Estimated task transfer scores for CIFAR100 task-incremental learning
experiment

CIFAR100 Task | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
1 | 0.96 | 0.15 | 0.06 | 0.07 | 0.04
2 | -0.56 | 0.91 | 0.01 | -0.01 | 0.01
3 | 0.00 | -0.13 | 1.00 | 0.01 | 0.00
4 | 0.00 | 0.00 | -0.12 | 1.00 | 0.01
5 | 0.01 | 0.09 | 0.10 | -0.09 | 1.00

Table 7: Estimated task difficulty values for CIFAR100 task-incremental
learning experiment

CIFAR100 Task | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Difficulty | 0.0 | 0.3 | 0.3 | 0.3 | 0.3

In Table 8, we compared the estimated transfer efficiency of multiple
algorithms between the MNIST domain and CIFAR-100 domain. Each column was
colored from green to red relative to the values in the column. Between the
MNIST and CIFAR100 experiment there is broad agreement about the rank ordering
of algorithms with respect to transfer efficiency. This indicates that latent
property analysis might be useful to estimate the relative performance of a
lifelong learning algorithm from one dataset to another.

Table 8: Estimated transfer efficiencies scores for different algorithms
across MNIST and CIFAR100 task-incremental learning experiments

Algorithm | MNIST | CIFAR100
---|---|---
EWC_online | 0.96 | 0.35
EWC | 0.86 | 0.60
MAS | 0.32 | 0.33
L2 | 0.74 | 0.12
Naive_Rehearsal_Low | 1.02 | 0.57
Naive_Rehearsal_High | 0.92 | 0.58
NormalNN | 0.99 | 0.52
SI | 0.85 | 0.39

###  C.3 Additional Discussion

Our contributions include: (i) the introduction of the first explainable
algorithm-agnostic surrogate performance model of lifelong learning, (ii)
quantitative validation of the optimization procedure used to estimate the
metrics, (iii) qualitative validation of the lifelong learning surrogate
performance model using data produced by several popular lifelong learning
approaches on benchmark datasets including CIFAR100, MNIST, and Atari.

Our approach to task transfer score recovery is dataset and algorithm-
agnostic, and therefore generally applicable to all domains of lifelong
learning. The algorithm latent properties seem to be predictive of how the
algorithm may perform on other datasets.

A limitation of _CLAMP_ is that it makes a number of assumption about lifelong
learning with the goal of explaining in part the performance of lifelong
learning algorithms over diverse tasks. For example, we assume linear
experience transfer between tasks. In future work, it would be interesting to
explore relaxing the linear task transfer assumptions and consider non-linear
transfer.