11institutetext: Deparment of Computer Science, University of Pisa
22institutetext: 22email<EMAIL_ADDRESS>33institutetext: 33email:
<EMAIL_ADDRESS>
44institutetext: 44email<EMAIL_ADDRESS>
# Avalanche RL: a Continual Reinforcement Learning Library

Nicolò Lucchesi  Antonio Carta  Vincenzo Lomonaco

###### Abstract

Continual Reinforcement Learning (CRL) is a challenging setting where an agent
learns to interact with an environment that is constantly changing over time
(the stream of experiences). In this paper, we describe Avalanche RL, a
library for Continual Reinforcement Learning which allows to easily train
agents on a continuous stream of tasks. Avalanche RL is based on PyTorch [23]
and supports any OpenAI Gym [4] environment. Its design is based on Avalanche
[16], one of the more popular continual learning libraries, which allow us to
reuse a large number of continual learning strategies and improve the
interaction between reinforcement learning and continual learning researchers.
Additionally, we propose Continual Habitat-Lab, a novel benchmark and a high-
level library which enables the usage of the photorealistic simulator Habitat-
Sim [28] for CRL research. Overall, Avalanche RL attempts to unify under a
common framework continual reinforcement learning applications, which we hope
will foster the growth of the field.

###### Keywords:

Continual Learning Reinforcement Learning Reproducibility.

##  1 Introduction

Recent advances in data-driven algorithms, the so-called Deep Learning
revolution, has shown the possibility for AI algorithms to achieve
unprecedented performances on a narrow set of specific tasks. On the contrary,
humans are able to quickly learn new tasks and generalize to novel scenarios.
Continual Learning (CL) in the same way seeks to develop data-driven
algorithms able to incrementally learn behaviors from a stream of data.
Reinforcement Learning (RL) is yet another Machine Learning paradigm which
formulates the learning process as a sequence of interactions between an agent
and the environment. The agent must learn off of this interaction how to
achieve a goal of a particular task by taking actions in the environment while
receiving a (scalar) reward. Continual Reinforcement Learning (CRL) combines
the non-stationarity assumption of a stream of data with the RL setting,
having an agent learn multiple tasks in sequence.

While still in its early stages, CRL has seen a rising interest in
publications in recent years (according to Dimensions [10] data). To support
this growth, we focus on benchmarks and tools, introducing AvalancheRL: we
extend Avalanche [16], the staple framework for Continual or Lifelong
Learning, to support Reinforcement Learning in order to seamlessly train
agents on a continuous stream tasks.

Existing RL libraries [25, 21, 6, 24] do not focus on lifelong applications
and force us to write custom code to develop continual solutions. Avalanche
gives us re-usability by providing pre-implemented CL strategies as well as
code structure when experimenting with them, but lacked support altogether
when coming to RL. Related CRL projects instead either focus on providing a
specific benchmark [33] or combine multiple frameworks results [22], limiting
the overall flexibility and methods customization options.
Avalanche RL attempts to address both problems aiming to offer a malleable
framework encompassing a variety of RL algorithms with fine-grained control
over their internals, leveraging pre-existing CL techniques to learn
efficiently from the interaction with multiple environments. In particular, we
support any environment exposing the OpenAI Gym gym.Env interface.

The availability of compelling benchmarks has always lead the progress of
data-driven algorithms [14, 13, 5], therefore our second effort is aimed at
providing a challenging dataset for realistic Continual Reinforcement
Learning.
Habitat-Lab allows an embodied agent to roam a photorealistic (typically
indoor) scene in the attempt of solving a particular task; unfortunately, it
does not offer support for the continual scenario. Therefore, we developed
Continual Habitat-Lab, a high-level library enabling the usage of Habitat-Sim
[28] for CRL, allowing the creation of sequences of tasks while integrating
with Avalanche RL.

We first outline the design principles that guided the development of
Avalanche RL (Section 2), describe its structure (Figure 1) and go over the
main features of the framework with code examples (Section 3). We then
introduce Continual Habitat-Lab and describe its integration with Avalanche RL
(Section 4).

All the source code of the work hereby presented is publicly available on
GitHub for both Avalanche RL111https://github.com/continualAI/avalanche-rl and
Continual Habitat-Lab222https://github.com/NickLucche/continual-habitat-lab.

Figure 1: Avalanche RL core-functionalities overview. The Benchmarks module
capabilities, providing access to a stream of environments, are addressed in
Section 3.1. Data is obtained through (parallel, Section 3.3) interaction with
the stream and it is consumed by the algorithm in the learning process, as
motivated in Section 3.2. Streams can be easily created through benchmark
generators (right-hand side).

##  2 Design Principles

Avalanche RL is built as an extension of Avalanche [16], and it retains the
same design principles and a similar API. The target users are practitioners
and researchers, and therefore the library must be simple, allowing to setup
an experiment with a few lines of code, as well as highly customizable. As a
result, Avalanche RL provides high-level APIs with ready-to-use components, as
well as low-level features that allow heavy customization of existing
implementations by leveraging an exhaustive callback system (Section 3.2).
Avalanche RL codebase is comprises 5 main modules: Benchmarks, Training,
Evaluation, Models, and Logging. We give a brief overview of them in the
remainder of this section, but we refer the reader to [16] for more details
about the general architecture of Avalanche.

####  2.0.1 Benchmarks

maintains a uniform API for data handling, generating a stream of data from
one or more datasets, conveniently divided into temporal experiences; this is
the core abstraction over the task stream formalism which is distinctive of CL
and it is accessible through a Scenario object. In order to create benchmarks
more easily, this module provides benchmark generators which allow one to
specify particular configurations through a simple API.

####  2.0.2 Training

provides all the necessary utilities concerning model training. It includes
simple and efficient ways of implementing new strategies as well as a set pre-
implemented CL baselines and state-of-the-art algorithms. A Strategy abstracts
a general learning algorithm implementing a training and an evaluation loop
while consuming experiences from a benchmark. Continual behaviors can be added
when needed through Plugins: they operate latching on the callback system
defined by Strategies and are designed in such a modular way so that they can
be easily composed to provide hybrid behaviors.

####  2.0.3 Evaluation

provides all the utilities and metrics that can help evaluate a CL algorithm.
Here we can find pluggable metric monitors such as (Train/Test/Batch)
Accuracy, RAM, CPU and GPU usage, all designed with the same modularity
principles in mind.

####  2.0.4 Models

contains several model architectures and pre-trained models that can be used
for continual learning experiments (similar to torchvision.models), from
simple customizable networks to implementation of state-of-the-art models.

####  2.0.5 Logging

includes advanced logging and plotting features with the purpose of
visualizing the metrics of the Evaluation module, such as highly readable
output, file and TensorBoard support.

###  2.1 Notation

We adopt the well renowned notation from [32] for Reinforcement Learning
related formulations while we make use of the formalization introduced in [15]
regarding Continual Learning.
In particular, we refer to the RL problem as consisting of a tuple of five
elements commonly denoted as
$<\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma>$ in the MDP
formulation, where $\mathcal{S}$ and $\mathcal{A}$ are sets of states and
actions, respectively. $\mathcal{R}$ or $r()$ is the reward function, with
$r(s,a,s^{\prime})$ being the expected immediate reward for transition from
state $s\in\mathcal{S}$ to $s^{\prime}\in\mathcal{S}$ under action
$a\in\mathcal{A}$. $\mathcal{P}$ or $p()$ is the transition function defining
the dynamics of the environment, with $p(s^{\prime},r|s,a)$ denoting the
probability of transitioning from $s$ into $s^{\prime}$ with scalar reward $r$
under $a$. Finally, $\gamma$ represents the discount factor which weights the
importance of immediate and future rewards.
An agent follows a policy $\pi$, which maps states to action probabilities. In
Deep RL, learned policies are parameterized function (such as a neural
network) which we indicate with $\pi_{\theta}$.
We refer to a Dataset as a collection of samples $\\{x_{i}\\}_{i}^{N}$,
optionally with labels $\\{<x_{i},y_{i}>\\}_{i}^{N}$ in the case of supervised
learning. We then denote a general task to be solved by some agent with $\tau$
and define the data relative to that task with $D_{\tau}$.

##  3 Avalanche RL

CRL applications in Avalanche RL are implemented by modeling the interaction
between core components: the task-stream abstraction (i.e., the continuously
changing environment) and the RL strategy (i.e., the agent and its learning
algorithm).

Avalanche RL implements these two components in the Benchmarks and Training
module, respectively. In the remainder of this section, we describe the
environment and the implementation of its continual shift in Section 3.1.
Then, in Section 3.2, we describe the implementation of RL algorithms and
their integration in the Training module. Section 3.3 and 3.4 highlight some
important implementation details and useful features offered by the framework,
such as the automatic parallelization of the RL environment.

###  3.1 Benchmarks: Stream of Environments

Most continual learning frameworks [15] assume that the stream of data is made
of static datasets of a fixed size. Instead, in CRL problems the stream
consists of different environments, and samples are obtained through the
interaction between the agent and the environment.

To support streams of environments, Avalanche RL defines a stream
$S=\\{e_{1},e_{2},..\\}$ as a sequence of experiences $e_{i}$, where each
experience provides access to an environment with which the agent can interact
to generate state transitions (samples) online. Over time, this means that the
agent learns by interacting with a stream of environments
$\\{\mathcal{E}_{1},\mathcal{E}_{2},..\\}$, as in Figure 1. In the source
code, RLExperience is the class which defines the CRL experience.

Using this task-stream abstraction, it is easy to define CRL benchmarks as a
set of parallel streams of environments. Notice that each experience may be a
small shift, such as a change in the background, as well as a completely
different tasks, such as a different game. Different tasks may provide a task
label which can be used by the agent to distinguish among them. The RLScenario
is the class responsible for the CRL benchmark’s definition, and it can be
thought as a container of streams.

RL Environments implement a common interface, which is the one of OpenAI Gym
environments. This common interface allows to abstract away the interaction
with the environment, decoupling the data generation process from the data
sampling and freeing the user from the hassle of manually re-writing the data-
fetching loop.
New CRL benchmarks can be easily created using the gym_benchmark_generator,
which allows to define an RLScenario by providing any sequence of Gym
environments (including custom ones). We can see an example in Fig. 2, in
which we instantiate an RLScenario handling a stream of tasks which gives
access to two randomly sampled environments.
Note that unlike static datasets, the environment can be used to produce an
endless amount of data. Therefore, the interaction with the experience must be
explicitly limited by some number of steps or episodes rather than epochs,
which we can express during the creation of a Strategy as in Section 3.2.

(a) Benchmark creation

(b) Minimal training setup

Figure 2: Example of Avalanche RL usage. (a) defines a task stream alternating
two randomly sampled environments for 4 experiences. n_parallel_envs specifies
the number of parallel actors (Section 3.3). The second scenario instead
creates a stream of 2 Atari games with pre-processing attached. (b) puts
everything together, instantiating a pre-implemented model (Section 3.4) and
creating an “A2C agent” which is trained on the stream of games. The agent
will perform 10000 Update steps per-experience while gathering 5 data samples
at every Rollout step (Section 3.2). Evaluation will take place with the
specified parameters.

As the Atari game suite [3] has become the main benchmark for RL algorithms in
recent years, we also provide a tailored atari_benchmark_generator (Fig. 2)
which takes care of adding common pre-processing techniques (e.g. frame
stacking) as Gym Wrappers around each environment. This allows to minimize the
time in between experiments as one can easily reproduce setups such as the one
in [12] (sampling random Atari games to learn in sequence) by simply
specifying a few arguments when creating a scenario. The benchmark interface
also promotes the pattern of environment wrapping, which is Gym’s intended way
of organizing data processing methods to favor reproducibility.
Reproducibility of experiments in particular is of great importance to
Avalanche and one of the main reasons that drove us to propose an end-to-end
framework for CRL.

###  3.2 Training: Reinforcement Learning Strategies

Avalanche RL provides several learning algorithms implementations which have
been implemented to be highly modular and easily customizable. The framework
offers full-access to their internals in order to provide fine-grained
customization options and specific support for continual learning techniques.
There are two main patterns to adapt a learning algorithm: subclassing and
Plugins (as introduced in Section 2). In particular, Avalanche RL implements
most continual learning strategies as Plugins. The modularity of the
implementation allows to combine many RL strategies with popular CL
strategies, such as replay buffers, regularization methods, and so on. As far
as we are aware, Avalanche is the only library that allows the seamless
composition of different learning algorithms. RL strategies inherit from
RLBaseStrategy, a class which provides a common “skeleton” for both on and
off-policy algorithms and abstracts many of the most repetitive patterns,
including environment interaction, tracking metrics, CPU-GPU tensor relocation
and more. RLBaseStrategy also provides callbacks which can be used by plugins.
Inspired by the open-source framework stable-baseline3 [25] (sb3), RL
strategies are divided into two main steps: rollout collection and update.
Unlike sb3, we grouped both on and off-policy algorithms under this simple
workflow.
The rollout stage abstracts the data gathering process which iterates the
following steps:

  1. 1.

$a_{t}\sim\pi_{\theta}(s_{t})$: sample_rollout_action, to be implemented by
the specific algorithm, returns the action to perform during a rollout step.

  2. 2.

play action and observe next state and reward: s’, r, done,
info=env.step($a_{t}$) referring to Gym interface.

  3. 3.

store state transition in some data structure: Step. Store multiple Steps in a
Rollout. These data structures are optimized for insertion speed and lazily
delay “re-shaping” operations until they are needed by the update phase.

  4. 4.

test rollout terminal condition, number of steps or episodes to run.

The update step is instead entirely delegated to the specifics of the
algorithm: it boils down to implementing a method which has access to the
rollouts collected at the previous stage and must define and compute a loss
function which is then used to execute the actual parameters update. To enable
the user with fine-grained control over the strategy workflow, we added
callbacks which are executed just before and after the two stages.
At a higher level, the workflow we described happens within a single
experience. To learn from a stream, the process is repeated for each
experience in the stream, a behavior which is implemented by the
RLBaseStrategy.
To summarize, one can implement a RL algorithm by sub-classing RLBaseStrategy
and implementing the sample_rollout_action and update step. For example, A2C
can be implemented in less than 30 lines of code
333https://github.com/ContinualAI/avalanche-
rl/blob/master/avalanche_rl/training/strategies/actor_critic.py.
Alternatively, customization of any algorithm is always possible by
implementing a plugin, which allows to “inject” additional behavior, or by
subclassing any of the available strategies. All the algorithm implementations
expose their internals through class attributes, so one can for instance
access the loss externally (e.g. from plugins) simply with strategy.loss.
Along with the release of our framework we provide an implementation of A2C
and DQN [18], including popular “variants” with target network [19] and
DoubleDQN [9].

###  3.3 Parallel Actors Interaction: VectorizedEnv

Since the data gathering supports any environment exposing the Gym interface,
we are also able to automatically parallelize the agent-environment
interaction in a transparent way to the user. This common practice [17, 25, 6]
relies on using multiple Actors, each owning a local copy of the environment
in which they perform actions, while synchronizing updates on a shared
network; varying the amount of local resources available to each worker we can
obtain different degrees of asynchronicity [7], allowing to scale computations
on multiple CPUs.

To implement this behavior we leveraged Ray [20], a framework for parallel and
distributed computing with a focus on AI applications. Ray abstracts away the
parallel (and distributed) execution of code, sharing data between master and
workers by serializing numpy arrays, which, in the case of execution on a
single machine, are written once to shared memory in read-only mode and only
referred to by actors.

This feature is opaque to the user, as it happens entirely inside a
VectorizedEnv: this component wraps a single Gym environment and exposes the
same interface, while under the hood it instantiates a pool of actors and
handles results gathering and synchronization, acting as master. The API of
our implementation was inspired by the work of sb3, although we opted to use
Ray as a backend instead of Python’s multiprocessing library due to
distributed setting support.
RLBaseStrategy takes care of wrapping any environment with a VectorizedEnv, so
the user can exploit parallel execution by simply specifying the number of
workers/environment replicas, as shown in Figure 2.

###  3.4 Additional Features

To complement the features we described in the previous sections we also
implemented a series of utility components which one expects from a
serviceable framework. Most of the changes listed in this section are not as
important when taken singularly but as a whole they contribute significantly
to Avalanche RL functionalities and as such they are hereby reported.

  * •

Models from seminal papers such as [19, 9, 18, 17] have been re-implemented in
Pytorch and are available in the Models module.

  * •

Evaluation Metrics: RLBaseStrategy automatically records gathered rewards and
episode lengths during training, smoothing scalars with a window average by
default. Additionally, one can record any significant value (e.g. loss,
$\epsilon$-greedy’s $\epsilon$) with minimal effort thanks to improved metrics
builders.

  * •

Continual Control Environments: classic control environments provided by Gym
have been wrapped in order to expose hard-coded parameters (e.g. gravity,
force..) which can now be modified to obtain varying conditions. This is
useful for rapidly testing out algorithms on well renowned problems.

  * •

Extended available Plugins, including EWC [12] and a ReplayMemory-based one
inspired by works from [27] and [11].

  * •

Miscellaneous tools such as environment wrappers for easily re-mapping actions
keys (useful when learning multiple games with a single network) or reducing
the action set and an additional logger with improved readability. Avalanche
RL is compatible with Avalanche logging methods, such as Tensorboard [1].

##  4 Continual Habitat Lab

Continual-Habitat-Lab (CHL) is a high-level library for FAIR’s simulator
Habitat [28]: inspired by Habitat-Lab, we created a library with the goal of
adding support for continual learning. CHL defines the abstraction layer
needed to work with a stream of tasks $\\{\tau_{1},\tau_{2}..\\}$, the core of
CL systems.
We designed the library to be a shallow wrapper on top of Habitat-Sim
functionalities and API while “steering” its intended usage toward learning
applications, enforcing the data generation process to be carried out through
online interaction and dropping the need for a pre-computed Dataset of
positions altogether.
We also revisited the concept of Task to make it simpler and yet give it more
control over the environment: while the next-state transition function
$p(s^{\prime}|s,a)$ is implemented by the dynamics of the simulator (Habitat-
Sim), we bundled the reward function $r$ into the task definition. To define a
Task one must hence define a reward function $r(s,a,s^{\prime})\rightarrow r$,
a goal test function $g(s)\rightarrow\\{T,F\\}$ and an action space
$\mathcal{A}$ as defined by Gym.
As Task is meant to be the main component through which the user can inject
logic and behavior to be learned by the agent, we give direct access to the
simulator at specific times through callbacks (e.g. to change environment
condition, lighting, add objects..).

In order to to natively support CRL a TaskIterator is assigned to the handling
of the stream of tasks, hiding away the logic behind task sampling and
duration while giving access to the current active task to be used by the
environment.
We leveraged the multitude of 3D scenes datasets compatible with Habitat-Sim
with the goal of specifying changing environment conditions, a most important
feature to CL. To do so, we bundled the functionalities regarding scene switch
in a sole component named SceneManager. It provides utilities for loading and
switching scenes with a few configurable behaviors: scene swapping can happen
on task change or after a number of episodes or or actions is reached, even
amid a running episode, maintaining current agent configuration and avoiding
any expensive simulator re-instantions.

To offer a easily configurable system we re-designed the configuration system
from scratch basing it on the popular OmegaConf library for Python: apart from
providing a unified configuration entry-point which can be created
programmatically or from a yaml file, the system dynamically maps Task and
Actions parameters to configuration options. This allows the user to change
experiments conditions by changing class arguments directly from the
configuration file.

Continual Habitat Lab is integrated with Avalanche RL through a specialized
benchmark generator (habitat_benchmark_generator) that takes care of
synchronizing the stream of tasks defined in the CHL configuration with the
one served to a Strategy. It does so by defining an experience each time a
task or scene is changed, while serving the same object reference to the
Habitat-Sim environment.

##  5 Conclusion and Future Work

In this paper we presented two novel libraries for Continual Reinforcement
Learning: Avalanche RL and Continual Habitat Lab. We believe that these
libraries can be helpful for the CRL community by extending and adapting work
from the Continual Learning community on supervised and unsupervised continual
learning (Avalanche) while also integrating a realistic simulator (Habitat-
Sim) to benchmark CRL algorithms on complex embodied real-life scenarios.

In particular, Avalanche RL allows to easily train and evaluate agents on a
continual stream of tasks defined as a sequence of any Gym Environment. It is
based on implementing a simple API upon the interaction of RL algorithms and
task-streams, while still offering a fine-grained control over their
internals.
Using Avalanche allows to exploit and extend the large amount of work done by
the Continual Learning community while providing RL algorithms that are highly
modular and easily extensible. The library implements a large set of highly
desirable features, such as parallel environment interaction, and provides
implementations for popular baselines such as EWC [12], including benchmarks,
learning strategies and architectures, all of which can be easily instantiated
with a single line of code.
Avalanche RL can improve code reusability, ease-of-use, modularity and
reproducibility of experiments, and we strongly believe that the whole CRL
community would benefit from a collective effort such as Avalanche RL as a
tool to speed-up the research in the field.

Having the goal of providing a shared and collaborative open-source codebase
for CRL applications, Avalanche RL is constantly looking to add and refine
functionalities. In the short term, we plan to implement a broader range of
state-of-the art RL algorithms, including (but not limited to) PPO [30], TRPO
[29] and SAC [8]. Additionally, we are also looking to increment the number of
CL strategies such as pseudo-rehersal [26, 2].
We are aiming to keep on expanding the supported simulators targeting a wider
range of applications, from robotics to games engines [31] to widen the CRL
benchmarks suite. Finally, we are expecting to merge Avalanche RL into
Avalanche, striving to provide a single end-to-end framework for all continual
learning applications.

## References

  * [1] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015), https://www.tensorflow.org/, software available from tensorflow.org
  * [2] Atkinson, C., McCane, B., Szymanski, L., Robins, A.V.: Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. CoRR abs/1812.02464 (2018), http://arxiv.org/abs/1812.02464
  * [3] Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research Vol. 47, 253–279 (2012). https://doi.org/10.1613/jair.3912, http://arxiv.org/abs/1207.4708, cite arxiv:1207.4708
  * [4] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. CoRR abs/1606.01540 (2016), http://arxiv.org/abs/1606.01540
  * [5] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [6] Denoyer, L., la Fuente, A.D., Duong, S., Gaya, J., Kamienny, P., Thompson, D.H.: Salina: Sequential learning of agents. CoRR abs/2110.07910 (2021), https://arxiv.org/abs/2110.07910
  * [7] Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., Kavukcuoglu, K.: Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures (2018)
  * [8] Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR abs/1801.01290 (2018), http://arxiv.org/abs/1801.01290
  * [9] van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. CoRR abs/1509.06461 (2015), http://arxiv.org/abs/1509.06461
  * [10] Hook, D.W., Porter, S.J., Herzog, C.: Dimensions: Building context for search and evaluation. Frontiers in Research Metrics and Analytics 3, 23 (2018). https://doi.org/10.3389/frma.2018.00023, https://www.frontiersin.org/article/10.3389/frma.2018.00023
  * [11] Isele, D., Cosgun, A.: Selective experience replay for lifelong learning. CoRR abs/1802.10269 (2018), http://arxiv.org/abs/1802.10269
  * [12] Kirkpatrick, J., Pascanu, R., Rabinowitz, N.C., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. CoRR abs/1612.00796 (2016), http://arxiv.org/abs/1612.00796
  * [13] Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (canadian institute for advanced research) http://www.cs.toronto.edu/~kriz/cifar.html
  * [14] LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010), http://yann.lecun.com/exdb/mnist/
  * [15] Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., Díaz-Rodríguez, N.: Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information Fusion 58, 52–68 (2020). https://doi.org/https://doi.org/10.1016/j.inffus.2019.12.004, https://www.sciencedirect.com/science/article/pii/S1566253519307377
  * [16] Lomonaco, V., Pellegrini, L., Cossu, A., Carta, A., Graffieti, G., Hayes, T.L., Lange, M.D., Masana, M., Pomponi, J., van de Ven, G., Mundt, M., She, Q., Cooper, K., Forest, J., Belouadah, E., Calderara, S., Parisi, G.I., Cuzzolin, F., Tolias, A., Scardapane, S., Antiga, L., Amhad, S., Popescu, A., Kanan, C., van de Weijer, J., Tuytelaars, T., Bacciu, D., Maltoni, D.: Avalanche: an end-to-end library for continual learning (2021)
  * [17] Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. CoRR abs/1602.01783 (2016), http://arxiv.org/abs/1602.01783
  * [18] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning (2013)
  * [19] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (Feb 2015), http://dx.doi.org/10.1038/nature14236
  * [20] Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M.I., Stoica, I.: Ray: A distributed framework for emerging ai applications (2018)
  * [21] Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Paul, W., Jordan, M.I., Stoica, I.: Ray: A distributed framework for emerging AI applications. CoRR abs/1712.05889 (2017), http://arxiv.org/abs/1712.05889
  * [22] Normandin, F., Golemo, F., Ostapenko, O., Rodríguez, P., Riemer, M.D., Hurtado, J., Khetarpal, K., Zhao, D., Lindeborg, R., Lesort, T., Charlin, L., Rish, I., Caccia, M.: Sequoia: A software framework to unify continual learning research. CoRR abs/2108.01005 (2021), https://arxiv.org/abs/2108.01005
  * [23] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), https://pytorch.org/
  * [24] Plappert, M.: keras-rl. https://github.com/keras-rl/keras-rl (2016)
  * [25] Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., Dormann, N.: Stable baselines3. https://github.com/DLR-RM/stable-baselines3 (2019)
  * [26] Robins, A.V.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7, 123–146 (1995)
  * [27] Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T.P., Wayne, G.: Experience replay for continual learning. CoRR abs/1811.11682 (2018), http://arxiv.org/abs/1811.11682
  * [28] Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., Batra, D.: Habitat: A platform for embodied ai research (2019)
  * [29] Schulman, J., Levine, S., Moritz, P., Jordan, M.I., Abbeel, P.: Trust region policy optimization. CoRR abs/1502.05477 (2015), http://arxiv.org/abs/1502.05477
  * [30] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017)
  * [31] Schwarz, J., Altman, D., Dudzik, A., Vinyals, O., Teh, Y.W., and, R.P.: Towards a natural benchmark for continual learning (2018), https://marcpickett.com/cl2018/CL-2018_paper_48.pdf
  * [32] Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, second edn. (2018), http://incompleteideas.net/book/the-book-2nd.html
  * [33] Wolczyk, M., Zajac, M., Pascanu, R., Kucinski, L., Milos, P.: Continual world: A robotic benchmark for continual reinforcement learning. CoRR abs/2105.10919 (2021), https://arxiv.org/abs/2105.10919