# ARLO: A Framework for Automated Reinforcement Learning

Marco Mussi
Politecnico di Milano
<EMAIL_ADDRESS>
&Davide Lombarda
Politecnico di Milano, ML cube
<EMAIL_ADDRESS>
&Alberto Maria Metelli
Politecnico di Milano
<EMAIL_ADDRESS>
&Francesco Trovò
Politecnico di Milano
<EMAIL_ADDRESS>
&Marcello Restelli
Politecnico di Milano
<EMAIL_ADDRESS>

###### Abstract

Automated Reinforcement Learning (AutoRL) is a relatively new area of research
that is gaining increasing attention. The objective of AutoRL consists in
easing the employment of Reinforcement Learning (RL) techniques for the
broader public by alleviating some of its main challenges, including data
collection, algorithm selection, and hyper-parameter tuning. In this work, we
propose a general and flexible framework, namely ARLO: Automated Reinforcement
Learning Optimizer, to construct automated pipelines for AutoRL. Based on
this, we propose a pipeline for offline and one for online RL, discussing the
components, interaction, and highlighting the difference between the two
settings. Furthermore, we provide a Python implementation of such pipelines,
released as an open-source library. Our implementation has been tested on an
illustrative LQG domain and on classic MuJoCo environments, showing the
ability to reach competitive performances requiring limited human
intervention. We also showcase the full pipeline on a realistic dam
environment, automatically performing the feature selection and the model
generation tasks.

_K_ eywords Automated Reinforcement Learning $\cdot$ AutoRL $\cdot$ AutoML

##  1 Introduction

Reinforcement Learning (RL, Sutton and Barto,, 2018) has recently achieved
successful results in solving several complex control problems, including
autonomous driving (Wang et al.,, 2018), robot manipulators (Nguyen and La,,
2019), and finance (Zhang et al.,, 2020). These outstanding achievements are
rooted in the employment of powerful training algorithms combined with complex
model representations, such as deep neural networks (Arulkumaran et al.,,
2017). Unfortunately, empirical experience suggests that this class of
approaches heavily depends on fine-tuning, where an inaccurate choice of the
hyper-parameters makes the difference between learning the optimal policy and
not learning at all (Buşoniu et al.,, 2018). This represents an indubitable
limitation, making this powerful tool not immediately usable by non-expert
users. While this scenario is common even in general Machine Learning (ML,
Bishop and Nasrabadi,, 2006), the inherent complexity of RL, due to the
sequential nature of the problem, exacerbates this issue even more.

The research effort towards the democratization of ML has reached a mature
level of development for supervised learning. Indeed, several Automated
Machine Learning (AutoML) frameworks and corresponding libraries have been
developed and tested, such as the ones proposed by Feurer et al., (2015,
2020); LeDell and Poirier, (2020); Olson et al., (2016). AutoML is intended to
automate the whole ML pipeline, starting from the preliminary operations on
the data, ending with the trained and evaluated final model. This way, the
complete ML process can be regarded, by the non-expert user, as a black-box,
abstracting from the unnecessary details and favoring the adoption of ML as a
production tool. For a detailed review of the currently available AutoML
frameworks, we refer the reader to the recent survey by He et al., (2021).
Conversely, RL is currently far from being a tool usable by a non-expert user
since a complete and reliable Automated Reinforcement Learning (AutoRL)
pipeline is currently missing. This automation gap between RL and supervised
learning is even more severe from a theoretical perspective since, to the best
of our knowledge, a general and flexible notion of AutoRL pipeline has not
been formalized yet.

Recently, a surge of scientific works in the RL field (Parker-Holder et al.,,
2022; Afshar et al.,, 2022) attempted to tackle either specific stages of the
RL pipeline _individually_ (e.g., feature construction, policy generation), or
focus on specific application scenarios. While providing a vast analysis of
the available approaches for each single stage, they review the state-of-the-
art to solve single tasks individually and do not propose a full pipeline and
do not study the peculiarities characterizing the _interaction_ between such
stages. On the other hand, a naïve adaptation of the existing automated
pipelines designed for AutoML to the RL setting is not a viable approach since
they fail to capture the unique characteristics of RL related to the presence
of an interacting environment and the sequential nature of the learning
problem.

Contributions In this paper, we make a step towards the formalization of an
AutoRL framework. The contributions of this work can be synthesized as
follows.

  * •

We propose a general and flexible formalization of a _pipeline_ for AutoRL.
Grounding on such a definition, we instantiate it for two different scenarios:
_offline_ and _online_ RL.111The reader might be tempted to address the
offline RL setting with AutoML, given the fixed available dataset and, thus,
the similarity with supervised learning. We stress that this choice is
inappropriate as the peculiarities of RL are still crucial, especially the
sequential properties of the problem. The former assumes that the RL process
is carried out based on a fixed _batch_ of data. The latter takes into account
the availability of an interactive environment.

  * •

We describe the individual _stages_ of the two pipelines and their respective
characteristics, highlighting the _interactions_ between them and focusing on
their inputs and outputs. Furthermore, we discuss the corresponding _units_ ,
i.e., possible implementations of stages, and introduce a general approach to
select the best-tuned unit in a finite set.

  * •

We provide an implementation of the framework in an open-source Python
library, called ARLO.222The library is available at https://github.com/arlo-
lib/ARLO. The library contains the implementation of all the stages, the two
RL pipelines, and the needed tools to run, optimize, and evaluate the
pipelines.

  * •

Finally, we test the implementation on an illustrative LQG and a MuJoCo
environment, showing the ability to reach optimal performances without
requiring any manual adjustment by humans. At last, we provide an experiment
on a realistic dam environment with a pipeline performing the data generation,
feature selection, policy generation, and policy evaluation stages.

Given the wide variety of RL problems and solutions, we restrict our
formalization to the case of _stationary_ and _fully observable_ environments.
We leave the extension to more complex settings (e.g., multi-objective, multi-
agent, lifelong) as a future work.

Limitations and Broader Impact Statement The goal of AutoRL is to bring RL
closer to the non-expert user. This represents a source of opportunities and
risks. On the one hand, making RL usable to a wide audience contributes to the
_democratization_ of the field, overcoming the need for specific education and
opening it to the large public. On the other hand, such an abstract approach
tends to compromise the transparency of the learning process and traceability
of the resulting model. Shadowing the underlying principles, AutoRL might pose
the risk of misuse of RL approaches, leading to results not in line with
expectations. Furthermore, AutoRL, even more than RL, requires huge amounts of
data and computation that might represent a limit of the framework.

Outline The paper is structured as follows. In Section 2, we present the
fundamental notions of Markov Decision Processes and the basics of RL. In
Section 3, we introduce a general notion of pipeline, stage, and unit. In
Section 4, we present the online and offline pipelines for RL. In Section 5,
we describe the details of the components included in the two pipelines. In
Section 6, we report the results of the tests performed on standard benchmarks
and on a realistic environment. In Section 7, we highlight the conclusions of
our works and we propose future research lines.

##  2 Preliminaries

A Markov Decision Process (MDP, Puterman,, 2014) is defined as a tuple
$\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma,\mu_{0})$, where
$\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions,
$P(s^{\prime}|s,a)$ is the state transition model, specifying the probability
to land in state $s^{\prime}$ starting from state $s$ and performing action
$a$, $R(s,a)$ is the reward function, defining the expected reward when the
agent is in state $s$ and performs action $a$, $\gamma\in[0,1]$ is the
discount factor, and $\mu_{0}(s)$ is the initial state distribution. The
agent’s behavior is defined in terms of a policy $\pi(a|s)$ defining the
probability of performing action $a$ in state $s$.

Interaction Protocol The initial state is sampled from the initial-state
distribution $s_{0}\sim\mu_{0}$, the agent selects an action based on its
policy $a_{0}\sim\pi(\cdot|s)$, the environment provides the agent with the
reward $R(s_{0},a_{0})$, and the state evolves according to the transition
model $s_{1}\sim P(\cdot|s_{0},a_{0})$. The process is repeated for $T$ steps,
where $T\in\mathbb{N}\cup\\{+\infty\\}$ is the (possibly infinite) horizon.

Objective The goal of RL consists in learning an _optimal_ policy $\pi(a|s)$,
i.e., a policy maximizing the expected discounted sum of the rewards, a.k.a.
the _expected return_ (Sutton and Barto,, 2018):

$J(\pi)\coloneqq\mathbb{E}^{\pi}\bigg{[}\sum_{t=0}^{T-1}\gamma^{t}R(s_{t},a_{t})\bigg{]},$
(1)

where the expectation $\mathbb{E}^{\pi}[\cdot]$ is computed w.r.t. the
randomness of environment and policy.

Environments and Datasets We introduce the notion of _environment_ and
_dataset_. Formally, an environment $\mathcal{E}$ is a device to interact with
the underlying MDP, that, given a state $s_{t}$ and an action $a_{t}$, it
provides the next state $s^{\prime}_{t}\sim P(\cdot|s_{t},a_{t})$ and the
reward $r_{t}=R(s_{t},a_{t})$. An environment is a _generative model_ if it
allows to freely choose the state $s_{t}$ at each step, or a _forward model_
if, instead, we can perform steps in the MDP ($s_{t+1}=s^{\prime}_{t}$) or
start again sampling $s_{t}$ from the initial-state distribution $\mu_{0}$. A
dataset $\mathcal{D}\coloneqq\\{\tau_{i}\\}_{i=1}^{n}$ is a set of
trajectories $\tau_{i}$, where each _trajectory_ is a sequence
$\tau_{i}=(s_{i}^{0},a_{i}^{0},r_{i}^{1},\ldots,s_{i}^{T_{i}-1},a_{i}^{T_{i}-1},r_{i}^{T_{i}},s_{i}^{T_{i}})$
and $T_{i}$ is the length of the trajectory.

Online vs. Offline RL We distinguish between two main groups of RL algorithms:
_online_ and _offline_ RL. The online RL algorithms (Sutton and Barto,, 2018)
aim at learning a policy $\pi$ by directly interacting with an environment
$\mathcal{E}$. Typically they employ the last available policy to collect data
and leverage the experience to improve it. Conversely, the offline RL paradigm
(Levine et al.,, 2020) consists in carrying out the policy learning on a
dataset $\mathcal{D}$ previously collected.333Even in this case, we may have
an environment $\mathcal{E}$ to test the performance of the learned policy.
Commonly, it is a less costly version, e.g., in terms of computational or real
costs, of the environment where the final policy will be applied. The ability
to learn a (near-)optimal policy heavily depends on the exploration properties
of the dataset $\mathcal{D}$.

Regarding offline RL, several works covered its peculiarities. In Levine et
al., (2020) the authors survey the field of offline RL, presenting open
problems, unique challenges and limitations. In Paine et al., (2020) the
authors focused on the evaluation problem present in offline RL, namely on the
evaluation of a learnt policy without resorting to an environment.
Furthermore, this work showcases how offline RL algorithms are not robust with
respect to hyper-parameters tuning.

##  3 Framework

In this section, we present the abstract formalization of the proposed AutoRL
pipeline, detailing the notions of _pipeline_ , _stage_ , and _unit_.

Stages and Pipelines A stage $\psi$ represents a single component of the
pipeline with a specific _purpose_. For instance, the portion of the pipeline
in charge of performing feature engineering is regarded as a stage. A stage
$\psi$ interacts with the other stages of the pipeline by means of an
_interface_ , defining its inputs and outputs. We denote a stage’s inputs with
$\mathrm{In}(\psi)$ and its outputs with $\mathrm{Out}(\psi)$. A pipeline is a
sequence of $m\in\mathbb{N}$ stages $\Psi=(\psi_{1},\dots,\psi_{m})$. The
possibility of staking specific stages in sequence depends, in general, on
problem-dependent constraints.

Units A unit constitutes the actual _implementation_ of the stages
corresponding to algorithms that are in charge of generating the output
required by the corresponding stage.444From a software engineering
perspective, a stage is an abstract class, while a unit a concrete class. We
define three relevant types of units: _fixed_ , _tunable_ , and _automatic_.

$\mathfrak{A}\left(h^{*}\right)$In($\displaystyle\psi_{i}$)Out($\displaystyle\psi_{i}$)

(a) Fixed Unit.

$\mathfrak{A}(h),$In($\displaystyle\psi_{i}$)Out($\displaystyle\psi_{i}$)$h\in\mathcal{H}$

(b) Tunable Unit.

$\mathfrak{A}_{1}(h),$$\mathfrak{A}_{k}(h),$Out($\displaystyle\psi_{i}$)In($\displaystyle\psi_{i}$)…$h\in\mathcal{H}_{k}$$h\in\mathcal{H}_{1}$

(c) Automatic Unit.

Figure 1: The three types of units.

_Fixed Unit_ A fixed unit (Figure 1(a)) corresponds to an algorithm
$\psi=\mathfrak{A}(h)$, where $\mathfrak{A}(h)$ denotes algorithm
$\mathfrak{A}$ that generates the stage output, instanced with hyper-
parameters $h\in\mathcal{H}$ selected from an hyper-parameter set
$\mathcal{H}$.

_Tunable Unit_ A tunable unit (Figure 1(b)) is described by a tuple
$\psi=(\mathfrak{A},\mathcal{H},\mathfrak{T},\ell)$ where
$\mathfrak{A}(\cdot)$ is an algorithm, $\mathcal{H}$ is a _hyper-parameters
set_ , $\mathfrak{T}$ is a _tuner_ (e.g., genetic algorithm, particle swarm,
Bayesian optimizer), and $\ell(\mathfrak{A},h)\in\mathbb{R}$ is a tuning
_performance index_ mapping an algorithm $\mathfrak{A}(\cdot)$ and hyper-
parameters $h\in\mathcal{H}$ pair to a real number. The _tuning optimization
problem_ can be formulated as finding the hyper-parameters
$h^{*}\in\mathcal{H}$ maximizing the performance index $\ell$. Formally:

$\displaystyle
h^{*}\in\operatorname*{arg\,max}_{h\in\mathcal{H}}\ell(\mathfrak{A},h).$

This optimization is addressed by the tuner $\mathfrak{T}$. When the stage
corresponding to the tunable unit is executed, it reduces to the fixed unit
$\mathfrak{A}(h^{*})$, and, subsequently, it generates the block outputs.

_Automatic Unit_ An _automatic unit_ (Figure 1(c)) is a set of tunable units
paired with a performance index, i.e., $\psi=(\\{\psi_{j}\\}_{j=1}^{k},\ell)$,
where $\psi_{j}=(\mathfrak{A}_{j},\mathcal{H}_{j},\mathfrak{T}_{j},\ell_{j})$,
for $j\in\\{1,\ldots,k\\}$, and $\ell(\mathfrak{A},h)\in\mathbb{R}$ is a
performance index for algorithm $\mathfrak{A}$ with hyper-parameters $h$. The
goal of an automatic unit consists in selecting the best tuned algorithm among
the available ones, by ranking them based on the additional performance index
$\ell$. We define the _automatic optimization problem_ as follows:

$\displaystyle
j^{*}\in\operatorname*{arg\,max}_{j\in\\{1,\dots,k\\}}\ell(\mathfrak{A}_{j},h^{*}_{j}),\qquad\text{where}\qquad
h^{*}_{j}\in\operatorname*{arg\,max}_{h\in\mathcal{H}_{j}}\ell_{j}(\mathfrak{A}_{j},h),\quad
j\in\\{1,\dots,k\\}.$

When the stage corresponding to the automatic unit is executed, it reduces the
automatic unit a fixed one $\mathfrak{A}_{j^{*}}(h^{*}_{j^{*}})$, and,
subsequently, it generates the corresponding output.

The problem of jointly finding the best algorithm and its related hyper-
parameter configuration is also referred in the AutoML community as CASH
(Combined Algorithm Selection and Hyper-parameter Optimization Problem,
Thornton et al.,, 2013).

Intuitively, a fixed unit is a human hand-crafted unit in which an algorithm
is selected and the related hyper-parameters are specified. No automatic
operations nor evaluation are performed here. In a tunable unit, the algorithm
is specified but the task to find the best hyper-parameter configuration is
demanded to the pipeline. In an automatic unit, both the choice of the best
algorithm and the best hyper-parameter configuration is demanded to the
pipeline.

##  4 AutoRL Pipelines

In this section, we present the main methodological contribution of the paper,
discussing the two AutoRL pipelines: _online_ and _offline_. We focus on how
to build these pipelines describing the stages’ interactions. The detailed
description of each individual stage is reported in Section 5. A graphical
representation of the pipelines is provided in Figure 2.

Online Pipeline The _Online AutoRL Pipeline_ (Figure 2(a)) takes as input an
environment $\mathcal{E}$ that is fed to the Feature Engineering stage, which
modifies its state/action representations and the reward to facilitate the
learning performed in the next stages. It outputs a transformed environment
$\mathcal{E}^{\prime}$, based on the features created in this stage.
Subsequently, the environment $\mathcal{E}^{\prime}$ is used to learn an
estimate $\hat{\pi}^{*}$ of the optimal policy through the Policy Generation.
Finally, the Policy Evaluation phase provides an estimate of the performance
$\eta(\hat{\pi}^{*})$, based on a performance index $\eta$.

Offline Pipeline In the _Offline AutoRL Pipeline_ (Figure 2(b)), differently
from the online one, two additional preliminary stages are included: Data
Generation and Data Preparation. If an environment $\mathcal{E}$ is provided
as input, the Data Generation stage creates a dataset $\mathcal{D}$. This
stage is omitted if a dataset $\mathcal{D}$ is already available, e.g., in the
case the dataset $\mathcal{D}$ comes from a real process. In such a case, the
environment $\mathcal{E}$ is employed for the evaluation of the policy
performance only. The Data Preparation stage modifies the dataset
$\mathcal{D}$, by applying corrections over the individual instances (i.e.,
the rows of the dataset) obtaining $\mathcal{D}^{\prime}$. Then, the
environment $\mathcal{E}$ and dataset $\mathcal{D}^{\prime}$ pass through the
Feature Engineering stage, which, similarly to its online counterpart,
generates a dataset $\mathcal{D}^{\prime\prime}$ and an environment
$\mathcal{E}^{\prime}$ with transformed states, actions, and reward. After
that, the dataset $\mathcal{D}^{\prime\prime}$ is used for learning an
estimate of the optimal policy $\hat{\pi}^{*}$ through the Policy Generation
stage. Differently from the online one, this stage uses the dataset
$\mathcal{D}^{\prime\prime}$, while the environment $\mathcal{E}^{\prime}$ is
employed for estimating $\eta(\hat{\pi}^{*})$ in the Policy Evaluation stage.

$\mathcal{E}$EngineeringFeature$\mathcal{E}^{\prime}$PolicyPolicyEvaluationGeneration$\hat{\pi}^{*}$$\eta\left(\hat{\pi}^{*}\right)$

(a) Online Pipeline.

$\mathcal{E}$GenerationData$\mathcal{D}$PreparationData$\mathcal{E}$$\mathcal{D}^{\prime}$EngineeringFeature$\mathcal{E}^{\prime}$$\mathcal{D}^{\prime\prime}$PolicyPolicy$\hat{\pi}^{*}$Evaluation$\displaystyle(\mathcal{D}$,$\displaystyle\mathcal{E})$Generation$\eta\left(\hat{\pi}^{*}\right)$

(b) Offline Pipeline.

Figure 2: The Online (a) and Offline (b) AutoRL Pipelines.

##  5 Stages and Units

We now provide examples of units for each of the stages, highlighting the
differences between the online and offline pipelines. For each stage, we
define its goal, performance index for tunable or automatic units, and
implementation selected from the state-of-the-art methodologies.

###  5.1 Data Generation

The Data Generation stage takes as input an environment $\mathcal{E}$ and
returns the unaltered environment $\mathcal{E}$ and a dataset $\mathcal{D}$
generated by interacting with the environment. The goal of this stage is to
create a dataset that is retrieved by exploring the state space as much as
possible. Based on the type of environment, i.e., generative or forward model,
the resulting dataset is made of transitions or trajectories.

In principle, this stage should output a dataset as “informative” as possible,
i.e., that represents exhaustively the corresponding environment. As
performance index for evaluating the quality of a Data Generation unit, we
adopt the _entropy_ of the state-action visitation distribution $d_{\pi}(s,a)$
generated by the policy $\pi(a|s)$, that is proportional to:

$-\int_{s\in\mathcal{S}}\int_{a\in\mathcal{A}}{d_{\pi}(s,a)\log
d_{\pi}(s,a)}da\ ds.$

A straightforward implementation of Data Generation consists in collecting
data with the random uniform policy. However, this approach is not guaranteed
to explore the state space effectively (Mutti et al.,, 2021; Endrawis et al.,,
2021). In the pipeline, we consider the state-of-the-art solutions proposed by
Pathak et al., (2019), and Mutti et al., (2021). The former employs Proximal
Policy Optimization (PPO, Schulman et al.,, 2017) using the estimated variance
of the MDP dynamics as reward, as a proxy for the entropy. Instead, the latter
provides a novel policy search algorithm maximizing a $K$-nearest neighbours-
based estimate of the state distribution entropy.

###  5.2 Data Preparation

This phase uses a dataset $\mathcal{D}$, coming either from a real-world
environment or generated in the Data Generation stage, and returns a dataset
$\mathcal{D}^{\prime}$ with the same state-action features and reward, but
with a possibly different number of entries. The goal of this phase is to
optimize an existing dataset in order to be processed better from in
subsequent stages. Data Preparation includes data augmentation, data
imputation, and data scaling, and can embed further domain-specific sub-stages
(e.g., for images, audio data), and/or consistency checks (e.g., filling
missing values).

No single automatic unit is deemed adoptable due to the difficulty of defining
a general enough performance index for this stage. However, domain-specific
performance indexes are available, e.g., for the data imputation sub-stages,
we may rely on the indexes defined by Jadhav et al., (2019).

Possible implementations of this stage include the techniques for classical ML
preprocessing, such as imputation from a dataset of trajectories via KNN
imputation or Bayesian Multiple Imputation (Lizotte et al.,, 2008). Moreover,
for pixel-based observations (e.g., the Gym Atari environments) data
augmentation techniques, e.g., cropping, reflection, scaling, were employed in
Ye et al., (2020). Other approaches viable for feature-based representations
are presented in Laskin et al., (2020), where experiments on the OpenAI
Procgen Benchmark and on the MuJoCo environments.

$\mathcal{E}$$\mathcal{E}^{\prime}$FeatureGenerationRewardShapingFeatureSelection$\mathcal{D}$$\mathcal{F}^{\prime}$DataGenerationEnvironmentEngineering$\mathcal{E}$

(a) Feature Engineering stage in Online Pipeline.

$\mathcal{E}$$\mathcal{E}^{\prime}$FeatureGenerationRewardShapingFeatureSelection$\mathcal{D}^{\prime}$$\mathcal{F}^{\prime}$EnvironmentEngineering$\mathcal{E}$$\mathcal{D}^{\prime\prime}$

(b) Feature Engineering stage in Offline Pipeline.

Figure 3: The offline and online Feature Engineering stages.

###  5.3 Feature Engineering

The Feature Engineering stage displays significant differences between online
and offline pipelines (Figure 3). Offline pipelines (Figure 3(b)) take as
input an environment $\mathcal{E}$ and a dataset $\mathcal{D}^{\prime}$ and
return a feature-adjusted environment $\mathcal{E}^{\prime}$ and dataset
$\mathcal{D}^{\prime\prime}$. Conversely, online pipelines (Figure 3(a)) take
as input an environment $\mathcal{E}$ and return a feature-adjusted
environment $\mathcal{E}^{\prime}$. In both cases, this stage requires an
internal dataset for feature engineering that, for the online case, has to be
generated.

The core task of this stage is to select and generate a set of features that
properly model the state-action space of the problem and perform reward
shaping actions to facilitate the following learning phase. Feature
Engineering stage includes one or more of the following _sub-stages_ :

  * •

Feature Generation, in charge of creating new features. This sub-stage makes
use of techniques such as radial basis functions, tile coding, or coarse
coding (Sutton and Barto,, 2018).

  * •

Feature Selection, aimed at selecting a meaningful subset of features, either
to reduce the computation requirements or to regularize the following policy
learning phase. Viable options are Mutual Information-based selection (Beraha
et al.,, 2019), correlation-based filtering methods, and tree-based variable
selection (Castelletti et al.,, 2011).

  * •

Reward Shaping, performing specific transformations on the reward function,
possibly preserving the optimal policy, to speed up the convergence of an RL
algorithm (Ng et al.,, 1999). For instance, in presence of sparse reward
functions, reward shaping can be regarded as a form of _curriculum learning_
(Portelas et al.,, 2020).

These sub-stages return a transformation that is applied to the environment
through the Environment Engineering stage. In the offline case, the same
transformation is applied to the dataset, while for the online case the
internal dataset is disregarded.

We consider as a performance index for the complete feature engineering stage
the mutual information between the current state-action pair $(s,a)$ features
and the next-state reward $(s^{\prime},r)$ features (Kraskov et al.,, 2004;
Gao et al.,, 2017) regularized, e.g., by the number of selected
features.666For instance, one may use the ratio between the mutual information
and the number of selected features.

###  5.4 Policy Generation

The Policy Generation stage is in charge of the training phase of the RL
learning algorithm. More specifically, it takes as input an environment
$\mathcal{E}^{\prime}$ or a dataset $\mathcal{D}^{\prime\prime}$, in the
online and offline RL pipelines, respectively, to output an estimate
$\hat{\pi}^{*}$ of the optimal policy.

Among the most common choices of performance indexes for this stage, we
mention the _expected return_ , i.e., the expected discounted sum of the
rewards, the _average reward_ , i.e., the long-term expected average reward,
and the _total reward_ i.e., expected cumulative sum of the rewards (in the
case the environment is episodic, Puterman,, 2014). For specific applications,
e.g., risk-averse setting, one may adopt the mean-variance, mean-volatility,
and CVaR (Pratt,, 1978; Bisi et al.,, 2021).

Many works deal with hyper-parameter optimization for RL algorithms. In Franke
et al., (2021) a framework based on Population Based Training (PBT, Jaderberg
et al.,, 2017) is proposed to tune off-policy RL algorithms. In Parker-Holder
et al., (2021) a new time-varying bandit algorithm was presented for tuning RL
algorithms. Hyper-parameter tuning is a widely researched topic and the
techniques developed by ML algorithms can be used for RL algorithms as well.
Nevertheless, the sample inefficiency of tuning techniques is a common
problem, not unique to RL. Another issue is the sensitivity to hyper-
parameters configurations, which increases the difficulty of benchmarking
tuning algorithms due to the difficulty of obtaining reproducible results.
Further methods were proposed by Zhang et al., (2021); Lee et al., (2021);
Team et al., (2021); Saphal et al., (2021); Falkner et al., (2018).

The specific implementation of the Policy Generation stage depends on the
selected RL algorithm. For offline pipelines, we mention, among the others,
Least Squares Policy Iteration (LSPI, Lagoudakis and Parr,, 2003), Fitted
Q-Iteration (FQI, Ernst et al.,, 2005). For online pipelines, a large surge of
RL algorithms have been developed in the recent years. We mention, among the
most popular ones, Deep Q-Networks (DQN, Schaul et al.,, 2016), Deep
Deterministic Policy Gradient (DDPG, Lillicrap et al.,, 2016), Trust Region
Policy Optimization (TRPO, Schulman et al.,, 2015), Soft Actor Critic (SAC,
Haarnoja et al.,, 2018), and Proximal Policy Optimization (PPO, Schulman et
al.,, 2017).

###  5.5 Policy Evaluation

The Policy Evaluation stage takes as input the policy $\hat{\pi}^{*}$ produced
by the Policy Generation phase and an environment $\mathcal{E}^{\prime}$, and
produces as output an estimation of a performance index $\eta(\hat{\pi}^{*})$.

Regarding the performance index used in this stage, the options are the same
we mentioned for Policy Generation. Notice that the performance index chosen
in this stage may differ from the one of the Policy Generation one. For
instance, it is a common practice to train RL algorithms using a discounted
objective and evaluate the resulting policies using an undiscounted one (Duan
et al.,, 2016). Notice that, due to the nature of the task, only fixed units
are used in this stage.

##  6 Experimental Results

In this section, we use the Python implementation of ARLO on $3$ RL problems.
In addition to the presented stages, the library also allows to create newly
defined stages, if needed, and a set of analysis tools. The implementation of
the framework is available at https://github.com/arlo-lib/ARLO. The
implemented methods are reported in Appendix A. The Policy Generation stages
have been integrated with the MushroomRL (D’Eramo et al.,, 2021)
library.777The ARLO library includes an easy procedure to integrate algorithms
coming from other RL libraries. The optimization of the tunable units has been
performed using a genetic algorithm as described in B. A comprehensive
description of all the features available in the ARLO library, as well as
details on how to integrate already developed methods for RL, are provided at
https://arlo-lib.github.io/arlo-lib.

In Sections 6.1 and 6.2, we present the results our online pipelines whose
Policy Generation stages contain tunable units to select the best hyper-
parameters over two simulated problems. In Section 6.3, we consider an offline
pipeline including tunable Feature Engineering and Policy Generation stages on
a realistic dam control problem. The experimental details are reported in the
supplementary material.

For the different experiments we considered different seeds. These did not
only influence the RL algorithm, but were used for all the components making
up the considered pipeline.

###  6.1 Linear Quadratic Gaussian Regulator

In this experiment, we address a Linear-Quadratic Gaussian Regulator (LQG,
Dorato et al.,, 1994) by the state dynamics $s_{t+1}=As_{t}+Ba_{t}+\sigma$,
where $s_{t}$ is the state at time $t$, $a_{t}$ is the action at time $t$, $A$
is the state dynamic matrix, $B$ is the action dynamic matrix, and $\sigma$ is
a Gaussian white noise. The reward function is
$r_{t+1}=-s_{t}^{T}Qs_{t}-a_{t}^{T}Ra_{t}$, where $Q$ and $R$ are the state
and action cost weight matrices respectively888The details about the hyper-
parameters configuration space, the tuning procedure, and the compute
requirements for the LQG experiment can be found in the Appendix B.1.. The
discount factor is equal to $\gamma=0.9$ and the time horizon is $T=15$.

We employ the _Soft-Actor Critic_ (SAC, Haarnoja et al.,, 2018) algorithm. To
tune its hyper-parameters, we create an online RL pipeline, using the expected
return (Eq. (1)) as performance index, and a genetic algorithm (like in Sehgal
et al.,, 2019) as tuning algorithm. The results are obtained after $50$
generations of the genetic algorithm, each using a population of $20$ agents.

Table 1: Results achieved tuning SAC hyper-parameters on an LQG environment. Method | Default | Tuned
---|---|---
Van Dooren, (1981) | $-7.2\ (4.9)$
$1^{st}$ Seed | $-59.0\ (24.0)$ | $-8.6\ (4.7)$
$2^{nd}$ Seed | $-67.4\ (16.1)$ | $-8.2\ (5.1)$
$3^{rd}$ Seed | $-52.4\ (12.5)$ | $-8.7\ (4.7)$
Table 2: Results achieved tuning DDPG hyper-parameters on HalfCheetah-v3 environment. Method | Default | Tuned
---|---|---
Islam et al., (2017) | $3725.3\ (512.8)$
$1^{st}$ Seed | $1157.7\ (45.6)$ | $3407.2\ (952.1)$
$2^{nd}$ Seed | $850.8\ (78.9)$ | $4624.6\ (110.9)$
$3^{rd}$ Seed | $956.2\ (34.2)$ | $3076.9\ (77.9)$

Results We compare the results provided by the ARLO framework with the optimal
solution (Van Dooren,, 1981). In Table 1, we report the estimated expected
return, averaged over $100$ episodes (with the standard deviation in
brackets), for the default configuration and the corresponding tuned policy
over three different seeds. Even if the performance of the tuned algorithms
does not match the one of the optimal solution, the _default_ hyper-parameter
configuration of SAC is notably under-performing ($\approx 5$ times worse)
compared to the tuned configuration ($\approx 1.2$ times worse). This result
suggests that the proposed framework can generate solutions compatible with
the optimal one without exploiting specific domain knowledge about the
problem.

###  6.2 HalfCheetah

In the second experiment, we apply the online RL pipeline to the MuJoCo
HalfCheetah-v3 environment from OpenAI Gym (Brockman et al.,,
2016).999https://www.gymlibrary.ml/pages/environments/mujoco/half_cheetah. As
learning algorithm for the Policy Generation we employ the _Deep Deterministic
Policy Gradient_ (DDPG, Lillicrap et al.,, 2016), whose hyper-parameter tuning
is known to be a challenging task (Islam et al.,, 2017)101010The details about
the hyper-parameters configuration space, the tuning procedure, and the
compute requirements for the HalfCheetah experiment can be found in the
Appendix B.2.. The hyper-parameters of DDPG have been tuned using the
undiscounted cumulative reward as a performance index and, as a tuner, a
genetic algorithm. We employ a discount factor $\gamma=1$ and the time horizon
to $T=1000$.

Results In Table 2, we report the estimated total reward, averaged over $100$
episodes, for the default and tuned configurations over three different seeds
(the standard deviation is provided in brackets). The provided performances
are in line with the literature ones (Islam et al.,, 2017) and show that the
proposed pipeline provides an automatic way of achieving competitive
performance. In Figure 4, we report the different hyper-parameters selected
during the learning phase by individuals (agents) used in the genetic
algorithm optimization procedure throughout the tuning procedure. These
results show how some of the parameters have a strong influence on the reward
obtained by the agents, i.e., the actor and critic learning rate and the steps
per fit (Figures 4(a), 4(b), and 4(c), respectively), which implies that the
value of the parameter concentrates around the optimal value after a few
generations of the genetic algorithm. Conversely, those which do not influence
the outcome of the optimization procedure, i.e., the steps (Figure 4(d)),
continue to explore the available range until the end of the generations.

$0$$20$$40$$10^{-4}$$10^{-3}$$10^{-2}$GenerationActor learning rate

(a) Actor learning rate.

$0$$20$$40$$10^{-3}$$10^{-2}$GenerationCritic learning rate

(b) Critic learning rate.

$0$$20$$40$$10^{0}$$10^{1}$$10^{2}$$10^{3}$$10^{4}$GenerationSteps per fit

(c) Steps per fit.

$0$$20$$40$$0.5$$1$$1.5$$\cdot 10^{4}$GenerationNumber of steps

(d) Number of steps.

Figure 4: Values of the hyper-parameters generated by the genetic optimization
procedure over the $50$ generations. The orange line corresponds to the best
found value.

###  6.3 Dam

To showcase the capabilities of our framework, we propose an experiment with a
more complex offline RL pipeline that includes Data Generation, Feature
Engineering, Policy Generation, and Policy Evaluation stages 111111The details
about the hyper-parameters configuration space, the tuning procedure, and the
compute requirements for the Dam experiment can be found in the Appendix B.3..
The selected environment consists of the control of a water reservoir (dam)
that models the dynamics of a real alpine lake (Castelletti et al.,, 2011).
The agent observes the current level of the lake and the sequence of the most
recent $30$ daily inflows. The actuation consists of the amount of daily water
release. The goal of the agent is to trade-off between avoiding floods and
fulfilling the downstream water demand. The dataset is generated using a
random uniform policy. The Feature Engineering stage performs forward feature
selection via mutual information (as presented in Beraha et al.,, 2019) to
identify a subset of the available inflows features. The Policy Generation
stage uses the Fitted Q-Iteration (FQI, Ernst et al.,, 2005) algorithm. The
hyper-parameters of FQI are fixed to a hand-tuned configuration as the one
presented by Tirinzoni et al., (2018). The objective of this experiment is to
show in a realistic environment that tuning the hyper-parameters of a Feature
Engineering stage is beneficial for the final performance.

Table 3: Results achieved tuning the hyper-parameters of a Feature Engineering stage. Method | Discounted Reward
---|---
Baseline |  $-1649.85$ ($112.88$)
Tuned Configuration |  $-1224.67$ ($124.41$)

Results In Table 3, we report the estimated expected return over $10$ episodes
for the baseline configuration (standard deviation in brackets), in which all
the features have been considered, and for the tuned configuration, in which
only a subset of the features was selected automatically by the pipeline. We
observe that the result achieved by the tuned agent significantly outperforms
the baseline one, meaning that the feature selection techniques select only
the most informative feature for the problem, with beneficial effects on the
successive learning phase.

##  7 Conclusions and Limitations

Conclusions This paper introduced the ARLO framework for automating
reinforcement learning by proposing two pipelines, one for the online setting
and one for the offline setting. Moreover, we showcased the capabilities of
such a framework by creating a Python library, and we tested its performance
in both simulated and realistic settings. While the proposed framework in its
current formulation is flexible and allows adding customized stages, the
complete democratization of RL is far from being achieved. First, the
procedures to optimize the different stages revealed to be computationally
demanding. Thus, adding tools to predict and control the amount of
computational time required by a pipeline is of paramount importance to
obtaining a flexible tool. Another interesting development, going in the
opposite direction of what we have just mentioned, consists in including a
“whole pipeline optimization” procedure, which _jointly_ optimize the entire
learning process. This direction requires a preliminary development of less
computationally demanding algorithms for each stage of the pipeline. Finally,
we focused our attention on fully-observable, stationary, single agent,
single-objective settings. Developing a more general pipeline to relax some or
all the above assumptions would ease the application of RL algorithms in a
more wide spectrum of real-world problems.

Limitations The goal of AutoRL is to bring RL closer to the non-expert user.
This represents a source of opportunities and risks. On the one hand, making
RL usable to a wide audience contributes to the _democratization_ of the
field, overcoming the need for specific education and opening it to the large
public. On the other hand, such an abstract approach tends to compromise the
transparency of the learning process and traceability of the resulting model.
Shadowing the underlying principles, AutoRL might pose the risk of misuse of
RL approaches, leading to results not in line with expectations. Furthermore,
AutoRL, even more than RL, requires huge amounts of data and computation that
might represent a limit of the framework.

## References

  * Afshar et al., (2022) Afshar, R. R., Zhang, Y., Vanschoren, J., and Kaymak, U. (2022).  Automated reinforcement learning: An overview.  CoRR, abs/2201.05000.
  * Akiba et al., (2019) Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019).  Optuna: A next-generation hyperparameter optimization framework.  In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD).
  * Arulkumaran et al., (2017) Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017).  A brief survey of deep reinforcement learning.  CoRR, abs/1708.05866.
  * Baxter and Bartlett, (2001) Baxter, J. and Bartlett, P. L. (2001).  Infinite-horizon policy-gradient estimation.  Journal of Artificial Intelligence Research, 15:319–350.
  * Beraha et al., (2019) Beraha, M., Metelli, A. M., Papini, M., Tirinzoni, A., and Restelli, M. (2019).  Feature selection via mutual information: New theoretical insights.  In Proceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE.
  * Bishop and Nasrabadi, (2006) Bishop, C. M. and Nasrabadi, N. M. (2006).  Pattern recognition and machine learning, volume 4.  Springer.
  * Bisi et al., (2021) Bisi, L., Sabbioni, L., Vittori, E., Papini, M., and Restelli, M. (2021).  Risk-averse trust region optimization for reward-volatility reduction.  In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence (IJCAI), pages 4583–4589.
  * Brockman et al., (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016).  Openai gym.  abs/1606.01540.
  * Buşoniu et al., (2018) Buşoniu, L., de Bruin, T., Tolić, D., Kober, J., and Palunko, I. (2018).  Reinforcement learning for control: Performance, stability, and deep approximators.  Annual Reviews in Control, 46:8–28.
  * Castelletti et al., (2011) Castelletti, A., Galelli, S., Restelli, M., and Soncini-Sessa, R. (2011).  Tree-based variable selection for dimensionality reduction of large-scale control systems.  In Proceedings of the IEEE Symposium on Adaptive Dynamic Programming And Reinforcement Learning (ADPRL), pages 62–69.
  * D’Eramo et al., (2017) D’Eramo, C., Nuara, A., Pirotta, M., and Restelli, M. (2017).  Estimating the maximum expected value in continuous reinforcement learning problems.  In Proceedings of the Conference on Artificial Intelligence (AAAI), volume 31.
  * D’Eramo et al., (2021) D’Eramo, C., Tateo, D., Bonarini, A., Restelli, M., and Peters, J. (2021).  Mushroomrl: Simplifying reinforcement learning research.  J. Mach. Learn. Res., 22:131:1–131:5.
  * Dorato et al., (1994) Dorato, P., Cerone, V., and Abdallah, C. (1994).  Linear-quadratic control: an introduction.  Simon & Schuster, Inc.
  * Duan et al., (2016) Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016).  Benchmarking deep reinforcement learning for continuous control.  In Proceedings of the International Conference on Machine Learning (ICML), pages 1329–1338.
  * Endrawis et al., (2021) Endrawis, S., Leibovich, G., Jacob, G., Novik, G., and Tamar, A. (2021).  Efficient self-supervised data collection for offline robot learning.  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 4650–4656.
  * Ernst et al., (2005) Ernst, D., Geurts, P., and Wehenkel, L. (2005).  Tree-based batch mode reinforcement learning.  J. Mach. Learn. Res., 6:503–556.
  * Falkner et al., (2018) Falkner, S., Klein, A., and Hutter, F. (2018).  BOHB: robust and efficient hyperparameter optimization at scale.  In Proceedings of the International Conference on Machine Learning (ICML), volume 80, pages 1436–1445.
  * Feurer et al., (2020) Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. (2020).  Auto-sklearn 2.0: Hands-free automl via meta-learning.  arXiv preprint arXiv:2007.04074.
  * Feurer et al., (2015) Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., and Hutter, F. (2015).  Efficient and robust automated machine learning.  In Proceedings of the Neural Information Processing Systems (NeurIPS), pages 2962–2970.
  * Franke et al., (2021) Franke, J. K. H., Köhler, G., Biedenkapp, A., and Hutter, F. (2021).  Sample-efficient automated deep reinforcement learning.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–23.
  * Gao et al., (2017) Gao, W., Kannan, S., Oh, S., and Viswanath, P. (2017).  Estimating mutual information for discrete-continuous mixtures.  In Proceedings of the Neural Information Processing Systems (NeurIPS), volume 30.
  * Geurts et al., (2006) Geurts, P., Ernst, D., and Wehenkel, L. (2006).  Extremely randomized trees.  Mach. Learn., 63(1):3–42.
  * Haarnoja et al., (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018).  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In Proceedings of the International Conference on Machine Learning (ICML), volume 80, pages 1856–1865.
  * He et al., (2021) He, X., Zhao, K., and Chu, X. (2021).  Automl: A survey of the state-of-the-art.  Knowledge-Based Systems, 212:106622.
  * Islam et al., (2017) Islam, R., Henderson, P., Gomrokchi, M., and Precup, D. (2017).  Reproducibility of benchmarked deep reinforcement learning tasks for continuous control.  arXiv preprint arXiv:1708.04133.
  * Jaderberg et al., (2017) Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. (2017).  Population based training of neural networks.
  * Jadhav et al., (2019) Jadhav, A., Pramod, D., and Ramanathan, K. (2019).  Comparison of performance of data imputation methods for numeric dataset.  Applied Artificial Intelligence, 33(10):913–933.
  * Kraskov et al., (2004) Kraskov, A., Stögbauer, H., and Grassberger, P. (2004).  Estimating mutual information.  Phys. Rev. E, 69:066138.
  * Lagoudakis and Parr, (2003) Lagoudakis, M. G. and Parr, R. (2003).  Least-squares policy iteration.  The Journal of Machine Learning Research, 4:1107–1149.
  * Laskin et al., (2020) Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. (2020).  Reinforcement learning with augmented data.  In Proceedings of the Neural Information Processing Systems (NeurIPS).
  * LeDell and Poirier, (2020) LeDell, E. and Poirier, S. (2020).  H2O AutoML: Scalable automatic machine learning.  Proceedings of the ICML Workshop on Automated Machine Learning (AutoML).
  * Lee et al., (2021) Lee, K., Laskin, M., Srinivas, A., and Abbeel, P. (2021).  SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning.  In Proceedings of the International Conference on Machine Learning (ICML), volume 139, pages 6131–6141.
  * Levine et al., (2020) Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020).  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643.
  * Lillicrap et al., (2016) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016).  Continuous control with deep reinforcement learning.  In Proceedings of the International Conference on Learning Representations (ICLR).
  * Lizotte et al., (2008) Lizotte, D. J., Gunter, L., Laber, E., and Murphy, S. A. (2008).  Missing data and uncertainty in batch reinforcement learning.  In Proceedings of the Neural Information Processing Systems (NeurIPS).
  * Mnih et al., (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015).  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533.
  * Mutti et al., (2021) Mutti, M., Pratissoli, L., and Restelli, M. (2021).  Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate.  In Proceedings of the Conference on Artificial Intelligence (AAAI), pages 9028–9036.
  * Ng et al., (1999) Ng, A. Y., Harada, D., and Russell, S. (1999).  Policy invariance under reward transformations: Theory and application to reward shaping.  In Proceedings of the International Conference on Machine Learning (ICML), volume 99, pages 278–287.
  * Nguyen and La, (2019) Nguyen, H. and La, H. M. (2019).  Review of deep reinforcement learning for robot manipulation.  In Proceedings of the IEEE International Conference on Robotic Computing (IRC), pages 590–595.
  * Olson et al., (2016) Olson, R. S., Bartley, N., Urbanowicz, R. J., and Moore, J. H. (2016).  Evaluation of a tree-based pipeline optimization tool for automating data science.  In Proceedings of the ACM Genetic and Evolutionary Computation Conference (GECCO), pages 485–492.
  * Paine et al., (2020) Paine, T. L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N. (2020).  Hyperparameter selection for offline reinforcement learning.  arXiv preprint arXiv:2007.09055.
  * Parker-Holder et al., (2021) Parker-Holder, J., Nguyen, V., Desai, S., and Roberts, S. J. (2021).  Tuning mixed input hyperparameters on the fly for efficient population based autorl.  CoRR, abs/2106.15883.
  * Parker-Holder et al., (2022) Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., Hutter, F., and Lindauer, M. (2022).  Automated reinforcement learning (autorl): A survey and open problems.  CoRR, abs/2201.03916.
  * Pathak et al., (2019) Pathak, D., Gandhi, D., and Gupta, A. (2019).  Self-supervised exploration via disagreement.  In Proceedings of the International Conference on Machine Learning (ICML), pages 5062–5071.
  * Portelas et al., (2020) Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P.-Y. (2020).  Automatic curriculum learning for deep rl: A short survey.  arXiv preprint arXiv:2003.04664.
  * Pratt, (1978) Pratt, J. W. (1978).  Risk aversion in the small and in the large.  In Uncertainty in economics, pages 59–79. Elsevier.
  * Puterman, (2014) Puterman, M. L. (2014).  Markov decision processes: discrete stochastic dynamic programming.  John Wiley & Sons.
  * Saphal et al., (2021) Saphal, R., Ravindran, B., Mudigere, D., Avancha, S., and Kaul, B. (2021).  SEERL: sample efficient ensemble reinforcement learning.  In Proceedings of the ACM International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1100–1108.
  * Schaul et al., (2016) Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016).  Prioritized experience replay.  In Proceedings of the International Conference on Learning Representations (ICLR).
  * Schulman et al., (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. (2015).  Trust region policy optimization.  In Proceedings of the International Conference on Machine Learning (ICML), volume 37, pages 1889–1897.
  * Schulman et al., (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Sehgal et al., (2019) Sehgal, A., La, H., Louis, S., and Nguyen, H. (2019).  Deep reinforcement learning using genetic algorithm for parameter optimization.  In Proceedings of the IEEE International Conference on Robotic Computing (IRC), pages 596–601.
  * Singh et al., (2003) Singh, H., Misra, N., Hnizdo, V., Fedorowicz, A., and Demchuk, E. (2003).  Nearest neighbor estimates of entropy.  American journal of mathematical and management sciences, 23(3-4):301–321.
  * Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018).  Reinforcement learning: An introduction.  MIT press.
  * Team et al., (2021) Team, O. E. L., Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., McAleese, N., Bradley-Schmieg, N., Wong, N., Porcel, N., Raileanu, R., Hughes-Fitt, S., Dalibard, V., and Czarnecki, W. M. (2021).  Open-ended learning leads to generally capable agents.  CoRR, abs/2107.12808.
  * Thornton et al., (2013) Thornton, C., Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2013).  Auto-weka: Combined selection and hyperparameter optimization of classification algorithms.  In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 847–855.
  * Tirinzoni et al., (2018) Tirinzoni, A., Sessa, A., Pirotta, M., and Restelli, M. (2018).  Importance weighted transfer of samples in reinforcement learning.  In Proceedings of the International Conference on Machine Learning (ICML), pages 4936–4945.
  * Van Dooren, (1981) Van Dooren, P. (1981).  A generalized eigenvalue approach for solving riccati equations.  SIAM Journal on Scientific and Statistical Computing, 2(2):121–135.
  * Wang et al., (2018) Wang, S., Jia, D., and Weng, X. (2018).  Deep reinforcement learning for autonomous driving.  arXiv preprint arXiv:1811.11329.
  * Ye et al., (2020) Ye, C., Khalifa, A., Bontrager, P., and Togelius, J. (2020).  Rotation, translation, and cropping for zero-shot generalization.  In Proceedings of the IEEE Conference on Games (CoG), pages 57–64.
  * Zhang et al., (2021) Zhang, B., Rajan, R., Pineda, L., Lambert, N. O., Biedenkapp, A., Chua, K., Hutter, F., and Calandra, R. (2021).  On the importance of hyperparameter optimization for model-based reinforcement learning.  In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130, pages 4015–4023.
  * Zhang et al., (2020) Zhang, Z., Zohren, S., and Roberts, S. (2020).  Deep reinforcement learning for trading.  The Journal of Financial Data Science, 2(2):25–40.

Supplementary Material for the Paper: "ARLO: A Framework for Automated
Reinforcement Learning"

##  Appendix A Library

ARLO is a Python library implementing the framework described in this paper.
It contains all the automation capabilities described in the main paper. It
also provides the implementation of specific stages for each phase of the two
pipelines we introduced in Section 4.

The RL algorithms present in the implementation are wrappers of those
implemented in MushroomRL (D’Eramo et al.,, 2021). Moreover, we structured the
library so that one has the option to implement wrappers for any other RL
library, e.g., Stable Baselines, RLLib, Tensorforce.

Supported Units In Table 4, we list the currently implemented units for each
stage. As mentioned before, this is a non-exhaustive list of the possible
methods that can be included in the proposed framework, but only those which
we used for experimental purposes. See Section 5 for some suggestions about
the methods which are appropriate for an extension for each stage.

Table 4: Supported units in the current ARLO implementation. Stage | Implementations
---|---
Data Generation | Random Uniform Policy
MEPOL (Mutti et al.,, 2021)
Data Preparation | Mean Imputation
1-NN Imputation
Feature Engineering | Recursive Feature Selection
Forward Feature Selection via Mutual Information (Beraha et al.,, 2019)
| Nystroem Map Feature Generation
Policy Generation | Fitted-Q Iteration (FQI, Ernst et al.,, 2005)
Double Fitted-Q Iteration (DoubleFQI, D’Eramo et al.,, 2017)
Least Squares Policy Iteration (LSPI, Lagoudakis and Parr,, 2003)
Deep Q-Network (DQN, Mnih et al.,, 2015)
Proximal Policy Optimization (PPO, Schulman et al.,, 2017)
Deep Deterministic Policy Gradient (DDPG, Lillicrap et al.,, 2016)
Soft Actor Critic (SAC, Haarnoja et al.,, 2018)
| GPOMDP (Baxter and Bartlett,, 2001)

Used Libraries ARLO requirements, in terms of libraries, are: catboost
(v1.0.3), gym (v0.19.0), joblib (v1.1.0), matplotlib (v3.5.0), mushroom_rl
(v1.7.0), numpy (v1.22.0), optuna (v2.10.0), plotly (v5.4.0), scikit_learn
(v1.0.2), scipy (v1.7.3), torch (v1.10.1), xgboost (v1.5.1).

##  Appendix B Details on the Experiments

All the experiments were run on a Linux-based server with an AMD Ryzen $9$
$5950X$ 16-Core Processor with 128GB DDR4 RAM running Python $3.8.8$ on CentOS
$8.5.2111$.

Hyper-parameter tuning The pseudo-code of the genetic tuner is detailed in
Algorithm 1. The hyper-parameter tuning of the genetic algorithms are run for
$50$ generations, each one including $20$ agents. Throughout each generation,
elitism is performed, i.e., the best agent of the generation is preserved, and
the new generation is created via tournament selection. More specifically, we
take the best agent, out of a subset of $3$ agents of the previous generation,
and we repeat such an operation until $19$ agents are selected (as the
remaining spot is reserved for the best performing agent in the previous
generation).

Algorithm 1 Genetic Tuner

1: Randomly initialise first generation

2: for $i\in[0,\ldots,n\\_generations)$ do

3: Fit and evaluate each agent in the generation $i$

4: Select the best agent and add it to the new generation $i+1$

5: for $j\in[0,\ldots,n\\_agents-1)$ do

6: Add best agent, out of a random subset of $3$, to the new generation $i+1$

7: end for

8: Mutate the new generation $i+1$

9: end for

Each hyper-parameter is mutated with probability $0.5$ and two different types
of mutation can take place:

  * •

for categorical hyper-parameters and for the ones having a discrete support,
we sample from a uniform distribution over the possible values;

  * •

for numerical, i.e., continuous domains, we sample hyper-parameters from a
uniform distribution over $0.8$ and $1.2$ times of the current value of the
hyper-parameter.

ARLO implements the Genetic Algorithm presented above as well as the hyper-
parameter tuning solutions from Optuna (Akiba et al.,, 2019).

In all the experiments, we chose reasonable hyper-parameters configuration
spaces so that they weren’t neither too small, to avoid exploring a space that
was too little and thus finding solutions quite far off, nor too large, to
avoid increasing the total computational time (as some hyper-parameters have a
great impact on the training time of the Policy Generation units).

Environment Whenever an environment is used for the training of a RL
algorithm, a deep copy of such an environment is provided to each agent in the
generation, while in the case a dataset is used for the training of a RL
algorithm, we provided each agent with a bootstrapped dataset coming from the
original one. The environments currently available are presented in Table 5.

Table 5: Supported environments in the current ARLO implementation. Source | Type | Environment
---|---|---
Gym | Classic Control | Grid World
Mountain Car
Cart Pole
MuJoCo | Inverted Pendulum
Walker2d
HalfCheetah
Ant
Hopper
Humanoid
Swimmer
Other | Controller | LQG

Loss Function As loss function for guiding the tuning procedure we used the
empirical expected return defined in Equation (1). The specific loss functions
used in the different experiments for policy evaluation are detailed in the
following sections.

###  B.1 Linear Quadratic Gaussian Regulator

In this experiment, we consider a Linear Quadratic Gaussian (LQG) Regulator
characterized as follows:

$A=\begin{bmatrix}1&0\\\ 0&1\end{bmatrix},\hskip
14.22636ptB=\begin{bmatrix}1&0&0\\\ 0&0&1\end{bmatrix},\hskip
14.22636ptQ=0.7\cdot\begin{bmatrix}1&0\\\ 0&1\end{bmatrix},\hskip
14.22636ptR=0.3\cdot\begin{bmatrix}1&0&0\\\ 0&1&0\\\ 0&0&1\end{bmatrix}.$

Along each dimension, and for each time step $t$, the action $a_{t}$ and the
observation $s_{t}$, can take values in $[-3.5,3.5]$, while the discount
factor and the time horizon were set to $\gamma=0.9$, and $T=15$,
respectively. We used a noise standard deviation as follows:

$\sigma=0.1\cdot\begin{bmatrix}1&0\\\ 0&1\end{bmatrix}.$

We perform three experiments using three different seeds: $2$, $42$, $2022$.
We tune the hyper-parameters of SAC through the genetic algorithm described
above, using as metric the empirical expected return. For all the seeds we
consider, we used the hyper-parameters configuration space reported in Table
6. Notice that if only a single value is specified for its domain, it means
that the hyper-parameter is considered fixed. The three runs performed (one
for each seed) took on average $44\ (\pm 17.2)$ hours each.

Table 6: Hyper-parameters configuration space for the Linear Quadratic Gaussian Regulator experiment. Hyper-parameter | Search Space
---|---
Actor Learning Rate | {$10^{-5},10^{-4},10^{-3},10^{-2}$}
Actor Network | One layer with 16 neurons and ReLU activation
Critic Learning Rate | {$10^{-5},10^{-4},10^{-3},10^{-2}$}
Critic Loss | MSE
Critic Optimizer | Adam
Critic Network | One layer with 16 neurons and ReLU activation
Batch Size | {8, 16, 32, 64, 128}
Initial Replay Size | {10, 100, 300, 500, 1000, 5000}
Max Replay Size | {3000, 10000, 30000, 100000}
Warmup Transitions | {50, 100, 500}
Tau | 0.005
Alpha Learning Rate | {$10^{-5},10^{-4},10^{-3}$}
Log Std Min | -20
Log Std Max | 3
N Epochs | [1, 30]
N Episodes | [1, 1600]
N Episodes Per Fit | [1, 500]

$0$$5$$10$$15$$20$$25$$30$$35$$40$$45$$50$$-60$$-50$$-40$$-30$$-20$$-10$Generations$J(\pi)$

Figure 5: SAC best agent performance for each generation.

In Figure 5 we report the value of the performance over time of the best agent
of each generation for this experiment. It shows how the performances are
almost constant in the last $\approx 25$ generations, meaning that the
optimization procedure converged to a solution near to a local minimum point.
The scripts needed to run these three experiments (each corresponding to a
different seed) are available at https://github.com/arlo-
lib/ARLO/tree/main/experiments/LQG.

###  B.2 HalfCheetah

In this second experiment we used the simulated environment of HalfCheetah to
run experiment on the model generation stage. The MDP corresponding to this
environment is assumed to have a discount factor and a time horizon of
$\gamma=1$, and $T=1000$, respectively. We perform three experiments using
three different seeds: $2$, $42$, and $2022$. We tune the hyper-parameters of
DDPG using the genetic algorithm described above, considering as metric the
Average Reward.

Table 7: Hyper-parameters configuration space for the HalfCheetah experiment. Hyper-parameter | Search Space
---|---
Actor Learning Rate | [$10^{-5},10^{-2}$]
Actor Network | Two layers with 128 neurons and ReLU activations
Critic Learning Rate | [$10^{-5},10^{-2}$]
Critic Loss | MSE
Critic Optimizer | Adam
Critic Network | Two layers with 128 neurons and ReLU activations
Batch Size | [8, 256]
Initial Replay Size | [1000, 20000]
Max Replay Size | [10000, 1500000]
Tau | 0.001
Policy delay | 1
Policy | OrnsteinUhlenbeckPolicy($\sigma$=0.2, $\theta$=0.15, dt=10-2)
N Epochs | [1, 50]
N Steps | [1000, 15000]
N Steps Per Fit | [1, 10000]

For all three seeds, we consider the hyper-parameters configuration space
reported in Table 7. The three runs performed (one for each seed), took on
average $124.7\ (\pm 8.8)$ hours each. The scripts needed to run these three
experiments (each corresponding to a different seed) are available at
https://github.com/arlo-lib/ARLO/tree/main/experiments/HalfCheetah-v3.

###  B.3 Dam

In this experiment, we consider the control of a water reservoir (dam) that
models the dynamics of a real alpine lake, as described by Castelletti et al.,
(2011). The observation space is a continuous space with $31$ dimensions, each
of which taking values in $\mathbb{R}^{+}$. This state space features
represent the inflow values for the previous month. The action space is
sampled to get a discrete space with $8$ actions, each one corresponding to a
different amount of water released in a day. The discount factor and the time
horizon have been set to $\gamma=0.999$ and $T=360$, respectively. In this
experiment, once we extract the dataset, we perform forward feature selection
via mutual information, as described by Beraha et al., (2019). Hyper-
parameters search space for the tested tunable feature selection unit is
reported in Table 8.

Table 8: Hyper-parameters configuration space for the Feature Engineering stage of the Dam experiment. Hyper-parameter | Search Space
---|---
K |  $\\{$1, 2, 3, 4, 5, 10, 20, 50$\\}$
N Features |  $\\{$1, 2, $\ldots$, 31$\\}$
Table 9: Hyper-parameters used in the Policy Generation stage of the Dam experiment. Hyper-parameter | Value
---|---
N Iterations | 60
N Estimators | 100
Criterion | MSE
Min Samples Split | 10

Once feature selection is performed, we fit a Policy Generation unit, i.e.,
FQI, using an Extremely Randomized Trees Regressor (Geurts et al.,, 2006) with
the hyper-parameters present in Table 9. The entire run took around $2$ hours.
The script needed to run this experiment is available at
https://github.com/arlo-lib/ARLO/tree/main/experiments/Dam/dam.py.

##  Appendix C Additional Experiments

In this section, we present two additional experiments, the former
implementing an offline pipeline (Section C.1), and the latter an online
pipeline (Section C.2).

###  C.1 Offline Pipeline

The objective of this experiment to evaluate the capabilities of using the
complete Offline Pipeline we defined in Section 4 in the Dam experiment
described above (Section B.3).

$\mathcal{E}$GenerationData$\mathcal{D}$PreparationData$\mathcal{E}$$\mathcal{D}^{\prime}$EngineeringFeature$\mathcal{E}^{\prime}$$\mathcal{D}^{\prime\prime}$PolicyPolicy$\hat{\pi}^{*}$EvaluationGeneration$\eta\left(\hat{\pi}^{*}\right)$Automatic
UnitTunable UnitFixed UnitNot used

Figure 6: Types of the units adopted in the offline pipeline experiment. Table 10: Hyper-parameters configuration space of XGBoost. Hyper-parameter | Search Space
---|---
N Iterations | [2, 60]
N Estimators | [5, 250]
Min Child Weight | [1, 100]
Subsample | [0.5, 1]
Learning Rate | [$10^{-3}$, 0.4]
Max Depth | [4, 15]
Table 11: Hyper-parameters configuration space of ExtraTrees. Hyper-parameter | Search Space
---|---
N Iterations | [2, 60]
N Estimators | [5, 250]
Criterion | MSE
Min Samples Split | [1, 50]

The scheme of the used pipeline and the topology of the different stages is
presented in Figure 6. We use a fixed unit to generate the data, and a tunable
unit to perform the Feature Engineering stage, as the ones we used for the dam
experiment in Section 6. Subsequently, we use an automatic Policy Generation
unit composed two tunable units using different versions of Fitted-Q
Iteration, one with XGBoost as regressor, and the other with Extremely
Randomized Trees. The two hyper-parameters configurations spaces are those
presented in Table 10 and Table 11, respectively.

Table 12: Results obtained for the additional experiment over the full offline pipeline. Method | Empirical Expected Return
---|---
Baseline | $-1224.67\ (124.41)$
Tuned Configuration | $-1047.97\ (213.37)$

In Table 12 we report the empirical expected return over $10$ episodes
(standard deviation in brackets). Even though we have improved over the
previously obtained result by about $20\%$, we point out that obtaining a
statistically significant result would require a huge computational effort,
indeed the entire run took around $16$ hours. We leave to future experiments
the test on a larger number of samples to assess the statistical significance
of this result. The script needed to run this experiment is available at
https://github.com/arlo-
lib/ARLO/blob/main/experiments/Dam/hp_tuning_fqi_dam.py.

###  C.2 Online Pipeline

Differently from the previous experiments we run on online pipelines, in which
we used a single tunable unit for the Policy Generation stage, in what
follows, we apply an automatic unit to this stage, to tune the hyper-
parameters of different algorithms. To test this out we run an automatic
Policy Generation unit for the Linear Quadratic Gaussian Regulator, in which
we tune both Soft Actor Critic (SAC, Haarnoja et al.,, 2018) and Proximal
Policy Optimization (PPO, Schulman et al.,, 2017). To test the performance we
consider the Discounted Reward (defined in Equation 1) in Policy Evaluation
stage. Figure 7 represents the pipeline and the topology of the units used in
the experiment.

$\mathcal{E}$EngineeringFeature$\mathcal{E}^{\prime}$PolicyPolicyEvaluationGeneration$\hat{\pi}^{*}$$\eta\left(\hat{\pi}^{*}\right)$Automatic
UnitTunable UnitFixed UnitNot used

Figure 7: Types of the units adopted in the online pipeline experiment. Table 13: Performance obtained using the automatic Policy Generation unit on the Linear Quadratic Gaussian Regulator. Method | Empirical Expected Return
---|---
Van Dooren, (1981) | $-7.2\ (4.9)$
Best SAC Tuned Configuration |  $-8.2$ ($4.7$)
Best Automatic Block Tuned Configuration |  $-7.4$ ($5.0$)

Table 13 shows the results in terms of discounted reward. By tuning the hyper-
parameters of two different Policy Generation tunable units the pipeline
further improved the results presented in Section 6, reaching a level of
performance in line with the one of the optimal solution by Van Dooren,
(1981).


# Performance Analysis of Universal Robot Control
System Using Networked Predictive Control

Mahsa Noroozi1 and Lorenz Kies2  Institute of Communications Technology
Leibniz Universität Hannover
Hanover, Germany
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Networked control systems are feedback control systems with system components
distributed at different locations connected through a communication network.
Since the communication network is carried out through the internet and there
are bandwidth and packet size limitations, network constraints appear. Some of
these constraints are time delay and packet loss. These network limitations
can degrade the performance and even destabilize the system. To overcome the
adverse effect of these communication constraints, various approaches have
been developed, among which a representative one is networked predictive
control. This approach proposes a controller, which compensates for the
network time delay and packet loss actively. This paper aims at implementing a
networked predictive control system for controlling a robot arm through a
computer network. The network delay is accounted for by a predictor, while the
potential of packet loss is mitigated using redundant control packets. The
results will show the stability of the system despite a high delay and a
considerable packet loss. Additionally, improvements to previous networked
predictive control systems will be suggested and an increase in performance
can be shown. Lastly, the effects of different system and environment
parameters on the control loop will be investigated.

###### Index Terms:

Networked predictive control systems, time delay, packet loss.

##  I Introduction

The use of a shared communication network to connect system components results
in the concept of Networked Control Systems (NCSs). NCSs are feedback control
systems containing plant, actuators, sensors, and controllers which may be
located in geographically dispersed facilities. The general architecture of an
NCS is shown in Fig. 1. NCSs have been utilized in various areas, such as
process control, industrial automation, vehicle industry, and robot
manipulators [1, 2].

Some advantages of using network architectures in NCSs are low installation
and maintenance costs, reduced system wiring, and increased system
flexibility. Even though the use of network architecture has many inherent
benefits, it has some weaknesses due to the limited capacity of the network.
As a result of traffic congestion in the network, there will be usually some
constraints including network-induced delay and packet loss. Due to the
existence of such constraints in communication network, the analysis and
design of an NCS is complex. These constraints can also degrade the
performance and even destabilize the system as well.

Figure 1: Basic structure of an NCS

To mitigate the effects of network imperfections, diverse control methods
including augmented deterministic discrete-time model [3], queuing method [4],
optimal stochastic control method [5], event-based method [6], and predictive
control [7] have been developed. Since periodically sending data at a high
frequency can result in a heavy load on the network, [8] used an adaptive
event-triggered controller to reduce network traffic. They also included
measurement uncertainties in their model and introduced dynamic output
feedback control to demonstrate the performance. To compensate some of the
constraints in NCS design, Networked Predictive Control (NPC) has attracted
more and more attention. Networked predictive control systems are NCSs that
use predictive controllers as an active compensation method for network
communication constraints. [9] studied the design, simulation, and
implementation of networked predictive control.

Delays occurring in the forward and feedback channels in NCSs are
distinguished apart. Regarding the forward channel, the controller doesn’t
know the time taking the control signal to reach the actuator from controller.
Thus an exact correction can’t be achieved while calculating the control
signals. To measure the delays of the end-to-end communication, Round Trip
Time (RTT) delay can be used. RTT in NCSs is used to measure the total time
delay of a control cycle. Instead of considering forward and feedback delay
separately, [10, 11, 12] combined it to the RTT delay and based their
prediction and delay compensation on this measure. This eliminated the need
for synchronization between the two sites. [13] used NPC to teleoperate a
robot with significant variable RTT delay. They proposed a sparse multi
variable linear regression algorithm to predict the RTT delay with good
accuracy.

In a lot of earlier work, the delay was only in the order of a few sample
times of the system and rarely more than the system dynamics. All in all a lot
is to be gained from the advantages of NPC, making it a worthwhile effort to
develop tools for their design [14, 15]. A networked predictive control system
will be here implemented and variations in parameters and architecture will be
compared. The base architecture consists of a predictor, controller, forward
channel, delay compensator, plant, and feedback channel. The system to be
controlled is a universal robot arm.

The remainder of this paper is organized as follows. Firstly, an overview of
formal description, the structure of the NPC approach, and the implementation
of the system are given in section II. In section III, the evaluation results
with different features or parameters are shown. Finally, section IV concludes
the paper.

##  II Networked predictive control (NPC)

Networked Predictive Control Systems (NPCSs) are control systems where the
control loop is closed through a network with some extra components compared
to a regular control loop. The basic structure of an NPCS can be seen in Fig.
2. The basic setup shares some similarities with a regular control loop like
the one shown in Fig. 1, but there are a few additional features namely
predictor, delay compensator, and network, which encompasses the forward and
feedback channel.

Figure 2: Architecture for a Networked Predictive Control System

###  II-A Plant model

The plant used is a 6-axis cooperative universal robot arm. The inverse
kinematics are handled on site and the robot is controlled by acceleration in
all three spacial dimensions. Since all of these dimensions should behave the
same, only one is used to simplify the model. The resulting system description
is:

$\centering\dot{\boldsymbol{x}}(t)=\boldsymbol{f}(\boldsymbol{x}(t),\boldsymbol{u}(t))=\begin{bmatrix}\dot{x}_{1}(t)\\\
\dot{x}_{2}(t)\end{bmatrix}=\begin{bmatrix}x_{2}(t)\\\
u_{1}(t)\end{bmatrix}\@add@centering$ (1)

or as a linear system in state space description:

$\centering\dot{\boldsymbol{x}}(t)=\boldsymbol{A}\boldsymbol{x}(t)+\boldsymbol{B}\boldsymbol{u}(t)=\begin{bmatrix}0&1\\\
0&0\end{bmatrix}\boldsymbol{x}(t)+\begin{bmatrix}0\\\
1\end{bmatrix}\boldsymbol{u}(t)\@add@centering$ (2)

with $\boldsymbol{x}(t)$ as the state consisting of the position and velocity,
and $\boldsymbol{u}(t)$ as the control input. The output of the system is:

$\centering\boldsymbol{y}(t)=\boldsymbol{g}(\boldsymbol{x}(t),\boldsymbol{u}(t))=\begin{bmatrix}y_{1}(t)\\\
y_{2}(t)\end{bmatrix}=\begin{bmatrix}x_{1}(t)\\\
x_{2}(t)\end{bmatrix}\@add@centering$ (3)

The discrete sample time $T$ or the sample time of the robot is $8{ms}$. To
provide a more challenging environment that can better represent the problems
of a real system, Gaussian noise is added to the measurement. The noise is
generated by adding a random value that is chosen from a Gaussian normal
distribution to both of the output vector elements.

It is a very conservative assumption, as real sensors for the position and
speed of the robot arm would be more precise, the position readout from the
universal robot is accurate to $0.03mm$ [16].

###  II-B Controller

A Linear Quadratic Regulator (LQR) is used in this work as a controller. LQR
is a linear full state feedback controller, that is optimal in its solution
space determined by weights for errors in different parts of the state and
costs for control inputs. It is a state space description but without the
output equation:

$\centering\dot{\boldsymbol{x}}(t)=\boldsymbol{A}\boldsymbol{x}(t)+\boldsymbol{B}\boldsymbol{u}(t)\@add@centering$
(4)
$\centering\boldsymbol{u}(t)=\boldsymbol{K}\cdot\boldsymbol{x}(t)\@add@centering$
(5)

with $\boldsymbol{K}$ as a gain vector or matrix for the case of a multiple
input system.

The cost function that the controller minimizes is:

$\centering
J(\boldsymbol{Q},\boldsymbol{R})=\int\boldsymbol{x}(t)^{T}\boldsymbol{Q}\boldsymbol{x}(t)+\boldsymbol{u(t)}^{T}\boldsymbol{R}\boldsymbol{u(t)}\,dt\@add@centering$
(6)

where $\boldsymbol{Q}$ is the weight matrix for the different parts of the
state and $\boldsymbol{R}$ is the weights for the control channels [17].

The weight matrices and vectors were chosen as follows, $\boldsymbol{A}$ and
$\boldsymbol{B}$ are the same as in 2.

$\centering\boldsymbol{Q}=\text{diag}(1,0.5)\quad\text{and}\quad
R=0.1\@add@centering$ (7)

This results in the following gain vector $\boldsymbol{K}$:

$\centering\boldsymbol{K}^{T}=[3.1623,3.3652]\@add@centering$ (8)

were calculated using the Python control systems library [18].

###  II-C Network

The network is modeled using the forward and feedback channel. These can be
different physical channels or conceptual channels that are run through a
shared communication network like a bus system or a computer network. The
properties of these channels create the need for the predictor and the delay
compensator.

The simulation was done using a network emulator. The test network is made up
of 6 nodes as shown in Fig. 3. All traffic in NPCS has to pass through the
bottleneck between node $C$ and $D$ which has a limited capacity, a
transmission delay, and probably packet loss.

Figure 3: Network topology of the experiment

Hosts $E$ and $F$ are sending dummy traffic to each other with similar
characteristics to real internet traffic. This traffic uses part of the link
capacity between $C$ and $D$ causing packets from the plant and controller to
be lost or delayed.

####  II-C1 Delay functions

The function to model the delay of the communication channel is a sum of sine
waves with random phase shifts. The random values are generated from a
supplied seed for repeatability. The expression for the function is the
following:

$\centering d(t)=a_{0}+\sum_{i=0}^{n-1}\sin{(1.2^{i}\cdot f_{0}\cdot 2\pi\cdot
t+\varphi_{i})}\cdot\frac{c}{1.2^{i}}\@add@centering$ (9)

with $c$ as a normalisation factor to bring the maximum deviation with respect
to $a_{0}$ to the desired deviation $d$. The factor is calculated as follows:

$\centering c=d\cdot
1.77\cdot\left({\sum_{i=0}^{n-1}\frac{1}{1.2^{n}\cdot}}\right)^{-1}\@add@centering$
(10)

With the decreasing amplitudes of the higher frequencies, the function has a
fractal like behaviour, where the amount of higher frequencies only increases
the detail but not the general shape.

###  II-D Predictor

Due to the delay introduced by the network, the control values that arrive
will be out of date. The predictor uses a model of the plant to predict future
states. It needs to know the initial conditions through the feedback channel
and all inputs that will be applied to the plant up to the desired prediction
point, which it obtains from the control buffer. With this information it can
integrate the system’s differential equations and calculate the new state.

With increasing network delay, the predictor has to integrate farther into the
future. This means, that especially for longer delays, it is very important to
have an accurate model of the plant. To achieve this, it runs an online
parameter estimator that continually updates the parameters of the plant model
by observing how the plant state evolves over time and how it reacts to
control inputs. The result of this predictor controller cycle is a series of
plant states and control values.

Another way to improve the performance of the predictor is to use part of the
control values of the last iteration. To implement this, the controller needs
to know the RTT. So it knows the control values that the controller will have
used and control values that can be improved upon with newer information.
Since the control unit always predicts and calculates all states and control
inputs from zero delay up to the prediction horizon, the control packet will
get quite large. This is a waste of bandwidth as the plant should only need
one of those values or rarely a few more if other control packets are lost. To
mitigate this, the control unit sends a section of the series with the values
that are most likely to get used. To know these values, it needs to know the
packet arrival time, which can be calculated using RTT. Then, it keeps a small
window of values around that time point, while discarding the beginning and
end of the series.

###  II-E Delay Compensator

Since the control unit’s prediction of when the control information will
arrive at the plant may be wrong, it can calculate a sequence of control
values for different arrival times. The delay compensator takes this sequence
of control values, buffers, and chooses the appropriate control value from the
most recent packet and apply it to the plant. Every control packet contains a
series of control values spanning a small time window around the predicted
arrival time. When packets are lost, the delay compensator uses the following
values of the series of the last control packet it received. If it reaches the
end of the series, it will hold the last value until it receives a new packet.

##  III Results

In this section, we evaluate and verify the proposed design. We compare either
the different features or variations of a parameter within a range. A large
part of the development and experiments were done as offline simulations as
opposed to online real time simulations.

###  III-A Metrics

The parameters of the control unit are following:

timeStep | 0.008 s
---|---
controllerHorizon | 100
delayMargin | 16
useIntegrator | True
useOldControl | True
useEstimator | True

The time step is the sample time of the plant and controller, the controller
horizon is the prediction horizon of the control unit, and the delay margin is
the length of a sequence of control values. The boolean variables enable or
disable the respective features. The combination of controller horizon and
time step results in a total of $0.8s$ look ahead time. This is greater than
the maximum RTT that can reasonably be expected.

####  III-A1 Offline experiments

Different parameters are used for the delay functions. The base frequency is
$0.4Hz$, base delay is equal to $125ms$, delay variation is set to $100ms$,
and there are $20$ sub-frequencies. It means that the average RTT is $250ms$
and the propagation time for fiber optics is around $66ms$.

####  III-A2 Real time experiments

While the offline simulations enabled quick developments and were generally
easier to execute, they are farther from reality and therefore yielded less
meaningful results. The discrete event simulation was replaced here by a real
time simulation running on a simulated network. The experiments were
essentially the same as the offline simulations with a few network specific
additions. The base parameters of the bottleneck between $C$ and $D$ were the
capacity of $25Mbps$, delay of $125ms$, and loss of $0\%$.

###  III-B Robustness and Performance

The results are evaluated by comparing them to a local delay free controller,
which may have deviations due to a bad plant model, measuring uncertainties,
or inappropriate controller gains. The comparison will only consist of errors
due to the network and network error compensation. The deviation is
accumulated to a scalar value by taking the sum of the squares of the
difference between ideal and actual value. This sum is then divided by the
total evaluated time to take experiment run time out of the equation:

$\centering
RSS=\frac{1}{n}\cdot\sum_{i=0}^{n-1}(x_{1,ideal}(i)-x_{1}(i))^{2}\@add@centering$
(11)

$n$ is the number of samples, $x_{1,ideal}(k)$ is the ideal state resulting
from local control and $x_{1}(k)$ is the actual state. Only the first part of
the state, the position, was evaluated.

####  III-B1 Parameter Estimator

Fig. 4 shows the performance of the parameter estimator. As the figure shows,
the estimator tracks the changing parameters with reasonable accuracy. All of
the estimates show noticeable noise, this is due to the measurement noise. The
effect of noise could be reduced by increasing the forgetting factor $\lambda$
to give more weight to older values, however this would also cause the
estimator to react slower to changes in parameters.

(a) Offline

(b) Real time

Figure 4: Parameter tracking of the estimator

The estimates for $\theta_{2}$ and $\theta_{3}$ show oscillations that
resemble the reference frequency. This is because the estimator has trouble
differentiating between the different sources that influence the prediction
expression. Similarly to the estimator following the noise, this can also be
mitigated by increasing $\lambda$, but would increase the reaction time.

In general, the performance for each parameter seems to scale with the
magnitude of the parameter’s effect on the system and the proximity of the
shared forgetting factor to the optimal individual forgetting factor. Despite
these shortcomings, the estimator still improves the performance of the
control unit, as the estimated parameters are generally closer to the actual
values than the constant approximations without the estimator.

The estimator performance in real time experiments was also very similar to
that of the offline simulations, as can be seen in Fig. 4(b).

####  III-B2 Using the Predictor

Fig. 5 shows the reference tracking compared to the ideal delay free system.
At the start, the error is very noticeable, as the controller will only send
control values after it has received the first system state. This means that
the system will do nothing for one RTT, so that the first $5$ seconds are
ignored in performance evaluation. It can also be seen that the ideal state
does not track the reference or even an amplitude and phase shifted version of
the reference. This is because the controller for the ideal system is not
designed to compensate for some errors of the plant including the errors
resulting from the change of parameters. It doesn’t implement additional
features to actually track the reference. But a very good performance can be
seen as the actual state behaves very close to the delay free ideal.

(a) Offline

(b) Real time

Figure 5: State vs ideal state over time using predictor

The real time version was able to show similarly promising results as the
offline version. The RSS value was only about twice that of the offline
version, which is still a good result. Fig. 5(b) shows the first $15$ seconds
of the experiment, showing a similar performance to the offline equivalent
Fig. 5(a).

Without the predictor, the control unit effectively assumes that there is no
delay. The performance of the system without predictor can be seen in Fig. 6.
It is evident that the controls do not lead to the system following the
reference and the system seems unstable.

Figure 6: State vs ideal state over time without predictor

####  III-B3 Performance over Varying Average Delay

Another effect to examine was the impact of different RTTs on performance.
Fig. 7 shows the effect of the average RTT on the performance. Note that in
the offline simulation the change of the variation was proportional to the
average delay. As expected, the error increases with increasing delay, and it
does so quite drastically at higher delays. This is to be expected, as
inaccuracies in the initial state or parameters can compound exponentially.

Figure 7: Performance over varying average delays

The real time experiment was slightly different to the offline version. In the
real time version, only the base delay of the bottleneck between $C$ and $D$
was set to the delay values of $[50,75,100,125,150,175,200]ms$. The total
delay however is dependent on the delay that was resulted from buffer times
and other network phenomena. This means that the actual RTT was slightly
higher. In a similar trend to the previous experiments, the qualitative
picture is the same as the offline version but with worse performance in
general.

####  III-B4 Variation of Delay Margin

To investigate the hypothesised trade off between performance and bandwidth
when changing the delay margin, the system performance and outgoing bandwidth
of the controller were measured with different delay margins. As shown in Fig.
8, the bandwidth grows linearly with the delay margin, since the individual
packet size is linearly dependent on the delay margin.

Figure 8: Real time bandwidth usage

As can be seen in Fig. 9(a) and 9(b), the performance increases with
increasing delay margin with quickly diminishing returns. This is because the
delay margin improves the performance in two ways: it compensates for changing
delays, and it provides redundancy for packet loss. If the delay varies very
slightly and the packet always arrives within a few steps of the predicted
arrival time, the delay margin does not need to be much larger than the
tolerance window.

(a) Real time performance

(b) Offline performance

Figure 9: Performance over varying delay margin

####  III-B5 Robustness against Packet Loss

Another factor to examine was the robustness of the system against packet
loss. For this, the packet loss in the bottleneck between $C$ and $D$ was
increased in steps. Note that the packet loss was applied in both ways,
meaning that a packet loss of $25\%$ means that only $75\%$ of the state
packets arrive at the controller to generate control packets of which another
$25\%$ would be lost on the way back to the plant for an arriving total of
$56,25\%$.

Fig. 10 shows inconsistent trend, but the system is robust against packet
loss. This is due to high redundancy of the system with a delay margin of
$16$. The system should be able to deal with losing a majority of the packets
as long as the control sequences of the remaining packets overlap.

Figure 10: Performance over Varying Packet Loss Rates

##  IV Conclusion

A networked predictive control system is implemented that can stabilize a
control loop which is closed through a network by compensating for adverse
effects like variable time delays and packet loss in a universal robot. The
resulting controller was able to stabilise the system despite a high delay of
$250ms$ and a considerable packet loss of up to $25\%$. The offline
simulations showed promising results that could be verified in real time
simulations. It was shown that the predictor is able to provide useful
predictions despite an uncertainty in the plant state and parameters. The
introduced estimator was able to provide further improvements to the
capabilities of the predictor by continually accounting for changes in the
plant model. Other suggested improvements such as reusing old control values
to increase the accuracy of the prediction or predicting the state of the
system using an integrator instead of using matrix multiplication were shown
to be effective. Efforts to cut down on excessive redundancy by introducing
the delay margin were shown to have only a small negative effect on
performance while significantly reducing the bandwidth needed. The remaining
redundancy in the system was shown to be high enough to make it very robust
against packet loss or packet reordering, which behaved very similarly in this
context. Since the simulation experiments proved promising, the next logical
step will be to go for full real-world experiments. The simulated plant can be
replaced with the real robot arm. Routing the data can be through internet
between two sites that are separated by a large geographical distance.

## References

  * [1] M. Li and Y. Chen, “Challenging research for networked control systems: A survey,” _Transactions of the Institute of Measurement and Control_ , vol. 41, no. 9, pp. 2400–2418, 2019. [Online]. Available: https://doi.org/10.1177/0142331218799818
  * [2] X. Zhang, Q. Han, X. Ge, D. Ding, L. Ding, D. Yue, and C. Peng, “Networked control systems: a survey of trends and techniques,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 7, no. 1, pp. 1–17, 2020.
  * [3] Y. Halevi and A. Ray, “Integrated communication and control systems: Part i—analysis,” 1988.
  * [4] R. Luck and A. Ray, “Experimental verification of a delay compensation algorithm for integrated communication and control systems,” _International Journal of Control_ , vol. 59, no. 6, pp. 1357–1372, 1994\.
  * [5] J. Nilsson _et al._ , “Real-time control systems with delays,” 1998.
  * [6] N. Xi and T.-J. Tarn, “Planning and control of internet-based teleoperation,” in _Telemanipulator and Telepresence Technologies V_ , vol. 3524. International Society for Optics and Photonics, 1998, pp. 189–195.
  * [7] G. Liu, J. Mu, D. Rees _et al._ , “Networked predictive control of systems with random communication delay,” in _Proceedings of UKACC International Conference on Control_ , 2004.
  * [8] J. Zhang, S. Chai, and B. Zhang, “Model-Based Event-Triggered Dynamic Output Predictive Control of Networked Uncertain Systems with Random Delay,” _International Journal of Systems Science_ , vol. 51, no. 1, pp. 20–34, 2020\.
  * [9] J. Zhang, Y. Xia, and P. Shi, “Design and stability analysis of networked predictive control systems,” _IEEE Transactions on Control Systems Technology_ , vol. 21, no. 4, pp. 1495–1501, 2012.
  * [10] M. Li and Y. Chen, “Robust adaptive sliding mode control for switched networked control systems with disturbance and faults,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 1, pp. 193–204, 2018.
  * [11] N. Vafamand, M. H. Khooban, T. Dragičević, and F. Blaabjerg, “Networked fuzzy predictive control of power buffers for dynamic stabilization of dc microgrids,” _IEEE Transactions on Industrial Electronics_ , vol. 66, no. 2, pp. 1356–1362, 2018.
  * [12] Z.-H. Pang, C.-G. Xia, J. Sun, G.-P. Liu, and Q.-L. Han, “Active fault-tolerant predictive control of networked systems subject to actuator faults and random communication constraints,” _International Journal of Control_ , pp. 1–7, 2021.
  * [13] D. Chen, X. Tang, N. Xi, Y. Wang, and H. Li, “Stability analysis for internet based teleoperated robot using prediction control,” in _2011 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems_. IEEE, 2011, pp. 138–143.
  * [14] X.-M. Zhang, Q.-L. Han, and X. Yu, “Survey on recent advances in networked control systems,” _IEEE Transactions on industrial informatics_ , vol. 12, no. 5, pp. 1740–1752, 2015.
  * [15] M. S. Mahmoud, “Networked control systems analysis and design: An overview,” _Arabian journal for science and engineering_ , vol. 41, no. 3, pp. 711–758, 2016.
  * [16] M. Pollák, M. Kočiško, D. Paulišin, and P. Baron, “Measurement of unidirectional pose accuracy and repeatability of the collaborative robot ur5,” _Advances in Mechanical Engineering_ , vol. 12, no. 12, p. 1687814020972893, 2020.
  * [17] A. Shaiju and I. R. Petersen, “Formulas for Discrete Time LQR, LQG, LEQG and Minimax LQG Optimal Control Problems,” _IFAC Proceedings Volumes_ , vol. 41, no. 2, pp. 8773–8778, 2008.
  * [18] S. Fuller, B. Greiner, J. Moore, R. Murray, R. van Paassen, and R. Yorke, “The python control systems library (python-control),” in _2021 60th IEEE Conference on Decision and Control (CDC)_. IEEE, 2021, pp. 4875–4881.