# Entropy Regularized Reinforcement Learning with Cascading Networks

Riccardo Della Vecchia∗<EMAIL_ADDRESS>
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL
Lille, France Alena Shilova∗<EMAIL_ADDRESS>
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL
Lille, France Philippe Preux<EMAIL_ADDRESS>
Inria, Univ. Lille, CNRS
Lille, France Riad Akrour∗<EMAIL_ADDRESS>
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 \- CRIStAL
Lille, France

###### Abstract

Deep Reinforcement Learning (Deep RL) has had incredible achievements on high
dimensional problems, yet its learning process remains unstable even on the
simplest tasks. Deep RL uses neural networks as function approximators. These
neural models are largely inspired by developments in the (un)supervised
machine learning community. Compared to these learning frameworks, one of the
major difficulties of RL is the absence of i.i.d. data. One way to cope with
this difficulty is to control the rate of change of the policy at every
iteration. In this work, we challenge the common practices of the
(un)supervised learning community of using a fixed neural architecture, by
having a neural model that grows in size at each policy update. This allows a
closed form entropy regularized policy update, which leads to a better control
of the rate of change of the policy at each iteration and help cope with the
non i.i.d. nature of RL. Initial experiments on classical RL benchmarks show
promising results with remarkable convergence on some RL tasks when compared
to other deep RL baselines, while exhibiting limitations on others.

**footnotetext: These authors contributed equally to this work.

Keywords: Policy iteration, Entropy regularization, Cascade neural networks,
Mirror descent

##  1 Introduction

Reinforcement learning (RL) formulates a general machine learning problem in
which an agent has to take a sequence of decisions to maximize a supervision
signal (Sutton and Barto, 2018; Szepesvari, 2010). RL as a learning framework,
especially when combined with neural function approximators, has made large
practical breakthroughs over the last years on complex, high dimensional
problems (Silver et al., 2016; Mnih et al., 2013), and its range of
application continues to grow to new domains such as drug discovery (Bengio et
al., 2021). Surprisingly however, even simple RL tasks can exhibit one of the
main disadvantages of current deep RL algorithms: their training is unstable
and does not exhibit convergence to an optimal policy. Rather, deep RL
discovers good policies along the way with often large performance
oscillations in-between.

So what makes the training process of a neural model by RL more brittle than
say, the training of the same model to regress a continuous signal or classify
images? Certainly, one of the major challenges of RL, and specifically online
RL111We note that offline, batch, RL also violates the independent and
identically distributed assumption since there is still a mismatch between the
distribution of training data and the data generated by the learned policy,
which is inherent to the sequential nature of RL in general. However, this
work will only focus on the online RL setting., is that the agent’s decisions
influence the data gathering process, violating the typical assumption of the
aforementioned supervised learning tasks that data is independent and
identically distributed (Bishop, 2006). To cope with this challenge, several
RL algorithms constrain the agent’s behavior to only slowly change. In
trajectory optimization and optimal control, a new policy is made close to the
policy around which the dynamics of the system have been approximated through
mixing (Todorov and L., 2005; Tassa et al., 2014) or by a Kullback-Leibler
(KL) constraint (Levine and Abbeel, 2014). In policy gradient, a key
breakthrough was the use of natural gradient that follows the steepest descent
in policy space rather than parameter space (Bagnell and Schneider, 2003;
Peters and Schaal, 2008; Bhatnagar et al., 2009), i.e. seeking maximal
objective improvement with minimal policy change.

Constraining successive policies to be close to each others in approximate
policy iteration is justified in the seminal work of Kakade and Langford 2002
by the mismatch between what the policy update should optimize (advantage
function in expectation of its own state distribution) and what is optimized
in practice (advantage function in expectation of the state distribution of
the data gathering policy). As in optimal control, closeness can be achieved
by mixing policies (Kakade and Langford, 2002; Pirotta et al., 2013), limiting
deviation of their probability ratio to one (Schulman et al., ) or
constraining their KL-divergence (Schulman et al., 2015; Abdolmaleki et al.,
2018; Tangkaratt et al., 2018; Akrour et al., 2018). In the latter case, when
KL-divergence is used to regularize the policy update, a recent line of
research has drawn similarities between these algorithms and the convex
optimizer mirror descent (Neu et al., 2017; Geist et al., 2019; Abbasi-Yadkori
et al., 2019), which provides as a result convergence guaranties to an optimal
policy. Of course, practical implementations of these algorithms may still
fail to find the optimal policy due to two difficulties: i) learning the
Q-function of the current policy from sample data and ii) solving the entropy
regularized policy update in the neural parameter space.

In this paper, we investigate the use of Cascade Neural Networks (Cascade-NN,
(Fahlman and Lebiere, 1989)) in the context of entropy regularized policy
iteration to address ii. Unlike typical neural models, Cascade-NNs can grow in
size during learning. Importantly, when new neurones are added, old ones are
frozen which allows us to perform the policy update in closed form and
completely eliminate the second source of errors discussed above. In the
remainder of the paper, we will first describe in a preliminaries section
Cascade-NNs and entropy regularized RL (Section 2), before describing our
approach in Section 3. We discuss related work in Section 5 after comparing
our algorithm in Section 4 to deep RL baselines on classical RL benchmarks.

##  2 Preliminaries

In this section, we introduce our notations and briefly describe the framework
of entropy regularized RL. We then present the general architecture of
Cascade-NNs as described in the seminal paper of (Fahlman and Lebiere, 1989)
before describing how it is used in our algorithm.

###  2.1 Notation

We consider Markov Decision Problems (MDPs) defined as a tuple
$(\mathcal{S},\mathcal{A},r,P,\gamma)$ to model the interactions of the agent
with the environment. $\mathcal{S}$ is a finite set of states, $\mathcal{A}$
is a finite set of actions, $r$ is the unknown reward function
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ and $P$ is the unknown
probability transition function
$P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta_{\mathcal{S}}$, where
$\Delta_{\mathcal{S}}$ is the set of distributions over ${\mathcal{S}}$.

Let the policy $\pi:\mathcal{S}\rightarrow\Delta_{\mathcal{A}}$ be a mapping
between a state to a distribution over actions. For every such policy $\pi$,
one can compute the value function

$V_{\pi}(s)=\mathbb{E}\Biggl{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\Biggm{|}s_{0}=s\Biggr{]},$

where the expectation is taken w.r.t. all states and actions following $s$. We
also define the state-action value function as

$Q_{\pi}(s,a)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim
P(s,a)}V_{\pi}(s^{\prime}).$

###  2.2 Entropy regularized policy iteration

RL considers the problem of finding the optimal policy in the MDP’s unknown
environment. One way of doing so is to use policy iteration methods that
successively perform i) a policy evaluation step to compute $Q_{\pi}$ and ii)
a policy improvement step, yielding a new policy $\pi^{\prime}$ that picks at
every state $s$ an action in $\arg\max_{a}Q_{\pi}(s,a)$. Repeatedly performing
these two steps will converge in finite time to an optimal policy,
irrespective of the choice of the initial policy (Sutton and Barto, 2018).
However, we notice that at any iteration, the policy $\pi^{\prime}$ obtained
after a policy improvement step does not explore actions that do not maximize
the previous Q-function $Q_{\pi}(s,.)$, for every state $s$. Hence, to
correctly estimate $Q_{\pi^{\prime}}(s,.)$ for such actions in practice, one
would need to introduce another data gathering policy that is more explorative
than $\pi^{\prime}$. An alternative is to regularize the policy update step to
maintain the stochasticity of $\pi^{\prime}$. This is usually performed by
adding a KL-divergence term between $\pi$ and $\pi^{\prime}$, ensuring that
exploration does not vanish too quickly.

In contrast to these algorithms, in this paper, we investigate solving the
soft policy update exactly for every state, using an incrementally increasing
neural architecture. Given $\hat{Q}_{\pi_{i}}$, the approximation of the
Q-function of $\pi_{i}$ at iteration $i$ of the algorithm, our policy update
at every state $s$ is similar to that of MPO (Abdolmaleki et al., 2018) and
Politex (Abbasi-Yadkori et al., 2019) and is given by the following
optimization problem:

$\begin{split}\pi_{i+1}(s)&=\arg\max_{\pi}\mathbb{E}_{a\sim\pi(.|s)}\hat{Q}_{\pi_{i}}(s,a)-\frac{1}{\eta}\text{KL}(\pi(s)\|\pi_{i}(s)),\\\
&\propto\exp\left(\eta\hat{Q}_{\pi_{i}}(s,\cdot)\right)\pi_{i}(.|s),\end{split}$
(1)

where $\text{KL}(\pi(s)\|\pi_{i}(s))$ is the KL-divergence between the
distributions over $\mathcal{A}$ that are $\pi(s)$ and $\pi_{i}(s)$ at state
$s$. By recursion and assuming that $\pi_{0}$ is the uniform distribution over
$\mathcal{A}$, we get

$\displaystyle\pi_{i+1}(s)\propto\exp\left(\eta\sum_{k=0}^{i}\hat{Q}_{\pi_{k}}(s,\cdot)\right).$
(2)

We thus see that the entropy regularized policy update can be solved in closed
form, and policies have a very simple form provided we can keep track of all
previously estimated Q-functions. Of course, this might appear too cumbersome
in practice if we consider that each Q-function is a separate neural network.
For this reason, other papers approximated the update, e.g. in MPO
(Abdolmaleki et al., 2018), by fitting a Gaussian neural policy minimizing the
divergence to $\exp\left(\eta\hat{Q}_{\pi_{i}}(s,\cdot)\right)\pi_{i}(.|s)$.
Alternatively, in (Lazic et al., 2021), authors have considered keeping only
the last few Q-functions or keeping a large replay memory of past MDP
transitions, and fitting a single network to the sum of Q-functions. In
contrast, our policy will take the exact form of Eq. (2). However, we will not
train independent neural networks for each $\hat{Q}_{\pi_{k}}$, but leverage
the Cascade-NN architecture and only add a few neurons at every iteration. The
rationale is that since the policy will not change drastically between
iterations neither will their Q-function and one might need only a few more
neurons to learn $\delta_{k}=\hat{Q}_{\pi_{k}}-\hat{Q}_{\pi_{k-1}}$.

###  2.3 Cascade-NN

][t]0.3

$i_{1}$$i_{2}$$o$

(a) No hidden layers

][t]0.3

$i_{1}$$i_{2}$$h_{1}$$o$

(b) 1 hidden layer

][t]0.35

$i_{1}$$i_{2}$$h_{1}$$h_{2}$$o$

(c) 2 hidden layers

Figure 1: Incremental growth in Cascade-NN architecture and freezing of
weights. Dashed lines show learnable parameters after each addition, while
solid lines represent frozen weights. In a Cascade-NN, when new hidden nodes
are added, all inputs to older nodes are frozen so as to freeze older
features.

Cascade Neural Network (Cascade-NN) or also called Cascade-Correlation
Networks (Fahlman and Lebiere, 1989) is a special type of neural network
architecture that is growing at each epoch by $n$ (typically $n=1$) new
neurons that are connected to the input of the NN and all previously created
hidden neurons. At first, it has no hidden layer (Figure 1(a)), and gradually
more and more neurons get added to its structure (Figures 1(b) and 1(c)).
Historically, these newly added neurons were trained following a three phase
process. Firstly, a batch of neurons larger then $n$ is added and the output
of each of those new neurons is trained to maximize the correlation with the
current residual error of the model. Secondly, the neurons are ranked
according to their correlation and only the $n$-top neurons are kept for the
final phase. Thirdly, new neurons are connected to the output layer and the
weights of the output layer are updated in order to minimize the residual
error. Importantly, throughout all phases older neurons are frozen, which
means that one can easily compute the sum of all past outputs $\sum_{k}o_{k}$
by simply summing all past weights of the (linear) output layer. An
illustration of the freezing procedure is given in Figure 1, showing all re-
trainable parameters at every iteration. Here $i_{1}$ and $i_{2}$ correspond
to data inputs, $h_{1}$ and $h_{2}$ to two hidden neurons and $o$ to the
output. Solid edges show the frozen connections, while dashed edges are for
trainable parameters.

In (Fahlman and Lebiere, 1989), the authors emphasize the following advantages
of Cascade-NN with respect to classical MLP neural networks:

  * •

Non-parametric training, there might be less hyper-parameters to tune such as
the depth, width and connectivity of NN, its training (and growth) is stopped
automatically as soon as a stopping criterion is satisfied;

  * •

Fast learning, freezing all the layers except the last one helps to optimize
the parameters without doing back-propagation, in this way all neurons have
their independent goal and they can ”settle into distinct useful roles”;

  * •

Incremental learning, especially useful when the model constantly receives
some new information (data) in a stream manner, in this case old features are
preserved, while new features enrich the feature extraction for the newly
obtained data.

Prior work already explored the use of Cascade-NN in RL. Notably, (Girgin and
Preux, 2008) combined features trained to maximize correlation with the
Bellman residual before using LSPI (Lagoudakis and Parr, 2003) to find an
optimal policy given the current set of features. However, (Girgin and Preux,
2008) did not leverage the properties of Cascade-NN to perform entropy
regularized policy update in closed form and investigate the stability of this
approach in a deep RL context, which is the main contribution of this paper.
Regarding the learning of the Q-function, we do not use LSTDQ (Lagoudakis and
Parr, 2003). Instead, we implemented a more streamlined training procedure
using a (neural) fitted Q-iteration scheme, similar to DQN (Mnih et al.,
2015)—although using the Bellman operator of the current policy instead of the
Bellman optimality operator in DQN—for simultaneously training the features
and learning the Q-function of the current policy. However, we discuss how
Cascade-NN techniques presented here can be used in the context of our work at
the end of Section 3.

##  3 The MicaRL algorithm

Algorithm 1 Pseudo code of the MicaRL Algortihm

Set $\hat{Q}_{\pi_{0}}$ to the zero function

for Iteration i in $\\{1,\dots,\text{NB\\_ITER}{}\\}$ do

Collect NB_SAMP transitions of type $(s,a,r,s^{\prime},a^{\prime})$ from the
environment following policy
$\pi_{i}(s)\propto\exp\left(\eta\sum_{k=0}^{i-1}\hat{Q}_{\pi_{k}}(s,\cdot)\right)$

Add $n$ neurons to the current Cascade-NN

Set $\delta^{0}$ to the zero function

for Epoch e in $\\{1,\dots,E\\}$ do

Compute target
$r+\gamma(\hat{Q}_{\pi_{k-1}}+\delta^{e-1})(s^{\prime},a^{\prime})-\hat{Q}_{\pi_{k-1}}(s,a)$
for every transition $(s,a,r,s^{\prime},a^{\prime})$

Obtain $\delta^{e}$ by fitting the target using stochastic gradient descent

end for

Set $\hat{Q}_{\pi_{k}}=\hat{Q}_{\pi_{k-1}}+\delta^{E}$

end for

In this section, we introduce our algorithm MicaRL. MicaRL is a deep RL
algorithm that follows the policy iteration scheme. To approximate
$Q_{\pi_{k}}$, we use the adapted Cascade-NN model depicted in Fig. 2. This
neural model has two heads giving the last Q-function and the sum of the last
Q-functions. Alg. 1 shows the pseudo-code for MicaRL. Starting with a zero
function $\hat{Q}_{\pi_{0}}$, at every iteration $i>0$ we collect a dataset
$\mathcal{D}^{\pi_{i}}$ of transition samples of type
$(s,a,r,s^{\prime},a^{\prime})$, where actions are sampled from the current
policy $\pi_{i}\propto\exp(\eta\sum_{k=0}^{i-1}\hat{Q}_{\pi_{k}})$ following
the entropy regularized policy update of Eq. (2), and the next state
$s^{\prime}$ and rewards are given by the environment. Using the transition
dataset $\mathcal{D}^{\pi_{i}}$, we learn the Q-function approximation
$\hat{Q}_{\pi_{k}}$ following a standard (neural) fitted Q-iteration learning
scheme. In our implementation, we learn
$\delta_{i}=Q_{\pi_{i}}-Q_{\pi_{i-1}}$, instead of $Q_{\pi_{i}}$. Starting
with a zero function $\delta^{0}_{i}$, at each epoch $e$, we compute the
targets $T^{e}_{i}(s,a)$ for every
$(s,a,r,s^{\prime},a^{\prime})\in\mathcal{D}^{\pi_{i}}$,

$T^{e}_{i}(s,a)=r+\gamma(\hat{Q}_{\pi_{i-1}}+\delta^{e-1}_{i})(s^{\prime},a^{\prime})-\hat{Q}_{\pi_{i-1}}(s,a).$
(3)

We then update the learnable weights $\xi^{(i)}$and $W_{\delta}^{(i)}$ (see
implementation details below) of the Cascade-NN for one epoch on the dataset
$\mathcal{D}^{\pi_{i}}$ by minimizing the loss

$\min_{\xi^{(i)},W_{\delta}^{(i)}}\sum_{(s,a,r,s^{\prime},a^{\prime})\sim\mathcal{D}^{\pi_{i}}}\left(\delta^{e}_{i}(s,a)-T^{e}_{i}(s,a)\right)^{2},$
(4)

using stochastic gradient descent. Finally, the process of computing targets
and fitting $\delta^{e}_{i}$ is repeated until reaching a given number of
epochs $E$.

As for the policy update, we simply add the new Q-function
$Q_{\pi_{i}}=Q_{\pi_{i-1}}+\delta^{E}_{i}$ to the head of the Cascade-NN
accumulating all past Q-functions, completing the definition of ${\pi_{i+1}}$.
Our implementation of MicaRL, including all experimental results, are obtained
using an on-policy setting. We note however that the extension of our
algorithm to the off-policy setting is trivial (one simply needs to replace
$a^{\prime}$ in a transition sample to match the current policy), but its
investigation is left for future work.

$s$$h_{1}$$h_{2}$$q$$\Sigma q$ Figure 2: Our Q-function and policy network.
The Cascade-NN has two heads, one storing the last Q-function—used to compute
the targets of the neural fitted Q step in Alg. 1, while the output $\Sigma q$
accumulates all past weight matrices of node $q$, and is used by the softmax
policy. Note that all nodes in the figure can be multi-dimensional, including
hidden nodes/layers. See the implementation details for more information.

#### Implementation details.

Our Cascade-NN at any iteration $i$ is taking state $s\in\mathcal{S}$ as an
input and outputs either a vector $\sum_{k=0}^{i-1}\hat{Q}_{\pi_{k}}(s,\cdot)$
or a vector $\hat{Q}_{\pi_{i-1}}(s,\cdot)$. For a given Cascade-NN, we start
with zero hidden neuron and we grow the architecture at each iteration
$i\in\\{1,\ldots,\text{NB\\_ITER}{}\\}$. In particular, at the beginning of
iteration $i$ we have $(i-1)n$ hidden neurons and during the iteration we grow
it by $n$ to reach a total of $in$ neurons at the end of $i$-th iteration.
Those hidden neurons are used to extract features. Further, what we consider
as features is the direct inputs of output layers. At the beginning of the
first iteration, the input is connected directly to the output layers, thus
the input acts as the feature before the training. At any iteration, the input
stays connected to the output layers and thus is a part of feature vectors. At
the beginning of iteration $i$, its $(i-1)n$ old hidden neurons plus the input
vector represent the feature vector function
${\phi^{(i-1)}}:\mathcal{S}\rightarrow\mathbb{R}^{(i-1)n+\dim\mathcal{S}}$
that for any state $s\in\mathcal{S}$ returns its corresponding feature vector
$\phi^{(i-1)}(s)$ from iteration $i-1$. Further, those neurons are frozen and
not trained. Moreover, we have access to the two output layers described
above. The weight matrix
$W^{(i-1)}\in\mathbb{R}^{|\mathcal{A}|\times\left((i-1)n+\dim\mathcal{S}\right)}$
corresponds to the sum of Q-values so that the matrix-vector product
$W^{(i-1)}\phi^{(i-1)}(s)$ gives the vector
$\sum_{k=0}^{i-1}\hat{Q}_{\pi_{k}}(s,\cdot)$ of size $|\mathcal{A}|$ (for
simplicity, we omit bias term). Therefore, this output layer can be used to
compute values necessary for policy $\pi_{i}$ from Eq. (2). The weight matrix
$W_{Q}^{(i-1)}\in\mathbb{R}^{|\mathcal{A}|\times\left((i-1)n+\dim\mathcal{S}\right)}$
corresponds to the approximation Q-value of policy $\pi_{i-1}$, that is
$\hat{Q}_{\pi_{i-1}}(s,\cdot)=W_{Q}^{(i-1)}\phi^{(i-1)}(s)$. During iteration
$i$, $n$ new neurons are generated in a cascade manner as described in Section
2.3. They are responsible for computing a new component of the features, in
particular they correspond to a vector function
$\tilde{\phi}^{(i)}_{\xi^{(i)}}:\mathbb{R}^{n(i-1)+\dim\mathcal{S}}\rightarrow\mathbb{R}^{n}$
that takes features from the previous iteration $\phi^{(i-1)}(s)$ as an input
and outputs $\tilde{\phi}^{(i)}_{\xi^{(i)}}(\phi^{(i-1)}(s))$, which is
further concatenated222We use cat to denote concatenation with respect to the
last mode of a tensor, so in case of matrices $A\in\mathbb{R}^{n\times k}$,
$B\in\mathbb{R}^{n\times m}$, then
$C=\textsc{cat}(A,B)\in\mathbb{R}^{n\times(k+m)}$ is the matrix with first $k$
columns from $A$ and last $m$ columns from $B$. with $\phi^{(i-1)}(s)$ to
constitute
$\phi^{(i)}(s)=\textsc{cat}\left(\phi^{(i-1)}(s),\tilde{\phi}^{(i)}_{\xi^{(i)}}(\phi^{(i-1)}(s))\right)\in\mathbb{R}^{ni+\dim\mathcal{S}}$.
Moreover, the new output layer is initialized with a weight matrix
$W_{\delta}^{(i)}\in\mathbb{R}^{|\mathcal{A}|\times(ni+\dim\mathcal{S})}$ that
should represent
$\delta_{i}(s,\cdot)=\hat{Q}_{\pi_{i}}(s,\cdot)-\hat{Q}_{\pi_{i-1}}(s,\cdot)=W_{\delta}^{(i)}\phi^{(i)}(s)$.
Parameters $\xi^{(i)}$ and $W_{\delta}^{(i)}$ are optimized to minimize the
loss defined in Eq. (4). At the end of iteration $i$, once $\xi^{(i)}$ and
$W_{\delta}^{(i)}$ are optimized, $\hat{Q}_{\pi_{i}}$ can be evaluated with
$\hat{Q}_{\pi_{i-1}}(s,\cdot)+\delta_{i}(s,\cdot)=W_{Q}^{(i-1)}\phi^{(i-1)}(s)+W_{\delta}^{(i)}\phi^{(i)}(s)$,
therefore by setting
$W_{Q}^{(i)}=\textsc{cat}(W_{Q}^{(i-1)},O^{|\mathcal{A}|\times
n})+W_{\delta}^{(i)}$, where $O^{|\mathcal{A}|\times n}$ is a zero matrix of
dimension $|\mathcal{A}|\times n$, the approximation
$\hat{Q}_{\pi_{i}}(s,\cdot)$ is naturally obtained from
$W_{Q}^{(i)}\phi^{(i)}(s)$. Similarly,
$\sum_{k=0}^{i}\hat{Q}_{\pi_{k}}(s,\cdot)=W^{(i)}\phi^{(i)}(s)$ where
$W^{(i)}=\textsc{cat}(W^{(i-1)},O^{|\mathcal{A}|\times n})+W_{Q}^{(i)}$.

In our implementation, the total number of weights grows as
$\mathcal{O}(i^{2})$ since new neurons connect to all previous neurons. This
might be too prohibitive in practice, and we will experiment in future work
with variants where new neurons only connect to a fixed number of past
neurons. For example, following the correlation ideas of the original Cascade-
NN (see Section 2.3), one might select the most promising past neurons
according to the correlation between their activation and the current Bellman
residual.

##  4 Experiments

We evaluate MicaRL on four different gym environments: CartPole-v1,
Acrobot-v1, a discrete action-space version of Pendulum-v1 and MountainCar-v0
and we compare it with A2C and DQN agents as implemented in rlberry Domingues
et al. (2021). In our experiments we use the default implementations of the
agents: for the value function and the policy network in A2C, and for the
Q-value function network of DQN, we take multilayer percecptrons with two
hidden layers and 64 hidden units each. Our experiments show that MicaRL is in
general more stable than both A2C and DQN and is achieving performances
superior or comparable to DQN on three out of four environments, as it is
evident from Figures 3 and 4. We plot curves averaged over five different
seeds, using the default seeding handler from (Domingues et al., 2021), and
the shaded area indicates one standard deviation. Figure 3 is a zoom of the
reward curves near the asymptotic values reached by the best performing
algorithms. For this reason, for Pendulum-v1 we do not see the curve
corresponding to A2C, since its performance is quite poor on this environment.
In Figure 4, we plot a more detailed study in which we do not just show
rewards but also losses during training, average normalized entropy (across
visited states) and the KL-divergence between two consecutive policies for A2C
and MicaRL (not for DQN since it is not a policy iteration algorithm). Note
that the way the loss is computed differs from one algorithm to another:
MicaRL and DQN compute Bellman residuals of different Bellman operators, while
A2C tries to minimize the difference between critic output and Monte Carlo
estimate of the value function based on the dataset. This explains different
levels of loss values in loss plots.

Figure 3: Zoom of the reward curves around their asymptotic values on
CartPole-v1 (left), Acrobot-v1 (center), and Pendulum-v1 (right).

Figure 4: MicaRL and baselines on CartPole-v1 (top-left corner), Acrobot-v1
(top-right corner), Pendulum-v1 (bottom-left corner) and MountainCar-v0
(bottom-right corner).

We note that the number of neurons $n$ to add at every iteration is important.
For small $n$, we typically lose the good properties of over-parameterization
when learning $\delta$. Especially, since in our implementation we do not use
the Cascade-NN idea discussed in Section 2.3 of training a number of candidate
neurons $>n$ before picking the $n$ most promising ones. As a result, we found
that using small $n$, e.g. $n=1$, would often lead to dead neurons and more
generally, to attraction to poor local optima. We address this issue by using
a larger than usual $n$, typically $n\in\\{10,20,50\\}$. We try also
combinations of the other hyper-parameters: the strength parameter of the KL-
regularizer in Eq. (1) is $\eta\in\\{0.01,0.1,0.5,1,5\\}$ and the number of
epochs per iteration $E\in\\{64,128\\}$. Hyper-parameter search is done in
grid search manner with the help of Ray Tune software Liaw et al. (2018).
Results are quite similar for different choices of epochs, therefore we
further keep it fixed $E=64$. In contrast, our algorithm is more sensitive to
the choice of $\eta$ and slightly less to the choice of $n$. Further, we
report separately for each environment the best choice of $n$ and $\eta$. In
addition, each trial is conducted with a number of iterations equal to
$\text{NB\\_ITER}=100$, $\text{batch-size}=64$ and in each iteration we take
$\text{NB\\_SAMP}=10\,000$ steps in the environment. Therefore, the total
number of steps in the environment is equal to $1\,000\,000$ for each
experiment, which is the quantity on the x-axis in Figures 4 and 3. The final
architecture of MicaRL contains $\text{NB\\_ITER}\times n$ hidden neurons.

The strong stability of MicaRL is probably due to the second term in Eq. (1)
that acts as a regularizer. Nonetheless, MicaRL achieves impressive and
surprising performances. For example on CartPole-v1, MicaRL is able to find a
much better approximation of the Q-value function with respect to our
baselines and the reward remains stable at its maximum value after a brief
training phase. On Acrobot-v1 and Pendulum-v1, MicaRL is performing as good,
or better, than DQN, and significantly better than A2C. MountainCar-v0 is
instead known to be a difficult environment and MicaRL is not able to learn
anything, the same for A2C. In contrast, DQN has a good performance on this
environment as well.

#### CartPole-v1

We test the performance of MicaRL against the baselines of A2C and DQN on
CartPole-v1 in Figure 4. The best performance of MicaRL is observed for $n=10$
and $\eta=0.1$. The training loss of MicaRL is getting closer to zero than the
losses of the two other algorithms, showing that MicaRL succeeds in better
approximating Q-value functions. Furthermore, A2C and DQN loss curves are
plateauing and oscillating, proving them to be unstable. The curves of the
rewards for A2C and DQN also exhibit an oscillating behaviour. Sometimes they
manage to reach the optimum value, which is 500 for this environment, but on
average they do not manage to maintain this performance in all episodes. The
stability of MicaRL is also confirmed by the KL-divergence plot where at the
end of the training the difference between successive policies becomes
increasingly negligible, meaning that the algorithm is converging to a policy.
Interestingly, the normalized entropy (having value between 0 for a
deterministic policy and 1 for the uniform policy) of the policy is staying
quite high, around $0.65$, for MicaRL. This reflects the fact that under a
near optimal policy that balances the pole correctly, there are a lot of
states where both actions are viable, meaning that our algorithm not only
finds an optimal policy, but several optimal ones. A2C exhibits an even higher
entropy, but since its reward is still oscillating, one cannot draw the same
conclusions. Overall, MicaRL achieves the best results, while being more
stable than other baselines.

#### Acrobot-v1 and Pendulum-v1

In the case of Acrobot-v1, MicaRL performs the best with $n=50$, $\eta=1$,
while in case of Pendulum-v1 with $n=50$ and $\eta=0.1$. Results on Acrobot-v1
and a discrete version of Pendulum-v1 are almost identical to DQN in terms of
the rewards and both significantly outperform A2C. We note that due to the
intrinsic stability of MicaRL, it shows even less oscillating behaviour. The
loss curves of MicaRL and DQN are very similar for Acrobot-v1, while, we note
that for Pendulum-v1 MicaRL is fitting much better than DQN the loss function.

#### MountainCar-v0

We show the performance of MicaRL on MountainCar-v0 with $n=50$ and
$\eta=0.1$, but the results for all the other hyperparameters are very
similar. Due to its very sparse reward, MountainCar-v0 is usually a very hard
task for reinforcement learning agents. In this case, the normalized entropy
of the policy is very close to one. This indicates that the policy is sampling
actions from the uniform distribution, which seemingly prevents the discovery
of any postive reward. As all rewards that the agent observes are $-1$, the
learned Q-function is a completely flat function, which according to Eq. (2)
results in a uniform distribution, explaining why the entropy stays close to
one. All in all, while entropy regularization is a good heuristic for
maintaining high entropy and sustaining exploration, we can see that it does
not fully address the exploration problem in RL.

#### Summary of results.

To summarize, we can categorize the results in three. In i) the best case
scenario, on CartPole-v1, we observed formidable convergence on all metrics:
the cumulative reward reaching and staying at its maximal value, the mean
squared Bellman residual going to $10^{-3}$, and the KL-divergence between
successive policies reaching zero. In ii) the middle case on Pendulum-v1 and
Acrobot-v1, while results are on par with the state-of-the-art with perhaps an
ever so slightly higher stability of the cumulative rewards, there still
remains a persistent Bellman error despite the ever growing size of the
Cascade-NN and an abundance of data on relatively simple problems. This is
somehow unsatisfying and indicates that research is still needed to discover
better policy evaluation algorithms. Finally, in iii) the failure case on
MountainCar-v0, where the policies at all iterations are close to uniform over
the action space, suggesting that while entropy regularization is a good
exploration mechanism with theoretical convergence guarantees, in practice,
other exploration methods might still be necessary in some more challenging
sparse reward cases.

##  5 Related Work

Examples of deep RL algorithms implementing entropy regularization include
TRPO (Schulman et al., 2015), SAC (Haarnoja et al., 2018) and MPO (Abdolmaleki
et al., 2018), but the closest to our work is Politex (Abbasi-Yadkori et al.,
2019; Lazic et al., 2021). Similarly to MicaRL, in Politex, the new policy is
defined as a Boltzmann distribution over the sum of all past state-action
value estimates, resulting from a KL-divergence regularization on the policy
update, which makes the learning process less noisy. For example, the
experimental results of (Lazic et al., 2021) show good convergence results,
outperforming other policy optimization algorithms. However, to perform this
policy update in closed form, Abbasi-Yadkori et al. (2019) considered learning
separate state-action value networks which is a computationally expensive
process requiring to store all past NNs corresponding to different
Q-functions. The more recent implementation of (Lazic et al., 2021) avoids
keeping different NNs and instead relies on one NN and experience replay
buffer to approximate directly the average behaviour of all previous state-
action value functions.

The strategy of Politex requires the knowledge of all previously trained
state-action value functions. Implementing them with NNs is challenging as the
candidate NN should be able to approximate equally good all the old functions
together with the new one. This particular problem known as catastrophic
forgetting is studied in the field of Incremental Learning. Among the
approaches of Incremental Learning, one could distinguish three main
directions: architectural, regularization and rehearsal strategies.
Architectural strategies, as used in MicaRL, suggest modifying (e.g.
expanding) the NN structure in order to preserve the good performance on the
old tasks and achieve good precision on the new one (Rusu et al., 2016;
Fahlman and Lebiere, 1989). Regularization techniques may be divided in two
groups. The first group is weight regularization, which is typically done by
introducing the additional component in the loss whose goal is to penalize the
change of the weights that are important for the old tasks (e.g. (Kirkpatrick
et al., 2017; Aljundi et al., 2018)). Second group is knowledge distillation
(Li and Hoiem, 2017; Lee et al., 2019), which was firstly used for transfer
learning, but can be efficiently applied to Incremental Learning, by bringing
the ”knowledge” from the old model to the new one, forcing the output of the
new model to be more consistent with the output of old models. Finally,
rehearsal methods (Rebuffi et al., 2017) or pseudo-rehearsal methods (Mellado
et al., 2019) alleviate the problem of catastrophic forgetting by reusing the
data of the old tasks when learning the new one. Pseudo-rehearsal methods
slightly differ from rehearsal methods as they generate data from the old
tasks instead of storing it. More thorough overview of Incremental Learning
methods can be found in (Luo et al., 2020).

##  6 Conclusion

In this paper, we consider entropy regularized reinforcement learning
algorithms and how to implement them efficiently in practice. These algorithms
were shown to be more stable than the classical approaches as they constrain
the current policy to be closer to the previous ones. In our study, we
concentrate on the Politex algorithm that builds the policy from the output of
all Q-value functions of past policies. Implementing those functions in
practice is not trivial, and straightforward approaches of assigning one
neural network per Q-value function is only possible for very small tasks and
cannot scale. Instead, we suggest using MicaRL based on Cascade-NN, a neural
architecture that grows each iteration by $n$ neurons, capable of preserving
past information while new neurons can leverage increasingly richer feature
representations. Our preliminary results show that this novel approach can
successfully compete with the state-of-the-art for most of the considered use
cases, while exhibiting impressive convergence on all considered metrics for
CartPole-v1. We believe these are very promising preliminary results
suggesting that MicaRL and its non-parametric approach to Q-function
approximation is worth further investigation. In future work, we would like to
understand what makes the algorithm better approximate the Q-function on
CartPole-v1 than the other problems, and research the integration of more
sophisticated exploration mechanisms in MicaRL to tackle sparse reward
problems.

Acknowledgments

The authors would like to acknowledge the financial support of the French
Ministry of Higher Education and Research, Inria, the Hauts-de-France region.
Philippe Preux is also supported by the Métropole Européenne de Lille, through
the AI chair Apprenf number R-PILOTE-19-004-APPRENF. Riccardo Della Vecchia is
thankful for the funding received by the CHIST-ERA Project Causal
eXplainations in Reinforcement Learning – CausalXRL.
333https://www.chistera.eu/projects/causalxrl Alena Shilova acknowledges the
funding coming by the Challenge HPC-BigData INRIA Project LAB.
444https://project.inria.fr/hpcbigdata/ We also thank the Scool team at Inria
Lille Nord Europe.

## References

  * Abbasi-Yadkori et al. (2019) Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz.  Politex: Regret bounds for policy iteration using expert prediction.  In _International Conference on Machine Learning_ , pages 3692–3702. PMLR, 2019.
  * Abdolmaleki et al. (2018) Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller.  Maximum a posteriori policy optimisation.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Akrour et al. (2018) R. Akrour, A. Abdolmaleki, H. Abdulsamad, J. Peters, and G. Neumann.  Model-free trajectory-based policy optimization with monotonic improvement.  _Journal of Machine Learning Resource (JMLR)_ , 2018.
  * Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 139–154, 2018.
  * Bagnell and Schneider (2003) J. A. Bagnell and J. C. Schneider.  Covariant Policy Search.  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2003.
  * Bengio et al. (2021) Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio.  Flow network based generative models for non-iterative diverse candidate generation.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Bhatnagar et al. (2009) Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, and Mark Lee.  Natural actor-critic algorithms.  _Automatica_ , 2009.
  * Bishop (2006) Christopher M. Bishop.  _Pattern Recognition and Machine Learning_.  Springer-Verlag New York, 2006.
  * Domingues et al. (2021) Omar Darwiche Domingues, Yannis Flet-Berliac, Edouard Leurent, Pierre Ménard, Xuedong Shang, and Michal Valko.  rlberry - A Reinforcement Learning Library for Research and Education, 10 2021.  URL https://github.com/rlberry-py/rlberry.
  * Fahlman and Lebiere (1989) Scott Fahlman and Christian Lebiere.  The cascade-correlation learning architecture.  _Advances in neural information processing systems_ , 2, 1989.
  * Geist et al. (2019) Matthieu Geist, Bruno Scherrer, and Olivier Pietquin.  A theory of regularized Markov decision processes.  In _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , 2019.
  * Girgin and Preux (2008) Sertan Girgin and Philippe Preux.  Basis function construction in reinforcement learning using cascade-correlation learning architecture.  In _2008 Seventh International Conference on Machine Learning and Applications_ , pages 75–82. IEEE, 2008.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning (ICML)_ , 2018.
  * Kakade and Langford (2002) Sham Kakade and John Langford.  Approximately optimal approximate reinforcement learning.  In _International Conference on Machine Learning (ICML)_ , 2002.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Lagoudakis and Parr (2003) Michail G Lagoudakis and Ronald Parr.  Least-squares policy iteration.  _The Journal of Machine Learning Research_ , 4:1107–1149, 2003.
  * Lazic et al. (2021) Nevena Lazic, Dong Yin, Yasin Abbasi-Yadkori, and Csaba Szepesvari.  Improved regret bound and experience replay in regularized policy iteration.  In _International Conference on Machine Learning_ , pages 6032–6042. PMLR, 2021.
  * Lee et al. (2019) Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee.  Overcoming catastrophic forgetting with unlabeled data in the wild.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 312–321, 2019.
  * Levine and Abbeel (2014) Sergey Levine and Pieter Abbeel.  Learning neural network policies with guided policy search under unknown dynamics.  In _Advances in Neural Information Processing Systems (NeurIPS)_. 2014.
  * Li and Hoiem (2017) Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12):2935–2947, 2017.
  * Liaw et al. (2018) Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica.  Tune: A research platform for distributed model selection and training.  _arXiv preprint arXiv:1807.05118_ , 2018.
  * Luo et al. (2020) Yong Luo, Liancheng Yin, Wenchao Bai, and Keming Mao.  An appraisal of incremental learning methods.  _Entropy_ , 22(11):1190, 2020.
  * Mellado et al. (2019) Diego Mellado, Carolina Saavedra, Steren Chabert, Romina Torres, and Rodrigo Salas.  Self-improving generative artificial neural network for pseudorehearsal incremental class learning.  _Algorithms_ , 12(10):206, 2019.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller.  Playing atari with deep reinforcement learning.  _CoRR_ , 2013.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Neu et al. (2017) Gergely Neu, Anders Jonsson, and Vicenç Gómez.  A unified view of entropy-regularized markov decision processes.  _CoRR_ , 2017.
  * Peters and Schaal (2008) J. Peters and S. Schaal.  Natural Actor-Critic.  _Neurocomputation_ , 2008.
  * Pirotta et al. (2013) Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello.  Safe policy iteration.  In _International Conference on Machine Learning (ICML)_ , 2013.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2001–2010, 2017.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * (31) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _CoRR_.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Michael Jordan, and Pieter Abbeel.  Trust Region Policy Optimization.  _International Conference on Machine Learning (ICML)_ , 2015.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, January 2016.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Szepesvari (2010) Csaba Szepesvari.  _Algorithms for Reinforcement Learning_.  Morgan & Claypool, 2010.
  * Tangkaratt et al. (2018) Voot Tangkaratt, Abbas Abdolmaleki, and Masashi Sugiyama.  Guide actor-critic for continuous control.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Tassa et al. (2014) Yuval Tassa, Nicolas Mansard, and Emo Todorov.  Control-limited differential dynamic programming.  In _International Conference on Robotics and Automation (ICRA)_ , 2014\.
  * Todorov and L. (2005) E. Todorov and Weiwei L.  A generalized Iterative LQG Method for Locally-Optimal Feedback Control of Constrained Nonlinear Stochastic Systems.  In _American Control Conference (ACC)_ , 2005.

##  A Number of added neurons

We ran an additional set of experiments to study the performance of MicaRL
with respect to its dependence on $n$, the number of neurons added per
iteration, in Figure 5. In general, we expect better performance as we
increase the number of neurons since it should be easier in this case to
approximate the Q function. This is confirmed by our simulations for
$n\in\\{10,20,50\\}$, where we see that more neurons seem to indicate faster
learning, as least initially, while in some cases, the mean squared Bellman
residual seem to increase with time for higher $n$, perhaps due to an
overfitting problem. Asymptotically however, the curves of the rewards achieve
approximately the same values.

Figure 5: Dependence on the number of added neurons per iteration $n$ on
CartPole-v1 (top-left corner), Acrobot-v1 (top-right corner), Pendulum-v1
(bottom-left corner) and MountainCar-v0 (bottom-right corner).


# Enabling Kernel Bypass Networking on gem5

Siddharth Agarwal, Minwoo Lee, Ren Wang, and Mohammad Alian  S. Agarwal is
with the Department of Electrical and Computer Engineering, University of
Illinois, Urbana-Champaign, IL, USA. E-mail<EMAIL_ADDRESS>R. Wang is
with Intel Labs, OR, USA. E-mail<EMAIL_ADDRESS>M. Lee and M. Alian are
with the Department of Electrical Engineering and Computer Science, University
of Kansas, KS, USA. E-mail<EMAIL_ADDRESS>
###### Abstract

Full-system simulation of computer systems is critical to capture the complex
interplay between various hardware and software components in future systems.
Modeling the network subsystem is indispensable to the fidelity of the full-
system simulation due to the increasing importance of scale-out systems. The
network software stack has undergone major changes over the last decade, and
kernel-bypass networking stacks and data-plane networks are rapidly replacing
the conventional kernel network stack. Nevertheless, the current state-of-the-
art architectural simulators still use kernel networking which precludes
realistic network application scenarios. In this work, we enable kernel bypass
networking stack on gem5, the state-of-the-art full-system architectural
simulator. We extend gem5’s NIC hardware model and device driver to enable the
support for userspace device drivers to run the DPDK framework. We also
implement a network load generator hardware model in gem5 to generate various
traffic patterns and perform per-packet timestamp and latency measurements
without introducing packet loss. Our experimental results show that DPDK’s
simulated network bandwidth scales with the number of cores and NIC ports. As
two use cases, we analyze the sensitivity of (1) network performance to
several microarchitectural parameters, and (2) direct cache access (DCA)
technology to DPDK burst size.

###### Index Terms:

Network, Kernel Bypass, DPDK, gem5

##  1 Introduction

The evolution of networking technology enabled hundreds of gigabytes per
second inter-server data transmission rates in datacenters, and terabit per
seconds network interfaces are on the horizon [7]. Proper handling of such
high network rates in the processor microarchitecture and memory hierarchy is
necessary to deliver high-quality end-to-end performance for emerging exascale
applications. Unfortunately, the current tools for modeling and evaluating
future system architectures have outdated models for the networking subsystem
and at most can deliver several tens of gigabits per second network data rates
to the processor and memory hierarchy. For instance, gem5’s baseline network
interface card (NIC) model only delivers $\sim$10Gbps network bandwidth with a
single NIC running iperf TCP test ($\sim$20Gbps with 4 NICs). FireSim, which
is an FPGA-based cycle-accurate simulator, supports networked simulation that
delivers $\sim$1Gbps bandwidth per NIC [5].

With such low network bandwidth, the architectural simulation cannot be used
for evaluating future terabit per second networked systems. In this work, we
filled this gap in the gem5 simulator by enabling the DPDK software stack on
gem5 that bypasses the Linux network software stack and delivers the maximum
network bandwidth that a given processor architecture and memory hierarchy can
sustain. In other words, we enable full-system gem5 to simulate networked
systems in which the network stack is no longer the bottleneck; instead, in
our setup, the processor and memory are the bottlenecks in network packet
delivery.

In this paper, we describe the shortcomings of gem5’s current network stack in
evaluating future networked systems and explain our extensions to the gem5
hardware model and Linux kernel to enable a DPDK software stack with a polling
mode driver (PMD) network interface. We also add a network traffic generator
hardware model to gem5 that can be used to inject packets to the simulated
network with configurable rate, packet size, and traffic pattern. The hardware
traffic generator is widely used in the industry to stress test the network
subsystem without experiencing any packet loss. Our experimental results show
that the bandwidth of our DPDK PMD interface scales with the number of
simulated network ports and processor cores. Each simulated network port
pinned to a simulated ARM core – loosely modeled after an ARM Cortex-A72 core
– in a quad-core setup can sustain network receive and transmit bandwidth of
$\sim$25Gbps when running L2fwd DPDK application without any packet loss. As
illustrated in Fig.1, a single NIC port running L2Fwd DPDK application
sustains $\sim$53Gbps while iperf only sustains $\sim$10Gbps. The complete
simulation setup and extensions are open sourced in the following git
repository https://github.com/agsiddharth/CAL-DPDK-GEM5. The authors are also
working on integrating the changes to the vanilla release of gem5.

Figure 1:  (a) Baseline dual-mode gem5 with Linux kernel software stack for
evaluating networked systems, (b) what is enabled in this work: kernel bypass
software stack with hardware load generator model in gem5.

##  2 Background and Motivation

Network packet processing in the Linux kernel suffers from the following
bottlenecks: frequent system calls for packet transmission and reception,
frequent buffer copies within kernel software stack and between kernel and
userspace buffers, and long-latency interrupt processing and notification.
Kernel bypass software stacks alleviate these bottlenecks by (1) reducing
context switches between userspace and kernel space; (2) using large buffer
allocations, huge pages, and zero-copy interfaces to reduce buffer management
and data movement overheads; (3) using polling for RX and TX completion
notification. Many kernel bypass protocols have been proposed and implemented,
including Data Plane Development Kit (DPDK) [4].

DPDK provides a userspace API for application developers. It reserves pinned
hugepages and allows the NIC to DMA directly into the application’s buffers.
Since it is polling based, it eliminates context switching overheads as well.
DPDK application can be implemented in two modes:

Run to completion mode where the packet processing loop is: (1) retrieve RX
packets through polling mode driver (PMD) RX API, (2) process packets on the
same logical core, (3) send pending packets through PMD TX API

Pipeline mode, which lets cores pass packets between each other via a ring
buffer to process packets.

Current gem5 Network Stack.

The current NIC simulation object in gem5 loosely models Intel® 8254x NIC
series at a minimal functional level. Fig. 1(a) shows a two-node full-system
simulated system connected through the simulated NIC and Ethernet Link.
Default gem5 uses Kernel interrupt-driven network stack and can only sustain
up to $\sim$20Gbps network bandwidth using four powerful, multi-core, O3 ARM
cores running at 3GHz frequency (See Sec. 4 for detailed experimental
methodology). Such low network bandwidth does not sufficiently exercise the
hardware and software stack of future networked systems modeled with gem5.
Thus, the current gem5 model is not useful for evaluating networked systems
supporting hundreds of gigabits per second network throughput.

Hardware Traffic Generators. One of the main concerns when evaluating
networked systems is to load the system-under-test with real traffic and
measure network bandwidth and per-packet, round-trip network latency without
introducing extra latency or packet drop a the load generator node. The
practice in the industry is to use hardware traffic generators that utilize
FPGA line cards to generate packets with configurable traffic patterns, sizes,
and protocols and provide detailed network statistics per transmitted and
received packets [6].

##  3 Linux Kernel Bypass in gem5

This section discusses the changes we made to gem5 and DPDK framework to
enable Kernel bypass networking and implement the hardware load generator
model in gem5. We do not make any changes in the Linux kernel.

Figure 2:  First 8 bytes of the PCI configuration space that includes PCI
Command Register.

###  3.1 Changes to gem5

The changes in gem5 are limited to PCI model to enable Userspace I/O (UIO)
driver and the NIC model to enable byte granular PCI configuration space
accesses.

####  3.1.1 Enable Userspace I/O Driver

uio_pci_generic driver in Linux enables a userspace application to directly
access the address space of a PCI device. DPDK uses this driver to gain the
userspace application access to the PCI config space and implement a polling
mode driver. The default gem5 does not enable the uio_pci_generic driver
during boot as the PCI Command Register is not fully implemented in gem5.
Fig.2 shows the first 8 bytes of the PCI configuration space that includes the
16-bit Command Register at offset 0x04. The baseline gem5 implements bits 0-9
of the Command Register but does not implement bit-10, which is the interrupt
disable bit. We implement interrupt disable bit in gem5 PCI model so the Linux
kernel can disable the interrupts for the PCI devices on gem5 which is
necessary to support uio_pci_generic driver.

####  3.1.2 Enable Byte-Granular Access to PCI Configuration Space

The default gem5 only supports 16-bit accesses to the Command Register shown
in Fig.2. In fact, this is rational since the size of the Command register is
16-bits. However, we observed that DPDK accesses the Command register using
8-bit memory accesses. Such byte-granular accesses are being ignored in gem5,
and therefore DPDK cannot properly read and write the upper half of the
Command Register (offset 0x05 of the PCI config space). We extended the
readConfig and writeConfig functions in the gem5 PCI model to enable byte-
granular accesses to the Command Register.

####  3.1.3 Implement Interrupt Mask Register in the NIC model

The last modification in gem5 is to implement Interrupt Mask Register in the
i8254xGBe device model. Interestingly, this register is included in the
i8254xGBe model, but the read and write methods for accessing the register are
not implemented in the current gem5 release. We implemented the read and write
methods to enable DPDK to launch its polling mode driver.

####  3.1.4 Enable the NIC Model to Correctly Operate with a PMD

NIC devices keep a handful of available descriptors (usually 32$\sim$64
descriptors) that can be populated upon receiving a packet on an on-chip cache
which is called descriptor cache. Descriptor cache improves the performance as
the NIC does not need to fetch available descriptors from the CPU memory on
demand. The NIC gradually writes back the descriptor cache to the CPU memory
(using DMA), and then the CPU is notified of received packets.

The current gem5’s NIC model writes back the received descriptors based on a
threshold set by the Linux kernel. Once the number of used descriptors is
higher than a threshold, NIC initiates a writeback. When using a PMD, the
threshold registers in the NIC model are not properly set, and thus the NIC
starts writing back the descriptors when all of them are used. This means that
packets are DMAed to the CPU memory in large batches (32$\sim$64 packets),
which causes unrealistic pressure on the CPU memory subsystem and increases
the possibility of packet drops at high receive rates. We implemented a
parameter for the NIC where the user can control the threshold of descriptor
writebacks in gem5.

###  3.2 Changes to DPDK

The DPDK Environment Abstraction Layer (EAL) relies on vendor ID checks to
match a device and a PMD. We modify the DPDK source to skip these checks and
force the matching of the gem5 device to the e1000 PMD. Unmodified DPDK cannot
fetch the correct vendor ID when running on gem5 and therefore fails to call
the proper PMD driver. We suspect this is because some manufacturer-specific
information is missing in the gem5 NIC model. Note that skipping the vendor ID
test does not have any impact on gem5 simulations as the current gem5 release
has only the e1000 NIC model. If new NIC models are added to gem5, the DPDK
framework should be recompiled after hard-coding the PMD driver to use a
different NIC model.

###  3.3 Hardware Load Generator Model.

The hardware load generator model can generate packets at arbitrary rates and
sizes. We implement a simulation object called EtherLoadGen that has a single
Ethernet port and can directly connect to the NIC port of a simulated node as
shown in Fig.1(b). Therefore, for simple network benchmarking, one does not
need to run distributed or dual mode gem5 simulations and a single system
simulation is enough. This significantly improves the simulation speed. The
parameters of EtherLoadGen are packet rate, packet size, and protocol. They
can be statically set while launching a simulation or a packet trace can be
passed to the simulator to be replayed by EtherLoadGen. In the static mode,
EtherLoadGen create Ethernet packets with the specified size and send them at
a fixed rate to the Ethernet port. The static protocol that we support for now
is plain Ethernet packets. If a trace file is provided, EtherLoadGen will read
from the trace file and generate traffic based on the timestamps, sizes, and
protocol in the trace file.

EtherLoadGen adds a timestamp to each outgoing packet at a configurable offset
and compares the timestamp with the current tick on incoming packets to
compute per-packet round-trip latency. EtherLoadGen reports mean, median,
standard deviation, and tail latency of network packets in the statistics
file. It also produces a packet drop percentage and a histogram of packet
forwarding latency. The load generator model enables simple network
benchmarking in gem5 without the need to simulate multiple system nodes. This
is similar to the practice in the industry for using hardware load generators
to evaluate the performance of the network [1].

EtherLoadGen also supports a bandwidth test mode where it gradually increases
the bandwidth to find the maximum sustainable bandwidth of a server, which is
the maximum bandwidth that a server can sustain without packet drops.

##  4 Evaluation

TABLE I: Simulation configuration.

Parameters | Baseline Values
---|---
Core freq: | 2GHz
Superscalar | 3 ways
ROB/IQ/LQ/SQ entries | 384/128/128/128
Int & FP physical registers | 128 & 192
Branch predictor/BTB entries | BiMode/2048
Caches (size, assoc): I/D/L2 | 32KB,2/64KB,2/2MB,16ways
L1I/L1D/L2 latency,MSHRs | 1/2/12 cycles, 2/6/16 MSHRs
DRAM/mem size | DDR4-3200-8x8/2GB
iocache | 24 cycles, 16 MSHRs
Network latency/BW | 1$\mu$/200Gbps
DPDK | Version 20.11.3
Operating system | Linux Linaro (kernel 5.4.0)
gem5 version | v21.1.0.2

###  4.1 Methodology

Table I shows the baseline gem5 configuration we used for the experiments. We
use iperf and L2Fwd for Kernel and DPDK experiments. In this section, we first
explain how to build DPDK on gem5 full-system disk-image and write scripts to
run a simple L2Fwd DPDK application and load it with the simulated hardware
packet generator. Then we show the results that validate the correctness of
our Kernel bypass setup. Next, we use iperf and L2Fwd to compare the
scalability of Kernel stack and DPDK on gem5. Lastly, we perform a sensitivity
analysis on microarchitecture configurations for network bandwidth when using
Kernel stack and DPDK.

To build a kernel and disk image for gem5, we use the buildroot tool. The
Kernel needs to be compiled with support for huge pages, and the kernel module
uio_pci_generic. Listing 1 shows the Kernel config option needed to be enabled
in buildroot tool for DPDK.

⬇

1CONFIG_HUGETLBFS=y

2CONFIG_HUGETLB_PAGE=y

3CONFIG_UIO=y

4CONFIG_PCI=y

5CONFIG_UIO_PCI_GENERIC=m

Listing 1: Kern CONFIG options needed for DPDK.

⬇

1 modprobe uio_pci_generic

2 dpdk-devbind.py -b uio_pci_generic 00:02.0

3 echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

4 dpdk-testpmd

Listing 2: Bash script for setting up the environment for running DPDK
applications on gem5.

Listing 2 shows the bash script for bringing up the Userspace IO (UIO) driver
(line 1), binding it to a NIC port (line 2), allocating huge pages, and
lastly, starting the testpmd application. As shown in the listing, the
procedure for running DPDK applications on gem5 is identical to running DPDK
apps on bare metal hardware.

###  4.2 Experimental Results

In this section, we show experimental results that verify the functionality of
DPDK enabled hardware/software stack on gem5 and illustrate the scalability of
our kernel bypass network software stack on gem5. Since validating gem5’s
performance is out of the scope of this paper, we do not compare the network
bandwidth of a bare metal system with our gem5 setup and only report the
performance of gem5. There are on-going efforts to validate gem5 performance
for different ISAs [2].

To verify DPDK’s functionality on the simulated system, we modify L2Fwd sample
application to print the content of the packets received from the network. We
always receive the correct content regardless of the packet size and network
configuration. This experimentally verifies the correct functionality of the
kernel bypass stack on gem5.

Fig. 3(a) shows the maximum sustainable bandwidth of a single gem5 node when
equipped with up to 4 NICs, running iperf and L2Fwd. We define the maximum
sustainable bandwidth as the maximum bandwidth without packet drop. As
explained in Sec.3.3, EtherLoadGen supports a mode where it gradually
increases the packet rate and monitors the responses received from the server
to find the maximum sustainable bandwidth. Fig.3(a) compares the bandwidth
achieved using Linux kernel stack (iperf configuration) and DPDK (L2Fwd
configuration). Fig.3(a) has two highlights: (1) DPDK delivers much higher
bandwidth compared with kernel stack. As shown in the figure, L2Fwd sustains
5.4$\times$ and 4.9$\times$ more bandwidth compared with iperf using 1 and 4
NICs, respectively. (2) adding more NICs scales DPDK’s network bandwidth
better than the Linux kernel. As shown in the figure, moving from 3 NICs to 4
NICs, the DPDK software stack has 24.1% higher bandwidth, while the Linux
kernel stack only sustains 5.3% higher network bandwidth.

Figure 3: Linux kernel and DPDK userspace software stack (a) scalability and
(b) sensitivity to architectural parameters.

##  5 Example Use Cases

###  5.1 Micro Architectural Sensitivity Analysis

In this subsection, we analyze the sensitivity of the network bandwidth to
various microarchitectural configurations. We perform the analysis for both
Linux kernel and kernel bypass network stacks and compare the results. We
start with a baseline node configuration shown in Table I (2GHz CPU
configuration in Fig.3(b)). Then increase the CPU frequency (3GHz CPU), reduce
the PICe bus latency (low latency PCIe), double the number of memory channels
(2x Mem Ch), double the size of ROB and LSQ (2xROB/LSQ), double the number of
load-store functional units in the processor pipeline (2xLSUs), double the
size of L1 data and instruction caches (2xL1D/I), double the size of L2 and
LLC caches (2xL2/LLC), and lastly enable direct cache access [3] to place
received network data on the LLC instead of DRAM (DCA). Fig.3(b) shows the
maximum sustainable bandwidth for all the mentioned configurations. Note that
the enhancements are accumulative, meaning that we apply each optimization on
top of the previous one, i.e., the DCA configuration runs at 3GHz, has low
latency PCIe, twice the number of memory channels, twice ROB, LSQ, L1D, L1D,
L2, LLC sizes of the configuration listed in Table I.

As shown in Fig.3(b), different microarchitectural parameters impact DPDK and
Linux kernel stacks differently. For example, increasing CPU frequency alone
improves DPDK bandwidth by 1.2% but improves Linux kernel stack bandwidth by
32.5%. This is due to the CPU intensity of the kernel stack compared to the
userspace DPDK stack. Fig.3(b) clearly shows that conducting architectural
research using an old software stack can lead to incorrect assumptions and
optimizations.

###  5.2 Sensitivity of Direct Cache Access Performance to Burst Size

As another usecase for the userspace networking and network load generator,
Fig.4 plots the impact of L2Fwd burst size on the writeback rate of L2 and L3
caches when receiving 1024 packets in a short time interval. Fig.4(a) L2Fwd
aggregate forwarding in burst of 32 packets while in Fig.4(b), L2Fwd waits
until 1024 packets are received and then start the forwarding. The simulated
node implements a non-inclusive L2 with DCA enabled. As shown, a large batch
size results in LLC contention at the beginning of the burst arrival as many
packets will be DMA transferred to the LLC (in the ring buffer) within a short
interval. A shorter batch size overlaps the processing of the received packets
with the DMA from NIC to LLC and since L2 is non-inclusive, demand misses from
L2 make space in the LLC for incoming packets; therefore, Fig.4(a) has lower
LLC writeback rate.

Figure 4: Sensitivity of L2 and LLC writebacks to the packet processing batch
size.

##  6 Conclusion

In this paper, we introduced gem5’s userspace networking stack. We explained
the changes we made to enable gem5 to run DPDK applications. We show that the
bandwidth of L2Fwd DPDK application running on gem5 follows the same trend
when running on bare metal hardware. We showed that the bandwidth of DPDK
applications running on gem5 scales significantly better than applications
that use the Linux kernel stack. Using gem5 running iperf and L2Fwd, we
performed a sensitivity analysis on microarchitecture optimizations and showed
that kernel bypass network applications are sensitive to different
microarchitecture optimizations compared with Linux kernel network
applications. We released the source code, scripts, and instructions to create
disk images, install DPDK, and run DPDK applications in the following git
repository https://github.com/agsiddharth/CAL-DPDK-GEM5..

## Acknowledgement

This research was in part supported by an NSF grant (CNS-2213807).

## References

  * [1] Spirent AION.  Driving innovation and accelerating next-gen technology deployments with award-winning L2-7 Network and Cloud testing solutions.  https://www.spirent.com/products/aion.
  * [2] Ayaz Akram and Lina Sawalha.  Validation of the gem5 simulator for x86 architectures.  In 2019 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pages 53–58, 2019.
  * [3] Mohammad Alian, Yifan Yuan, Jie Zhang, Ren Wang, Myoungsoo Jung, and Nam Sung Kim.  Data direct I/O characterization for future I/O system exploration.  In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 160–169. IEEE, 2020.
  * [4] Intel®.  DPDK Intel NIC Performance Report Release 17.08.  http://fast.dpdk.org/doc/perf/DPDK_17_08_Intel_NIC_performance_report.pdf.
  * [5] Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanovic.  FireSim: FPGA-accelerated cycle-exact scale-out system simulation in the public cloud.  In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pages 29–42, 2018.
  * [6] Keysight Technologies.  High-Volume Traffic Generator Products Catalog.  https://www.keysight.com/us/en/assets/7121-1065/catalogs/High-Volume-Traffic-Generator-Products-Catalog.pdf.
  * [7] Mark Wade, Erik Anderson, Shahab Ardalan, Pavan Bhargava, Sidney Buchbinder, Michael L Davenport, John Fini, Haiwei Lu, Chen Li, Roy Meade, et al.  TeraPHY: a chiplet technology for low-power, high-bandwidth in-package optical I/O.  IEEE Micro, 40(2):63–71, 2020.