ifaamas [AAMAS ’23]Proc. of the 22nd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2023)May 29 – June 2, 2023 London, United
KingdomA. Ricci, W. Yeoh, N. Agmon, B. An (eds.) 2023 2023 432 Eindhoven
University of Technology Eindhoven University of Technology University of
Alberta University of Twente University of Alberta & Alberta Machine
Intelligence Institute (Amii) Eindhoven University of Technology University of
Luxembourg & University of Twente

# Automatic Noise Filtering with Dynamic Sparse Training
in Deep Reinforcement Learning

Bram Grooten<EMAIL_ADDRESS>,  Ghada Sokar<EMAIL_ADDRESS>,
Shibhansh Dohare<EMAIL_ADDRESS>,  Elena Mocanu<EMAIL_ADDRESS>,
Matthew E. Taylor<EMAIL_ADDRESS>,  Mykola Pechenizkiy
<EMAIL_ADDRESS>and  Decebal Constantin Mocanu<EMAIL_ADDRESS>
###### Abstract.

Tomorrow’s robots will need to distinguish useful information from noise when
performing different tasks. A household robot for instance may continuously
receive a plethora of information about the home, but needs to focus on just a
small subset to successfully execute its current chore. Filtering distracting
inputs that contain irrelevant data has received little attention in the
reinforcement learning literature. To start resolving this, we formulate a
problem setting in reinforcement learning called the extremely noisy
environment (ENE), where up to 99% of the input features are pure noise.
Agents need to detect which features provide task-relevant information about
the state of the environment. Consequently, we propose a new method termed
Automatic Noise Filtering (ANF), which uses the principles of dynamic sparse
training in synergy with various deep reinforcement learning algorithms. The
sparse input layer learns to focus its connectivity on task-relevant features,
such that ANF-SAC and ANF-TD3 outperform standard SAC and TD3 by a large
margin, while using up to 95% fewer weights. Furthermore, we devise a transfer
learning setting for ENEs, by permuting all features of the environment after
1M timesteps to simulate the fact that other information sources can become
relevant as the world evolves. Again, ANF surpasses the baselines in final
performance and sample complexity. Our code is available online.111See
https://github.com/bramgrooten/automatic-noise-filtering

###### Key words and phrases:

deep reinforcement learning; noise filtering; sparse training

##  1\. Introduction

Future robots will likely perceive a plethora of information about the state
of the world, but only parts of it are going to be relevant to their current
task. For instance, a household robot receiving abundant information about all
objects and processes in the house.222For example: cleanliness of floors,
furniture, cupboards, kitchen utensils; CO2, CO levels and temperature in each
room; up-to-date stock of all food and non-food items in the fridge and/or
basement; mood, nourishment, and health of all inhabitants; etc. For its
current task, e.g. making pancakes, only a small subset of these information
sources, or features, are relevant. Agents should automatically detect which
features are task-relevant, without humans having to predefine this. Other
examples may be: a hearing aid distinguishing between voices and auditory
noise, a surgical robot receiving all possible information about the patient,
or a self-driving car that needs to ignore distracting billboards.

Figure 1. Performance of SAC on Humanoid-v3 environments expanded with a
different number of pure noise features. Once the environment contains too
much noise, SAC struggles to learn a decent policy. Standard dense networks
cannot filter through the noise well enough on this problem.

To illustrate the current situation: Soft Actor-Critic (SAC) Haarnoja et al.
(2018) fails to learn a decent policy on an environment with 90% added noise
features, see Figure 1. We simulate the noisy real-world environment by adding
synthetic noise features to an existing state space. This allows us to study
the problem in a controlled environment to understand where we stand and what
can be done. We need to invent methods that can effectively filter through the
noise while learning to perform the environment’s task. Our research question
becomes: How can we design RL agents to learn and perform well in an extremely
noisy environment?

Dynamic Sparse Training (DST), a class of methods stemming from the Sparse
Evolutionary Training (SET) algorithm Mocanu et al. (2018), is promising in
this regard. By starting from a randomly sparsified network and subsequently
pruning and growing connections (weights) during training, DST searches for
the optimal network topology. DST is able to perform efficient feature
selection for unsupervised learning, as shown by (Atashgahi et al., 2022;
Sokar et al., 2022a). Further, Vischer et al. (2022) discovered that sparse
networks can find minimal task representations in deep RL by pruning redundant
input dimensions. Not long after, Sokar et al. (2022b) successfully applied
DST in deep RL, reducing the number of parameters without compromising
performance.

This leads us to a plausible approach to our research question. We think that
the adaptability of DST can improve an agent’s sparse network structure such
that task-relevant features are emphasized by receiving more connections than
noise features. The combination of sparsity and adaptability enables the agent
to filter through the noise more effectively, outperforming dense network
approaches. The underlying hypothesis follows:

The Adaptability Hypothesis: A sparse neural network layer can adapt the
location of its connections (weights) to gain a better performance faster than
a dense layer can adapt the weight values to achieve the same gain.

Note that newly grown connections still need to adjust their weight values
through gradient descent, but we hypothesize that this generally happens
quicker than a dense network modifies all of its weights. Relocated weights
may receive a more informative gradient when connected to task-relevant
features. Briefly, the hypothesis states: dropping and growing connections is
easier than adjusting the weights. This is inspired by our own brain’s
plasticity, which also dynamically drops and grows synapses Bach-y Rita et al.
(1969); Castaldi et al. (2020); Pereda (2014).

To verify our hypothesis, we propose a new algorithm called Automatic Noise
Filtering (ANF), which can easily be combined with deep RL methods. It has a
sparse input layer with adapting connectivity through dynamic sparse training.
We compare ANF to two strong baseline deep RL algorithms: SAC Haarnoja et al.
(2018) and TD3 Fujimoto et al. (2018), which have fully dense layers
throughout their networks. We devise the extremely noisy environment (ENE),
further defined in Section 2, which expands the state space of an existing RL
environment with a large number of noise features. We apply this approach to
four continuous control tasks from MuJoCo Gym Todorov et al. (2012); Brockman
et al. (2016).

#### Contributions.

  * •

We formulate a problem setting termed the extremely noisy environment (ENE),
where up to 99% of the input features consist of pure noise. Agents need to
detect the task-relevant features autonomously.

  * •

We propose Automatic Noise Filtering (ANF), a dynamic sparse training method
that outperforms baseline deep RL algorithms by a large margin, especially on
environments with high noise levels.

  * •

We devise a transfer learning setting of extremely noisy environments and show
that ANF has better performance and forward transfer than the baselines SAC
and TD3.

  * •

We show that highly sparse ANF agents with up to 95% fewer parameters can
still surpass their dense baselines on the extremely noisy environments.

  * •

We extend the ENE by adjusting the noise distribution in two ways, increasing
the difficulty. ANF maintains its advantage on these challenging
extensions.333See an illustrative video here: https://youtu.be/vS47UnsTQk8

#### Outline.

In Section 2 we formulate the problem setting. Section 3 gives an overview of
the background and related work. Our method is introduced in Section 4, along
with the first experiments. In Section 5 we explore the transfer learning
setting. Sections 6 through 9 provide further analysis, where we perform an
ablation study and discover how far we can extend our problem and algorithm.
Finally, Section 10 concludes the paper. Additional results, details, and
discussion are in the Appendix.

##  2\. Problem Formulation

We introduce a problem setting where agents have to act in environments that
contain a lot of noise. As the noise features generally greatly outnumber the
task-relevant features in this setting, we simply call it the extremely noisy
environment (ENE).

Extremely noisy environment. To create an ENE, we take any reinforcement
learning environment that generates feature vectors as states. The ENE expands
this feature vector by concatenating many additional features consisting only
of pure noise, sampled from any given distribution. An agent is not told which
features are useful (task-relevant) and which are useless (noise), so it has
to learn to ignore the distracting noise features by itself, see Figure 2.

Figure 2. Extremely noisy environments (ENEs) contain many noise
$\mathbin{\vbox{\hbox{\scalebox{1.8}{$\bullet$}}}}$ features and some relevant
$\mathbin{\vbox{\hbox{\scalebox{1.8}{$\bullet$}}}}$ features. We use ENEs
where up to $99\%$ of the features are noise. Our method Automatic Noise
Filtering (ANF) learns to predominantly connect with the input neurons that
provide useful information and outperforms dense baselines by a large margin,
especially in the noisiest environments. Figure 3. In permutated extremely
noisy environments (PENE) the order of the input features gets shuffled after
a certain number of timesteps. Our ANF agents automatically adjust their
network structure to this new environment. They show superior forward transfer
compared to fully dense methods, even though ANF agents might need to prune
and regrow many connections in the sparse input layer. We hypothesize that
adapting the location of sparse connections is easier than adjusting the
weights of all connections in a dense network.

In our main experiments, the noise features produce pure Gaussian noise,
sampled i.i.d. from $\mathcal{N}(0,1)$. The fraction of noise features in an
ENE is denoted by $n_{f}\in[0,1)$. For example, for $n_{f}=0.5$ we enlarge the
original state space of a MuJoCo Gym environment by lengthening the state
feature vector by a factor of 2. In general, the dimensionality of the new
state space is

$d_{ene}=\left\lceil\dfrac{d_{og}}{1-n_{f}}\right\rceil$

where $d_{og}$ is the number of dimensions in the original state space. As
$n_{f}$ increases to $1$, the dimensionality of the ENE expands.

Transfer learning setting. Next to the ENE, we introduce an even more
challenging problem setting where, after every $T_{p}$ timesteps, all input
features are permuted at random. This permutation simulates the fact that
other features can become relevant over time. Previously irrelevant features
might suddenly become relevant, for example, when a household robot gets a new
task.444Features could also gradually become more relevant, as the world
evolves (i.e. concept drift). This is outside the scope of our research, we
focus on the sudden change. In our case, the change in environment is not
announced to the agent. Agents need to detect the change and transfer their
representations quickly to adapt to the new instantiation of the permutated
extremely noisy environment (PENE), see Figure 3. A previously task-relevant
feature may or may not still be relevant after the permutation, inducing the
need to rediscover the distribution of the features and filter through the
noise.

##  3\. Background and Related Work

Our proposed ANF algorithm is based on dynamic sparse training (DST). In this
section, we briefly overview the related work of DST in reinforcement learning
(RL) and existing noise filtering methods.

Sparse training. Dynamic sparse training is a subfield of the sparse training
regime Mocanu et al. (2021), where weights deemed superfluous are pruned away
to increase the efficiency of a neural network. In dense-to-sparse training,
dense networks are gradually pruned to higher sparsity levels throughout
training Han et al. (2015); Frankle and Carbin (2019); Liu et al. (2020b). In
sparse-to-sparse training, where DST belongs, a network begins with a high
sparsity level from scratch Mocanu et al. (2018); Bellec et al. (2018). The
existing connections can either stay fixed (static sparse training) or be
pruned and regrown during training (dynamic sparse training).

In supervised learning, especially computer vision, many promising results
have been achieved with sparsity over the last few years Liu et al. (2021);
Evci et al. (2020); Chen et al. (2021). These algorithms benefit from
potential performance boosts, decreased computational costs, and better
generalization (Liu et al., 2019; Chen et al., 2022). Furthermore, DST has
been used successfully for an efficient feature selection algorithm Atashgahi
et al. (2022), which inspired our project.

DST in RL. Applying sparse training in reinforcement learning is useful, as
real-world applications often deal with latency constraints Degrave et al.
(2022), which limits the number of parameters. Unfortunately, in the area of
RL it seems that applying sparse training is more challenging than in
supervised learning, as the achievable sparsity levels without loss in
performance are generally lower Graesser et al. (2022). Only a few papers have
applied sparse training to deep RL so far. In the offline RL setting, Arnob et
al. (2021) have reached 95% sparsity with almost no performance degradation.
While this is impressive, we believe that offline RL is more similar to
supervised learning than online RL. Moreover, it does not support learning in
changing environments (Silver et al., 2021). Therefore, we focus on the online
RL setting throughout the paper, and even go into the transfer learning
setting Taylor and Stone (2009).

To the best of our knowledge, the first work applying DST to online RL is from
Sokar et al. (2022b). They outperform dense networks with the algorithms DS-
TD3 and DS-SAC, which combine sparse evolutionary training (SET) (Mocanu et
al., 2018) with TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018).
The methods of Sokar et al. (2022b) form the foundation of our ANF algorithm.

Sokar et al. Sokar et al. (2022b) reached a global sparsity level of 50%,
which was later improved upon by Graesser et al. (2022); Tan et al. (2023),
who experimented with sparsity levels up to 99%. They showed that the sparsity
level reachable without loss of performance largely depends on the
environment. Graesser et al. Graesser et al. (2022) compared DST methods such
as SET Mocanu et al. (2018) and RigL Evci et al. (2020) in many deep RL
environments. Their performance proved to be quite similar, so we choose to
use only SET.

Noise in RL. There exist different types of noise that an agent may encounter.
Let us characterize the two main categories:

  * •

Type 1: uncertainty in perception, for example when an automated vehicle
cannot clearly see a traffic sign since the sun is right next to it.

  * •

Type 2: distracting, task-irrelevant percepts, for example the bright colors
of a billboard when crossing Times Square in New York City.

Type 1 noise, i.e. measurement errors, is often researched by adding noise on
top of existing features to produce more robust agents (Bemd, 2022; Vinitsky
et al., 2020; Sun et al., 2021; Moos et al., 2022). This type of noise is
outside the scope of this work. Instead, we focus on type 2 noise and
investigate it by adding synthetic features alongside the existing features,
creating a state space of higher dimensionality. The goal is to discover
algorithms that can perform tasks well while having access to all available
features, without having to pre-select the task-relevant ones. Feature
selection should be carried out automatically by the RL agents.

Figure 4. Performance of Automatic Noise Filtering (ANF) compared to its
baselines for different noise fractions $n_{f}$. The curves show average
return in the last 10% of training over 5 seeds, with shaded regions
representing 95% confidence intervals. When the number of noise features in an
environment increases, the performance of the standard fully-dense networks of
SAC deteriorates much faster than ANF-SAC. Similar graphs for ANF-TD3 are
shown in Figure 15 of Appendix D.1.

To the best of our knowledge, the first work to make an existing RL
environment noisier by adding extra features was FS-NEAT Whiteson et al.
(2005). It introduced an evolutionary algorithm to select relevant features.
Most of the follow-up work takes this evolutionary approach Kroon and Whiteson
(2009); Bishop and Miikkulainen (2013); Acunto (2012), while we use the
efficiency of deep learning, stochastic gradient descent, and dynamic sparse
training.

Our work extends environments that provide the current state as a feature
vector. However, it is worth mentioning that environments with visual (pixel)
inputs have likewise been augmented to include a noisy challenge, such as
distracting backgrounds Stone et al. (2021). Other methods that have some
similarities to our approach include recognizing distractor signals Rafiee et
al. (2020), reducing state dimensionality Curran et al. (2016); Botteghi et
al. (2021), and identifying fake features in federated learning Li et al.
(2021).

##  4\. Automatic Noise Filtering

In this section, we explain how our ANF algorithm works, after which we show
and interpret the results of our main experiments. ANF is a simple method that
can be applied to any MLP-based deep RL algorithm. It is built upon the DS-TD3
and DS-SAC algorithms of Sokar et al. (2022b), which use sparse evolutionary
training (SET) from Mocanu et al. (2018) as the underlying dynamic sparse
training method.

In both the actor and critic networks, ANF begins by randomly pruning the
input layer to the desired sparsity level $s_{i}$. During training, we drop
weak connections of the input layer (weights with the smallest magnitude)
after every topology-change period $\Delta T$. After dropping a certain
fraction $d_{f}$ of the existing weights, ANF randomly grows the same number
of connections to maintain the sparsity level $s_{i}$. By giving new
connections enough time to increase their weights, ANF detects task-relevant
features without explicit supervision. We provide pseudocode for ANF-SAC in
Appendix A.

One aspect that sets ANF apart from the previous works on non-noisy settings
Sokar et al. (2022b); Graesser et al. (2022) is that we only sparsify the
input layer. This helps us to pinpoint the support of DST on our Adaptability
Hypothesis. Furthermore, in extremely noisy environments it is essential to
filter through the large fraction of noise. Dynamic sparse training can
perform this filtering elegantly. It works well to focus the DST principle on
the first layer only, as this is where the distinction between relevant and
noise features is made. In Section 9 we investigate models that also have
sparse hidden layers.

Another difference between ANF and DS-TD3/SAC Sokar et al. (2022b) is that we
mask the running averages of first and second raw moments of the gradient
within the Adam optimizer Kingma and Ba (2015) for non-existing connections.
When connections are dropped and later regrown, they do not have access to
previous information if implemented in a truly sparse manner. This aspect has
been overlooked in the implementation of some sparsity research papers that
apply Adam and only simulate true sparsity with binary masks on top of the
weight matrices. Our research also utilizes such binary masks while keeping
the truly sparse implementation in mind. See Appendix C for further
discussion.

Experimental setup. We integrate our ANF method in two popular deep RL
algorithms: SAC and TD3. This means we compare the algorithms ANF-SAC and ANF-
TD3 with their fully-dense counterparts as baselines. Furthermore, we compare
to the closely related DS-SAC and DS-TD3, which both use their default global
sparsity level of $50\%$. All neural networks have two hidden layers of 256
neurons with the ReLU activation function. After a hyperparameter search for
ANF, we set the input layer sparsity $s_{i}$ to $80\%$, the topology-change
period $\Delta T=1000$ timesteps, and the drop fraction $d_{f}=0.05$. Further
hyperparameter settings replicate prior work Sokar et al. (2022b); Haarnoja et
al. (2018); Fujimoto et al. (2018). See Appendix B for additional details.

Our experiments are carried out in four continuous control environments from
the MuJoCo Gym suite: Humanoid-v3, HalfCheetah-v3, Walker2d-v3, and Hopper-v3.
We first run an experiment without any added noise features as a baseline and
then start increasing the noise level. The fraction of noise features,
$n_{f}$, ranges over the set $\\{0.8,0.9,0.95,0.98,0.99\\}$. Note that the
state spaces of these settings increase by
$5\times,10\times,20\times,50\times$, and $100\times$, respectively.

We train our agents for 1 million timesteps and evaluate them by running 10
test episodes after every 5000 timesteps. We measure the average return over
the last 10% of training, as done in Graesser et al. (2022), for overview
graphs such as Figure 4. Throughout the paper, we run 5 random seeds for every
setting. In the graphs, we show the average curve as well as a 95% confidence
interval.

Figure 5. Learning curves on 3 environments with 98% noise features. This
level of noise is too high for standard SAC and TD3 to learn a decent policy.
ANF can still filter through the distraction and find a well-performing
policy. Table 1. State and action space dimensions.

Environment | State dim. | Action dim. | State dim. | State dim.
---|---|---|---|---
| Original | Original | ENE ($n_{f}\\!=\\!.8$) | ENE ($n_{f}\\!=\\!.99$)
Humanoid-v3 | 376 | 17 | 1880 | 37600
HalfCheetah-v3 | 17 | 6 | 85 | 1700
Walker2d-v3 | 17 | 6 | 85 | 1700
Hopper-v3 | 11 | 3 | 55 | 1100

Results. First of all, the horizontal lines in Figure 4 show that even in
environments without noise ANF-SAC is able to reach similar or better
performance than SAC and DS-SAC for Humanoid-v3 and HalfCheetah-v3. By
adjusting the connectivity of the input layer, ANF is able to select the set
of most important features, the so-called minimal task representation Vischer
et al. (2022).

Furthermore, when the noise level increases our ANF method outperforms the
dense baseline by a significant margin on all environments. Especially in the
noisiest environments, when $n_{f}\geq.95$, a large gap is visible between
ANF-SAC and SAC for HalfCheetah, Walker2d, and Hopper. The Humanoid
environment is an exception, as ANF outperforms its baseline much earlier here
but then struggles with the high noise levels as well. Table 1 shows that
Humanoid-v3 differs noticeably from the other three environments by the size
of its state space.

The learning curves in Figure 6 indicate that SAC and TD3 are unable to learn
a decent policy within 1M timesteps in this challenging extremely noisy
environment. ANF learns to ignore the distracting noise and reaches a
performance level similar even to SAC and TD3 in the environment without
noise.555Which is a return of $\sim$4500, see SAC’s dashed line in Figure 4,
Humanoid-v3.

Figure 6. Learning curves for Humanoid-v3 with 90% noise features. While
standard SAC and TD3 are too distracted by the noise to learn the task, ANF
finds the task-relevant features and is able to improve.

In the environments HalfCheetah, Walker2d, and Hopper, we continue to observe
this behavior up to a noise fraction of 98%, as shown in Figure 5. ANF
outperforms its baselines by a large margin in each environment.

Topology shift. To analyze what is actually happening, we visualize the
connectivity of ANF. In Figure 7, we present a graph that shows the
development of the network’s topology over time. The graph clearly
demonstrates a topology shift in the input layer: on the one hand, the average
number of connections to task-relevant features rises, while on the other
hand, noise features receive fewer weights. Together with the increased
performance shown in Figure 4, this fully supports our Adaptivity Hypothesis.

Figure 7. Average number of connections in the input layer of one of ANF-TD3’s
critic networks, on HalfCheetah-v3 with 90% noise features. At the start of
training every input neuron has around $256\cdot 0.2\approx 51$ connections,
because the input layer sparsity is 80% and connections are allocated
uniformly at random. During training, ANF gradually prunes connections from
the noise features and grows connections to the relevant features. A similar
graph for ANF-SAC is shown in Figure 16 of Appendix D.1. Figure 8. Performance
of ANF-TD3 and its baselines on permuted extremely noisy environments (PENE)
with 95% noise features. After every 1M timesteps, the environment’s features
are shuffled with a random permutation. ANF is able to cope with this
challenge, while the fully dense networks of TD3 are struggling. DS-TD3
performs decently, but ANF has an advantage by focusing its sparsity on the
input layer. Similar graphs for ANF-SAC and other environments are shown in
Appendix D.2.

##  5\. Transfer Learning

During a robot’s lifetime, it may happen that other information sources become
relevant to its task. Moreover, the agent may receive an entirely new task,
which can require it to focus its attention on totally different state
features.

We simulate this change in a permutated extremely noisy environment (PENE), as
described in Section 2. The PENE rearranges all input features with a fixed
permutation after every $T_{p}$ timesteps. For our experiments, this means
that the relevant and noise features are now mixed instead of concatenated.
The agents will have to rediscover which input neurons are receiving task-
relevant signals. Note that the PENE setting does not announce the change in
environment to the agent.

Experimental setup. We set $T_{p}$ to 1M timesteps, such that agents have
enough time to learn. We run on the same four environments with a noise
fraction of $n_{f}=0.95$. In these experiments, we now train for 4 million
timesteps, meaning that agents encounter four different instances (sub-
environments) of feature permutations. Similar to the experiments of Section
4, we compare ANF-SAC and ANF-TD3 with their fully dense baselines and DS-
SAC/TD3. We show 95% confidence intervals over 5 seeds.

Results. Figure 8 shows the results for ANF-TD3 on Humanoid and HalfCheetah.
See Appendix D.2 for the graphs of the remaining algorithms and environments.
It is evident that the performance drops considerably after each permutation
of features. However, ANF is able to recover faster than the dense baselines
in all environments. The method does not need to be adjusted for the
challenging PENE setting; ANF keeps adapting the sparse input layer as before.

For Humanoid, some beneficial internal representations may be transferred
forward, as the performance increases much earlier in the third sub-
environment (between 2M and 3M timesteps) than when it is trained from scratch
(between 0 and 1M timesteps). However, on the fourth sub-environment some ANF
agents struggled a bit: each random seed determines not only the
initialization of the agent, but also the random permutations of the
environment. Thus, some sub-environments can be more challenging than others.

Maintaining plasticity. Agents that have to learn continually must be able to
maintain plasticity. Standard methods are unable to do so, as shown by Dohare
et al. (2021). Since the connections of the input layer can drop and grow
dynamically, ANF ensures that the agent has sufficient adaptability to adjust
to a new environment. We analyze this plasticity by looking into the
connectivity of the input layer once more, as done earlier in Figure 7 for the
ENE experiments.

Figure 9. Average number of connections in the input layer of an ANF-SAC
critic network, on HalfCheetah-v3 with $n_{f}=0.95$. Every 1M timesteps, the
PENE permutes the order of the features. The ANF agent adjusts its network
structure quickly, growing connections to the task-relevant features, which
are now sent to different input neurons. A similar graph for ANF-TD3 is shown
in Figure 21 of Appendix D.2.

Now, in Figure 9, we see that the average number of connections to task-
relevant features quickly recovers after an environment change in the PENE. At
every 1M steps, the PENE shuffles the features, which makes the average number
of connections to relevant features drop considerably, close to the initial
value. This is because many task-relevant signals are now coming in at input
neurons that were previously receiving noise.

The fact that ANF has not pruned all connections to the irrelevant noise
features after training on the first sub-environment is actually an advantage
in this PENE setting. It means that ANF may be able to reach a high number of
connections to new task-relevant features faster, as they already have some
‘spare’ connections waiting.

Figure 10. ANF-SAC and its dense baseline on ENEs with louder noise. The noise
features are sampled from $\mathcal{N}(0,\sigma^{2})$. Noise amplitude
$\sigma$ is increased exponentially, notice the log-scale on the horizontal
axis. ANF tolerates louder noise much better than standard SAC, maintaining a
high performance up to $\sigma=16$. In this experiment $n_{f}\\!=\\!0.9$. The
graph for ANF-TD3 is given in Figure 23 of Appendix D.3.

##  6\. Louder noise

All of our experiments so far have been executed with noise features sampled
from the standard Gaussian distribution of $\mathcal{N}(0,1)$. But what would
happen if we increase the standard deviation, i.e. the noise amplitude? We
expect the louder noise to be more distracting, increasing the difficulty of
the ENE. We hope to discover whether ANF can cope with this additional
challenge.

Experimental setup. We run ANF and its dense baselines on the ENE of
HalfCheetah-v3, but the noise is now sampled from $\mathcal{N}(0,\sigma^{2})$.
We let the standard deviation $\sigma$ increase exponentially, ranging over
the set $\\{1,2,4,8,16\\}$. The ENE contains 90% of these louder noise
features ($n_{f}=0.9$).

Results. From the experimental results, we can conclude that louder noise does
make the ENE more challenging. In Figure 10, it is clearly visible that as the
noise amplitude increases, the performance decreases. Fortunately for ANF,
this decrease is much less pronounced compared to its dense baseline. In fact,
the final return of ANF-SAC on an ENE with noise amplitude $\sigma=16$ is
almost the same as SAC’s performance on the standard $\sigma=1$ environment.
ANF can cope well with even the loudest noise.666See
https://youtu.be/vS47UnsTQk8 for a video comparing the actual motion of
HalfCheetah when controlled by ANF-SAC vs. SAC with different noise
amplitudes.

##  7\. Imitating real features

Upon closer inspection of the data distribution of the original state
features, we discovered that these are far from a standard Gaussian
distribution. In Appendix E, we present visualizations of the distributions of
task-relevant features before and after training.777The challenging
distribution shift of RL is clearly visible! To get closer to real-world
noise, we want the noise features of our ENEs to imitate the original
features. We expect that this increases the difficulty of our extremely noisy
environments, as the noise is now much more similar to the task-relevant
features.

Experimental setup. For each of the original features, we make a histogram of
its final distribution (after training an agent in the standard environment),
as shown in Appendix E. In the experiments of this section, the ENE samples
from these histograms888By first sampling a bin according to the histogram’s
probability mass function, and then sampling a value uniformly at random
within the chosen bin. to generate noise features for the next state. We
repeatedly sample from the distribution of each original feature until we have
enough noise features. We run this experiment on HalfCheetah-v3, with 90% of
these noise features that imitate the task-relevant features.

Results. In Figure 11, we see that the imitated noise indeed raises the
difficulty of the ENE. The performance of both ANF-TD3 and TD3 decreases
considerably compared to the standard $\mathcal{N}(0,1)$ noise. However, ANF
is still able to outperform its dense baseline by a large margin, even in this
challenging ENE.

Figure 11. Learning curves for ANF-TD3 and its dense baseline on the
challenging ENE, where the noise features imitate the task-relevant features.
This increases the difficulty, but ANF still achieves the highest return. The
ENE has 90% of these realistic noise features in this experiment. See Figure
24 in Appendix D.4 for the graph of ANF-SAC.

##  8\. Does ANF need to be dynamic?

In this section, we perform an ablation study to show that the dynamic network
topology updates of ANF are a necessary component of the algorithm. Removing
these dynamic updates during training would give a static sparse training
algorithm. This algorithm starts with a randomly sparsified input layer just
like ANF, but it does not drop or regrow any connections during training.

Experimental setup. We compare ANF to its fixed-connectivity counterpart,
which we call Static-ANF. In addition, we compare the standard dense
algorithms of SAC and TD3. We run on Humanoid-v3 with 90% noise features.

Results. The graphs in Figure 12 show that ANF indeed needs to be dynamic, as
it significantly outperforms its static version. Intuitively, this is
consistent with the concept shown in Figure 7, where ANF changes its
connectivity to emphasize its focus on the task-relevant features. This
emphasis is lost if one removes ANF’s ability to dynamically adjust the
network structure.

Nevertheless, it is remarkable to see that Static-ANF is able to surpass
standard dense TD3 in this setting. It seems that just having fewer
connections to the 90% noisy input features already helps to lower the overall
distraction.

Figure 12. Comparison of ANF to its static sparse counterpart and standard
(fully dense) TD3. ANF-TD3 dynamically updates the sparse network topology,
while the topology of the other two methods remains static. The learning
curves show that the dynamic updates are an essential part of ANF. This
experiment is run with 90% noise features. A similar graph for SAC is shown in
Figure 25 of Appendix D.5.

##  9\. How Sparse can we go?

We already showed that sparsifying the input layer can significantly improve
performance in extremely noisy environments. In this section, we investigate
whether we can further sparsify the agent’s networks by also pruning
connections in other layers. This would reduce the network size (total number
of parameters) even further, while hopefully maintaining performance.

Experimental setup. Instead of only having an 80% sparse input layer, the
networks now also have a sparse hidden layer for which the connectivity is
frequently adjusted with DST. We keep the output layer dense, just as in Sokar
et al. (2022b). The sparsity distribution is uniform, meaning that both the
input layer and the hidden layer have the same sparsity level. We compute the
required layer sparsity levels such that the global sparsity level (over the
full network) is at $s$, where $s$ ranges over $\\{.80,.90,.95,.98\\}$. We
compare with our standard ANF algorithm, which has a global sparsity of 74.6%
for $n_{f}\\!=\\!0.9$ (actor network on Humanoid). We run on Humanoid-v3 and
HalfCheetah-v3, as the achievable sparsity level before performance
degradation differs significantly between these two environments.

Results. We see in Figure 13 that on extremely noisy environments, nearly the
same performance can be reached with further sparsified networks, meaning that
a large proportion of the parameters can be pruned. In Figure 13 (left), we
see that ANF-TD3 with a global sparsity of 80% still surpasses standard dense
TD3 in final return on HalfCheetah-v3. As the global sparsity increases, the
performance gradually decreases. This is quite different for Humanoid-v3, in
Figure 13 (right). Notice that ANF-TD3 can go up to a global sparsity level of
95%, with barely any performance degradation. This means it can use $20\times$
fewer parameters than standard TD3, reducing the network size considerably, as
shown in Table 2.

Figure 13. Performance of ANF-TD3 and its sparser versions, on 90% noise
features. Sparser-ANF also prunes weights in the hidden layer, instead of only
the input layer. Further sparsifying the network does not improve performance,
but can drastically reduce the size of the network. The graphs for ANF-SAC are
shown in Figure 26 of Appendix D.6. Table 2. Comparison of different
algorithms, along with their parameter counts for the actor network. The
critic networks have comparable numbers of parameters.

Algorithm | Environment | Return ($\uparrow$) | # Params. ($\downarrow$)
---|---|---|---
ANF-TD3 | Humanoid | 4968.3 | 262,400
TD3 | Humanoid | 817.3 | 1,032,448
Sparser(80%)-ANF-TD3 | Humanoid | 4963.1 | 206,489
Sparser(95%)-ANF-TD3 | Humanoid | 4806.4 | 51,622
ANF-TD3 | HalfCheetah | 11086.4 | 75,776
TD3 | HalfCheetah | 9452.6 | 110,592
Sparser(80%)-ANF-TD3 | HalfCheetah | 10640.3 | 22,118
Sparser(95%)-ANF-TD3 | HalfCheetah | 8357.9 | 5,529

##  10\. Conclusion & Limitations

In this work, we formulated the problem setting of extremely noisy
environments and showed that our Automatic Noise Filtering algorithm succeeds
at this challenge where standard deep RL methods struggle. By using dynamic
sparse training, the ANF algorithm adjusts its network topology to focus on
task-relevant features.

Our experiments provide an initial empirical verification of our Adaptability
Hypothesis, which roughly states that dropping and growing sparse connections
is easier than adjusting dense weights. Further research is necessary to grant
more conclusive evidence, as our work is limited to continuous control tasks
that have feature vectors as states. We exclusively studied SAC and TD3;
integrating ANF in other deep RL methods is open for future work.

The input layer sparsity is an important hyperparameter for ANF. Making this
an adaptive parameter could be beneficial. Further, we believe expanding ANF’s
compatibility towards other neural network types is a promising future
research direction. Combining ANF with convolutional NNs or transformers would
open the possibility of operating on noisy image or video data.

This publication is part of the project AMADeuS (with project number 18489) of
the Open Technology Programme, which is partly financed by the Dutch Research
Council (NWO). This research used the Dutch national e-infrastructure with the
support of the SURF Cooperative, using grant no. EINF-3098. Part of this work
has taken place in the Intelligent Robot Learning (IRL) Lab at the University
of Alberta, which is supported in part by research grants from the Alberta
Machine Intelligence Institute (Amii); a Canada CIFAR AI Chair, Amii; Compute
Canada; Huawei; Mitacs; and NSERC. We thank Joan Falcó Roget, Mickey
Beurskens, Anne van den Berg, and Rik Grooten for the fruitful discussions.
Finally, we thank the anonymous reviewers and Antonie Bodley for their
thorough proofreading and useful comments.

## References

  * (1)
  * Acunto (2012) Kevin Acunto. 2012\.  _Feature selection for scalable reinforcement learning_.  Ph.D. Dissertation. State University of New York at Binghamton.  URL: https://www.proquest.com/openview/73f1a695503d3e32f0fa49cae57cb411.
  * Arnob et al. (2021) Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. 2021.  Single-Shot Pruning for Offline Reinforcement Learning.  _arXiv preprint arXiv:2112.15579_ (2021).  URL: https://arxiv.org/abs/2112.15579.
  * Atashgahi et al. (2022) Zahra Atashgahi, Ghada Sokar, Tim van der Lee, Elena Mocanu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. 2022\.  Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders.  _Machine Learning_ 111, 1 (2022), 377–414.  URL: https://link.springer.com/article/10.1007/s10994-021-06063-x.
  * Bach-y Rita et al. (1969) Paul Bach-y Rita, Carter C Collins, Frank A Saunders, Benjamin White, and Lawrence Scadden. 1969\.  Vision Substitution by Tactile Image Projection.  _Nature_ 221, 5184 (1969), 963–964.  URL: https://www.nature.com/articles/221963a0.
  * Bellec et al. (2018) Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. 2018.  Deep Rewiring: Training very sparse deep networks.  _International Conference on Learning Representations_ (2018).  URL: https://arxiv.org/abs/1711.05136.
  * Bemd (2022) Wouter van den Bemd. 2022\.  _Robust Deep Reinforcement Learning for Greenhouse Control and Crop Yield Optimization_.  Master’s thesis. Eindhoven University of Technology.  URL: https://research.tue.nl/en/studentTheses/robust-deep-reinforcement-learning-for-greenhouse-control-and-cro.
  * Bishop and Miikkulainen (2013) Julian Bishop and Risto Miikkulainen. 2013.  Evolutionary Feature Evaluation for Online Reinforcement Learning. In _2013 IEEE Conference on Computational Intelligence in Games (CIG)_. 1–8.  https://doi.org/10.1109/CIG.2013.6633648
  * Botteghi et al. (2021) Nicolò Botteghi, Khaled Alaa, Mannes Poel, Beril Sirmacek, Christoph Brune, Abeje Mersha, and Stefano Stramigioli. 2021\.  Low Dimensional State Representation Learning with Robotics Priors in Continuous Action Spaces. In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 190–197.  URL: https://arxiv.org/abs/2107.01667.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016.  OpenAI Gym.  _arXiv preprint arXiv:1606.01540_ (2016).  URL: https://www.gymlibrary.dev/.
  * Castaldi et al. (2020) Elisa Castaldi, Claudia Lunghi, and Maria Concetta Morrone. 2020\.  Neuroplasticity in adult human visual cortex.  _Neuroscience & Biobehavioral Reviews_ 112 (2020), 542–552.  URL: https://www.sciencedirect.com/science/article/pii/S0149763419303288.
  * Chen et al. (2021) Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. 2021\.  Chasing Sparsity in Vision Transformers: An End-to-End Exploration.  _Advances in Neural Information Processing Systems_ 34 (2021).  URL: https://arxiv.org/abs/2106.04533.
  * Chen et al. (2022) Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. 2022.  Sparsity Winning Twice: Better Robust Generalization from More Efficient Training.  _International Conference on Machine Learning_ (2022).  URL: https://openreview.net/forum?id=SYuJXrXq8tw.
  * Curci et al. (2021) Selima Curci, Decebal Constantin Mocanu, and Mykola Pechenizkiyi. 2021.  Truly Sparse Neural Networks at Scale.  _arXiv preprint arXiv:2102.01732_ (2021).  URL: https://arxiv.org/abs/2102.01732.
  * Curran et al. (2016) William Curran, Tim Brys, David Aha, Matthew Taylor, and William D Smart. 2016.  Dimensionality Reduced Reinforcement Learning for Assistive Robots. In _2016 AAAI Fall Symposium Series_.  URL: https://www.aaai.org/ocs/index.php/FSS/FSS16/paper/download/14076/13660.
  * Degrave et al. (2022) Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. 2022\.  Magnetic control of tokamak plasmas through deep reinforcement learning.  _Nature_ 602, 7897 (2022), 414–419.  URL: https://www.nature.com/articles/s41586-021-04301-9.
  * Dohare et al. (2021) Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. 2021.  Continual Backprop: Stochastic Gradient Descent with Persistent Randomness.  _arXiv preprint arXiv:2108.06325_ (2021).  URL: https://arxiv.org/abs/2108.06325.
  * Evci et al. (2020) Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020.  Rigging the Lottery: Making All Tickets Winners. In _International Conference on Machine Learning_. PMLR, 2943–2952.  URL: https://arxiv.org/abs/1911.11134.
  * Frankle and Carbin (2019) Jonathan Frankle and Michael Carbin. 2019.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.  _International Conference on Learning Representations_.  URL: https://arxiv.org/abs/1803.03635.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger. 2018.  Addressing Function Approximation Error in Actor-Critic Methods. In _International Conference on Machine Learning_. PMLR, 1587–1596.  URL: https://arxiv.org/abs/1802.09477.
  * Graesser et al. (2022) Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. 2022.  The State of Sparse Training in Deep Reinforcement Learning. In _International Conference on Machine Learning_. PMLR, 7766–7792.  URL: https://arxiv.org/abs/2206.10369.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In _International Conference on Machine Learning_. PMLR, 1861–1870.  URL: https://arxiv.org/abs/1801.01290.
  * Han et al. (2015) Song Han, Jeff Pool, John Tran, and William Dally. 2015\.  Learning both Weights and Connections for Efficient Neural Networks.  _Advances in Neural Information Processing Systems_ 28 (2015).  URL: https://arxiv.org/abs/1506.02626.
  * Hoefler et al. (2021) Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021.  Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks.  _Journal of Machine Learning Research_ 22, 241 (2021), 1–124.  URL: https://arxiv.org/abs/2102.00554.
  * Hooker (2021) Sara Hooker. 2021\.  The Hardware Lottery.  _Commun. ACM_ 64, 12 (2021), 58–65.  URL: https://dl.acm.org/doi/10.1145/3467017.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization.  _International Conference for Learning Representations_ (2015).  URL: https://arxiv.org/abs/1412.6980.
  * Kroon and Whiteson (2009) Mark Kroon and Shimon Whiteson. 2009.  Automatic Feature Selection for Model-Based Reinforcement Learning in Factored MDPs. In _2009 International Conference on Machine Learning and Applications_. IEEE, 324–330.  URL: https://ieeexplore.ieee.org/document/5381529.
  * Li et al. (2021) Yuchen Li, Yifan Bao, Liyao Xiang, Junhan Liu, Cen Chen, Li Wang, and Xinbing Wang. 2021.  Privacy Threats Analysis to Secure Federated Learning.  _arXiv preprint arXiv:2106.13076_ (2021).  URL: https://arxiv.org/abs/2106.13076.
  * Liu et al. (2020b) Junjie Liu, Zhe Xu, Runbin Shi, Ray Cheung, and Hayden So. 2020b.  Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers.  _International Conference for Learning Representations_ (2020).  URL: https://arxiv.org/abs/2005.06870.
  * Liu et al. (2020a) Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. 2020a.  Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware.  _Neural Computing and Applications_ 33 (2020), 2589–2604.  URL: https://arxiv.org/abs/1901.09181.
  * Liu et al. (2019) Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy. 2019.  On improving deep learning generalization with adaptive sparse connectivity.  _arXiv preprint arXiv:1906.11626_ (2019).  URL: https://arxiv.org/abs/1906.11626.
  * Liu et al. (2021) Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. 2021.  Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training. In _International Conference on Machine Learning_. PMLR, 6989–7000.  URL: https://arxiv.org/abs/2102.02887.
  * Mocanu et al. (2021) Decebal Constantin Mocanu, Elena Mocanu, Tiago Pinto, Selima Curci, Phuong H Nguyen, Madeleine Gibescu, Damien Ernst, and Zita A Vale. 2021.  Sparse Training Theory for Scalable and Efficient Agents.  _Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems_ (2021).  URL: https://arxiv.org/abs/2103.01636.
  * Mocanu et al. (2018) Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. 2018.  Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.  _Nature communications_ 9, 1 (2018), 1–12.  URL: https://arxiv.org/abs/1707.04780.
  * Moos et al. (2022) Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. 2022\.  Robust Reinforcement Learning: A Review of Foundations and Recent Advances.  _Machine Learning and Knowledge Extraction_ 4, 1 (2022), 276–315.  URL: https://www.mdpi.com/2504-4990/4/1/13.
  * Pereda (2014) Alberto E Pereda. 2014\.  Electrical synapses and their functional interactions with chemical synapses.  _Nature Reviews Neuroscience_ 15, 4 (2014), 250–263.  URL: https://www.nature.com/articles/nrn3708.
  * Rafiee et al. (2020) Banafsheh Rafiee, Zaheer Abbas, Sina Ghiassian, Raksha Kumaraswamy, Richard S Sutton, Elliot A Ludvig, and Adam White. 2020\.  From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learning.  _Adaptive Behavior_ (2020).  URL: https://arxiv.org/abs/2011.04590.
  * Silver et al. (2021) David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. 2021.  Reward is enough.  _Artificial Intelligence_ 299 (2021), 103535.  URL: https://www.sciencedirect.com/science/article/pii/S0004370221000862.
  * Sokar et al. (2022a) Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, and Decebal Constantin Mocanu. 2022a.  Where to Pay Attention in Sparse Training for Feature Selection?. In _Advances in Neural Information Processing Systems_.  URL: https://openreview.net/forum?id=xWvI9z37Xd.
  * Sokar et al. (2022b) Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. 2022b.  Dynamic Sparse Training for Deep Reinforcement Learning.  _The 31st International Joint Conference on Artificial Intelligence_ (2022).  URL: https://arxiv.org/abs/2106.04217.
  * Stone et al. (2021) Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. 2021.  The Distracting Control Suite – A Challenging Benchmark for Reinforcement Learning from Pixels.  _arXiv preprint arXiv:2101.02722_ (2021).  URL: https://arxiv.org/abs/2101.02722.
  * Sun et al. (2021) Ke Sun, Yi Liu, Yingnan Zhao, Hengshuai Yao, Shangling Jui, and Linglong Kong. 2021\.  Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observations.  _arXiv preprint arXiv:2109.08776_ (2021).  URL: https://arxiv.org/abs/2109.08776.
  * Tan et al. (2023) Yiqin Tan, Pihe Hu, Ling Pan, and Longbo Huang. 2023\.  RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch.  _International Conference on Learning Representations_ (2023).  URL: https://arxiv.org/abs/2205.15043.
  * Taylor and Stone (2009) Matthew E Taylor and Peter Stone. 2009.  Transfer learning for reinforcement learning domains: A survey.  _Journal of Machine Learning Research_ 10, 7 (2009).  URL: https://www.jmlr.org/papers/v10/taylor09a.html.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  MuJoCo: A physics engine for model-based control. In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 5026–5033.  URL: https://mujoco.org/.
  * Vinitsky et al. (2020) Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. 2020\.  Robust Reinforcement Learning using Adversarial Populations.  _arXiv preprint arXiv:2008.01825_ (2020).  URL: https://arxiv.org/abs/2008.01825.
  * Vischer et al. (2022) Marc Aurel Vischer, Robert Tjarko Lange, and Henning Sprekeler. 2022.  On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning.  _ICLR_ (2022).  URL: https://arxiv.org/abs/2105.01648.
  * Whiteson et al. (2005) Shimon Whiteson, Peter Stone, Kenneth O Stanley, Risto Miikkulainen, and Nate Kohl. 2005\.  Automatic Feature Selection in Neuroevolution. In _Proceedings of the 7th annual conference on Genetic and evolutionary computation_. 1225–1232.  URL: https://dl.acm.org/doi/10.1145/1068009.1068210.
  * Zhou et al. (2020) Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. 2020\.  Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch.  _International Conference on Learning Representations_ (2020).  URL: https://arxiv.org/abs/2102.04010.

## Appendix

##  Appendix A ANF algorithm

In this section we provide pseudocode of the Automatic Noise Filtering (ANF)
algorithm. In Algorithm 1 we show the implementation of ANF-SAC, but keep in
mind that ANF can be applied to any MLP-based deep RL method. The novel parts
of our proposed method ANF are colored violet. The dynamic sparse training
components, already introduced by Sokar et al. (2022b), are colored medium-
blue.999Colors from https://xkcd.com/color/rgb/. The rest (in black) is the
standard SAC algorithm Haarnoja et al. (2018).

Our code is open-source and can be found at:
https://github.com/bramgrooten/automatic-noise-filtering.

Algorithm 1 ANF-SAC

1:input layer sparsity $s_{i}$, topology-change period $\Delta T$, drop
fraction $d_{f}$, initial collect steps $b_{init}$, train every $k$ env steps,
minibatch size $n$, learning rate $\lambda$, target smoothing coefficient
$\tau$, max env steps $T$

2:Initialize the actor network $\pi$ and two critic networks $Q_{1}$, $Q_{2}$,
with weights $\phi,\theta_{1},\theta_{2}$.

3:Randomly prune the input layer of all 3 networks to sparsity level $s_{i}$.

4:Duplicate the two critics to create two target networks, with weights
$\bar{\theta}_{1}=\theta_{1}$, $\bar{\theta}_{2}=\theta_{2}$.

5:Initialize the replay buffer $\mathcal{B}$ with $b_{init}$ random actions.

6:for $t=1$ to $T$ do

7: Sample action $a$ from the policy (actor network) based on current state
$s$: $a\sim\pi_{\phi}(\cdot|s)$

8: Take a step in the environment and observe reward $r$ and new state
$s^{\prime}$.

9: Store transition tuple $(s,a,r,s^{\prime})$ in $\mathcal{B}$.

10: if $t$ mod $k==0$ then

11: Sample minibatch of $n$ transitions from $\mathcal{B}$.

12: Update the weights according to SAC’s objective functions $J_{Q}$,
$J_{\pi}$:

13:
$\theta_{i}\leftarrow\theta_{i}-\lambda\hat{\nabla}_{\theta_{i}}J_{Q}(\theta_{i})$
for $i\in\\{1,2\\}$

14: $\phi\leftarrow\phi-\lambda\hat{\nabla}_{\phi}J_{\pi}(\phi)$

15: Update the target networks:

16: $\bar{\theta_{i}}\leftarrow\tau\theta_{i}+(1-\tau)\bar{\theta_{i}}$ for
$i\in\\{1,2\\}$

17: end if

18: if $t$ mod $\Delta T==0$ then

19: Update the topology of the networks:

20: Prune fraction $d_{f}$ of the smallest magnitude weights.

21: Grow fraction $d_{f}$ new weights randomly, initialize at value 0.

22: Mask Adam’s running avg. of the 1st & 2nd raw moment of the gradient for
pruned weights:

23: $m\leftarrow 0$, $v\leftarrow 0$.

24: end if

25:end for

##  Appendix B Experimental Details

In this section we present an overview of the hyperparameters used in our
experiments.

Algorithms: Throughout the experiments in the paper we compared our proposed
dynamic sparse algorithms (ANF-SAC and ANF-TD3) with the fully-dense
counterparts (SAC and TD3), and the static sparse variants (Static-ANF-SAC and
Static-ANF-TD3). The complete list of hyperparameters can be found in Table 3.
Aiming for a fair comparison, we tried to maximize the number of shared
hyperparameters. Table 3 also includes the parameters used for the experiments
described in Section 9, where an increasing sparsity level is incorporated in
ANF beyond the first layer, leading to the Sparser-ANF-SAC and Sparser-ANF-TD3
versions.

Environments: We used as the foundation for our extremely noisy environments
(ENEs) four continuous control tasks (Humanoid-v3,101010Please note that
according to the environment’s documentation
(https://www.gymlibrary.dev/environments/mujoco/humanoid/#observation-space)
versions 1,2,3 of Humanoid contain an issue with the contact forces (the
corresponding features always give 0). Humanoid-v4 solves this, but came out
too late for our research. HalfCheetah-v3, Walker-v3, and Hopper-v3) as shown
in Figure 14. See Table 4 for the parameters corresponding to the ENEs built
on top of these four worlds.

Table 3. Hyperparameters. Table format from Haarnoja et al. (2018). Parameter | Value
---|---
Shared by all algorithms |
optimizer | Adam Kingma and Ba (2015)
learning rate ($\lambda$) | $1\cdot 10^{-3}$
weight decay | $2\cdot 10^{-4}$
discount ($\gamma$) | $0.99$
nonlinearity | ReLU
replay buffer size | $10^{6}$
initial collect steps ($b_{init}$) | $25,000$
network type | MLP
number of hidden layers | $2$
number of neurons per hidden layer | $256$
minibatch size ($n$) | $100$
target smoothing coefficient ($\tau$) | $0.005$
train every $k_{c}$ env steps (critic), $k_{c}=$ | $1$
gradient steps per training step $=$ | $1$
SAC and ANF-SAC |
SAC type (Gaussian / Deterministic) | Gaussian
temperature ($\alpha$ in Haarnoja et al. (2018)) | $0.2$
automatic temperature tuning | False
train every $k_{a}$ env steps (actor), $k_{a}=$ | $1$
target update interval ($k_{tar}$) | $1$
TD3 and ANF-TD3 |
train every $k_{a}$ env steps (actor), $k_{a}=$ | $2$
target update interval ($k_{tar}$) | $2$
std. dev. of exploration noise ($\sigma$ in Fujimoto et al. (2018)) | $0.1$
std. dev. of sampling noise ($\widetilde{\sigma}$ in Fujimoto et al. (2018)) | $0.2$
sampling noise clip ($c$ in Fujimoto et al. (2018)) | $0.5$
ANF |
sparsity of the input layer ($s_{i}$) | $0.8$
drop fraction ($d_{f}$) | $0.05$
new weights init value | $0$
sparse layers | input layer
topology-change period ($\Delta T$), env steps | $1000$
Static-ANF |
topology-change period ($\Delta T$), env steps |  $\infty$ (no change)
Sparser-ANF |
global sparsity | varying (Section 9)
sparsity distribution (over sparse layers) | uniform
sparse layers | input & hidden
Figure 14. The MuJoCo Gym Todorov et al. (2012); Brockman et al. (2016) environments used as a base for our ENEs. Table 4. Extremely noisy environment (ENE) parameters. Parameter | Value | Experiments in
---|---|---
noise fraction ($n_{f}$) | varying | Section 4
noise distribution | $\mathcal{N}(0,\sigma^{2})$ | Sections 4-6, 8-9
| Imitate | Section 7
noise amplitude ($\sigma$) | $1$ | Sections 4-5, 8-9
| varying | Section 6
permutation period ($T_{p}$) | 1M | Section 5

##  Appendix C Sparse Hardware and Software support

As discussed in Sokar et al. (2022b), research on sparsity is moving in three
simultaneous directions as a collaborative community effort. Firstly, hardware
that can benefit from sparse neural networks. In order to support a sparsity
level of 50% as a first step, NVIDIA produced the A100 Zhou et al. (2020).
Secondly, software libraries that support neural network implementations which
are truly sparse. Supervised learning has begun to receive attention in this
direction Liu et al. (2020a); Curci et al. (2021). Thirdly, algorithmic
approaches, which are the subject of our research, are intended to offer
sparse network methods whose performance is at least at the level of dense
models Hoefler et al. (2021). We will be able to produce faster, memory-
efficient, and energy-efficient deep neural networks with parallel efforts in
the three dimensions. Further discussion of this can be found in Hooker
(2021); Mocanu et al. (2021).

##  Appendix D Additional Results

In this appendix we share results that did not fit into the main body of the
paper. Many figures present outcomes of additional algorithms or environments,
while some graphs such as Figure 17 and 22 show extra analysis of the network
connectivity.

###  D.1. Automatic Noise Filtering graphs

Additional results on the main ANF experiments are shown in Figure 15.

Figure 15. Performance of ANF compared to its baselines for different
fractions of noise features $n_{f}$. When the environments contain a lot of
noise (high $n_{f}$) the standard dense networks of TD3 seem to struggle.
Similar graphs for ANF-SAC are shown in Figure 4 of Section 4. Figure 16.
Average number of connections in the input layer of one of ANF-SAC’s critic
networks, on HalfCheetah-v3 with 90% noise features. At the start of training
every input neuron has around $256\cdot 0.2\approx 51$ connections, because
the input layer sparsity is 80% and connections are allocated uniformly at
random. During training, ANF gradually prunes connections from the noise
features and grows connections to the relevant features. A similar graph for
ANF-TD3 is shown in Figure 7 of Section 4. Figure 17. Showing the connectivity
of the actor network’s input layer for ANF-TD3 on HalfCheetah-v3 with 90%
noise features. At the start of training (iteration 0) every input neuron has
around $256\cdot 0.2\approx 51$ connections, because the input layer sparsity
is 80% and connections are allocated uniformly at random. During training,
ANF-TD3 gradually removes connections from the noise features and adds
connections to the relevant features (the 17 leftmost input neurons in this
graph). Note that a dense network always has 256 connections to every input
neuron, and thus gets distracted more easily by all the noise features.

A noteworthy observation for Figure 17 is the fact that apparently not all 17
input features are deemed to be useful, as some received less than the
original 51 connections. This phenomenon has been shown in standard RL
environments recently by Vischer et al. (2022), and it is interesting to see
that it still holds up in extremely noisy environments. Original state
features that fall outside the minimal task representation are considered just
as irrelevant as the noise features, according to Figure 17. We see that ANF
filters not only through synthetic features, but also de-emphasizes redundant
features.

The full version of Table 1 with all noise fractions used is shown below in
Table 5.

Table 5. State and action space dimensions. Environment | State dim. | Action dim. | State dim. | State dim. | State dim. | State dim. | State dim.
---|---|---|---|---|---|---|---
| Original | Original | ENE ($n_{f}\\!=\\!.8$) | ENE ($n_{f}\\!=\\!.9$) | ENE ($n_{f}\\!=\\!.95$) | ENE ($n_{f}\\!=\\!.98$) | ENE ($n_{f}\\!=\\!.99$)
Humanoid-v3 | 376 | 17 | 1880 | 3760 | 7520 | 18,800 | 37,600
HalfCheetah-v3 | 17 | 6 | 85 | 170 | 340 | 850 | 1700
Walker2d-v3 | 17 | 6 | 85 | 170 | 340 | 850 | 1700
Hopper-v3 | 11 | 3 | 55 | 110 | 220 | 550 | 1100

###  D.2. Transfer Learning graphs

Extra graphs on transfer learning shown in Figures 18, 19, 20. They correspond
to Figure 8 in Section 5 of the paper. The connectivity graph of ANF-TD3 is
given in Figure 21.

Figure 18. Performance of ANF-TD3 and its baselines on permuted extremely
noisy environments (PENE) with 95% noise features. After every 1M timesteps
the features are shuffled with a random permutation. ANF is able to cope with
this challenge better than the fully dense networks of TD3. Figure 19.
Performance of ANF-SAC and its baselines on permuted extremely noisy
environments (PENE) with 95% noise features. After every 1M timesteps the
features are shuffled with a random permutation. ANF is able to cope with this
challenge better than the fully dense networks of SAC. Figure 20. Performance
of ANF-SAC and its baselines on permuted extremely noisy environments (PENE)
with 95% noise features. After every 1M timesteps the features are shuffled
with a random permutation. ANF is able to cope with this challenge, while the
fully dense networks of SAC are struggling. Figure 21. Average number of
connections in the input layer of one of ANF-TD3’s critic networks, on
HalfCheetah-v3 with 95% noise features. After every 1M timesteps the PENE
rearranges the order of the features by a random permutation. The ANF agent
adjusts its network structure quickly, growing connections with the task-
relevant features which are now coming in at different input neurons. A
similar graph for ANF-SAC is shown in Figure 9 of Section 5.  Figure 22.
Number of connections per input neuron (feature) of the input layer of ANF-
TD3’s actor network, on HalfCheetah-v3 with 95% noise features. After 1M
iterations the agent has been trained on the initial setup (all 17 relevant
features have index 0-16). The environment now changes and shuffles all input
features with a fixed permutation. After 2M iterations the agent has adjusted
to this new environment, and is able to find the new locations of the relevant
features (shown by the green vertical lines).

We see in Figure 22 that during training on the second sub-environment ANF
drops the connections to input neurons that previously had task-relevant
features (the leftmost features with index 0-16). It grows new connections to
input neurons that currently have task-relevant features, represented by the
green vertical lines.

###  D.3. Louder noise graphs

Additional results on the louder noise experiments for TD3 are shown in Figure
23.

Figure 23. Performance of ANF and its baseline on ENEs with louder noise. The
noise features are sampled i.i.d. from $\mathcal{N}(0,\sigma^{2})$. Noise
amplitude is increased exponentially, notice the log-scale on the horizontal
axis. This experiment uses 90% noise features. A similar graph for ANF-SAC is
shown in Figure 10 of Section 6.

###  D.4. Imitating real noise graphs

The graph for ANF-SAC in the imitated noise experiment is shown in Figure 24.

Figure 24. Learning curves for ANF-SAC and its baseline on the challenging
ENE, where the noise features imitate the task-relevant features. This
increases the difficulty, but ANF-SAC still learns much quicker than SAC. A
similar graphs for ANF-TD3 is shown in Figure 11 of Section 7.

###  D.5. Static ablation graphs

Graphs for the static ablation study are shown in Figure 25.

Figure 25. Comparison of ANF, which dynamically updates the sparse network
topology, to its static sparse counterpart and the standard (fully dense) SAC.
This experiment is run with 90% noise features. A similar graph for TD3 is
shown in Figure 12 of Section 8.

###  D.6. Sparsifying further

Additional results for the experiments where we sparsified the agents even
further are shown in Table 6, which is an extension of Table 2, and in Figure
26. Notice that for HalfCheetah we could only go up to 97% global sparsity
with ANF-SAC. This is because 98% turned out to be impossible for SAC on
HalfCheetah with noise fraction $n_{f}=0.9$. The actor network of SAC has two
output heads (in contrast to TD3, which only has one). This means that SAC’s
actor output layer has more connections (more than 2% of the total possible
number of connections). We keep the output layer dense in all these
experiments, meaning that for SAC on HalfCheetah, 97% global sparsity was the
highest we could go.

Figure 26. Performance of ANF-SAC and its sparser versions, on 90% noise features. Sparser-ANF also prunes weights in the hidden layer, instead of only the input layer. Further sparsifying the network surprisingly improves performance for Humanoid (up to a certain point), but not for HalfCheetah. In both cases, it drastically reduces the size of the network. The graphs for ANF-TD3 are shown in Figure 13 of Section 9. Table 6. Comparison of different algorithms, along with their parameter counts for the actor network. The critic networks have comparable numbers of parameters. Algorithm | Environment | Return ($\uparrow$) | # Params. ($\downarrow$)
---|---|---|---
ANF-TD3 | Walker2d-v3 | 4622.0 | 75,776
TD3 | Walker2d-v3 | 3554.8 | 110,592
Sparser(80%)-ANF-TD3 | Walker2d-v3 | 4060.7 | 22,118
Sparser(95%)-ANF-TD3 | Walker2d-v3 | 2677.4 | 5,529
ANF-TD3 | Hopper-v3 | 3329.7 | 71,936
TD3 | Hopper-v3 | 2941.8 | 94,464
Sparser(80%)-ANF-TD3 | Hopper-v3 | 3309.1 | 18,892
Sparser(95%)-ANF-TD3 | Hopper-v3 | 2807.2 | 4,723

###  D.7. Non-zero centered noise

One of our anonymous reviewers pointed out that the zero-centered
$\mathcal{N}(0,1)$ noise simplifies ANF’s process of pruning connections to
noise-features. We ran some extra experiments with non-zero-centered Gaussian
noise to show empirically that: (i) non-zero-centered noise is indeed more
challenging and (ii) ANF is still able to handle it better than dense
networks. The experiment is run on HalfCheetah-v3 with noise distribution
$\mathcal{N}(\mu,1)$, noise fraction $n_{f}=.98$, and 5 random seeds.

Table 7. Experiment with non-zero-centered noise $\mathcal{N}(\mu,1)$. Avg. returns | $\mu=0$ | $\mu=1$ | $\mu=-2$ | $\mu=4$
---|---|---|---|---
ANF-SAC | 9250.4 | 5642.2 | 5047.8 | 636.9
SAC | 6124.3 | 3744.4 | 419.1 |  $-$35.3

#### Theoretical perspective.

Even with non-zero-mean noise, we think the weights connected to noise-
features will stay close to zero (and thus get pruned by ANF). Note that
initial weights are small, and ANF’s newly grown weights start at 0. Since a
noise-feature is irrelevant, its connections will receive mixed signals
(positive gradient for some $<$state, action, reward$>$ tuples, negative
gradient for others). This means it barely gets a chance to grow a large
magnitude. In a simplified setting, we can prove a stronger claim;

Conjecture: Weights connected to noise-features converge to 0 with gradient-
descent, even for non-zero-centered noise.

###### Proof.

(Not a full proof, only for a simple setting.) We assume to be in the local
neighborhood of the optimum. Suppose we have a function approximator
$f(x_{1},x_{2})=w_{1}\cdot x_{1}+w_{2}\cdot x_{2}=y_{\text{approx}}$, which is
trying to estimate the true function $g(x_{1})=a\cdot x_{1}=y$. Note:
$g(\cdot)$ does not depend on $x_{2}$ (noise-feature).

Suppose we use mean-squared-error loss: $L=\frac{1}{2}(f(x_{1},x_{2})-y)^{2}$.
Then the gradient (partial derivative) of weight $w_{2}$ is:

$\frac{\textrm{d}L}{\textrm{d}w_{2}}=(f(x_{1},x_{2})-y)\cdot x_{2}=(w_{1}\cdot
x_{1}+w_{2}\cdot x_{2}-a\cdot x_{1})\cdot x_{2}.$

Assuming we’re near the optimum, i.e., $w_{1}-a\approx 0$, we can rewrite
$\frac{\textrm{d}L}{\textrm{d}w_{2}}\approx w_{2}\cdot(x_{2})^{2}$.

For any noise $x_{2}$: if $w_{2}$ is negative, its gradient is negative, and
vice versa. We minimize MSE-loss, so gradient-descent moves in the direction:
$-$gradient. Thus, $w_{2}$ will be updated toward zero. ∎

###  D.8. Matching the input-layer-sparsity-level with the noise-fraction

From the problem setting of extremely noisy environments (ENEs) it seems
beneficial to match the algorithm’s input-layer-sparsity-level with the noise-
fraction of the ENE. We ran this experiment but omitted it from the main body
of the paper for the following reasons:

  1. (i)

performance is similar (see Table 8 below, it’s challenging to prune all
connections to noise-features, so it may be useful to have some surplus of
connections for task-relevant features),

  2. (ii)

we did not want to assume that the agent knows the noise-fraction. (The input-
layer-sparsity-level could be an adaptive parameter, which is mentioned as
potential future work in the paper.)

Instead, we kept the input-layer-sparsity at a well-working 80% to have a
generally applicable algorithm.

Table 8. Experiments matching input-layer-sparsity-levels with noise-fractions (policy = ANF-SAC, env = HalfCheetah-v3, num_seeds = 5). Avg. returns | $n_{f}=.90$ | $n_{f}=.95$ | $n_{f}=.98$ | $n_{f}=.99$
---|---|---|---|---
matching input-layer-sparsity | 11041.1 | 10259.0 | 10090.3 | 8641.2
80% input-layer-sparsity | 11913.8 | 9920.5 | 9250.4 | 10007.2

##  Appendix E Distributions of original features

Here we present the distributions of the original state features of the MuJoCo
Gym environments. These histograms are generated by taking a trained or
untrained agent and letting it run on an evaluation environment for $10,000$
steps while recording the feature values. We used the orange (after-training)
distributions to generate realistic noise features in the experiments of
Section 7. Note that the distributions differ a lot depending on whether the
agent is trained or not; the non-stationarity of the data distribution in RL
is quite evident. See Figures 27, 28, 29, 30. The titles of the features are
taken from the environment’s documentation.111111See
https://www.gymlibrary.dev/environments/mujoco/.

Figure 27. Distributions of the 17 original features of HalfCheetah-v3, after
running 10K steps with a trained and an untrained model (ANF-TD3 on 90% noise
features). Note that the distributions tend to widen significantly after
training. Figure 28. Distribution of the 11 original features of Hopper-v3,
after running 10K steps with a trained and an untrained model (ANF-TD3 on 90%
noise features). Figure 29. Distribution of the 17 original features of
Walker2d-v3, after running 10K steps with a trained and an untrained model
(ANF-TD3 on 90% noise features). Figure 30. Distribution of the first 45
original features of Humanoid-v3. A figure showing histograms for all 376
dimensions is available for download online.