§ INTRODUCTION

Exploration is one of the essential challenges in reinforcement learning (RL). RL algorithms often use simple randomised methods, e.g. applying $\epsilon$-greedy policies <cit.>
or adding random noise to continuous actions <cit.>.
Such exploration techniques may be inefficient on tasks where rewards are sparse. One category of exploration techniques which has been found to be particularly effective in sparse-reward environments are intrinsic rewards <cit.>.
These additional rewards $r^i$ are computed by the agent and added to the extrinsic reward $r^e$ provided by the environment for a combined reward signal $r = r^e + \lambda r^i$ with some weighting factor $\lambda$. Intrinsic rewards incentivise the exploration of novel or underexplored parts of the environment commonly using self-supervised predictions in the environment <cit.> or (pseudo-) counts of states <cit.>.

Unfortunately, optimising for this combined feedback introduces three challenges. (1) Intrinsic rewards lead to non-stationary rewards as they are designed to diminish with more completed exploration. Such non-stationary reward shaping violates the Markov assumption and can cause the learning progress to be inconsistent. (2) Intrinsically-motivated exploration is sensitive to the scale $\lambda$. If intrinsic rewards are too large, they might heavily distort training and introduce non-stationary noise to the optimisation. On the other hand, we show that small intrinsic rewards have no sufficient impact and do not incentivise exploration as intended (<Ref>). (3) Intrinsically-motivated exploration is sensitive to the rate of decay which intrinsic rewards rely on throughout training. Similar to their scale, we show that slowly decaying intrinsic rewards disrupt training whereas quickly vanishing intrinsic rewards have insufficient impact on exploration (<Ref>).

These challenges lead to a significant dependency of intrinsic rewards on hyperparameters. Additionally, determining these hyperparameters for scale and rate of decay is task-dependent due to the scale of extrinsic rewards and required exploration in the respective task. Current approaches usually address the difficulties caused by sensitivity to hyperparameters using a large hyperparameter search to find effective parameterisation of a method. However, such a search can be considered an exploration by itself, and introduces bias in reported results focusing only on runs with best-identified hyperparameters and disregarding the considerable computational cost involved <cit.>. We argue that this bias is particularly harmful in approaches focusing on exploration as best-identified hyperparameters may steer exploration towards the solution of an environment, effectively shifting the achieved exploration from the proposed method to the hyperparameter search. All these properties make the practical application of such methods difficult <cit.> and motivate the need for more robust approaches.

Motivated by these challenges and success in off-policy RL <cit.>, we propose to separate the RL training into two separate policies. We train an exploration policy with the combined signal of extrinsic and intrinsic rewards. Simultaneously, we train an exploitation policy using only extrinsic rewards on the data collected by the exploration policy. We refer to this approach as Decoupled RL (DeRL)[We provide an implementation of DeRL and the Hallway environment at <https://github.com/uoe-agents/derl>].
Using such decoupling addresses challenges (1)–(3) of previous application of intrinsic rewards. The exploration policy is optimised using the combined objective of extrinsic and intrinsic rewards as in typical intrinsically-motivated RL, but it is only trained to generate data for the training of the exploitation policy. The exploitation policy is thereby decoupled from the challenges of training with intrinsic rewards and optimised to be an effective policy in the given environment. Our experiments show that DeRL leverages the benefits of intrinsically-motivated exploration while stabilising its inherent sensitivity to scale and rate of decay of intrinsic rewards.

We implement and evaluate two versions of DeRL built upon on-policy actor-critic and off-policy Q-learning with five types of intrinsic rewards <cit.> in two learning environments that focus on exploration. We analyse the sensitivity of DeRL and RL baselines to the scale and the rate of decay of intrinsic rewards to verify the general dependency of these methods on the hyperparameters of intrinsic rewards and show that DeRL is more robust to varying hyperparameters. Additionally, the exploitation policy of several DeRL algorithms is able to converge to higher evaluation returns using up to $\sim40\%$ fewer interactions and reaches higher returns in some tasks compared to intrinsically-motivated RL baselines. Such improved robustness and sample efficiency can justify the additional cost of training a second policy. However, we also observe that DeRL still suffers from variability in the off-policy optimisation of the exploitation policy in several tasks. We hypothesise that distribution shift caused by the divergence of and leads to these instabilities, and show that regularisers can be applied to restrict divergence of both policies <cit.>, reducing deviations in returns of exploration and exploitation policies and further improving robustness to hyperparameters of intrinsic rewards.

§ BACKGROUND

§.§ Markov Decision Process

We formulate an environment as a Markov Decision Process (MDP) <cit.> defined as a tuple $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$. $\mathcal{S}$ and $\mathcal{A}$ denote the sets of states and actions, respectively, and $\mathcal{P}: \mathcal{S} \times \mathcal{A} \mapsto \Delta(\mathcal{S})$ represents the transition function defining a probability distribution over the next state given current state and applied action. The agent receives rewards for a given transition following $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \mapsto \mathbb{R}$. The objective is to learn a policy $\pi: \mathcal{S} \mapsto \Delta(\mathcal{A})$ which maximises the expected discounted returns $\mathbb{E}_\pi \left[ \sum_{t=0}^\infty \gamma^t \mathcal{R}(s_t, a_t, s_{t+1}) \mid a_t \sim \pi(s_t) \right]$ with discount factor $\gamma \in [0, 1)$.

§.§ Intrinsically-Motivated Exploration

A variety of methods have been proposed to replicate the exploration incentive of curiosity in RL <cit.> guided by the idea that agents should be incentivised to explore novel or poorly understood parts of the environment. Therefore, intrinsic rewards are defined which reward the agent for such exploration. Over time, the agent should become less “curious” and exploitation will gradually take over.
There are two common branches of intrinsic rewards for exploration: (1) count-based and (2) prediction-based intrinsic rewards.

§.§.§ Count-based Intrinsic Rewards

Count-based intrinsic rewards are inverse proportional to the visitations of encountered states.

\begin{equation}
r_t^i \coloneqq \frac{1}{\sqrt{N(s_t)}}
\label{eq:count_curiosity}
\end{equation}

Thereby, agents are incentivised to visit states within the environment which are less frequently encountered. Likewise, agents are discouraged from visiting frequently encountered states which are deemed less valuable for exploration.
While this approach is easily applicable in small, discrete state spaces, pseudo-counts have to be computed for large or continuous state spaces where encountering any state multiple times is rare. These pseudo-counts can be computed using density models predicting visitations of states <cit.> or using locality-sensitive <cit.> hash functions <cit.>.

§.§.§ Prediction-based Intrinsic Rewards

A separate approach defines intrinsic rewards using predictions in the environment.
<cit.> proposed an intrinsic reward defined as the error of predicting the next state given the current state and action. However, stochastic and thereby unpredictable dynamics within the environment lead to the so-called “noisy TV problem” <cit.>, i.e. the exploration signal remains high in the presence of unpredictability, which remains a major challenge of these approaches.

Visualisation of Decoupled Reinforcement Learning (DeRL) training loop.

Intrinsic curiosity module (ICM): <cit.> propose to learn efficient state representations $\phi(s)$ and assign an intrinsic reward for the prediction error of the next state

\begin{equation}
\label{eq:icm_prediction}
r^i_t \coloneqq \left( \widehat{\phi}(s_{t+1}) - \phi(s_{t+1}) \right)^2
\end{equation}

where $\phi$ is a learned self-supervised state-representation trained using an inverse-dynamics objective: given a representation of the current state $\phi(s_t)$ and next state $\phi(s_{t+1})$ predict the applied action $a_t$. Through this representation, the model learns to encode information which can be affected by the agent's actions. The intrinsic reward is given by the error of the prediction $\widehat{\phi}(s_{t+1})$ of the next state given current state $s_t$ and applied action $a_t$.

Rewarding impact-driven exploration (RIDE): <cit.> propose to reward the agent for applying actions which lead to significant change in the environment. Such change is defined as the difference between embeddings of consecutive states, where the embedding function $\phi$ is trained using an inverse dynamics model identical to ICM <cit.>. In order to avoid the agent going back and forth between a group of states, an episodic state-count $N_{ep}$ is added to the objective.

\begin{equation}
\label{eq:ride_prediction}
r^i_t \coloneqq \frac{\left( \phi(s_{t+1}) - \phi(s_t) \right)^2}{\sqrt{N_{ep}(s_{t+1})}}
\end{equation}

Random network distillation (RND):
<cit.> propose a simplified prediction-based intrinsic reward which optimises a state representation function $\widehat{\phi}$ to mimic a randomly initialised, fixed target representation $\phi$.

\begin{equation}
\label{eq:rnd_prediction}
r^i_t \coloneqq \left( \widehat{\phi}(s_t) - \phi(s_t) \right)^2
\end{equation}

§ RELATED WORK

<cit.> propose a new objective for Meta-RL optimisation to decouple exploration and exploitation. Using such an objective, they train separate exploration and exploitation policies guided by task-specific information for fast adaptation to novel scenarios. For multi-agent RL, <cit.> learn separate exploration and exploitation policies using off-policy RL to focus coordinated exploration across multiple agents towards underexplored parts within the state space. However, a mixture of both policies is applied to explore whereas our work fully decouples both policies and their training. Furthermore, both of these approaches consider the meta-learning and multi-agent settings, respectively, and do not address the challenge of single-agent exploration we focus on.

Independently from our work, <cit.> propose to concurrently train an exploration policy using only intrinsic rewards and train a task policy using off-policy soft Double-DQN <cit.>. They apply a factored policy of both the task and exploration policies with optimisations focused on fast adaptation of the exploration policy. In contrast, we fully decouple both trained policies and find that training of the task policy using on-policy actor-critic algorithms with off-policy correction leads to higher returns and less sensitivity to hyperparameters in several tasks. Furthermore, we evaluate DeRL with several intrinsic rewards whereas <cit.> use a single count-based intrinsic reward which is also used for optimisitic initialisation <cit.>.

§ DECOUPLED REINFORCEMENT LEARNING

In this work, we propose to decouple exploration and exploitation into two separate policies to improve sample efficiency and reduce sensitivity to hyperparameters of intrinsic rewards. We train an exploration policy with the intent to explore the environment. Using the data collected by the exploration policy, we train a separate exploitation policy , as visualised in <Ref>. Separating exploration and exploitation in this way enables training of the exploration policy with intrinsic rewards without modifying the training objective of the exploitation policy.

Formally, an agent trains an exploration policy to maximise the sum of intrinsic and extrinsic rewards,

\begin{align}
\pi_\beta &\in \text{arg}\max_{\pi} \mathbb{E}\left[ \sum_{t=0}^\infty {\gamma}^t \left(r_t^e + \lambda r_{t}^{i} \right) \mid a_t \sim \pi(s_t) \right]\\
&= \text{arg}\max_{\pi} \mathbb{E}\left[G_{t}^{e+i} \mid a_t \sim \pi(s_t) \right]
\label{eq:derl_pibeh}
\end{align}

with $G^{e+i}$ denoting the discounted returns computed using the combination of extrinsic and intrinsic rewards with scaling factor $\lambda$ and discount factor $\gamma \in [0, 1)$. During training of , experience samples $(s_t, a_t, r_t^e, s_{t+1})$ with extrinsic rewards are collected in $\mathcal{D}$.

In addition to this typical intrinsically-motivated RL, we train a separate exploitation policy to maximise only expected cumulative extrinsic rewards using experience accumulated in $\mathcal{D}$ with $G^e_t$ denoting discounted extrinsic returns.

\begin{align}
\pi_e &\in \text{arg}\max_{\pi} \mathbb{E}\left[ \sum_{t=0}^\infty {\gamma}^t r_{t}^{e} \mid (s_t, a_t, r^e_t, s_{t+1}) \sim \mathcal{D} \right]\\
&= \text{arg}\max_{\pi} \mathbb{E}\left[G_{t}^{e} \mid (s_t, a_t, r^e_t, s_{t+1}) \sim \mathcal{D} \right]
\end{align}

Both exploration policy and exploitation policy can be trained using any RL algorithm given the defined objectives. We optimise the exploration policy as RL with intrinsic rewards <cit.>, whereas we train every $T_{Dec}$ timesteps on experience from $\mathcal{D}$ which is generated by 's interaction in the environment. Note that $\mathcal{D}$ only contains extrinsic rewards and is off-policy data for the optimisation of as it was generated by following . Therefore, training the exploitation policy using experience generated by the exploration policy requires off-policy RL. Off-policy RL is concerned with the optimisation of a policy using experience generated within the environment by following a separate behaviour policy.
Below, we propose two methods to apply such decoupled RL using an actor-critic and Q-learning framework.

§.§ Decoupled Actor-Critic

In order to use on-policy RL algorithms to train using $\mathcal{D}$, off-policy correction must be applied to account for differences in trajectory distributions of both and .

One technique for off-policy correction
is importance sampling (IS). In the following, we train using an on-policy actor-critic RL algorithm with state value function $V$, parameterised by $\theta$, and policy , parameterised by $\phi$. We optimise the latter by minimising the actor loss given in <Ref>

\begin{equation}
\label{eq:dec_policyloss}
\mathcal{L}(\phi) = \mathbb{E}\left[-\rho(a_t | s_t) \log \pi_e(a_t|s_t;\phi) \ A^e(s_t) \mid (s_t, a_t, r^e_t, s_{t+1}) \sim \mathcal{D}\right]
\end{equation}

with bootstrapped advantage estimates $A^e(s_t)$ and IS weights $\rho(a_t | s_t)$.

\begin{align}
A^e(s_t) &= \left(r^e_t + \gamma V(s_{t+1};\theta)-V(s_t;\theta)\right)\\
\rho(a_t | s_t) &= \frac{\pi_e(a_t|s_t; \phi)}{\pi_\beta(a_t|s_t)}
\end{align}

Similarly, the value loss for using IS weights can be defined as follows:

\begin{multline}
\label{eq:dec_valueloss}
\mathcal{L}(\theta) = \mathbb{E}\left[\rho(a_t | s_t) \ \left(V(s_t; \theta) - \left(r_t^e + \gamma V(s_{t+1};\theta)\right)\right)^2 \right.\\
\left. \mid (s_t, a_t, r^e_t, s_{t+1}) \sim \mathcal{D}\right]
\end{multline}

The IS weights, $\rho$, can cause inconsistent returns during off-policy training through exploding weights when $\pi_e(a_t|s_t; \phi) \gg \pi_\beta(a_t|s_t)$ or vanishing weights for $\pi_e(a_t|s_t; \phi) \ll \pi_\beta(a_t|s_t)$ <cit.>. In particular, such instabilities occur when one policy assigns approximately zero probability for some action. Various techniques have been proposed to address such exploding weights,
including clipping of importance weights <cit.> to minimise vanishing or exploding gradients. The pseudocode for Decoupled Actor-Critic optimisation of can be found in <Ref>.

Decoupled Actor-Critic

Initialise: parameters $\phi$, $\theta$ and $\mathcal{D} \gets \emptyset$
$i \gets 0$
$\text{ep} = 0, \ldots, N_{ep}$
$a_t \sim \pi_\beta(s_t)$
$s_{t+1}, r^e_t \gets$ environment step with $a_t$
Update $\pi_\beta$ using RL on intrinsic rewards (<Ref>)
$\mathcal{D} \gets \mathcal{D} \cup (s_t, a_t, r^e_t, s_{t+1})$
$i \gets i + 1$
$i \mod T_{Dec} = 0$
Update $\phi$ with <Ref> and $\mathcal{D}$
Update $\theta$ with <Ref> and $\mathcal{D}$
$\mathcal{D} \gets \emptyset$

§.§ Decoupled Deep Q-Networks

Instead of optimising using actor-critic algorithms with off-policy corrections, we can also apply off-policy algorithms such as Q-learning without the need for any correction. In this work, we consider optimising using Deep Q-Networks (DQN) <cit.>. For DQN optimisation, the following loss is minimised

\begin{align}
\mathcal{L}(\theta) = \mathbb{E}\left[\left(Q(s_t, a_t; \theta) - \widebar{Q}(s_t)\right)^2 \mid (s_t, a_t, r^e_t, s_{t+1}) \sim \mathcal{D}\right]
\label{eq:dec_dqnloss}
\end{align}

with target Q-values $\widebar{Q}(s_t)$ and $\widebar{\theta}$ denoting the parameters of the periodically updated target network.

\begin{equation}
\widebar{Q}(s_t) = (r_t^e + \gamma \max_{a'}Q(s_{t+1}, a';\widebar{\theta}))
\end{equation}

Pseudocode for Decoupled Deep Q-Networks of can be found in <Ref>. Note that $\mathcal{D}$ is only used for a single update in Decoupled Actor-Critic, whereas in Decoupled Deep Q-Networks $\mathcal{D}$ represents a replay buffer <cit.> which is continually filled with experience.

§ EVALUATION

We evaluate DeRL in two learning environments with a variety of RL algorithms and intrinsic rewards. In particular, we investigate the following three hypotheses: (1) Intrinsically-motivated RL is sensitive to varying scale $\lambda$ and rate of decay of intrinsic rewards, (2) DeRL is more robust than intrinsically-motivated RL baselines to varying scale and rate of decay, and (3) DeRL leads to similar or improved returns and sample efficiency compared to intrinsically-motivated RL baselines.

§.§ Algorithms

Baselines: As baselines, we consider on-policy RL algorithms Advantage Actor-Critic (A2C) <cit.> and Proximal Policy Optimisation (PPO) <cit.>. Both algorithms are trained using the combined reward $r_t = r^e_t + \lambda r^i_t$ with some weighting factor $\lambda$ and various intrinsic reward definitions as stated below.

DeRL: For our decoupled RL optimisation, we consistently train using A2C as we found it to be more robust than PPO. As intrinsic rewards, we use Count and ICM to train .
For the optimisation of , we consider A2C and PPO for Decoupled Actor-Critic and Decoupled Deep Q-Networks based on DQN. We refer to these algorithms as DeA2C, DePPO and DeDQN.

§.§ Intrinsic Rewards

Count-based: We consider two count-based intrinsic rewards computing intrinsic rewards following <Ref>. Count directly stores and increments state occurrences in a table. Hash-Count first groups states using the SimHash function <cit.>.

Prediction-based: Besides count-based intrinsic exploration definitions, we consider ICM <cit.>, RND <cit.>, and RIDE <cit.> as prediction-based approaches.
For details on these intrinsic rewards, see <Ref>.

Decoupled Deep Q-Networks

Initialise: parameters $\theta$ and $\mathcal{D} \gets \emptyset$
$i \gets 0$
$\text{ep} = 0, \ldots, N_{ep}$
$a_t \sim \pi_\beta (s_t)$
$s_{t+1}, r^e_t \gets$ environment step with $a_t$
Update $\pi_\beta$ using RL on intrinsic rewards (<Ref>)
$\mathcal{D} \gets \mathcal{D} \cup (s_t, a_t, r^e_t, s_{t+1})$
$i \gets i + 1$
$i \mod T_{Dec} = 0$
Update $\theta$ with <Ref> and $\mathcal{D}$

§.§ Environments

DeepSea is an environment proposed as part of the Behaviour Suite (Bsuite) for RL <cit.>, visualised in <Ref>. The environment targets the challenge of exploration and represents a $N \times N$ grid where the agent starts in the top left and has to reach a goal in the bottom right location. At each timestep, the agent moves one row down and can choose one out of two actions. For each row, both actions are randomly assigned to left and right movement. The agent observes the current location as a 2D one-hot encoding and receives a small negative reward of $\frac{-0.01}{N}$ for moving right and $0$ reward for moving left. Additionally, the agent receives a reward of $+1$ for reaching the goal and the episode ends after $N$ timesteps. The difficulty of the exploration in DeepSea can be adjusted using $N$: the larger $N$, the harder it becomes for the agent to reach the goal location for optimal returns of $0.99$. We evaluate all algorithms in the DeepSea task for $N \in \{10, 14, 20, 24, 30\}$.

DeepSea environment, from <cit.>.

Hallway is a new environment proposed as part of this work to represent domains where exploration and exploitation are misaligned,
visualised in <Ref>. In DeepSea, agents receive reward by reaching states at the end of the environment, so intrinsic rewards for exploration strongly align with extrinsic rewards from the environment. We hypothesise that tasks in which intrinsic and extrinsic rewards are not well aligned require carefully balanced exploration through intrinsic rewards. Motivated by this hypothesis, we design the Hallway environment in which an agent is located in a hallway starting on the left. A goal can be reached by moving $N_l$ cells to the right. In contrast to DeepSea, the goal is not necessarily located at the right end of the hallway, but there might be further $N_r$ empty cells to the right of the goal location. At each timestep, the agent can choose between three actions: move left, stay or move right. The agent receives a reward of $+1$ for reaching the goal for the first time and every time it stays at the goal location for $10$ steps. Therefore, the agent needs to learn to move to the goal and stay there for the remaining timesteps of the episode to collect further reward. Episodes end after $2 N_l$ steps and small negative reward of $-0.01$ is assigned for moving right or stay. Hallway tasks, in particular with $N_r > 0$, require exploration through intrinsic rewards to be carefully balanced because staying at the goal for optimal returns and exploration are not aligned.
We evaluate all algorithms in the Hallway environment with $N_l \in \{10, 20, 30\}$ and $N_r$ either being $0$ or equal to $N_l$.

§.§ Implementation Details

We compute n-step returns <cit.> to reduce the bias of value estimates in all algorithms. On-policy training uses four parallel, synchronous environments and an additional entropy regularisation term in the policy loss <cit.>. Double-DQN <cit.> targets are computed for DQN.
For details on the conducted hyperparameter search as well as all values used throughout experiments, see <Ref>[All appendices are available online at <https://arxiv.org/pdf/2107.08966.pdf>.].

We train all algorithms for 100,000 episodes and evaluate every 1,000 episodes for a total of 100 evaluations by applying the greedy (evaluation) policy in the respective task for 8 episodes. Following recent suggestions for evaluation in deep RL <cit.>, we report averaged evaluation returns and stratified bootstrap $95\%$ confidence intervals using 5,000 samples for the bootstrapping across five random seeds. Optimal returns are indicated using a dashed horizontal line. A weighting factor of $\lambda=1$ is used for the combined reward signal unless stated otherwise.

Hallway environment.

DeepSea 10

Hallway $N_l=N_r=10$

Average evaluation returns in DeepSea 10 and Hallway $N_l=N_r=10$ with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals. A method that is insentive to hyperparameters will have final average episodic return concentrated to the right for all hyperparameter values.

DeepSea 10

Hallway $N_l=N_r=10$

Average evaluation returns in DeepSea 10 and Hallway $N_l=N_r=10$ with varying rates of decay. Shading indicates 95% confidence intervals.

§.§ Hyperparameter Sensitivity

To investigate our first two hypotheses, we
train all baselines and DeRL algorithms on the combined reward $r = r^e + \lambda r^i$ in DeepSea $N=10$ and Hallway $N_l = N_r = 10$ with varying $\lambda$ and rates of decay. We confirm our first hypothesis that intrinsically-motivated RL is indeed highly sensitive to scale and decay rate of intrinsic rewards, not learning at all or reaching significantly lower evaluation returns for many hyperparameter values. In particular in the Hallway environment where exploration and extrinsic rewards are misaligned, all algorithms exhibit significant dependency on carefully tuned scale and rate of decay. We further confirm our second hypothesis that DeRL algorithms are more robust to varying scale and decay rate of intrinsic rewards, reaching higher returns across a wider range of hyperparameters. <Ref> show sensitivity of all baselines and DeRL algorithms with Count and ICM. Sensitivity analysis for all remaining intrinsic rewards can be found in <Ref>.

Scale of intrinsic rewards:
We consider $\lambda \in \{0.01, 0.1, 0.25, 0.5,\allowbreak 1.0, 2.0, 4.0, 10.0, 100.0\}$ to analyse the sensitivity to varying scale of intrinsic rewards. <Ref> shows average evaluation returns for all values of $\lambda$ for baselines and DeRL. Average returns and bootstrap confidence intervals are computed across five seeds before the average over all 100 evaluations is computed. In DeepSea $N=10$, DeA2C and DeDQN exhibit improved robustness by reaching close to optimal returns for almost all values of $\lambda$. In contrast, DePPO and the baselines are found to be more sensitive in particular to large values of $\lambda$. In Hallway, all algorithms exhibit larger variance for varying $\lambda$ compared to DeepSea with no significant learning being observed for large or small values of $\lambda$, with DeA2C and DePPO demonstrating slightly more robustness. These results indicate the sensitivity to values of $\lambda$. Even small deviations can make the difference between learning and not learning at all.

Decay of intrinsic rewards:
We also investigate the sensitivity of intrinsically-motivated baselines and DeRL algorithms to the rate of decay of intrinsic rewards. For count-based intrinsic rewards, the rate of decay can be determined by the increment of the state count $N(s)$. For a sensitivity analysis, we consider increments $\{0.01, 0.1, 0.2, 1.0, 5.0, 10.0, 100.0\}$. For deep prediction-based intrinsic rewards, we consider learning rates  $\{1e^{-9}, 1e^{-8}, 2e^{-8}, 1e^{-7}, 5e^{-7},\allowbreak 1e^{-6}, 1e^{-5}, 1e^{-4}, 1e^{-3}\}$ determining the rate of decay.
<Ref> shows average evaluation returns of baselines and DeRL with varying rates of decay in both DeepSea $N=10$ and Hallway $N_l=N_r=10$.
A2C is shown to be more robust to varying rates of decay in both environments compared to PPO. DePPO demonstrates larger sensitivity compared to A2C, but DeA2C and DeDQN are again shown to be the most robust algorithms, especially with Count intrinsic rewards, exhibiting high evaluation returns for most considered values in DeepSea $N=10$. Similar to $\lambda$ sensitivity, we observe very significant dependency on the rate of decay in the Hallway task with DeA2C exhibiting improved robustness to varying values.

Normalised evaluation returns for DeepSea (left) and Hallway (right). Returns for each task are normalised to be within [0, 1] before averaged returns and 95% confidence intervals are computed across all tasks and five random seeds.

Evaluation returns and log IS weights for DeA2C Count in DeepSea $N=14$.

Average evaluation returns and a single standard deviation in all DeepSea and Hallway tasks over 100,000 episodes. The highest achieved returns in each task are highlighted in bold together with all returns within a single standard deviation. For DeRL algorithms, evaluations are executed using the exploitation policy.

Alg     DeepSea 10     DeepSea 14     DeepSea 20     DeepSea 24     DeepSea 30     Hallway 10-0     Hallway 10-10     Hallway 20-0     Hallway 20-20     Hallway 30-0     Hallway 30-30

A2C     0.93(22)      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)     0.67(5)      0.49(9)      0.42(2)      0.50(3)      0.28(8)      0.42(8)

A2C Count     0.98(7)     0.94(16)     0.74(10)      0.11(15)      -0.01(0)     0.85(1)     0.85(2)      0.61(3)     0.55(6)      -0.33(15)      -0.06(7)

A2C Hash-Count     0.98(7)     0.96(15)      0.39(14)     0.53(12)      -0.01(0)     0.85(1)     0.85(3)      0.56(3)     0.55(6)      -0.34(15)      -0.13(11)

A2C ICM      0.87(20)      0.69(31)      0.54(23)     0.46(30)     0.08(12)     0.62(17)      0.57(17)      0.27(12)     0.78(27)     1.16(47)     0.64(38)

A2C RND      0.06(1)      0.19(2)      -0.01(0)      -0.01(0)      -0.01(0)     -0.12(2)      -0.07(0)      -0.20(1)      -0.24(0)      -0.24(1)      -0.12(0)

A2C RIDE      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)     0.85(4)     0.85(2)     0.70(0)     0.62(0)      0.37(4)      0.28(8)

PPO      0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)     0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

PPO Count      0.84(10)      0.70(17)      0.46(19)      0.17(18)     0.20(15)     -0.00(1)      -0.00(0)      -0.00(1)      -0.00(1)      -0.00(0)      -0.00(1)

PPO Hash-Count      0.86(8)      0.77(13)      0.34(14)      0.28(20)     0.12(13)     0.39(11)      0.10(7)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

PPO ICM      0.84(17)      0.28(17)      -0.00(3)      0.12(17)      -0.00(3)     0.05(15)      0.11(15)      0.02(16)      0.08(19)      -0.04(8)      -0.02(14)

PPO RND      0.26(12)      0.15(8)      -0.01(0)      -0.00(0)      -0.01(0)     -0.04(4)      -0.04(11)      -0.21(6)      -0.17(9)      -0.27(10)      -0.27(11)

PPO RIDE      0.73(8)      -0.00(0)      -0.00(2)      -0.01(0)      -0.01(0)      -0.10(3)      0.02(8)      -0.21(3)      -0.08(8)      -0.32(4)      -0.29(8)

DeA2C Count     0.98(10)      0.65(23)      0.42(16)      0.07(10)     0.09(8)     0.84(7)     0.84(9)      0.42(2)     0.70(1)      0.55(0)      0.22(2)

DeA2C ICM      0.86(19)      0.52(28)      0.27(24)      0.08(14)     0.05(11)     0.77(18)      0.80(17)      0.44(15)     0.53(20)      0.52(34)     0.97(51)

DePPO Count      0.61(20)     0.92(18)      -0.01(1)     0.63(27)      -0.01(0)     0.73(10)      0.80(8)      0.56(1)     0.55(4)      -0.20(17)      -0.06(7)

DePPO ICM      0.61(18)      0.37(17)      -0.00(1)      -0.01(0)      -0.00(0)      0.82(11)      0.81(11)      0.64(16)     0.57(7)      -0.01(25)      0.26(6)

DeDQN Count     0.98(9)     0.95(17)      0.40(8)     0.53(27)     0.10(10)     -0.13(4)      -0.15(4)      -0.05(5)      -0.12(8)      -0.17(7)      -0.10(6)

DeDQN ICM     0.94(20)      0.59(40)      0.16(12)      0.24(25)     0.05(12)     -0.09(9)      0.02(16)      -0.11(9)      -0.19(8)      -0.26(8)      -0.19(8)

§.§ Evaluation Returns

Lastly, we report evaluation returns of all algorithms across all DeepSea and Hallway tasks in <Ref>. Average returns and standard deviations are computed across all 100 evaluations after being averaged across five seeds to indicate achieved returns as well as sample efficiency. Additionally, we present normalised returns with 95% confidence intervals across both environments in <Ref>, and tables with maximum achieved evaluation returns at any evaluation and learning curves for each individual task in <Ref>.

In DeepSea, DeDQN performs best out of all algorithms (<Ref>). DeDQN converges to returns comparable to or higher than the best performing baselines exhibiting highest average evaluation returns in all tasks but DeepSea $20$. DeA2C and DePPO demonstrate similar returns and sample efficiency in some of these tasks (see <Ref>). In DeepSea $24$ (<Ref>) and harder Hallway tasks with $N_l = 20, N_r = 0$ and $N_l = N_r = 30$ (<Ref>), the exploitation policies of DeA2C and DePPO converge to the highest returns and are shown to be more sample efficient reaching high returns after up to $40\%$ fewer episodes of training compared to the best performing baselines.
Generally, we can see that DeA2C learns the optimal policy in the majority of Hallway tasks for some of the five executed runs, but fails to converge to such behaviour consistently. Instead, the majority of baselines and some DeRL runs learn to reach the goal but move back and forth between the goal and its left neighboured cell. Presumably, consistently staying at the goal is rarely discovered due to the small negative reward of staying at a cell.

However, we also observe some failure cases for DeRL algorithms.
DeDQN achieves low returns in the Hallway environment compared to both on-policy DeA2C and DePPO. Also, significant variance can be observed for baselines and DeRL algorithms in harder DeepSea and most Hallway tasks. Off-policy optimisation is theoretically independent of the policy generating training samples, and in DeA2C and DePPO IS weight correction is applied to correct for the off-policy training data. However, we believe distribution shift <cit.> is causing inconsistent returns when optimising the exploitation policy from data generated by .
<Ref> visualises unstable IS weights for DeA2C in the DeepSea task with $N = 14$ averaged over five seeds. These appear to correlate with some of the noticeable drops in returns throughout training, indicating the negative impact of divergence of exploration and exploitation policies on RL training of . Even when applying Retrace($\lambda$) <cit.> to clip IS weights, similar results are observed.

DeepSea $N=10$

Hallway $N_l=N_r=20$

Evaluation and training returns, IS weights and KL-divergence of exploration and exploitation policy for training of DeA2C with Count in (a) DeepSea 10 and (b) Hallway $N_l=N_r=20$ with intrinsic and extrinsic rewards or only intrinsic rewards as training signal for , and with KL-divergence constraints with coefficients $\alpha_\beta$ and $\alpha_e$. Evaluation and training returns are achieved using the exploitation and exploration policies, respectively. Shading indicates 95% confidence intervals.

§ EXPLORATION USING ONLY INTRINSIC REWARDS

Prior work on intrinsic rewards for exploration investigated the effectiveness of training using only intrinsic rewards and no extrinsic rewards from the environment <cit.>.
Motivated by these experiments, we also investigate the possibility of optimising the exploration policy using only intrinsic rewards. Such optimisation would likely lead to increased robustness to hyperparameters of intrinsic rewards as they would not be combined with extrinsic rewards of the environment. However, we also find that the optimisation of without extrinsic rewards causes further divergence of and . It should be noted that the evaluation policy is still trained using extrinsic rewards.

We conduct experiments in the DeepSea 10 and Hallway $N_l=N_r=20$ tasks training DeA2C with Count intrinsic rewards for 20,000 episodes. Results are averaged across three seeds and we directly compare optimising using either the sum of extrinsic and intrinsic rewards (orange) or only using intrinsic rewards (blue) in <Ref>. We find that training the exploration policy with only intrinsic rewards does lead to increased divergence of both policies seen in IS weights (top right) and the KL divergence (bottom right). Training of the exploitation policy appears to suffer from such differences in the more challenging Hallway task, but in DeepSea the exploitation policy was successfully trained to solve the task despite the exploration policy never reaching high returns. Our results show the feasibility of training using only intrinsic rewards but also the challenge of increased distribution shift.

§ DIVERGENCE CONSTRAINTS

In order to address distribution shift caused by diverging exploration and exploitation policies, we investigate the application of divergence constraints proposed in the literature of offline RL <cit.>.
These auxiliary objectives are introduced to the optimisation and enforce and to not diverge significantly by introducing a term $\alpha D(\pi_e, \pi_\beta)$
to the optimisation loss. This term is based on a distance measure $D$ between the distribution of policies and and some weighting hyperparameter $\alpha$.
A common distance measure is the Kullback-Leibler (KL) divergence, which has been applied in offline RL <cit.>, which can be written as following.

\begin{equation}
D_{KL} (\pi_e(s_t), \pi_\beta(s_t)) = \mathbb{E}_{a \sim \pi_e(\cdot | S)} \left[ \log \pi_e(a|s_t) - \log \pi_\beta(a|s_t) \right]
\label{eq:kl_divergence}
\end{equation}

For more distance measure candidates between two policies, see <cit.> which found these metrics to perform comparably.

In DeRL, divergence constraints can be directly applied to the optimisation of either the exploration or exploitation policy , i.e. can choose to keep close to and likewise can enforce to stay close to .
We consider either of these directions as well as a combination of both constraints.

We evaluate the application of KL constraints as regularisers in the policy loss of the exploration policy, $\alpha_\beta D_{KL}(\pi_\beta, \pi_e)$, and exploitation policy, $\alpha_e D_e(\pi_e, \pi_\beta)$, with varying weights $\alpha_\beta$ and $\alpha_e$, respectively. These constraints are applied in both settings introduced in <Ref> with being optimised using only intrinsic (green) or intrinsic and extrinsic rewards (red) with results shown in <Ref> for selected constraint coefficients. We find KL divergence constraints successfully address distribution shift and thereby keep both policies close to each other, even if is only trained using intrinsic rewards. Such minimised divergence also leads to reduced variability of returns in both tasks. These results indicate the feasibility of training using only intrinsic rewards and the effectiveness of divergence constraints to minimise distribution shift. We further evaluate the sensitivity of DeA2C with KL divergence constraints and show that such regularisation can improve robustness. For figures showing distribution shift, training and evaluation returns for a range of KL constraint coefficients, $\alpha_\beta$ and $\alpha_e$ as well the conducted sensitivity analysis, see <Ref>.

§ CONCLUSION

In this work, we proposed Decoupled RL (DeRL) which decouples exploration and exploitation into two separate policies. DeRL optimises the exploration policy with additional intrinsic rewards to incentivise exploration and trains the exploitation policy using only extrinsic rewards from data generated by the exploration policy.
Based on this general framework, we formulate Decoupled Actor-Critic and Decoupled Deep Q-Networks and evaluated in two sparse-reward environments.
Our results demonstrate that intrinsically-motivated RL is highly dependent on careful hyperparameter tuning of intrinsic rewards, indicating the need for more robust solutions. We show that decoupling exploration and exploitation is possible and does lead to significant benefits in robustness to varying scale and rate of decay of intrinsic rewards. Furthermore, we identify distribution shift as a challenge in separating the RL optimisation into two policies with separate optimisation objectives and investigate the application of divergence constraints to minimise such divergence of both policies. Our results demonstrate the effectiveness of divergence constraint regularisation and indicate improved sample efficiency of DeRL in some tasks by reaching high returns in fewer interactions in the environment.
Lastly, we demonstrated the feasibility of training the exploration policy using only intrinsic rewards. Alongside divergence constraints, such a training setting seems a promising directions for further research into decoupled exploitation and exploration.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Agarwal, Schwarzer, Castro, Courville, and
BellemareAgarwal et al2021]
authorpersonRishabh Agarwal, personMax
Schwarzer, personPablo Samuel Castro, personAaron
Courville, and personMarc G Bellemare.
Deep reinforcement learning at the edge of the
statistical precipice. In booktitleAdvances on Neural
Information Processing Systems.

[Andoni and IndykAndoni and Indyk2008]
authorpersonAlexandr Andoni and
personPiotr Indyk. year2008.
Near-optimal hashing algorithms for approximate
nearest neighbor in high dimensions.
journalCommun. ACM volume51,
number1 (year2008), pages117.

authorpersonAndrew G Barto.
Intrinsic motivation and reinforcement learning.
In booktitleIntrinsically Motivated Learning in
Natural and Artificial Systems. publisherSpringer,

[Bellemare, Srinivasan, Ostrovski, Schaul,
Saxton, and MunosBellemare et al2016]
authorpersonMarc Bellemare, personSriram
Srinivasan, personGeorg Ostrovski, personTom Schaul,
personDavid Saxton, and personRemi Munos.
Unifying count-based exploration and intrinsic
motivation. In booktitleAdvances on Neural Information
Processing Systems. pages1471–1479.

[Burda, Edwards, Pathak, Storkey, Darrell, and
EfrosBurda et al2018]
authorpersonYuri Burda, personHarri
Edwards, personDeepak Pathak, personAmos Storkey,
personTrevor Darrell, and personAlexei A Efros.
Large-scale study of curiosity-driven learning.
journalarXiv preprint arXiv:1808.04355

[Burda, Edwards, Storkey, and KlimovBurda
et al2019]
authorpersonYuri Burda, personHarrison
Edwards, personAmos Storkey, and personOleg
Klimov. year2019.
Exploration by random network distillation. In
booktitleInternational Conference on Learning

[Chentanez, Barto, and SinghChentanez
et al2005]
authorpersonNuttapong Chentanez,
personAndrew G Barto, and personSatinder P Singh.
Intrinsically motivated reinforcement learning. In
booktitleAdvances on Neural Information Processing
Systems. pages1281–1288.

[Christianos, Schäfer, and
AlbrechtChristianos et al2020]
authorpersonFilippos Christianos,
personLukas Schäfer, and personStefano V.
Albrecht. year2020.
Shared Experience Actor-Critic for Multi-Agent
Reinforcement Learning. In booktitleAdvances in Neural
Information Processing Systems, Vol. volume33.

[Degris, White, and SuttonDegris
et al2012]
authorpersonThomas Degris, personMartha
White, and personRichard S Sutton.
Off-policy actor-critic. In
booktitleInternational Conference on Machine Learning.

[Espeholt, Soyer, Munos, Simonyan, Mnih, Ward,
Doron, Firoiu, Harley, Dunning, et alEspeholt
et al2018]
authorpersonLasse Espeholt, personHubert
Soyer, personRemi Munos, personKaren Simonyan,
personVlad Mnih, personTom Ward,
personYotam Doron, personVlad Firoiu,
personTim Harley, personIain Dunning,
et al year2018.
Impala: Scalable distributed deep-rl with
importance weighted actor-learner architectures. In
booktitleInternational Conference on Machine Learning.
PMLR, pages1407–1416.

[Flet-Berliac, Ferret, Pietquin, Preux, and
GeistFlet-Berliac et al2021]
authorpersonYannis Flet-Berliac, personJohan
Ferret, personOlivier Pietquin, personPhilippe
Preux, and personMatthieu Geist.
Adversarially Guided Actor-Critic. In
booktitleInternational Conference on Learning

[Fujimoto, Van Hoof, and MegerFujimoto
et al2018]
authorpersonScott Fujimoto, personHerke
Van Hoof, and personDavid Meger.
Addressing function approximation error in
actor-critic methods. In booktitleInternational Conference
on Machine Learning. PMLR, pages1587–1596.

[Haarnoja, Zhou, Abbeel, and LevineHaarnoja
et al2018]
authorpersonTuomas Haarnoja, personAurick
Zhou, personPieter Abbeel, and personSergey
Levine. year2018.
Soft actor-critic: Off-policy maximum entropy deep
reinforcement learning with a stochastic actor. In
booktitleInternational Conference on Machine Learning.
PMLR, pages1861–1870.

authorpersonRonald A Howard.
Dynamic programming and Markov processes.

[Jaques, Ghandeharioun, Shen, Ferguson,
Lapedriza, Jones, Gu, and PicardJaques et al2019]
authorpersonNatasha Jaques, personAsma
Ghandeharioun, personJudy Hanwen Shen, personCraig
Ferguson, personAgata Lapedriza, personNoah Jones,
personShixiang Gu, and personRosalind Picard.
Way off-policy batch deep reinforcement learning of
implicit human preferences in dialog.
journalarXiv preprint arXiv:1907.00456

[Levine, Kumar, Tucker, and FuLevine
et al2020]
authorpersonSergey Levine, personAviral
Kumar, personGeorge Tucker, and personJustin Fu.
Offline reinforcement learning: Tutorial, review,
and perspectives on open problems.
journalarXiv preprint arXiv:2005.01643

[Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa,
Silver, and WierstraLillicrap et al2016]
authorpersonTimothy P Lillicrap,
personJonathan J Hunt, personAlexander Pritzel,
personNicolas Heess, personTom Erez,
personYuval Tassa, personDavid Silver, and
personDaan Wierstra. year2016.
Continuous control with deep reinforcement
learning. In booktitleInternational Conference on Learning

authorpersonLong-Ji Lin.
booktitleReinforcement learning for robots using
neural networks.
publisherCarnegie Mellon University.

[Liu, Raghunathan, Liang, and FinnLiu
et al2021b]
authorpersonEvan Z Liu, personAditi
Raghunathan, personPercy Liang, and personChelsea
Finn. year2021b.
Decoupling Exploration and Exploitation for
Meta-Reinforcement Learning without Sacrifices. In
booktitleInternational Conference on Machine Learning.
PMLR, pages6925–6935.

[Liu, Jain, Yeh, and SchwingLiu
et al2021a]
authorpersonIou-Jen Liu, personUnnat Jain,
personRaymond A Yeh, and personAlexander Schwing.
Cooperative exploration for multi-agent deep
reinforcement learning. In booktitleInternational
Conference on Machine Learning. PMLR, pages6826–6836.

[Mnih, Badia, Mirza, Graves, Lillicrap, Harley,
Silver, and KavukcuogluMnih et al2016]
authorpersonVolodymyr Mnih,
personAdria Puigdomenech Badia, personMehdi Mirza,
personAlex Graves, personTimothy Lillicrap,
personTim Harley, personDavid Silver, and
personKoray Kavukcuoglu. year2016.
Asynchronous methods for deep reinforcement
learning. In booktitleInternational Conference on Machine
Learning. pages1928–1937.

[Mnih, Kavukcuoglu, Silver, Rusu, Veness,
Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie,
Sadik, Antonoglou, King, Kumaran, Wierstra, Legg, and HassabisMnih
et al2015]
authorpersonVolodymyr Mnih, personKoray
Kavukcuoglu, personDavid Silver, personAndrei A.
Rusu, personJoel Veness, personMarc G. Bellemare,
personAlex Graves, personMartin Riedmiller,
personAndreas K. Fidjeland, personGeorg Ostrovski,
personStig Petersen, personCharles Beattie,
personAmir Sadik, personIoannis Antonoglou,
personHelen King, personDharshan Kumaran,
personDaan Wierstra, personShane Legg, and
personDemis Hassabis. year2015.
Human-level control through deep reinforcement
journalNature volume518,
number7540 (year2015), pages529–533.

[Munos, Stepleton, Harutyunyan, and
BellemareMunos et al2016]
authorpersonRémi Munos, personTom
Stepleton, personAnna Harutyunyan, and personMarc G
Bellemare. year2016.
Safe and efficient off-policy reinforcement
learning. In booktitleAdvances on Neural Information
Processing Systems, Vol. volume29.

[Osband, Doron, Hessel, Aslanides, Sezener,
Saraiva, McKinney, Lattimore, Szepesvari, Singh, et alOsband
et al2020]
authorpersonIan Osband, personYotam Doron,
personMatteo Hessel, personJohn Aslanides,
personEren Sezener, personAndre Saraiva,
personKatrina McKinney, personTor Lattimore,
personCsaba Szepesvari, personSatinder Singh,
et al year2020.
Behaviour suite for reinforcement learning. In
booktitleInternational Conference on Learning

[Ostrovski, Bellemare, van den Oord, and
MunosOstrovski et al2017]
authorpersonGeorg Ostrovski, personMarc G
Bellemare, personAäron van den Oord, and
personRémi Munos. year2017.
Count-based exploration with neural density
models. In booktitleInternational Conference on Machine
Learning. JMLR, pages2721–2730.

[Oudeyer and KaplanOudeyer and
authorpersonPierre-Yves Oudeyer and
personFrederic Kaplan. year2009.
What is intrinsic motivation? A typology of
computational approaches.
journalFrontiers in Neurorobotics
volume1 (year2009).

[Oudeyer, Kaplan, et alOudeyer
et al2008]
authorpersonPierre-Yves Oudeyer,
personFrederic Kaplan, et al
How can we define intrinsic motivation. In
booktitleConference on Epigenetic Robotics,
Vol. volume5. pages29–31.

[Oudeyer, Kaplan, and HafnerOudeyer
et al2007]
authorpersonPierre-Yves Oudeyer,
personFrdric Kaplan, and personVerena V Hafner.
Intrinsic motivation systems for autonomous mental
journalIEEE transactions on Evolutionary
Computation  volume11 (year2007),

[Papoudakis, Christianos, Schäfer, and
AlbrechtPapoudakis et al2021]
authorpersonGeorgios Papoudakis,
personFilippos Christianos, personLukas Schäfer,
and personStefano V. Albrecht.
Benchmarking Multi-Agent Deep Reinforcement
Learning Algorithms in Cooperative Tasks. In
booktitleAdvances in Neural Information Processing Systems,
Track on Datasets and Benchmarks.

[Pathak, Agrawal, Efros, and DarrellPathak
et al2017]
authorpersonDeepak Pathak, personPulkit
Agrawal, personAlexei A Efros, and personTrevor
Darrell. year2017.
Curiosity-driven exploration by self-supervised
prediction. In booktitleInternational Conference on Machine

[Raileanu and RocktäschelRaileanu and
authorpersonRoberta Raileanu and personTim
Rocktäschel. year2020.
RIDE: Rewarding impact-driven exploration for
procedurally-generated environments. In
booktitleInternational Conference on Learning

[Rashid, Peng, Böhmer, and WhitesonRashid
et al2020]
authorpersonTabish Rashid, personBei Peng,
personWendelin Böhmer, and personShimon
Whiteson. year2020.
Optimistic Exploration even with a Pessimistic
Initialisation. In booktitleInternational Conference on
Learning Representations.

authorpersonJürgen Schmidhuber.
Curious model-building control systems.
journalIEEE International Joint Conference on Neural
Networks  volume2 (year1991),

[Schulman, Wolski, Dhariwal, Radford, and
KlimovSchulman et al2017]
authorpersonJohn Schulman, personFilip
Wolski, personPrafulla Dhariwal, personAlec Radford,
and personOleg Klimov. year2017.
Proximal policy optimization algorithms.
journalarXiv preprint arXiv:1707.06347

[Silver, Lever, Heess, Degris, Wierstra, and
RiedmillerSilver et al2014]
authorpersonDavid Silver, personGuy Lever,
personNicolas Heess, personThomas Degris,
personDaan Wierstra, and personMartin Riedmiller.
Deterministic policy gradient algorithms. In
booktitleInternational Conference on Machine Learning.
PMLR, pages387–395.

[Strehl and LittmanStrehl and
authorpersonAlexander L Strehl and
personMichael L Littman. year2008.
An analysis of model-based interval estimation for
Markov decision processes.
journalJ. Comput. System Sci.
volume74 (year2008), pages1309–1331.

[Sutton and BartoSutton and Barto2018]
authorpersonRichard S Sutton and
personAndrew G Barto. year2018.
booktitleReinforcement learning: An introduction.
publisherMIT press.

[Taïga, Fedus, Machado, Courville, and
BellemareTaïga et al2019]
authorpersonAdrien Ali Taïga,
personWilliam Fedus, personMarlos C Machado,
personAaron Courville, and personMarc G Bellemare.
Benchmarking bonus-based exploration methods on the
arcade learning environment.
journalarXiv preprint arXiv:1908.02388

[Tang, Houthooft, Foote, Stooke, Chen, Duan,
Schulman, DeTurck, and AbbeelTang et al2017]
authorpersonHaoran Tang, personRein
Houthooft, personDavis Foote, personAdam Stooke,
personOpenAI Xi Chen, personYan Duan,
personJohn Schulman, personFilip DeTurck, and
personPieter Abbeel. year2017.
# Exploration: A study of count-based exploration
for deep reinforcement learning. In booktitleAdvances on
Neural Information Processing Systems. pages2753–2762.

[van Hasselt, Guez, and Silvervan Hasselt
et al2016]
authorpersonHado van Hasselt, personArthur
Guez, and personDavid Silver.
Deep Reinforcement Learning with Double
Q-Learning. In booktitleAAAI Conference on Artificial
Intelligence. publisherAAAI Press, pages2094–2100.

authorpersonChristopher John Cornish Hellaby Watkins.
titleLearning from delayed rewards.
thesistypePh.D. Dissertation. schoolKing's
College, Cambridge.

[Whitney, Bloesch, Springenberg, Abdolmaleki,
and RiedmillerWhitney et al2021]
authorpersonWilliam F Whitney, personMichael
Bloesch, personJost Tobias Springenberg, personAbbas
Abdolmaleki, and personMartin Riedmiller.
Decoupled Exploration and Exploitation Policies for
Sample-Efficient Reinforcement Learning.
journalarXiv preprint arXiv:2101.09458

[Wu, Tucker, and NachumWu
et al2019]
authorpersonYifan Wu, personGeorge Tucker,
and personOfir Nachum. year2019.
Behavior regularized offline reinforcement
journalarXiv preprint arXiv:1911.11361

[Zhong, Hanna, Schäfer, and AlbrechtZhong
et al2021]
authorpersonRujie Zhong, personJosiah P.
Hanna, personLukas Schäfer, and personStefano V.
Albrecht. year2021.
Robust On-Policy Data Collection for Data-Efficient
Policy Evaluation. In booktitleOffline Reinforcement
Learning Workshop at Neural Information Processing Systems Conference.

§ HYPERPARAMETER CONFIGURATION

A gridsearch was conducted to identify the best hyperparameter configuration for each algorithm in both environments. In order to keep the search feasible, a hyperparameter search for DeepSea was conducted in DeepSea $N=20$ and the same identified configuration was applied in all other DeepSea tasks. Similarly, for the Hallway environment a hyperparameter search was conducted in Hallway $N_l = N_r = 10$. Every gridsearch configuration is evaluated using three different random seeds with results being averaged across all seeds.

§.§ Baseline Hyperparameters

First, the best hyperparameters for A2C and PPO baselines were identified by training each configuration in the respective training task of both environments with Count intrinsic rewards. We chose to conduct the hyperparameter search with intrinsic rewards as both baselines do perform poorly in most tasks without any intrinsic rewards which make different hyperparameter configurations hardly distinguishable. The Count intrinsic reward was chosen as it does not require any hyperparameter tuning in contrast to other intrinsic reward definitions. For all considered hyperparameter combinations, we conduct training in the same way as described in <Ref> with $\lambda = 1$. The best hyperparameter configuration is chosen as the one which leads to highest maximum evaluation returns. If multiple configurations reach the same maximum evaluation returns then the one with the highest average evaluation returns computed over all 100 evaluations throughout training is considered the best. Such latter metric considers achieved returns alongside sample efficiency. Identified hyperparameters for A2C and PPO in both environments can be found in <Ref> and <Ref>. All considered values of hyperparameters are listed with the best-identified combination highlighted in bold.

Hyperparameters for A2C baseline.

Hyperparameter     DeepSea     Hallway

Normalise observations          False, True                                  False, True

Normalise rewards               False, True                                  False, True

Learning rate                   $3e^{-4}, \mathbf{1e^{-3}}$                           $\mathbf{3e^{-4}}, 1e^{-3}$

Nonlinearity                    Tanh, ReLU                                   Tanh, ReLU

Maximum gradient norm           $\mathbf{0.5}, 10.0, 40.0$                            $\mathbf{0.5}, 10.0, 40.0$

Entropy loss coefficient        $\mathbf{1e^{-4}}, 3e^{-4}, 7e^{-4}, 1e^{-3}$         $\mathbf{1e^{-4}}, 3e^{-4}, 7e^{-4}, 1e^{-3}$

Actor architecture              Fully connected (64, 64)                              Fully connected (64, 64)

Critic architecture             Fully connected (64, 64)                              Fully connected (64, 64)

N-steps                         5                                                     5

Adam $\epsilon$                 0.001                                                 0.001

Value loss coefficient          0.5                                                   0.5

Hyperparameters for PPO baseline.

Hyperparameter     DeepSea     Hallway

Normalise observations          False, True                                  False, True

Normalise rewards               False, True                                  False, True

Learning rate                   $3e^{-4}, \mathbf{1e^{-3}}$                           $\mathbf{3e^{-4}}, 1e^{-3}$

Nonlinearity                    Tanh, ReLU                                   Tanh, ReLU

Maximum gradient norm           $\mathbf{0.5}, 10.0, 40.0$                            $\mathbf{0.5}, 10.0, 40.0$

Entropy loss coefficient        $\mathbf{1e^{-4}}, 3e^{-4}, 7e^{-4}, 1e^{-3}$         $1e^{-4}, 3e^{-4}, \mathbf{7e^{-4}}, 1e^{-3}$

Actor architecture              Fully connected (64, 64)                     Fully connected (64, 64)

Critic architecture             Fully connected (64, 64)                     Fully connected (64, 64)

N-steps                         10                                           10

Adam $\epsilon$                 0.001                                        0.001

Value loss coefficient          0.5                                          0.5

Number of epochs                10                                           10

Number of minibatches           4                                            4

Clipping ratio                  0.1                                          0.1

Clip value loss                 True                                         True

§.§ Intrinsic Reward Hyperparameters

Following the hyperparameter search of A2C and PPO with Count intrinsic rewards, a search over hyperparameters of all parameterised intrinsic reward definitions was conducted. The setup of the hyperparameter search is identical to the one described above and the best identified hyperparameter configuration for the baseline algorithms are used in the gridsearch for intrinsic rewards. Best identified hyperparameters and all considered hyperparameter configurations can be found in <Ref>.

Hyperparameters for Hash-Count.

Hyperparameter     DeepSea     Hallway

A2C     Hash key dimensionality                 16, 32, 64, 128          16, 32, 64, 128

PPO     Hash key dimensionality                 16, 32, 64, 128          16, 32, 64, 128

Hyperparameters for ICM.

Hyperparameter     DeepSea     Hallway

4*[origin=c]90General     $\phi$ architecture     Fully connected (64, 64)     Fully connected (64, 64)

$\phi(s)$ dimensionality     16     16

Forward prediction architecture     Fully connected (64)     Fully connected (64)

Inverse prediction architecture     Fully connected (64)     Fully connected (64)

3*[origin=c]90A2C     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$     $1e^{-7}, 5e^{-7}, \mathbf{1e^{-6}}, 5e^{-6}, 1e^{-5}$

Forward loss coefficient        $0.5, 1.0, \mathbf{5.0}, 10.0$                             $0.5, 1.0, \mathbf{5.0}, 10.0$

Inverse loss coefficient        $0.5, \mathbf{1.0}, 5.0, 10.0$                             $\mathbf{0.5}, 1.0, 5.0, 10.0$

3*[origin=c]90PPO     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$     $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$

Forward loss coefficient        $0.5, 1.0, \mathbf{5.0}, 10.0$                             $\mathbf{0.5}, 1.0, 5.0, 10.0$

Inverse loss coefficient        $0.5, \mathbf{1.0}, 5.0, 10.0$                             $0.5, 1.0, 5.0, \mathbf{10.0}$

Hyperparameters for RND.

Hyperparameter     DeepSea     Hallway

General     $\phi$ architecture     Fully connected (64, 64)     Fully connected (64, 64)

$\phi(s)$ dimensionality     16     16

A2C     Learning rate                   $\mathbf{1e^{-7}}, 5e^{-7}, 1e^{-6}, 5e^{-6}, 1e^{-5}$     $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$

PPO     Learning rate                   $\mathbf{1e^{-7}}, 5e^{-7}, 1e^{-6}, 5e^{-6}, 1e^{-5}$     $1e^{-7}, \mathbf{5e^{-7}}, 1e^{-6}, 5e^{-6}, 1e^{-5}$

Hyperparameters for RIDE.

Hyperparameter     DeepSea     Hallway

5*[origin=c]90General     $\phi$ architecture     Fully connected (64, 64)     Fully connected (64, 64)

$\phi(s)$ dimensionality     16     16

Forward prediction architecture     Fully connected (64)     Fully connected (64)

Inverse prediction architecture     Fully connected (64)     Fully connected (64)

State count     Count     Count

3*[origin=c]90A2C     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$     $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$

Forward loss coefficient        $\mathbf{0.5}, 1.0, 5.0, 10.0$                             $0.5, 1.0, 5.0, \mathbf{10.0}$

Inverse loss coefficient        $0.5, 1.0, 5.0, \mathbf{10.0}$                             $\mathbf{0.5}, 1.0, 5.0, 10.0$

3*[origin=c]90PPO     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, \mathbf{5e^{-6}}, 1e^{-5}$     $\mathbf{1e^{-7}}, 5e^{-7}, 1e^{-6}, 5e^{-6}, 1e^{-5}$

Forward loss coefficient        $0.5, 1.0, 5.0, \mathbf{10.0}$                             $0.5, \mathbf{1.0}, 5.0, 10.0$

Inverse loss coefficient        $0.5, \mathbf{1.0}, 5.0, 10.0$                             $0.5, \mathbf{1.0}, 5.0, 10.0$

§.§ Decoupled Reinforcement Learning Hyperparameters

Based on the aforementioned hyperparameters, a gridsearch for all DeRL algorithms, DeA2C, DePPO and DeDQN, was conducted using the best identified A2C and Count intrinsic rewards to train the exploration policy. Identified configurations are listed in <Ref>. <Ref> shows hyperparameters for DeRL algorithms with ICM intrinsic rewards used to train the A2C exploration policy. The same general architecture is used for ICM when applied alongside DeRL as reported in <Ref>.

Hyperparameters for DeA2C.

Hyperparameter     DeepSea     Hallway

Importance sampling             Default IS weights, Retrace($\lambda$)                Default IS weights, Retrace($\lambda$)

Learning rate                   $3e^{-4}, \mathbf{1e^{-3}}$                                    $\mathbf{3e^{-4}}, 1e^{-3}$

Nonlinearity                    Tanh, ReLU                                            Tanh, ReLU

Maximum gradient norm           $\mathbf{0.5}$                                                 $\mathbf{0.5}$

Entropy loss coefficient        $0.0, \mathbf{1e^{-6}}, 1e^{-5}, 1e^{-4}$                      $0.0, 1e^{-6}, \mathbf{1e^{-5}}, 1e^{-4}$

Actor architecture              Fully connected (64, 64)                     Fully connected (64, 64)

Critic architecture             Fully connected (64, 64)                     Fully connected (64, 64)

N-steps                         5                                            5

Adam $\epsilon$                 0.001                                        0.001

Value loss coefficient          0.5                                          0.5

$T_{Dec}$                       1                                            1

Hyperparameters for DePPO.

Hyperparameter     DeepSea     Hallway

Importance sampling             Default IS weights                           Default IS weights

Learning rate                   $3e^{-4}, \mathbf{1e^{-3}}$                           $\mathbf{3e^{-4}}, 1e^{-3}$

Nonlinearity                    Tanh, ReLU                                   Tanh, ReLU

Maximum gradient norm           $\mathbf{0.5}$                                        $\mathbf{0.5}$

Entropy loss coefficient        $0.0, 1e^{-6}, 1e^{-5}, \mathbf{1e^{-4}}$             $0.0, \mathbf{1e^{-6}}, 1e^{-5}, 1e^{-4}$

Actor architecture              Fully connected (64, 64)                     Fully connected (64, 64)

Critic architecture             Fully connected (64, 64)                     Fully connected (64, 64)

N-steps                         10                                           10

Adam $\epsilon$                 0.001                                        0.001

Value loss coefficient          0.5                                          0.5

Number of epochs                10                                           10

Number of minibatches           4                                            4

Clipping ratio                  0.1                                          0.1

Clip value loss                 True                                         True

$T_{Dec}$                       1                                            1

Hyperparameters for DeDQN.

Hyperparameter     DeepSea     Hallway

Learning rate                   $1e^{-4}, 3e^{-4}, \mathbf{1e^{-3}}$                           $\mathbf{1e^{-4}}, 3e^{-4}, 1e^{-3}$

Soft target update $\tau$       0.01, 0.001                                           0.01, 0.001

Batch size                      128, 256, 512                                         128, 256, 512

N-steps                         5                                                     5

Nonlinearity                    Tanh, ReLU                                            Tanh, ReLU

Replay buffer                   Default, Prioritised                                  Default, Prioritised

Replay buffer capacity          100,000                                               100,000

Maximum gradient norm           $\mathbf{0.5}$                                                 $\mathbf{0.5}$

DQN architecture                Fully connected (64, 64)                              Fully connected (64, 64)

Adam $\epsilon$                 0.001                                                 0.001

$T_{Dec}$                       1                                                     1

Hyperparameters for DeRL with ICM.

Hyperparameter     DeepSea     Hallway

3*[origin=c]90DeA2C     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$     $1e^{-7}, 5e^{-7}, \mathbf{1e^{-6}}, 5e^{-6}, 1e^{-5}$

Forward loss coefficient        $0.5, 1.0, \mathbf{5.0}, 10.0$                             $0.5, 1.0, \mathbf{5.0}, 10.0$

Inverse loss coefficient        $0.5, \mathbf{1.0}, 5.0, 10.0$                             $\mathbf{0.5}, 1.0, 5.0, 10.0$

3*[origin=c]90DePPO     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$     $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$

Forward loss coefficient        $0.5, 1.0, \mathbf{5.0}, 10.0$                             $\mathbf{0.5}, 1.0, 5.0, 10.0$

Inverse loss coefficient        $0.5, \mathbf{1.0}, 5.0, 10.0$                             $0.5, 1.0, 5.0, \mathbf{10.0}$

3*[origin=c]90DeDQN     Learning rate                   $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$     $1e^{-7}, 5e^{-7}, 1e^{-6}, 5e^{-6}, \mathbf{1e^{-5}}$

Forward loss coefficient        $0.5, 1.0, \mathbf{5.0}, 10.0$                             $\mathbf{0.5}, 1.0, 5.0, 10.0$

Inverse loss coefficient        $0.5, \mathbf{1.0}, 5.0, 10.0$                            $0.5, 1.0, 5.0, \mathbf{10.0}$

§ EVALUATION RESULTS

In this section, we provide tables containing maximum evaluation returns for all baselines and DeRL algorithms in every task. As described in <Ref>, average returns and stratified bootstrap 95% confidence intervals are computed using 5,000 samples across five random seeds for the best identified hyperparameter configuration as reported in <Ref>. For maximum evaluation returns, we identify the single evaluation out of all 100 conducted evaluations with the maximum evaluation returns averaged across samples and report its value with the standard deviation across all samples. Within tables, the highest performing algorithms for each task are highlighted in bold together with every algorithm within a single standard deviation of the highest return. Besides these tables, we also provide learning curves for all baselines and DeRL algorithms in every evaluated task. In order to maintain visual clarity, we average two following evaluations (of the total of 100 evaluations throughout training) for a total of 50 plotted evaluation points for each algorithm.

§.§ DeepSea

Maximum evaluation returns in the DeepSea environment with a single standard deviation.

Algorithm \ Task     DeepSea 10     DeepSea 14     DeepSea 20     DeepSea 24     DeepSea 30

A2C     0.99(0)      0.00(0)      0.00(0)      -0.00(0)      -0.00(0)

A2C Count     0.99(0)     0.99(0)     0.99(0)     0.79(40)      -0.01(0)

A2C Hash-Count     0.99(0)     0.99(0)      0.64(48)     0.59(49)      -0.00(0)

A2C ICM     0.99(0)     0.99(0)      0.70(45)     0.79(40)     0.39(49)

A2C RND      0.06(24)      0.19(40)      -0.00(0)      -0.00(0)      -0.00(0)

A2C RIDE      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)

PPO      0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)

PPO Count     0.99(0)      0.79(40)      0.73(44)     0.59(49)     0.59(49)

PPO Hash-Count     0.99(0)     0.99(0)      0.59(49)     0.79(40)     0.59(49)

PPO ICM     0.99(0)      0.39(49)      0.31(46)     0.79(40)     0.19(40)

PPO RND      0.38(48)      0.19(40)      -0.00(0)      -0.00(0)      -0.00(0)

PPO RIDE      0.77(41)      -0.00(0)      0.15(36)      -0.00(0)      -0.00(0)

DeA2C Count     0.99(0)     0.99(0)      0.90(28)      0.19(40)     0.16(37)

DeA2C ICM     0.99(0)      0.82(37)      0.66(47)      0.39(49)     0.33(47)

DePPO Count     0.99(0)     0.99(0)      0.08(28)     0.82(37)      -0.00(0)

DePPO ICM     0.99(0)      0.66(47)      0.09(30)      -0.00(0)      -0.00(0)

DeDQN Count     0.99(0)     0.99(0)      0.44(49)     0.79(40)     0.19(40)

DeDQN ICM     0.99(0)     0.99(0)      0.30(46)     0.59(49)     0.44(49)

DeepSea $N=10$

DeepSea $N=14$

DeepSea $N=20$

DeepSea $N=24$

DeepSea $N=30$

Average Evaluation returns for A2C, PPO and DeRL with the highest achieving intrinsic reward in all DeepSea tasks. Shading indicates 95% confidence intervals.

DeepSea $N=10$ A2C

DeepSea $N=10$ PPO

DeepSea $N=10$ DeRL

DeepSea $N=14$ A2C

DeepSea $N=14$ PPO

DeepSea $N=14$ DeRL

DeepSea $N=20$ A2C

DeepSea $N=20$ PPO

DeepSea $N=20$ DeRL

DeepSea $N=24$ A2C

DeepSea $N=24$ PPO

DeepSea $N=24$ DeRL

DeepSea $N=30$ A2C

DeepSea $N=30$ PPO

DeepSea $N=30$ DeRL

Average evaluation returns for A2C (first column), PPO (second column) and DeRL (third column) with all intrinsic rewards for all DeepSea tasks. Shading indicates 95% confidence intervals.

§.§ Hallway

Maximum evaluation returns in the Hallway environment with a single standard deviation.

Algorithm \ Task     Hallway 10-0     Hallway 10-10     Hallway 20-0     Hallway 20-20     Hallway 30-0     Hallway 30-30

A2C      0.68(34)      0.51(42)     0.42(34)     0.50(25)     0.44(22)     0.46(7)

A2C Count     0.85(0)     0.85(0)     0.70(0)     0.60(0)     0.52(6)      0.14(24)

A2C Hash-Count     0.85(0)     0.85(0)     0.70(0)     0.60(0)     0.52(6)      0.22(27)

A2C ICM      0.68(34)      0.68(34)     0.66(101)     1.08(76)     1.58(150)     1.09(115)

A2C RND      0.12(35)      -0.07(9)      -0.17(15)      -0.24(20)     -0.24(28)      -0.12(24)

A2C RIDE     0.85(0)     0.85(0)     0.70(0)     0.62(4)     0.55(0)     0.43(6)

PPO      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

PPO Count      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

PPO Hash-Count      0.50(41)      0.27(39)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

PPO ICM      0.48(39)      0.40(41)      0.28(34)     0.39(32)     0.08(16)      0.10(20)

PPO RND      0.13(37)      0.47(44)      0.02(31)      0.17(40)      -0.04(32)      0.00(0)

PPO RIDE      0.10(35)      0.26(44)      -0.03(37)      0.13(41)      -0.19(28)      -0.01(40)

DeA2C Count     0.85(0)     0.85(0)     0.60(0)     0.70(0)     0.55(0)     0.43(6)

DeA2C ICM     0.85(0)     0.85(0)     1.08(76)     1.08(76)     1.38(168)     1.69(140)

DePPO Count     0.85(0)     0.85(0)     0.56(28)     0.60(0)     0.52(6)     0.43(6)

DePPO ICM     0.85(0)     0.85(0)     1.08(76)     0.62(4)     0.55(0)     0.43(6)

DeDQN Count      0.13(37)      0.11(37)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

DeDQN ICM      0.27(77)      0.39(82)      0.06(36)     0.35(112)      0.00(0)      0.00(0)

Hallway $N_l=10, N_r=0$

Hallway $N_l=20, N_r=0$

Hallway $N_l=30, N_r=0$

Hallway $N_l=10, N_r=10$

Hallway $N_l=20, N_r=20$

Hallway $N_l=30, N_r=30$

Average evaluation returns for A2C, PPO and DeRL with the highest achieving intrinsic reward in all Hallway tasks. Shading indicates 95% confidence intervals.

Hallway $N_l=10, N_r=0$ A2C

Hallway $N_l=10, N_r=0$ PPO

Hallway $N_l=10, N_r=0$ DeRL

Hallway $N_l=10, N_r=10$ A2C

Hallway $N_l=10, N_r=10$ PPO

Hallway $N_l=10, N_r=10$ DeRL

Hallway $N_l=20, N_r=0$ A2C

Hallway $N_l=20, N_r=0$ PPO

Hallway $N_l=20, N_r=0$ DeRL

Hallway $N_l=20, N_r=20$ A2C

Hallway $N_l=20, N_r=20$ PPO

Hallway $N_l=20, N_r=20$ DeRL

Average evaluation returns for A2C (first column), PPO (second column) and DeRL (third column) with all intrinsic rewards for all Hallway tasks with $N_l \in \{10, 20\}$. Shading indicates 95% confidence intervals.

Hallway $N_l=30, N_r=0$ A2C

Hallway $N_l=30, N_r=0$ PPO

Hallway $N_l=30, N_r=0$ DeRL

Hallway $N_l=30, N_r=30$ A2C

Hallway $N_l=30, N_r=30$ PPO

Hallway $N_l=30, N_r=30$ DeRL

Average evaluation returns for A2C (first column), PPO (second column) and DeRL (third column) with all intrinsic rewards for all Hallway tasks with $N_l=30$. Shading indicates 95% confidence intervals.

§ HYPERPARAMETER SENSITIVITY

In this section, we provide further plots and tables containing maximum and average achieved evaluation returns for both sets of hyerparameter sensitivity experiments. As described in <Ref>, we evaluate both baselines and DeRL algorithms with varying intrinsic reward coefficients, $\lambda$, and rate of decay of intrinsic rewards. Both experiments were conducted in DeepSea 10 and Hallway $N_l = N_r = 10$. We provide tables showing the maximum and average evaluation returns as in <Ref> and bar plots indicating the varying average evaluation returns for various parameterisation of intrinsic rewards. Within tables, the highest performing configuration for a single algorithm is highlighted in bold within each row with all configurations within a single standard deviation of the highest return.

§.§ Intrinsic Reward Scale

First, we evaluate all baselines and DeRL algorithms for various intrinsic reward coefficients $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$ in DeepSea and Hallway.

Average evaluation returns for baselines and DeRL with Count (upper row) and ICM (lower row) intrinsic rewards in DeepSea 10 with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

A2C Hash-Count

A2C RND

A2C RIDE

PPO Hash-Count

PPO RND

PPO RIDE

Average evaluation returns for A2C and PPO with Hash-Count, RND and RIDE intrinsic rewards in DeepSea 10 with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation in DeepSea 10 for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

A2C Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.79(30)      0.79(30)

A2C Hash-Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.79(30)      0.39(40)

A2C ICM     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.59(40)

A2C RND     0.99(0)      0.39(40)      0.19(30)      -0.00(0)      0.20(30)      -0.00(0)      0.19(30)      -0.00(0)      -0.01(0)

A2C RIDE     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)

PPO Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

PPO Hash-Count      0.59(40)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.79(30)

PPO ICM      0.00(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.39(40)

PPO RND      0.20(30)      -0.00(0)     0.39(40)      -0.00(0)     0.39(40)      -0.00(0)      -0.01(0)      0.19(30)      -0.01(0)

PPO RIDE      0.79(30)      0.79(30)      0.79(30)     0.99(0)     0.99(0)      0.79(30)     0.99(0)      0.79(30)      0.59(40)

DeA2C Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeA2C ICM     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DePPO Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.59(40)

DePPO ICM     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.59(40)      0.39(40)      0.19(30)

DeDQN Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeDQN ICM     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.79(30)

Average evaluation returns with a single standard deviation in DeepSea 10 for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

A2C Count      0.93(21)      0.96(17)     0.97(13)     0.98(8)     0.99(2)     0.98(7)      0.77(21)      0.47(16)      0.17(16)

A2C Hash-Count      0.94(20)      0.96(17)     0.97(11)     0.98(8)     0.98(6)     0.99(2)     0.91(15)      0.52(19)      0.09(13)

A2C ICM      0.92(22)      0.92(22)     0.93(20)     0.91(22)      0.90(21)      0.55(27)      0.82(21)      0.63(28)      0.05(13)

A2C RND     0.92(23)      0.37(10)      0.19(3)      -0.00(0)      0.19(2)      -0.01(0)      0.16(8)      -0.01(0)      -0.01(0)

A2C RIDE     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)

PPO Count     0.99(2)     0.98(4)     0.99(3)     0.93(11)      0.85(11)      0.87(13)      0.73(16)      0.56(25)      0.17(20)

PPO Hash-Count      0.59(2)     0.99(2)     0.98(4)     0.96(7)      0.85(12)     0.97(7)      0.81(20)      0.64(22)      0.16(21)

PPO ICM      -0.00(0)      0.85(26)     0.89(15)      0.81(18)      0.70(31)      0.79(18)      0.68(25)      0.59(20)      0.04(9)

PPO RND      0.20(0)      -0.01(0)      0.20(6)      -0.01(0)     0.29(11)      -0.01(0)      -0.01(0)      0.05(9)      -0.01(0)

PPO RIDE      0.73(14)      0.73(17)      0.77(9)     0.91(17)     0.94(12)      0.74(15)     0.91(17)      0.77(9)      0.57(6)

DeA2C Count      0.91(24)      0.95(18)     0.97(14)     0.96(15)     0.97(11)      0.94(13)     0.97(13)     0.93(15)      0.58(19)

DeA2C ICM      0.93(22)      0.92(24)      0.92(24)     0.94(21)      0.82(19)      0.93(17)     0.96(11)     0.95(13)      0.84(23)

DePPO Count      0.76(24)      0.56(20)      0.65(27)      0.80(19)     0.86(20)      0.71(20)      0.73(25)      0.54(22)      0.06(12)

DePPO ICM      0.55(24)     0.61(22)      0.58(22)      0.55(25)      0.60(23)      0.55(25)      0.20(18)      0.08(11)      0.00(3)

DeDQN Count      0.94(17)      0.96(15)     0.97(14)     0.98(10)     0.98(10)     0.98(10)     0.98(10)      0.77(21)     0.88(15)

DeDQN ICM      0.94(18)      0.92(21)      0.91(24)     0.93(21)      0.94(19)     0.96(14)     0.95(15)     0.96(14)      0.53(20)

Average evaluation returns for baselines and DeRL with Count (upper row) and ICM (lower row) intrinsic rewards in Hallway $N_l=N_r=10$ with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

A2C Hash-Count

A2C RND

A2C RIDE

PPO Hash-Count

PPO RND

PPO RIDE

Average evaluation returns for A2C and PPO with Hash-Count, RND and RIDE intrinsic rewards in Hallway $N_l=N_r=10$ with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation in Hallway $N_l=N_r=10$ for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

A2C Count      0.68(26)      0.51(34)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)      0.51(34)      0.80(0)

A2C Hash-Count      0.68(26)      0.51(34)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)      0.51(34)      0.80(0)

A2C ICM      0.17(26)      0.68(26)     0.85(0)     0.85(0)      0.51(34)      0.51(34)     0.85(0)     0.89(72)      0.28(40)

A2C RND      0.00(0)     0.34(34)      0.13(32)      -0.06(7)      -0.04(6)      -0.08(8)      -0.04(6)      -0.06(7)      -0.16(6)

A2C RIDE     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)

PPO Count     0.00(0)     0.00(0)     0.00(0)     0.00(0)     0.00(0)     0.00(0)      -0.06(4)     0.00(0)      -0.06(7)

PPO Hash-Count      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.17(26)     0.64(24)      0.32(32)      0.17(25)      0.14(27)

PPO ICM      0.00(0)     0.51(34)      0.26(38)      0.32(32)      0.33(33)      0.32(36)      0.33(34)      0.42(40)      0.24(40)

PPO RND      0.16(24)      0.28(37)      0.32(32)      0.48(32)      0.80(0)      0.61(31)      0.64(24)      0.60(30)     0.81(1)

PPO RIDE      0.10(28)      0.26(37)      0.26(37)      0.30(57)     0.43(37)      0.30(57)      0.29(57)      0.30(57)      0.26(37)

DeA2C Count      0.17(26)      0.68(26)     0.85(0)     0.85(0)     0.85(0)     0.85(0)      0.82(2)      0.84(2)      0.82(2)

DeA2C ICM      0.17(26)     0.85(0)      0.68(26)     0.85(0)     0.85(0)     0.85(0)      0.67(26)      0.34(34)      0.81(1)

DePPO Count      0.34(34)      0.68(26)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)      0.34(34)      0.80(0)

DePPO ICM      0.17(26)      0.17(26)      0.68(26)      0.51(34)     0.85(0)     1.04(28)     0.85(1)     1.04(28)     0.85(56)

DeDQN Count      0.26(42)      0.28(40)      0.32(60)      0.43(42)      0.43(42)      0.68(76)      0.47(40)     1.60(30)     1.23(38)

DeDQN ICM      0.32(60)      0.11(31)      0.25(60)      0.22(61)      0.34(57)      0.50(63)      0.66(26)      0.64(57)     0.70(54)

Average evaluation returns with a single standard deviation in Hallway $N_l=N_r=10$ for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

A2C Count      0.67(7)      0.51(3)     0.84(7)     0.85(1)     0.85(2)     0.84(10)      0.77(24)      0.32(18)      -0.09(18)

A2C Hash-Count      0.67(7)      0.51(2)     0.84(5)     0.85(2)     0.85(2)     0.84(10)      0.79(21)      0.27(18)      -0.10(19)

A2C ICM      0.17(2)      0.50(9)      0.62(13)     0.81(15)      0.44(14)      0.45(17)      0.61(28)      0.48(27)      -0.00(18)

A2C RND      0.00(0)     0.33(4)      0.13(2)      -0.06(0)      -0.04(0)      -0.12(0)      -0.04(0)      -0.08(1)      -0.16(1)

A2C RIDE     0.85(2)     0.85(2)     0.85(2)     0.85(1)     0.85(2)     0.85(2)     0.85(0)     0.85(2)     0.85(1)

PPO Count     0.00(0)     0.00(0)     0.00(0)     0.00(0)     0.00(0)      -0.02(1)      -0.08(1)      -0.07(3)      -0.12(2)

PPO Hash-Count      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.04(7)     0.40(9)      0.16(6)      0.03(6)      -0.03(5)

PPO ICM      0.00(0)     0.24(20)      0.01(9)      -0.02(9)      0.05(13)      0.04(12)      0.03(13)      -0.01(13)      -0.06(11)

PPO RND      0.00(2)      0.06(9)      -0.02(7)      0.12(7)      -0.04(17)      -0.05(9)      0.20(20)      0.00(17)     0.28(17)

PPO RIDE      -0.05(8)      -0.05(9)      -0.05(9)      0.00(14)      -0.04(10)      0.00(14)      -0.05(9)      -0.04(11)     0.01(11)

DeA2C Count      0.17(2)      0.67(7)     0.84(9)     0.84(8)     0.84(9)     0.84(8)      0.67(10)      0.82(12)      0.80(6)

DeA2C ICM      0.17(3)      0.17(24)      0.60(14)     0.81(14)      0.78(20)     0.77(20)      0.62(16)      0.30(9)      0.63(6)

DePPO Count      0.34(3)      0.45(13)     0.84(9)      0.80(10)      0.75(12)     0.84(9)      0.71(19)      0.16(16)      -0.07(19)

DePPO ICM      -0.02(4)      0.15(5)      0.39(13)      0.48(9)      0.80(12)     0.88(14)      0.81(12)     0.85(20)      0.61(17)

DeDQN Count      -0.11(4)      -0.17(5)      -0.15(5)      -0.13(8)      -0.16(7)      -0.15(11)      -0.13(10)      0.04(35)     0.17(33)

DeDQN ICM      -0.02(4)      -0.11(5)      -0.12(6)      -0.11(10)      -0.03(16)      0.02(14)      0.15(20)     0.23(18)      0.21(19)

§.§ Intrinsic Reward Decay

Second, we evaluate all baselines and DeRL algorithms with varying rate of decay of intrinsic rewards in DeepSea and Hallway. For count-based intrinsic rewards, the rate of decay can be determined by the increment of the state count $N(s)$. By default, the count is incremented by $1$ for each state occurrence. For this analysis, we consider increments $\{0.01, 0.1, 0.2, 1.0, 5.0, 10.0, 100.0\}$. For deep prediction-based intrinsic rewards, ICM, RND and RIDE, the rate of decay is determined by their learning rates. We consider learning rates  $\{1e^{-9}, 1e^{-8}, 2e^{-8}, 1e^{-7}, 5e^{-7}, 1e^{-6}, 1e^{-5}, 1e^{-4}, 1e^{-3}\}$.

Average evaluation returns for baselines and DeRL with Count (upper row) and ICM (lower row) intrinsic rewards in DeepSea 10 with varying count increments and learning rates. Shading indicates 95% confidence intervals.

A2C Hash-Count

A2C RND

A2C RIDE

PPO Hash-Count

PPO RND

PPO RIDE

Average evaluation returns for A2C and PPO with Hash-Count, RND and RIDE intrinsic rewards in DeepSea 10 with varying count increments and learning rates. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation in DeepSea 10 with count-based intrinsic rewards and varying count increments.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

A2C Count     0.99(0)     0.99(0)      0.79(30)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

A2C Hash-Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

PPO Count      0.59(40)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)      0.00(0)

PPO Hash-Count      0.79(30)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeA2C Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DePPO Count      0.79(30)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeDQN Count     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

Average evaluation returns with a single standard deviation in DeepSea 10 with count-based intrinsic rewards and varying count increments.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

A2C Count      0.08(15)      0.20(22)      0.52(19)     0.99(2)      0.96(17)      0.95(19)      0.92(23)

A2C Hash-Count      0.31(17)     0.93(16)     0.98(4)     0.98(6)     0.98(11)     0.97(13)      0.96(17)

PPO Count      0.08(13)      0.34(23)      0.57(20)      0.85(11)     0.99(2)     0.99(2)      0.00(0)

PPO Hash-Count      0.43(19)     0.81(21)      0.80(14)      0.85(12)      0.85(9)      0.79(5)     0.99(0)

DeA2C Count      0.44(22)      0.68(15)      0.91(21)     0.97(11)      0.94(20)      0.94(19)      0.90(24)

DePPO Count      0.15(21)      0.13(25)      0.28(25)     0.86(20)      0.56(26)      0.56(26)      0.39(32)

DeDQN Count     0.85(23)      0.73(21)      0.91(14)     0.98(10)      0.96(16)      0.92(21)      0.90(25)

Maximum evaluation returns with a single standard deviation in DeepSea 10 with prediction-based intrinsic rewards and varying learning rates.

Algorithm \ Learning rate     1e-09     1e-08     2e-08     1e-07     5e-07     1e-06     1e-05     0.0001     0.001

A2C ICM     -0.00(0)     -0.00(0)      -0.00(0)     -0.00(0)      0.39(40)     0.79(30)     0.99(0)     0.99(0)     0.99(0)

A2C RND     0.19(30)     -0.00(0)     -0.01(0)     0.20(30)      -0.00(0)      0.19(30)     0.39(40)      0.19(30)     0.39(40)

A2C RIDE     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)

PPO ICM     -0.01(0)     0.19(30)     -0.00(0)     0.39(40)     0.99(0)     0.59(40)     0.99(0)     0.99(0)     0.99(0)

PPO RND     0.19(30)     0.19(30)     0.19(30)     0.39(40)     0.59(40)     0.59(40)     0.59(40)      0.39(40)      0.39(40)

PPO RIDE     -0.00(0)     0.19(30)     0.20(30)     -0.00(0)     0.79(30)     0.79(30)     0.79(30)      0.39(40)      0.20(30)

DeA2C ICM     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)      0.39(40)     0.59(40)     0.99(0)     0.99(0)     0.99(0)

DePPO ICM     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)      0.19(30)      0.39(40)     0.99(0)     0.99(0)     0.99(0)

DeDQN ICM     -0.00(0)     0.19(30)     -0.00(0)     -0.00(0)      0.39(40)     0.79(30)     0.99(0)     0.99(0)     0.99(0)

Average evaluation returns with a single standard deviation in DeepSea 10 with prediction-based intrinsic rewards and varying learning rates.

Algorithm \ Learning rate     1e-09     1e-08     2e-08     1e-07     5e-07     1e-06     1e-05     0.0001     0.001

A2C ICM      -0.00(0)      -0.00(0)      -0.00(0)      -0.01(0)      0.27(14)      0.38(13)     0.90(21)     0.95(19)     0.96(16)

A2C RND      -0.00(2)      -0.01(0)      -0.01(0)     0.19(2)      -0.01(0)      0.16(7)     0.39(4)      0.19(3)      0.32(10)

A2C RIDE     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)     -0.00(0)

PPO ICM      -0.00(0)      0.02(7)      -0.00(0)      0.10(15)     0.56(28)     0.49(15)      0.70(31)     0.93(12)     0.98(7)

PPO RND      0.19(0)     0.19(2)     0.18(5)     0.29(11)     0.50(10)     0.53(10)     0.59(0)      0.20(2)      0.37(6)

PPO RIDE      -0.00(0)      0.16(7)     0.19(2)      -0.00(0)     0.49(22)     0.55(18)     0.75(11)      0.39(3)      0.18(6)

DeA2C ICM      -0.01(0)      -0.00(0)      -0.01(0)      -0.00(0)     0.29(16)     0.47(21)     0.82(19)     0.94(22)     0.93(22)

DePPO ICM      -0.00(0)      -0.00(0)      -0.00(0)      -0.00(0)      0.11(10)      0.21(12)      0.69(22)     0.73(22)      0.51(31)

DeDQN ICM      -0.01(0)      -0.00(2)      -0.01(0)      -0.01(0)      0.16(8)     0.63(21)     0.94(19)     0.95(17)     0.95(16)

Average evaluation returns for baselines and DeRL with Count (upper row) and ICM (lower row) intrinsic rewards in Hallway $N_l=N_r=10$ with varying count increments and learning rates. Shading indicates 95% confidence intervals.

A2C Hash-Count

A2C RND

A2C RIDE

PPO Hash-Count

PPO RND

PPO RIDE

Average evaluation returns for A2C and PPO with Hash-Count, RND and RIDE intrinsic rewards in Hallway $N_l=N_r=10$ with varying count increments and learning rates. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation in Hallway $N_l = N_r = 10$ with count-based intrinsic rewards and varying count increments.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

A2C Count      0.80(0)     0.80(0)      0.17(26)     0.85(0)      0.68(26)      0.68(26)     0.68(26)

A2C Hash-Count      0.34(34)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.51(34)

PPO Count      0.00(0)      0.00(0)     0.16(25)      0.00(0)      0.00(0)      0.00(0)      0.00(0)

PPO Hash-Count      0.16(24)      0.26(38)     0.80(0)      0.17(26)      0.00(0)      0.00(0)      0.00(0)

DeA2C Count     0.85(0)     1.00(30)      0.83(2)     0.85(0)     0.85(0)      0.34(34)      0.17(26)

DePPO Count     1.23(38)     0.80(0)      0.25(37)     0.85(0)      0.68(26)      0.51(34)      0.00(0)

DeDQN Count     1.02(60)     0.85(0)     1.23(38)      0.43(42)      0.24(40)      0.05(34)      0.26(42)

Average evaluation returns with a single standard deviation in Hallway $N_l = N_r = 10$ with count-based intrinsic rewards and varying count increments.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

A2C Count      -0.06(26)      -0.11(10)      0.10(9)     0.85(2)      0.67(7)      0.67(7)     0.67(7)

A2C Hash-Count      0.27(14)     0.82(16)     0.84(10)     0.85(2)     0.85(1)     0.84(7)      0.50(5)

PPO Count      -0.09(4)      -0.02(4)      -0.01(3)     0.00(0)     0.00(0)     0.00(0)     0.00(0)

PPO Hash-Count      -0.00(2)      0.00(3)     0.50(6)      0.04(7)      0.00(0)      0.00(0)      0.00(0)

DeA2C Count     0.84(7)     0.83(19)     0.81(12)     0.84(9)      0.67(6)      0.33(4)      0.17(2)

DePPO Count      -0.07(28)      -0.11(10)      -0.05(5)     0.75(12)      0.67(7)      0.42(12)      0.00(0)

DeDQN Count      0.25(28)     0.34(28)      0.03(30)      -0.16(7)      -0.14(5)      -0.07(2)      -0.11(5)

Maximum evaluation returns with a single standard deviation in Hallway $N_l = N_r = 10$ with prediction-based intrinsic rewards and varying learning rates.

Algorithm \ Learning rate     1e-09     1e-08     2e-08     1e-07     5e-07     1e-06     1e-05     0.0001     0.001

A2C ICM      -0.04(6)      -0.04(6)      -0.04(6)      0.15(28)      0.68(26)      0.51(34)     0.85(0)     0.85(0)      0.34(34)

A2C RND      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)      0.00(0)      -0.04(6)      -0.08(8)     0.17(25)

A2C RIDE      -0.11(1)      -0.11(1)      -0.11(1)     0.84(2)      0.66(28)     0.85(0)     0.85(0)      0.66(28)      0.17(26)

PPO ICM      0.40(40)      0.40(40)      0.22(40)      0.24(40)      0.40(40)      0.26(38)      0.33(33)      0.50(34)     0.67(26)

PPO RND     0.80(0)     0.80(0)     0.64(24)     0.44(38)     0.80(0)      0.23(39)      0.12(30)      0.24(40)      0.44(38)

PPO RIDE      0.44(36)      0.62(28)     0.65(24)     0.43(37)      0.44(37)      0.10(30)      0.26(37)      0.46(38)      0.51(34)

DeA2C ICM      0.00(0)      0.00(0)      0.00(0)     0.49(33)      0.67(26)     0.85(0)     0.85(0)     0.85(0)     0.85(0)

DePPO ICM      -0.04(6)      -0.04(6)      -0.04(6)     0.42(40)      0.48(38)      0.68(26)     0.85(0)      0.49(33)      0.00(0)

DeDQN ICM      -0.04(6)      0.00(0)      -0.04(6)     1.04(78)      0.34(57)      0.31(57)      0.30(38)      0.13(32)      0.17(26)

Average evaluation returns with a single standard deviation in Hallway $N_l = N_r = 10$ with prediction-based intrinsic rewards and varying learning rates.

Algorithm \ Learning rate     1e-09     1e-08     2e-08     1e-07     5e-07     1e-06     1e-05     0.0001     0.001

A2C ICM      -0.08(2)      -0.09(2)      -0.10(4)      -0.07(12)     0.56(26)      0.44(14)     0.83(11)     0.84(9)      0.34(0)

A2C RND      0.00(0)      -0.05(4)      -0.06(3)      -0.10(3)      -0.12(3)      -0.14(2)      -0.04(0)      -0.08(0)     0.16(3)

A2C RIDE      -0.11(0)      -0.11(0)      -0.11(0)     0.36(40)     0.58(19)     0.82(16)     0.85(2)      0.66(3)      0.17(0)

PPO ICM      -0.06(10)      -0.03(10)      -0.09(7)      -0.10(8)      -0.03(11)      -0.06(10)      0.05(13)      0.19(14)     0.32(13)

PPO RND      -0.04(10)      -0.02(12)     0.07(12)      -0.05(14)      -0.04(17)      -0.04(9)      -0.06(5)      -0.07(9)      -0.02(6)

PPO RIDE     0.26(14)     0.32(14)     0.29(16)     -0.04(10)      -0.03(15)      -0.08(6)      -0.08(6)      0.28(6)     0.51(2)

DeA2C ICM      0.00(0)      0.00(0)      0.00(0)     0.23(22)     0.61(17)     0.78(20)      0.70(18)     0.84(7)     0.68(3)

DePPO ICM      -0.04(0)      -0.10(4)      -0.14(6)     0.06(18)      0.25(13)      0.63(16)     0.65(19)      0.40(15)      0.00(0)

DeDQN ICM      -0.11(2)      -0.05(2)      -0.11(2)     0.19(28)      -0.03(16)      -0.05(13)      -0.12(6)      -0.11(5)      -0.15(5)

§ KL-DIVERGENCE CONSTRAINT REGULARISATION

In this section, we first provide figures showing evaluation returns of the evaluation policy (top left), training returns of the exploration policy (bottom left), IS weights (top right) and KL-divergence of exploration and exploitation policy (bottom right) for DeA2C trained with Count intrinsic rewards and KL-divergence constraints in the DeepSea $10$ and Hallway $N_l=N_r=20$ tasks for 20,000 episodes. Each figure corresponds to DeA2C being trained with KL-divergence constraint coefficients $\alpha_\beta$ and $\alpha_e$ for the regulariser terms for the exploration policy and exploitation policy , respectively. We present training with the exploration policy being trained using only intrinsic rewards (blue) or using the sum of intrinsic and extrinsic rewards (orange). Average returns and 95% confidence intervals are computed over 1,000 samples across three seeds and we consider regularisation coefficients $\alpha_\beta, \alpha_e \in \{0.0, 0.0001, 0.001, 0.01, 0.1\}$.

Following, we analyse the sensitivity of DeA2C trained with Count intrinsic rewards and KL-divergence constraints in DeepSea $10$ and Hallway $N_l=N_r=10$ tasks following the evaluation procedure outlined in <Ref>.

§.§ DeepSea 10

$\alpha_\beta=0.0, \alpha_e=0.0$

$\alpha_\beta=0.0, \alpha_e=0.0001$

$\alpha_\beta=0.0, \alpha_e=0.001$

$\alpha_\beta=0.0, \alpha_e=0.01$

$\alpha_\beta=0.0, \alpha_e=0.1$

$\alpha_\beta=0.0001, \alpha_e=0.0$

$\alpha_\beta=0.0001, \alpha_e=0.0001$

$\alpha_\beta=0.0001, \alpha_e=0.001$

$\alpha_\beta=0.0001, \alpha_e=0.01$

DeepSea 10 evaluation with divergence constraint regularisation coefficients $\alpha_\beta$ and $\alpha_e$.  Shading indicates 95% confidence intervals; Part 1

$\alpha_\beta=0.0001, \alpha_e=0.1$

$\alpha_\beta=0.001, \alpha_e=0.0$

$\alpha_\beta=0.001, \alpha_e=0.0001$

$\alpha_\beta=0.001, \alpha_e=0.001$

$\alpha_\beta=0.001, \alpha_e=0.01$

$\alpha_\beta=0.001, \alpha_e=0.1$

$\alpha_\beta=0.01, \alpha_e=0.0$

$\alpha_\beta=0.01, \alpha_e=0.0001$

$\alpha_\beta=0.01, \alpha_e=0.001$

$\alpha_\beta=0.01, \alpha_e=0.01$

$\alpha_\beta=0.01, \alpha_e=0.1$

$\alpha_\beta=0.1, \alpha_e=0.0$

DeepSea 10 evaluation with divergence constraint regularisation coefficients $\alpha_\beta$ and $\alpha_e$.  Shading indicates 95% confidence intervals; Part 2

$\alpha_\beta=0.1, \alpha_e=0.0001$

$\alpha_\beta=0.1, \alpha_e=0.001$

$\alpha_\beta=0.1, \alpha_e=0.01$

$\alpha_\beta=0.1, \alpha_e=0.1$

DeepSea 10 evaluation with divergence constraint regularisation coefficients $\alpha_\beta$ and $\alpha_e$.  Shading indicates 95% confidence intervals; Part 3

§.§ Hallway $N_l=N_r=20$

$\alpha_\beta=0.0, \alpha_e=0.0$

$\alpha_\beta=0.0, \alpha_e=0.0001$

$\alpha_\beta=0.0, \alpha_e=0.001$

$\alpha_\beta=0.0, \alpha_e=0.01$

$\alpha_\beta=0.0, \alpha_e=0.1$

$\alpha_\beta=0.0001, \alpha_e=0.0$

$\alpha_\beta=0.0001, \alpha_e=0.0001$

$\alpha_\beta=0.0001, \alpha_e=0.001$

$\alpha_\beta=0.0001, \alpha_e=0.01$

Hallway $N_l=N_r=20$ evaluation with divergence constraint regularisation coefficients $\alpha_\beta$ and $\alpha_e$.  Shading indicates 95% confidence intervals; Part 1

$\alpha_\beta=0.0001, \alpha_e=0.1$

$\alpha_\beta=0.001, \alpha_e=0.0$

$\alpha_\beta=0.001, \alpha_e=0.0001$

$\alpha_\beta=0.001, \alpha_e=0.001$

$\alpha_\beta=0.001, \alpha_e=0.01$

$\alpha_\beta=0.001, \alpha_e=0.1$

$\alpha_\beta=0.01, \alpha_e=0.0$

$\alpha_\beta=0.01, \alpha_e=0.0001$

$\alpha_\beta=0.01, \alpha_e=0.001$

$\alpha_\beta=0.01, \alpha_e=0.01$

$\alpha_\beta=0.01, \alpha_e=0.1$

$\alpha_\beta=0.1, \alpha_e=0.0$

Hallway $N_l=N_r=20$ evaluation with divergence constraint regularisation coefficients $\alpha_\beta$ and $\alpha_e$.  Shading indicates 95% confidence intervals; Part 2

$\alpha_\beta=0.1, \alpha_e=0.0001$

$\alpha_\beta=0.1, \alpha_e=0.001$

$\alpha_\beta=0.1, \alpha_e=0.01$

$\alpha_\beta=0.1, \alpha_e=0.1$

Hallway $N_l=N_r=20$ evaluation with divergence constraint regularisation coefficients $\alpha_\beta$ and $\alpha_e$.  Shading indicates 95% confidence intervals; Part 3

§.§ Sensitivity Analysis

Below, we analyse the sensitivity of DeA2C trained with Count intrinsic rewards and KL-divergence constraints in DeepSea $10$ and Hallway $N_l=N_r=10$ tasks following the evaluation procedure outlined in <Ref>. We consider both cases in which the exploration policy is trained using extrinsic and intrinsic rewards or is trained using only intrinsic rewards. All results are computed over five random seeds.

$\alpha_\beta=\alpha_e=0$, $r=r^e + \lambda r^i$

$\alpha_\beta=\alpha_e=0.1$, $r=r^e + \lambda r^i$

$\alpha_\beta=\alpha_e=0.1$, $r=\lambda r^i$

Average evaluation returns for DeA2C with Count in DeepSea 10 with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation of DeA2C with Count in DeepSea 10 for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$ and with or without KL-divergence constraints.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

DeA2C Count (standard)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=r^e + \lambda r^i$)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=\lambda r^i$)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

Average evaluation returns with a single standard deviation of DeA2C with Count in DeepSea 10 for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$ and with or without KL-divergence constraints.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

DeA2C Count (standard)     0.91(24)     0.95(18)     0.97(14)     0.96(15)     0.97(11)     0.94(13)     0.97(13)     0.93(15)     0.58(19)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=r^e + \lambda r^i$)     0.84(32)     0.89(28)     0.93(24)     0.96(17)     0.97(13)     0.97(13)     0.97(13)      0.74(16)     0.56(23)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=\lambda r^i$)     0.71(40)     0.83(34)     0.91(26)     0.95(19)     0.97(13)     0.97(13)     0.96(14)      0.44(24)     0.43(26)

$\alpha_\beta=\alpha_e=0$, $r=r^e + \lambda r^i$

$\alpha_\beta=0.01, \alpha_e=0.0001$, $r=r^e + \lambda r^i$

$\alpha_\beta=0.01, \alpha_e=0.0001$, $r=\lambda r^i$

Average evaluation returns for DeA2C with Count in Hallway $N_l=N_r=10$ with $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation of DeA2C with Count in Hallway $N_l=N_r=10$ for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$ and with or without KL-divergence constraints.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

DeA2C Count (standard)     0.17(26)     0.68(26)     0.85(0)     0.85(0)     0.85(0)     0.85(0)      0.82(2)     0.84(2)     0.82(2)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=r^e + \lambda r^i$)     0.17(34)     0.68(34)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.85(0)     0.84(2)     0.82(2)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=\lambda r^i$)     0.00(0)      0.33(40)      0.81(2)      0.83(2)      0.84(2)     0.85(0)      0.82(2)      0.81(2)     0.83(2)

Average evaluation returns with a single standard deviation of DeA2C with Count in Hallway $N_l=N_r=10$ for $\lambda \in \{0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 10.0, 100.0\}$ and with or without KL-divergence constraints.

Algorithm \ $\lambda$     0.01     0.1     0.25     0.5     1.0     2.0     4.0     10.0     100.0

DeA2C Count (standard)     0.17(2)     0.67(7)     0.84(9)     0.84(8)     0.84(9)     0.84(8)      0.67(10)     0.82(12)     0.80(6)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=r^e + \lambda r^i$)     0.17(2)     0.67(9)     0.84(9)     0.84(11)     0.84(7)     0.84(10)     0.83(10)     0.82(12)     0.80(6)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=\lambda r^i$)     0.00(0)      0.32(6)     0.78(12)     0.80(11)     0.83(19)     0.83(11)      0.67(10)     0.73(15)     0.80(9)

$\alpha_\beta=\alpha_e=0$, $r=r^e + \lambda r^i$

$\alpha_\beta=\alpha_e=0.1$, $r=r^e + \lambda r^i$

$\alpha_\beta=\alpha_e=0.1$, $r=\lambda r^i$

Average evaluation returns for DeA2C with Count in DeepSea 10 with varying count increments in $\{0.01, 0.1, 0.2, 1.0, 5.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation of DeA2C with Count in DeepSea 10 with varying count increments and with or without KL-divergence constraints.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

DeA2C Count (standard)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=r^e + \lambda r^i$)     0.59(40)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=\lambda r^i$)      0.59(40)      0.79(30)      0.79(30)     0.99(0)     0.99(0)     0.99(0)     0.99(0)

Average evaluation returns with a single standard deviation of DeA2C with Count in DeepSea 10 with varying count increments and with or without KL-divergence constraints.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

DeA2C Count (standard)     0.55(20)     0.68(15)     0.82(21)     0.98(10)     0.95(18)     0.94(18)     0.92(21)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=r^e + \lambda r^i$)     0.15(17)     0.68(15)     0.75(16)     0.97(13)     0.89(28)     0.86(30)     0.84(32)

DeA2C Count ($\alpha_\beta=\alpha_e=0.1$, $r=\lambda r^i$)     0.18(14)      0.34(20)      0.38(16)     0.97(13)     0.82(35)     0.79(36)     0.72(41)

$\alpha_\beta=\alpha_e=0$, $r=r^e + \lambda r^i$

$\alpha_\beta=0.01, \alpha_e=0.0001$, $r=r^e + \lambda r^i$

$\alpha_\beta=0.01, \alpha_e=0.0001$, $r=\lambda r^i$

Average evaluation returns for DeA2C with Count in Hallway $N_l=N_r=10$ with varying count increments in $\{0.01, 0.1, 0.2, 1.0, 5.0, 10.0, 100.0\}$. Shading indicates 95% confidence intervals.

Maximum evaluation returns with a single standard deviation of DeA2C with Count in Hallway $N_l=N_r=10$ with varying count increments and with or without KL-divergence constraints.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

DeA2C Count (standard)     0.85(0)     0.82(2)     0.82(2)     0.85(0)     0.57(40)     0.00(0)     0.00(0)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=r^e + \lambda r^i$)     0.85(0)     0.81(1)     0.83(2)     0.85(0)     0.68(26)     0.17(26)     0.00(0)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=\lambda r^i$)     0.85(0)     1.00(30)      0.80(0)      0.84(2)      0.32(32)     0.17(26)     0.00(0)

Average evaluation returns with a single standard deviation of DeA2C with Count in Hallway $N_l=N_r=10$ with varying count increments and with or without KL-divergence constraints.

Algorithm \ Count increment     0.01     0.1     0.2     1.0     5.0     10.0     100.0

DeA2C Count (standard)     0.84(7)     0.83(19)     0.81(12)     0.84(9)     0.67(5)     0.33(4)     0.17(2)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=r^e + \lambda r^i$)     0.84(7)     0.79(12)     0.81(12)     0.84(11)     0.67(7)     0.17(2)     0.00(0)

DeA2C Count ($\alpha_\beta=0.01, \alpha_e=0.0001$, $r=\lambda r^i$)     0.84(7)     0.83(19)      0.59(17)     0.83(10)      0.31(5)     0.16(3)     0.00(0)