# Introspective Experience Replay: Look Back When Surprised

Ramnath Kumar  Dheeraj Nagaraj

###### Abstract

In reinforcement learning (RL), experience replay-based sampling techniques
play a crucial role in promoting convergence by eliminating spurious
correlations. However, widely used methods such as uniform experience replay
(UER) and prioritized experience replay (PER) have been shown to have sub-
optimal convergence and high seed sensitivity respectively. To address these
issues, we propose a novel approach called Introspective Experience Replay
(IER) that selectively samples batches of data points prior to surprising
events. Our method builds upon the theoretically sound reverse experience
replay (RER) technique, which has been shown to reduce bias in the output of
Q-learning-type algorithms with linear function approximation. However, this
approach is not always practical or reliable when using neural function
approximation. Through empirical evaluations, we demonstrate that IER with
neural function approximation yields reliable and superior performance
compared to UER, PER, and hindsight experience replay (HER) across most tasks.

Machine Learning, ICML

##  1 Introduction

Reinforcement learning (RL) involves learning with dependent data derived from
trajectories of Markov processes. In this setting, the iterations of descent
algorithms designed for i.i.d. data co-evolve with the trajectory at hand,
leading to poor convergence. Experience replay (Lin, 1992) involves storing
the received data points in a large buffer and producing a random sample from
this buffer whenever the learning algorithm requires it. Therefore experience
replay is usually deployed with popular algorithms like DQN, DDPG, and TD3 to
achieve state-of-the-art performance (Mnih et al., 2015; Lillicrap et al.,
2015). It has been shown experimentally (Mnih et al., 2015) and theoretically
(Nagaraj et al., 2020) that these learning algorithms for Markovian data show
sub-par performance without experience replay.

The simplest and most widely used experience replay method is the uniform
experience replay (UER), where the data points stored in the buffer are
sampled uniformly at random every time a data point is queried (Mnih et al.,
2015). However, UER might pick uninformative data points most of the time,
which may slow down the convergence. For this reason, optimistic experience
replay (OER) and prioritized experience replay (PER) (Schaul et al., 2015)
were introduced, where samples with higher TD error (i.e., ‘surprise’) are
sampled more often from the buffer. Optimistic experience replay (originally
called “greedy TD-error prioritization”) was shown to have a high bias111We
use the term ‘bias’ here as used in (Schaul et al., 2015), which means biased
with respect to the empirical distribution over the replay buffer. This leads
to the algorithm predominantly selecting rare data points and ignoring the
rest. Prioritized experience replay was proposed to solve this issue (Schaul
et al., 2015) by using a sophisticated sampling approach. However, as shown in
our experiments outside of the Atari environments, PER still suffers from a
similar problem, and its performance can be highly sensitive to seed and
hyper-parameter. Although this speeds up the learning process in many cases,
its performance can be quite bad and erratic due to picking and choosing only
specific data points. The design of experience replay continues to be an
active field of research. Several other experience replay techniques like
Hindsight experience replay (HER) (Andrychowicz et al., 2017), Reverse
Experience Replay (RER) (Rotinov, 2019), and Topological Experience Replay
(TER) (Hong et al., 2022) have been proposed. An overview of these methods is
discussed in Section 2.

Even though these methods are widely deployed in practice, theoretical
analyses have been very limited. Recent results on learning dynamical systems
(Kowshik et al., 2021b, a) showed rigorously in a theoretical setting that RER
is the conceptually-grounded algorithm when learning from Markovian data.
Furthermore, this work was extended to the RL setting in (Agarwal et al.,
2021) to achieve efficient Q learning with linear function approximation. The
RER technique achieves good performance since reverse order sampling of the
data points prevents the build-up of spurious correlations in the learning
algorithm. In this paper, we build on this line of work and introduce
Introspective Experience Replay (IER). Roughly speaking, IER first picks top
$k$ ‘pivot’ points from a large buffer according to their TD error. It then
returns batches of data formed by selecting the consecutive points
_temporally_ before these pivot points. In essence, the algorithm _looks back
when surprised_. The intuition behind our approach is linked to the fact that
the agent should always associate outcomes to its past actions, just like in
RER. The summary of our approach is shown in Figure 1. This technique is an
amalgamation of Reverse Experience Replay (RER) and Optimistic Experience
Replay (OER), which only picks the points with the highest TD error.

Figure 1: An illustration of our proposed methodology when selecting $k$
batches in the CartPole environment. The red color is used to indicate the
batches being sampled from the replay buffer. The green samples are the un-
sampled states from the buffer. The arrow explicitly points the pivots and the
snapshot of the surprising state encountered.

Our main findings are summarized below:

Better Performance Against SOTA: Our proposed methodology (IER) outperforms
previous state-of-the-art baselines such as PER, UER and HER on most
environments (see Table 1, Section 5).

Conceptual Understanding: We consider a simple toy example where we understand
the differences between UER, RER, OER and IER (Section 3.2). This study
illustrates the better performance of our proposed method by showing a) why
naive importance sampling, like in OER suffers from poor performance, often
failing to learn any non-trivial policy and b) why techniques like UER and RER
are slow to learn.

Forward vs. Reverse: We show that the temporal direction (forward/reverse) to
consider after picking the pivot is non-trivial. We show empirically that IER
performs much better than its forward counterpart, IER (F) (see Table 3). This
gives evidence that causality plays a role in the success of IER.

Whole vs. Component Parts: Our method (IER) is obtained by combining two
methods RER (which picks the samples in the reverse order as received) and OER
(which greedily picks the samples with the largest TD error). We show that
neither of these components performs well compared to their amalgamation, IER
(Table 4).

Minimal hyperparameter tuning: Our proposed methodology uses minimal
hyperparameter tuning. We use the same policy network architecture, learning
rate, batch size, and all other parameters across all our runs for a given
environment. These hyperparameters are selected based on the setting required
to achieve SOTA performance on UER. However, we have few options available, a
Hindsight flag, leading to the H-IER sampler, and the Uniform Mixing fraction
leading to the U-IER sampler. Furthermore, for most of our experiments, we use
a mixing fraction of $0$.

Table 1: IER outperforms previous state-of-the-art baselines. These baselines
include samplers such as UER (Mnih et al., 2013), PER (Schaul et al., 2015),
and HER (Andrychowicz et al., 2017) across many environments. Results are from
13 different environments that cover a broad category of MDPs. These include a
few Atari environments (previously used to support the efficacy of UER, PER),
some Robotics environments (previously used to support the efficacy of HER),
and many other classes of environments, including Classic Control, Box 2D, and
Mujoco. More details about these experiments and their setup have been
discussed in Section 5.

Experience Replay Method | UER | PER | HER | IER
---|---|---|---|---
Best Performance Frequency | 1 | 0 | 1 | 11

##  2 Related Works and Comparison

###  2.1 Experience Replay Techniques

Experience replay involves storing consecutive temporally dependent data in a
(large) buffer in a FIFO order. Whenever a learning algorithm queries for
batched data, the experience replay algorithm returns a sub-sample from this
buffer such that this data does not hinder the learning algorithms due to
spurious correlations. The most basic form of experience replay is UER (Lin,
1992) which samples the data in the replay buffer uniformly at random. This
approach has significantly improved the performance of off-policy RL
algorithms like DQN (Mnih et al., 2015). Several other methods of sampling
from the buffer have been proposed since; PER (Schaul et al., 2015) samples
experiences from a probability distribution which assigns higher probability
to experiences with significant TD error and is shown to boost the convergence
speed of the algorithm. This outperforms UER in most Atari environments. HER
(Andrychowicz et al., 2017) works in the "what if" scenario, where even a bad
policy can lead the agent to learn what not to do and nudge the agent towards
the correct action. There have also been other approaches such as (Liu et al.,
2019; Fang et al., 2018, 2019) have adapted HER in order to improve the
overall performance with varying intuition. RER processes the data obtained in
a buffer in the reverse temporal order. We refer to the following sub-section
for a detailed review of this and related techniques. We will also consider
‘optimistic experience replay’ (OER), the naive version of PER, where at each
step, only top $B$ elements in the buffer are returned when batched data is
queried. This approach can become highly selective to the point of completely
ignoring certain data points, leading to poor performance. This is mitigated
by a sophisticated sampling procedure employed in PER. Other works such as
(Fujimoto et al., 2020; Pan et al., 2022) attempt to study PER and address
some of its shortcomings, and (Lahire et al., 2021) introduces ’large batch
experience replay’ (LaBER), which reduces the stochastic noise in gradients
while keeping the updates unbiased with respect to the empirical data
distribution in the replay buffer

###  2.2 Reverse Sweep Techniques

Reverse sweep or backward value iteration refers to methods that process the
data as received in reverse order. This has been studied in the context of
planning tabular MDPs (Dai & Hansen, 2007; Grześ & Hoey, 2013). We refer to
Section 4 for a brief overview of why these methods are considered. However,
this line of work assumes that the MDP and the transition functions are known.
Inspired by the behavior of biological networks, (Rotinov, 2019) proposed
reverse experience replay where the experience replay buffer is replayed in a
LIFO order. Since RER forms mini-batches with consecutive data points can be
unreliable with Neural approximation (i.e., it does not learn consistently
well across different environments). Therefore, the iterations of namelo are
‘mixed’ with UER. However, the experiments are limited and do not demonstrate
that this method outperforms even UER. A similar procedure named Episodic
Backward Update (EBU) is introduced in (Lee et al., 2019). However, to ensure
that the pure RER works well, the EBU method also seeks to change the target
for Q learning instead of just changing the sampling scheme in the replay
buffer. The reverse sweep was independently rediscovered as RER in the context
of streaming linear system identification in (Kowshik et al., 2021b), where
SGD with reverse experience replay was shown to achieve near-optimal
performance. In contrast, naive SGD was significantly sub-optimal due to the
coupling between the Markovian data and the SGD iterates. The follow-up work
(Agarwal et al., 2021) analyzed off-policy Q learning with linear function
approximation and reverse experience replay to provide near-optimal
convergence guarantees using the unique super martingale structure endowed by
reverse experience replay. (Hong et al., 2022) considers topological
experience replay, which executes reverse replay over a directed graph of
observed transitions. Mixed with PER enables non-trivial learning in some
challenging environments. Another line of work (Florensa et al., 2017; Moore &
Atkeson, 1993; Goyal et al., 2018; Schroecker et al., 2019) considers reverse
sweep with access to a simulator or using a fitted generative model. On the
other hand, our work only seeks on-policy access to the MDP.

##  3 Background and Proposed Methodology

We consider episodic reinforcement learning (Sutton & Barto, 2018), where at
each time step an agent takes actions $a_{t}$ in an uncertain environment with
state $s_{t}$, and receives a reward $r_{t}$. The environment then evolves
into a new state $s_{t+1}$ whose law depends only on $s_{t},a_{t}$. Our goal
is to (approximately) find the policy $\pi^{*}$ which maps the environmental
state $s$ to an action $a$ such that when the agent takes the action
$a_{t}=\pi^{*}(s_{t})$, the discounted reward
$\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right]$ is maximized. To
achieve this, we consider algorithms like DQN ((Mnih et al., 2015)), DDPG
((Lillicrap et al., 2015)) and TD3 ((Fujimoto et al., 2018)), which routinely
use experience replay buffers. In this paper, we introduce a new experience
replay method, IER, and investigate the performance of the aforementioned RL
algorithms with this modification. In this work, when we say “return”, we mean
discounted episodic reward.

Algorithm 1 Our proposed Introspective Experience Replay (IER) for
Reinforcement Learning

Input: Data collection mechanism $\mathbb{T}$, Data buffer $\mathcal{H}$,
Batch size $B$, grad steps per Epoch $G$, number of episodes $N$, Importance
function $I$, learning procedure $\mathbb{A}$, Uniform Mixing fraction $p$

$n\leftarrow 0$

while  _$n <N$_ do

$n\leftarrow n+1$ $\mathcal{H}\leftarrow\mathbb{T}(\mathcal{H})$ ;

// Add a new episode to the buffer $\mathcal{I}\leftarrow I(\mathcal{H})$ ;

// Compute importance of each data point in the buffer
$P\leftarrow\mathsf{Top}(\mathcal{I};G)$ ;

// Obtain index of top $G$ elements of $\mathcal{I}$ $g\leftarrow 0$ while
_$g <G$_ do

if  _$g <(1-p)G$_ then

$D\leftarrow\mathcal{H}[P[g]-B,P[g]]$ ;

// Load batch of previous $B$ examples from pivot $P[g]$

else

$D\leftarrow\mathcal{H}[\mathsf{Uniform}(\mathcal{H},B)]$ ;

// Randomly chose $B$ indices from buffer

end if

$g\leftarrow g+1$ $\mathbb{A}(D)$ ;

// Run the learning algorithm with batch data $D$

end while

end while

###  3.1 Methodology

We now describe our main method in a general way where we assume that we have
access to a data collection mechanism $\mathbb{T}$ which samples new data
points. This then appends the sampled data points to a buffer $\mathcal{H}$
and discards some older data points. The goal is to run an iterative learning
algorithm $\mathbb{A}$, which learns from batched data of batch size $B$ in
every iteration. We also consider an important metric $I$ associated with the
problem. At each step, the data collection mechanism $\mathbb{T}$ collects a
new episode and appends it to the buffer $\mathcal{H}$ and discards some old
data points, giving us the new buffer as
$\mathcal{H}\leftarrow\mathbb{T}(\mathcal{H})$. We then sort the entries of
$\mathcal{H}$ based on the importance metric $I$ and store the indices of the
top $G$ data points in an array $P=[P[0],\dots,P[G-1]]$. Then for every index
in $P$, we run the learning algorithm $\mathbb{A}$ with the batch
$D=(\mathcal{H}(P[i]),\dots,\mathcal{H}(P[i]-B+1))$. In some cases, we can
‘mix’222Mixing here denotes sampling with a given probability from one sampler
A, and filling the remaining samples of a batch with sampler B. this with the
standard UER sampling mechanism to reduce bias of the stochastic gradients
with respect to the empirical distribution in the buffer, as shown below. Our
experiments show that this amalgamation helps convergence in certain cases. We
describe this procedure in Algorithm 1.

In the reinforcement learning setting, $\mathbb{T}$ runs an environment
episode with the current policy and appends the transitions and corresponding
rewards to the buffer $\mathcal{H}$ in the FIFO order, maintaining a total of
$1E6$ data points, usually. We choose $\mathbb{A}$ to be an RL algorithm like
TD3 or DQN or DDPG. The importance function $I$ is the magnitude of the TD
error with respect to the current Q-value estimate provided by the algorithm
$\mathbb{A}$ (i.e.,
$I=|Q(s,a)-R(s,a)-\gamma\sup_{a^{\prime}}Q^{\mathsf{target}}(s^{\prime},a^{\prime})|$).
When the data collection mechanism ($\mathbb{T}$) is the same as in UER, we
will call this method IER. In optimistic experience replay (OER), we take
$\mathbb{T}$ to be the same as in UER. However, we query top $BG$ data points
from the buffer $\mathcal{H}$ and return $G$ disjoint batches each of size $B$
from these ‘important’ points. It is clear that IER is a combination of OER
and RER. Notice that we can also consider the data collection mechanism like
that of HER, where examples are labeled with different goals, i.e.
$\mathbb{T}$ has now been made different, keeping the sampling process exactly
same as before. In this case, we will call our algorithm H-IER. Our experiment
in Enduro and Acrobat depicts an example of this successful coalition. We also
consider the RER method, which served as a motivation for our proposed
approach. Under this sampling methodology, the batches are drawn from
$\mathcal{H}$ in the temporally reverse direction. This approach is explored
in the works mentioned in Section 2.2. We discuss this methodology in more
detail in Appendix F.

###  3.2 Didactic Toy Example

In this section, we discuss the working of IER on a simple environment such as
GridWorld-1D, and compare this with some of our baselines such as UER, OER,
RER, and IER (F). In this environment, the agent lives on a discrete
1-dimensional grid of size 40 with a max-timestep of 1000 steps, and at each
time step, the agent can either move left or right by one step. The agent
starts from the starting state (S; $[6]$), the goal of the agent is to reach
goal state (G; $[40]$) getting a reward of $+1$, and there is also a trap
state (T; $[3]$), where the agents gets a reward of $-2$. The reward in every
other state is $0$. For simplicity, we execute an offline exploratory policy
where the agent moves left or right with a probability of half and obtain a
buffer of size 30000. The rewarding states occur very rarely in the buffer
since it is hard to reach for this exploration policy. The episode ends upon
meeting either of two conditions: (i) the agent reaches the terminal state,
which is the goal state, or (ii) the agent has exhausted the max-timestep
condition and has not succeeded in reaching any terminal state. An overview of
our toy environment is depicted in Figure 2(a). Other hyperparameters crucial
to replicating this experiment are described in Appendix B.

In this example, reaching the goal state as quickly as possible is vital to
receive a positive reward and avoid the fail state. Therefore, it is essential
to understand the paths which reach the goal state. Figure 2(b) depicts the
number of times each state occurs in the buffer. Furthermore, the remaining
subplots of Figure 2 depict the Absolute Frequency of our off-policy algorithm
trained in this environment. A state’s “absolute frequency” is the number of
times the replay technique samples a given state during the algorithm’s run.
The experiments on this simple didactic toy environment do highlight a few
interesting properties:
Comparison of UER and IER: Since the goal state appears very rarely in buffer,
UER and RER rarely sample the goal state and hence do not manage to learn
effectively. While RER naturally propagates the information about the reward
back in time to the states that led to the reward, it does not often sample
the rewarding state.
Limitation of OER: While OER samples a lot from the states close to the goal
state, the information about the reward does not propagate to the start state.
We refer to the bottleneck in Figure 2(e) where some intermediate states are
not sampled.
Advantage of IER: IER prioritizes sampling from the goal state and propagates
the reward backward so that the entire path leading to the reward is now aware
of how to reach the reward. Therefore, a combination of RER and OER reduces
the sampling bias in OER by preventing the bottlenecks seen in Figure 2(e).
Bottleneck of IER (F): IER (F) has a more significant bottleneck when compared
to RER and chooses to sample the non-rewarding middle states most often. Also,
note that whenever IER (F) chooses the goal state as the pivot, it selects the
rest of the batch to overflow into the next episode, which begins at the
starting state. This does not allow the algorithm to effectively learn the
path which _led_ to the goal state.

(a) GridWorld-1D environment

(b) Buffer

(c) UER

(d) RER

(e) OER

(f) IER

(g) IER (F)

Figure 2: Gridworld-1D environment is depicted in Figure 2(a). Distribution of
states in the buffer (Figure 2(b)) and relative frequency of different
experience replay samplers on the didactic example of GridWorld-1D environment
(Figure 2(c);2(d);2(e);2(f);2(g)).

The toy example above models several salient features in more complicated RL
environments.

(i) In the initial stages of learning, the exploratory policy is essentially
random, and such a naive exploratory policy does not often lead to non-trivial
rewards.

(ii) Large positive and negative reward states (the goal and trap states), and
their neighbors provide the pivot state for IER and OER.

We show empirically that this holds in more complicated environments as well.
Figure 3 depicts the surprise vs. reward for the Ant environment. Here we see
a strong correlation between absolute reward and TD error (“Surprise factor”).

(a) UER

(b) RER

(c) IER

Figure 3: Relationship between absolute values of TD Error (Surprise factor)
and Reward for the Ant environment.

##  4 Understanding Reverse Replay

There are various conceptual ways we can look at RER and IER. This section
outlines some of the motivations behind using this technique. Theoretical
works such as (Agarwal et al., 2021; Kowshik et al., 2021b) have established
rigorous theoretical guarantees for RER by utilizing super martingale
structures. This structure is not present in forward replay techniques (i.e.,
the opposite of reverse replay) as shown in (Kowshik et al., 2021a). We refer
to Appendix D, where we show via an ablation study that going forward in time
instead of reverse does not work very well. We give the following explanations
for the success of IER.
Propogation of Sparse Rewards: In many RL problems, non-trivial rewards are
sparse and only received at the goal states. Therefore, processing the data
backward in time from such goal states helps the algorithm learn about the
states that led to this non-trivial reward. Our study (see Figure 3 and
Appendix E for further details) shows that in many environments IER picks
pivots which are the states with large (positive or negative) rewards,
enabling effective learning.
More spread than OER: OER, which greedily chooses the examples with the
largest TD error to learn from, performs very poorly since it is overtly
selective to the point of ignoring most of the states. To illustrate this
phenomenon, we refer to the didactic example in Section 3.2. One possible way
of viewing IER is that RER is used to reduce this affinity to picking a
minimal subset of states in OER. PER is designed to achieve a similar outcome
with a sophisticated and computationally expensive sampling scheme over the
buffer.
Causality: MDPs have a natural causal structure: actions and events in the
past influence the events in the future. Therefore, whenever we see a
surprising or unexpected event, we can understand why or how it happened by
looking into the past. Further theoretical work is needed to realize IER
better.

We further illustrate the same by referring to the straightforward didactic
example (Section 3.2), where we can see the effects of each of the experience
replay methods. We also demonstrate superior performance on more complicated
environments (Section 5), showcasing the robustness of our approach with
minimal hyperparameter tuning.

##  5 Experimental Results

In this section, we briefly discuss our experimental setup as well as the
results of our experiments.

#### Environments:

We evaluate our approach on a diverse class of environments, such as (i)
Environments with low-dimensional state space (including classic control and
Box-2D environments), (ii) Multiple joint dynamic simulations and robotics
environments (including Mujoco and Robotics environments), and (iii) Human-
challenging environments (such as Atari environments). Note that previous
seminal papers in the field of experience replay, such as (Mnih et al., 2013),
(Schaul et al., 2015), and (Andrychowicz et al., 2017), showed the efficacy of
their approach on a subset of these classes. For instance, UERand PER was
shown to work well on Atari games. Furthermore, HER was effective in the
Robotics environments such as FetchReach. In this work, we perform a more
extensive study to show the robustness and effectiveness of our model not just
in Atari and Robotics environments but also in Mujoco, Box2D, and Classic
Control environments. Due to computational limitations and the non-
reproducibility of some baselines, we could not extend our experiments to some
Atari environments. We refer to Appendix A for a brief description of the
environments used.

#### Hyperparameters:

Refer to Appendix B for the exact hyperparameters used. Across all our
experiments on various environments, we use a standard setting for all the
different experience replay buffers. This classic setting is set so we can
reproduce state-of-the-art performance using UER on the respective
environment. For most of our experiments, we set the uniform mixing fraction
($p$) from Algorithm 1 to be 0. We use a non-zero $p$ value only for a few
environments to avoid becoming overtly selective while training, as described
in Appendix B. For PER, we tune the $\alpha$ and $\beta$ hyperparameters used
in the (Schaul et al., 2015) paper across all environments other than Atari.
The default values of $\alpha=0.4$ and $\beta=0.6$ are robust on Atari
environments as shown by extensive hyperparameter search by (Schaul et al.,
2015). We detail the results from our grid search in Appendix C.2.

#### Metric:

To compare the different models, we use the Top-K seeds moving average return
as the evaluation metric across all our runs. Top-K seeds here mean we take
the average of $k=3$ seeds that gave the best performance out of a total of
$n=5$ trials. This measures the average performance of the top $k/n$ quantile
of the trajectories. This factors in the seed sensitivity and is robust
despite the highly stochastic learning curves encountered in RL, where there
is a non-zero probability that non-trivial policies are not learned after many
steps. It is common to use top-1 and Top-K trials to be selected from among
many trials in the reinforcement learning literature (see (Schaul et al.,
2015);(Sarmad et al., 2019);(Wu et al., 2017);(Mnih et al., 2016)).
Furthermore, we motivate our design choice of using the Top-K metric as our
metric of choice from our study in Appendix C.1. We show that the Top-K metric
is more stable and suitable for comparing the performances of different RL
agents, which generally suffer from highly volatile learning curves, as
described above. Moving average with a given window size is taken for learning
curves (with a window size of 20 for FetchReach and 50 for all others) to
reduce the variation in return which is inherently present in each epoch. We
argue that taking a moving average is essential since, usually, pure noise can
be leveraged to pick a time instant where a given method performs best
(Henderson et al., 2018). Considering the Top-K seed averaging of the last
step performance of the moving average of the learning curves gives our metric
- the Top-K seeds moving average return.

#### Comparison with SOTA:

Table 2 depicts our results in various environments upon using different SOTA
replay sampler mechanisms (UER, PER and HER). Our proposed sampler outperforms
all other baselines in most tasks and compares favorably in others. Our
experiments on various environments across various classes, such as classic
control, and Atari, amongst many others, show that our proposed methodology
consistently outperforms all other baselines in most environments, as
summarized in Table 1. Furthermore, our proposed methodology is robust across
various environments, as highlighted in Table 2. The learning curves for our
experiments have been depicted in the Appendix. (see Appendix C)

Table 2: Top-K seeds Moving Average Return results across various
environments. From our experiments, we note that IER outperforms previous SOTA
baselines in most environments. Appendix B depicts the hyperparameters used
for the experiment.

Dataset | UER | PER | HER | IER
---|---|---|---|---
CartPole | 153.14 $\pm$ 32.82 | 198.06 $\pm$ 3.68 | 173.84 $\pm$ 26.11 |  199.83 $\pm$ 0.31
Acrobot | -257.93 $\pm$ 184.28 | -233.45 $\pm$ 127.99 | -389.42 $\pm$ 113.22 |  -193.90 $\pm$ 57.56
Inverted Pendulum | -161.93 $\pm$ 10.55 | -155.05 $\pm$ 9.95 | -629.42 $\pm$ 815.24 |  -150.27 $\pm$ 9.63
LunarLander | -4.42 $\pm$ 20.06 | 11.21 $\pm$ 15.51 | 6.00 $\pm$ 10.02 |  12.32 $\pm$ 27.55
HalfCheetah | 10808.93 $\pm$ 1094.32 | 99.75 $\pm$ 1124.46 |  11072.54 $\pm$ 297.12 | 10544.88 $\pm$ 342.01
Ant | 3932.85 $\pm$ 1024.86 | -2699.84 $\pm$ 1.34 | 3803.17 $\pm$ 996.81 |  4203.21 $\pm$ 345.22
Reacher | -4.97 $\pm$ 0.31 | -5.42 $\pm$ 0.61 | -5.30 $\pm$ 0.50 |  -4.92 $\pm$ 0.27
Walker | 3597.03 $\pm$ 1203.79 | 1709.48 $\pm$ 1635.90 | 889.82 $\pm$ 1427.92 |  4349.29 $\pm$ 680.35
Hopper | 3072.65 $\pm$ 621.40 | 784.00 $\pm$ 1536.00 | 2685.72 $\pm$ 1221.81 |  3205.05 $\pm$ 406.35
Inverted Double Pendulum | 8489.54 $\pm$ 927.69 | 8654.78 $\pm$ 981.67 | 9002.05 $\pm$ 464.20 |  9241.61 $\pm$ 91.68
Fetch-Reach | -1.84 $\pm$ 0.57 | -49.90 $\pm$ 0.10 | -2.92 $\pm$ 1.79 |  -1.74 $\pm$ 0.24
Pong |  19.15 $\pm$ 1.32 | 17.02 $\pm$ 3.27 | 18.70 $\pm$ 1.02 | 19.10 $\pm$ 1.20
Enduro | 227.01 $\pm$ 319.15 | 565.23 $\pm$ 116.36 | 514.32 $\pm$ 132.39 |  586.32 $\pm$ 111.44

#### Forward vs. Reverse:

The intuitive limitation to the "looking forward" approach is that in many RL
problems, the objective for the agent is to reach a final goal state, where
the non-trivial reward is obtained. Since non-trivial rewards are only offered
in this goal state, it is informative to look back from here to learn about
the states that _lead_ to this. When the goals are sparse, the TD error is
more likely to be large upon reaching the goal state. Our algorithm selects
these as pivots, and IER (F) might select batches overflowing into the next
episode. Our studies on many environments (see Figure 3 and Appendix E) show
that the pivot points selected based on importance indeed have large (positive
or negative) rewards. Our experiments depicted in Table 3 show that IER
outperforms IER (F) in most environments.

Table 3: Top-K seeds Moving Average Return results across various environments
for Temporal Ablation study between IER (F) and IER (R; default). Note that we
use the base setting of IER in this section to avoid spurious comparisons
(i.e., with $p=0$ and no hindsight).

Dataset | Forward | Reverse
---|---|---
CartPole | 196.51 $\pm$ 6.26 |  199.83 $\pm$ 0.31
Acrobot | -423.15 $\pm$ 108.08 |  -313.03 $\pm$ 190.27
Inverted Pendulum |  -882.13 $\pm$ 521.77 | -1111.51 $\pm$ 559.83
LunarLander | -22.75 $\pm$ 30.19 |  12.32 $\pm$ 27.55
HalfCheetah | 9369.30 $\pm$ 454.40 |  10108.15 $\pm$ 919.27
Ant | 2963.71 $\pm$ 828.50 |  4203.21 $\pm$ 345.22
Reacher | -5.25 $\pm$ 0.33 |  -4.92 $\pm$ 0.27
Walker | 2213.89 $\pm$ 1684.03 |  2830.03 $\pm$ 881.51
Hopper | 393.64 $\pm$ 181.89 |  505.58 $\pm$ 266.52
Inverted Double Pendulum |  9260.27 $\pm$ 95.59 |  9241.61 $\pm$ 91.68
FetchReach | -17.73 $\pm$ 27.91 |  -2.28 $\pm$ 1.11
Pong | 17.92 $\pm$ 2.60 |  19.10 $\pm$ 1.20
Enduro |  600.87 $\pm$ 149.99 | 525.84 $\pm$ 146.39

#### Whole vs. Component Parts:

Our approach is an amalgamation of OER and RER. Here we compare these
individual parts with IER. Table 4 describes the Top-K seeds Moving Average
Return across various environments in this domain. As demonstrated, IER
outperforms its component parts OER nor RER . Furthermore, we also motivate
each of our design choices such as the pivot point selection (see Appendix C.8
where we compare our proposed approach with it’s variant where the pivot
points are randomly selected), and temporal structure (see Appendix C.9 where
we compare our proposed approach with it’s variant where the points are
randomly sampled instead of temporally looking backward after selecting a
pivot point, and also buffer batch size sensitivity (see Appendix C.7).

Table 4: Top-K seeds Moving Average Return results across various environments
for ablation study between RER, OER, and IER.

Dataset | RER | OER | IER
---|---|---|---
CartPole | 163.93 $\pm$ 40.03 | 162.36 $\pm$ 34.89 |  199.83 $\pm$ 0.31
Acrobot | -320.70 $\pm$ 144.05 | -472.63 $\pm$ 31.21 |  -193.90 $\pm$ 57.56
Inverted Pendulum | -735.38 $\pm$ 613.5 | -166.57 $\pm$ 17.23 |  -150.27 $\pm$ 9.63
LunarLander | 9.84 $\pm$ 13.23 | -16.08 $\pm$ 15.38 |  12.32 $\pm$ 27.55
HalfCheetah | 9449.39 $\pm$ 648.60 | 2237.91 $\pm$ 2824.72 |  10544.88 $\pm$ 342.01
Ant | 2168.47 $\pm$ 415.53 | -47.93 $\pm$ 20.47 |  4203.21 $\pm$ 345.22
Reacher | -5.91 $\pm$ 0.37 | -5.28 $\pm$ 0.58 |  -4.92 $\pm$ 0.27
Walker | 1578.33 $\pm$ 1313.11 | 207.51 $\pm$ 193.06 |  4349.29 $\pm$ 680.35
Hopper | 206.23 $\pm$ 318.49 | 660.24 $\pm$ 580.77 |  3205.05 $\pm$ 406.35
Inverted Double Pendulum | 8953.44 $\pm$ 456.95 | 7724.99 $\pm$ 1726.58 |  9241.61 $\pm$ 91.68
Fetch-Reach | -49.94 $\pm$ 0.07 | -47.72 $\pm$ 3.33 |  -1.74 $\pm$ 0.24
Pong | 18.58 $\pm$ 1.75 | 3.52 $\pm$ 21.33 |  19.10 $\pm$ 1.20
Enduro | 483.98 $\pm$ 75.45 | 361.21 $\pm$ 86.30 |  586.32 $\pm$ 111.44

##  6 Discussion

We summarize our results and discuss possible future steps.

#### Speedup:

IER shows a significant speedup in terms of time complexity over PER as
depicted in Table 5. On average IER achieves a speedup improvement of
$26.20\%$ over PER across a large umbrella of environment classes.As the
network becomes more extensive or complicated, our approach does have a higher
overhead (especially computing TD error). Future work can investigate how to
further reduce the computational complexity of our method by computing the TD
error fewer times at the cost of operating with an older TD error. We also
notice a speedup of convergence toward a local-optimal policy of our proposed
approach, as shown in a few environments. Furthermore, the lack of speedup in
some of the other experiments (even if they offer an overall performance
improvement) could be since the "surprised" pivot cannot be successfully
utilized to teach the agent rapidly in the initial stages. We refer to
Appendix C for the learning curves.

Table 5: Average Speedup in terms of time complexity over PER across various
environment classes.

Environment | Average Speedup
---|---
Classic Control | 32.66% $\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{\uparrow}$
Box-2D | 54.32% $\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{\uparrow}$
Mujoco | 18.09% $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$
Robotics | 55.56% $\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{\uparrow}$
Atari | 6.53% $\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{\uparrow}$

#### Issues with stability and consistency

Picking pivot points by looking at the TD error might cause us to sample rare
events much more often and hence cause instability compared to UER as seen in
some environments like HalfCheetah, LunarLander, and Ant, where there is a
sudden drop in performance for some episodes (see Appendix C). We observe that
our strategy IER corrects itself quickly, unlike RER, which cannot do this
(see Figure 7(e)). Increasing the number of pivot points per episode (the
parameter $G$) and the uniform mixing probability $p$ usually mitigates this.
In this work, we do not focus on studying the exact effects of $p$ and $G$
since our objective was to obtain methods that require minimal hyper-parameter
tuning. However, future work can systematically investigate the significance
of these parameters in various environments.

#### Why does IER outperform the traditional RER?

The instability333i.e., sudden drop in performance without recovery like
catastrophic forgetting and unreliability of pure RER with neural
approximation has been noted in various works (Rotinov, 2019; Lee et al.,
2019), where RER is stabilized by mixing it with UER . (Hong et al., 2022)
stabilizes reverse sweep by mixing it with PER. This is an interesting
phenomenon since RER is near-optimal in the tabular and linear approximation
settings (Agarwal et al., 2021). Two explanations of this are i) The loss
function used to train the neural network is highly non-convex, which hinders
the working of RER and ii) The proof given in (Agarwal et al., 2021) relies
extensively on ‘coverage’ of the entire state-action space - that is, the
entire state-action space is visited enough number of times - which might not
hold, as shown in the toy example in Section 3.2.

##  7 Conclusion

In conclusion, our proposed approach, Introspective Experience Replay (IER),
has shown significant promise as a solution for improving the convergence and
robustness of reinforcement learning algorithms. We have demonstrated through
extensive experiments that IER outperforms state-of-the-art techniques such as
uniform experience replay (UER), prioritized experience replay (PER), and
hindsight experience replay (HER) on a wide range of tasks, and also shows a
significant average speedup improvement in terms of time complexity over PER.
One of the key strengths of our proposed approach is its ability to
selectively sample batches of data points prior to surprising events. This
allows for the removal of bias and spurious correlations that can impede the
convergence of RL algorithms. Additionally, our method is based on the
theoretically rigorous reverse experience replay (RER) technique, which adds a
further level of rigor to our approach.

Overall, our proposed approach, Introspective Experience Replay (IER), is a
promising solution that offers significant improvements over existing methods
and has the potential to advance the field of reinforcement learning. We
believe that our proposed approach could be widely adopted in various RL
applications and have a positive impact on the field of AI.

## Reproducibility Statement

In this paper, we work with thirteen datasets, all of which are open-sourced
in gym (https://github.com/openai/gym). More information about the
environments is available in Appendix A. We predominantly use DQN, DDPG and
TD3 algorithms in our research, both of which have been adapted from their
open-source code. We also experimented with seven different replay buffer
methodologies, all of which have been adapted from their source
code444https://github.com/rlworkgroup/garage. More details about the models
and hyperparameters are described in Appendix B. All runs have been run using
the A100-SXM4-40GB, TITAN RTX, and V100 GPUs. Our source code is made
available for additional reference 555https://github.com/google-research/look-
back-when-surprised.

## References

  * Agarwal et al. (2021) Agarwal, N., Chaudhuri, S., Jain, P., Nagaraj, D., and Netrapalli, P.  Online target q-learning with reverse experience replay: Efficiently finding the optimal policy for linear mdps.  _arXiv preprint arXiv:2110.08440_ , 2021.
  * Andrychowicz et al. (2017) Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W.  Hindsight experience replay.  _Advances in neural information processing systems_ , 30, 2017.
  * Barto et al. (1983) Barto, A. G., Sutton, R. S., and Anderson, C. W.  Neuronlike adaptive elements that can solve difficult learning control problems.  _IEEE transactions on systems, man, and cybernetics_ , (5):834–846, 1983.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Dai & Hansen (2007) Dai, P. and Hansen, E. A.  Prioritizing bellman backups without a priority queue.  In _ICAPS_ , pp. 113–119, 2007.
  * Fang et al. (2018) Fang, M., Zhou, C., Shi, B., Gong, B., Xu, J., and Zhang, T.  Dher: Hindsight experience replay for dynamic goals.  In _International Conference on Learning Representations_ , 2018.
  * Fang et al. (2019) Fang, M., Zhou, T., Du, Y., Han, L., and Zhang, Z.  Curriculum-guided hindsight experience replay.  _Advances in neural information processing systems_ , 32, 2019.
  * Florensa et al. (2017) Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P.  Reverse curriculum generation for reinforcement learning.  In _Conference on robot learning_ , pp. 482–495. PMLR, 2017.
  * Fujimoto et al. (2018) Fujimoto, S., Hoof, H., and Meger, D.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pp. 1587–1596. PMLR, 2018.
  * Fujimoto et al. (2020) Fujimoto, S., Meger, D., and Precup, D.  An equivalence between loss functions and non-uniform sampling in experience replay.  _Advances in neural information processing systems_ , 33:14219–14230, 2020.
  * Goyal et al. (2018) Goyal, A., Brakel, P., Fedus, W., Singhal, S., Lillicrap, T., Levine, S., Larochelle, H., and Bengio, Y.  Recall traces: Backtracking models for efficient reinforcement learning.  _arXiv preprint arXiv:1804.00379_ , 2018.
  * Grześ & Hoey (2013) Grześ, M. and Hoey, J.  On the convergence of techniques that improve value iteration.  In _The 2013 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–8. IEEE, 2013.
  * Henderson et al. (2018) Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D.  Deep reinforcement learning that matters.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Hong et al. (2022) Hong, Z.-W., Chen, T., Lin, Y.-C., Pajarinen, J., and Agrawal, P.  Topological experience replay.  _arXiv preprint arXiv:2203.15845_ , 2022.
  * Kowshik et al. (2021a) Kowshik, S., Nagaraj, D., Jain, P., and Netrapalli, P.  Near-optimal offline and streaming algorithms for learning non-linear dynamical systems.  _Advances in Neural Information Processing Systems_ , 34, 2021a.
  * Kowshik et al. (2021b) Kowshik, S., Nagaraj, D., Jain, P., and Netrapalli, P.  Streaming linear system identification with reverse experience replay.  _Advances in Neural Information Processing Systems_ , 34, 2021b.
  * Lahire et al. (2021) Lahire, T., Geist, M., and Rachelson, E.  Large batch experience replay.  _arXiv preprint arXiv:2110.01528_ , 2021.
  * Lee et al. (2019) Lee, S. Y., Sungik, C., and Chung, S.-Y.  Sample-efficient deep reinforcement learning via episodic backward update.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Lin (1992) Lin, L.-J.  Self-improving reactive agents based on reinforcement learning, planning and teaching.  _Machine learning_ , 8(3):293–321, 1992.
  * Liu et al. (2019) Liu, H., Trott, A., Socher, R., and Xiong, C.  Competitive experience replay.  _arXiv preprint arXiv:1902.00528_ , 2019.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Mnih et al. (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pp. 1928–1937. PMLR, 2016.
  * Moore & Atkeson (1993) Moore, A. W. and Atkeson, C. G.  Prioritized sweeping: Reinforcement learning with less data and less time.  _Machine learning_ , 13(1):103–130, 1993.
  * Nagaraj et al. (2020) Nagaraj, D., Wu, X., Bresler, G., Jain, P., and Netrapalli, P.  Least squares regression with markovian data: Fundamental limits and algorithms.  _Advances in neural information processing systems_ , 33:16666–16676, 2020.
  * Pan et al. (2022) Pan, Y., Mei, J., Farahmand, A.-m., White, M., Yao, H., Rohani, M., and Luo, J.  Understanding and mitigating the limitations of prioritized replay.  In _The 38th Conference on Uncertainty in Artificial Intelligence_ , 2022.
  * Plappert et al. (2018) Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., et al.  Multi-goal reinforcement learning: Challenging robotics environments and request for research.  _arXiv preprint arXiv:1802.09464_ , 2018.
  * Rotinov (2019) Rotinov, E.  Reverse experience replay.  _arXiv preprint arXiv:1910.08780_ , 2019.
  * Sarmad et al. (2019) Sarmad, M., Lee, H. J., and Kim, Y. M.  Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5898–5907, 2019.
  * Schaul et al. (2015) Schaul, T., Quan, J., Antonoglou, I., and Silver, D.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ , 2015.
  * Schroecker et al. (2019) Schroecker, Y., Vecerik, M., and Scholz, J.  Generative predecessor models for sample-efficient imitation learning.  _arXiv preprint arXiv:1904.01139_ , 2019.
  * Schulman et al. (2015) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ , 2015.
  * Sutton (1995) Sutton, R. S.  Generalization in reinforcement learning: Successful examples using sparse coarse coding.  _Advances in neural information processing systems_ , 8, 1995.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Todorov et al. (2012) Todorov, E., Erez, T., and Tassa, Y.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , pp. 5026–5033. IEEE, 2012.
  * Wawrzyński (2009) Wawrzyński, P.  A cat-like robot real-time learning to run.  In _International Conference on Adaptive and Natural Computing Algorithms_ , pp. 380–390. Springer, 2009.
  * Wu et al. (2017) Wu, Y., Mansimov, E., Grosse, R. B., Liao, S., and Ba, J.  Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation.  _Advances in neural information processing systems_ , 30, 2017.

##  Appendix A Environments

For all OpenAI environments, data is summarized from
https://github.com/openai/gym, and more information is provided in the wiki
https://github.com/openai/gym/wiki. Below we briefly describe some of the
tasks we experimented on in this paper.

###  A.1 CartPole-v0

CartPole, as introduced in (Barto et al., 1983), is a task of balancing a pole
on top of the cart. The cart has access to position and velocity as its state
vector. Furthermore, it can go either left or right for each action. The task
is over when the agent achieves 200 timesteps without a positive reward
(balancing the pole) which is the goal state or has failed, either when (i)
the cart goes out of boundaries ($\pm$ 2.4 units off the center), or (ii) the
pole falls over (less than $\pm$ 12 deg). The agent is given a continuous
4-dimensional space describing the environment and can respond by returning
one of two values, pushing the cart either right or left.

###  A.2 Acrobot-v1

Acrobot, as introduced in (Sutton, 1995), is a task where the agent is given
rewards for swinging a double-jointed pendulum up from a stationary position.
The agent can actuate the second joint by one of three actions: left, right,
or no torque. The agent is given a six-dimensional vector comprising the
environment’s angles and velocities. The episode terminates when the end of
the second pole is over the base. Each timestep that the agent does not reach
this state gives a -1 reward, and the episode length is 500 timesteps.

###  A.3 Pendulum-v0

The inverted pendulum swingup problem, as introduced in (Lillicrap et al.,
2015) is based on the classic problem in control theory. The system consists
of a pendulum attached fixed at one end, and free at the other end. The
pendulum starts in a random position and the goal is to apply torque on the
free end to swing it into an upright position, with its center of gravity
right above the fixed point. The episode length is 200 timesteps, and the
maximum reward possible is 0, when no torque is being applied, and the object
has 0 velocity remaining at an upright configuration.

###  A.4 LunarLander-v2

The LunarLander environment introduced in (Brockman et al., 2016) is a classic
rocket trajectory optimization problem. The environment has four discrete
actions - do nothing, fire the left orientation engine, fire the right
orientation engine, and fire the main engine. This scenario is per
Pontryagin’s maximum principle, as it is optimal to fire the engine at full
throttle or turn it off. The landing coordinates (goal) is always at $(0,0)$.
The coordinates are the first two numbers in the state vector. There are a
total of 8 features in the state vector. The episode terminates if (i) the
lander crashes, (ii) the lander gets outside the window, or (iii) the lander
does not move nor collide with any other body.

###  A.5 HalfCheetah-v2

HalfCheetah is an environment based on the work by (Wawrzyński, 2009) adapted
by (Todorov et al., 2012). The HalfCheetah is a 2-dimensional robot with nine
links and eight joints connecting them (including two paws). The goal is to
apply torque on the joints to make the cheetah run forward (right) as fast as
possible, with a positive reward allocated based on the distance moved forward
and a negative reward is given for moving backward. The torso and head of the
cheetah are fixed, and the torque can only be applied to the other six joints
over the front and back thighs (connecting to the torso), shins (connecting to
the thighs), and feet (connecting to the shins). The reward obtained by the
agent is calculated as follows:

$r_{t}=\dot{x_{t}}-0.1*\left\|a_{t}\right\|_{2}^{2}$

###  A.6 Ant-v2

Ant is an environment based on the work by (Schulman et al., 2015) and adapted
by (Todorov et al., 2012). The ant is a 3D robot with one torso, a free
rotational body, and four legs. The task is to coordinate the four legs to
move in the forward direction by applying torques on the eight hinges
connecting the two links of each leg and the torso. Observations consist of
positional values of different body parts of the ant, followed by the
velocities of those individual parts (their derivatives), with all the
positions ordered before all the velocities. The reward obtained by the agent
is calculated as follows:

$r_{t}=\dot{x_{t}}-0.5*\left\|a_{t}\right\|_{2}^{2}-0.0005*\left\|s_{t}^{\text{contact}}\right\|_{2}^{2}+1$

###  A.7 Reacher-v2

The Reacher environment, as introduced in (Todorov et al., 2012), is a two-
jointed robot arm. The goal is to move the robot’s end effector (called
*fingertip*) close to a target that is spawned at a random positions. The
action space is a two-dimensional vector representing the torque to be applied
at the two joints. The state space consists of angular positions (in terms of
cosine and sine of the angle formed by the two moving arms), coordinates, and
velocity states for different body parts followed by the distance from target
for the whole object.

###  A.8 Hopper-v2

The Hopper environment, as introduced in (Todorov et al., 2012), sets out to
increase the number of independent state and control variables compared to
classic control environments. The hopper is a two-dimensional figure with one
leg that consists of four main body parts - the torso at the top, the thigh in
the middle, the leg at the bottom, and a single foot on which the entire body
rests. The goal of the environment is to make hops that move in the forward
(right) direction by applying torques on the three hinges connecting the body
parts. The action space is a three-dimensional element vector. The state space
consists of positional values for different body parts followed by the
velocity states of individual parts.

###  A.9 Walker-v2

The Walker environment, as builds on top of the Hopper environment introduced
in (Todorov et al., 2012), by adding another set of legs making it possible
for the robot to walker forward instead of hop. The hopper is a two-
dimensional figure with two legs that consists of four main body parts - the
torso at the top, two thighs in the middle, two legs at the bottom, and two
feet on which the entire body rests. The goal of the environment is to
coordinate both feel and move in the forward (right) direction by applying
torques on the six hinges connecting the body parts. The action space is a
six-dimensional element vector. The state space consists of positional values
for different body parts followed by the velocity states of individual parts.

###  A.10 Inverted Double-Pendulum-v2

Inverted Double-Pendulum as introduced in (Todorov et al., 2012) is built upon
the CartPole environment as introduced in (Barto et al., 1983), with the
infusion of Mujoco. This environment involves a cart that can be moved
linearly, with a pole fixed and a second pole on the other end of the first
one (leaving the second pole as the only one with one free end). The cart can
be pushed either left or right. The goal is to balance the second pole on top
of the first pole, which is on top of the cart, by applying continuous forces
on the cart. The agent takes a one-dimensional continuous action space in the
range [-1,1], denoting the force applied to the cart and the sign depicting
the direction of the force. The state space consists of positional values of
different body parts of the pendulum system, followed by the velocities of
those individual parts (their derivatives) with all the positions ordered
before all the velocities. The goal is to balance the double-inverted pendulum
on the cart while maximizing its height off the ground and having minimum
disturbance in its velocity.

###  A.11 FetchReach-v1

The FetchReach environment introduced in (Plappert et al., 2018) was released
as part of OpenAI Gym and used the Mujoco physics engine for fast and accurate
simulation. The goal is 3-dimensional and describes the desired position of
the object. Rewards in this environment are sparse and binary. The agent
obtains a reward of $0$ if the target location is at the target location
(within a tolerance of $5$ cm) and $-1$ otherwise. Actions are four-
dimensional, where 3 specifies desired gripper movement, and the last
dimension controls the opening and closing of the gripper. The FetchReach aims
to move the gripper to a target position.

###  A.12 Pong-v0

Pong, also introduced in (Mnih et al., 2013), is comparatively more accessible
than other Atari games such as Enduro. Pong is a two-dimensional sports game
that simulates table tennis. The player controls an in-game paddle by moving
vertically across the left and right sides of the screen. Players use this
paddle to hit the ball back and forth. The goal is for each player to reach
eleven points before the opponent, where the point is earned for each time the
agent returns the ball and the opponent misses.

###  A.13 Enduro-v0

Enduro, introduced in (Mnih et al., 2013), is a hard environment involving
maneuvering a race car in the National Enduro, a long-distance endurance race.
The goal of the race is to pass a certain number of cars each day. The agent
must pass 200 cars on the first day and 300 cars on all subsequent days.
Furthermore, as time passes, the visibility changes as well. At night in the
game, the player can only see the oncoming cars’ taillights. As the days’
progress, cars will become more challenging to avoid. Weather and time of day
are factors in how to play. During the day, the player may drive through an
icy patch on the road, which would limit control of the vehicle, or a patch of
fog may reduce visibility.

##  Appendix B Model and Hyperparameters

In this paper, we work with two classes of algorithms: DQN, DDPG and TD3. The
hyperparameters used for training our DQN algorithms in various environments
are described in Table 6. The hyperparameters used for training our DDPG
algorithms in various environments are described in Table 7. The
hyperparameters used for training DDPG are described in Table 7. Furthermore,
the hyperparameters used for training TD3 are described in Table 8.

Table 6: Hyperparameters used for training DQN on various environments.

Description | CartPole | Acrobot | LunarLander | Pong | Enduro | argument_name
---|---|---|---|---|---|---
General Settings |
Discount | $0.9$ | $0.9$ | $0.9$ | $0.99$ | $0.99$ | discount
Batch size | $512$ | $512$ | $512$ | $32$ | $32$ | batch_size
Number of epochs | $100$ | $100$ | $200$ | $150$ | $800$ | n_epochs
Steps per epochs | $10$ | $10$ | $10$ | $20$ | $20$ | steps_per_epoch
Number of train steps | $500$ | $500$ | $500$ | $125$ | $125$ | num_train_steps
Target update frequency | $30$ | $30$ | $10$ | $2$ | $2$ | target_update_frequency
Replay Buffer size | $1e^{6}$ | $1e^{6}$ | $1e^{6}$ | $1e^{4}$ | $1e^{4}$ | buffer_size
Algorithm Settings |
CNN Policy Channels | - | - | - | $(32,64,64)$ | $(32,64,64)$ | cnn_channel
CNN Policy Kernels | - | - | - | $(8,4,3)$ | $(8,4,3)$ | cnn_kernel
CNN Policy Strides | - | - | - | $(4,2,1)$ | $(4,2,1)$ | cnn_stride
Policy hidden sizes (MLP) | $(8,5)$ | $(8,5)$ | $(8,5)$ | $(512,)$ | $(512,)$ | pol_hidden_sizes
Buffer batch size | $64$ | $128$ | $128$ | $32$ | $32$ | batch_size
Exploration Settings |
Max epsilon | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | max_epsilon
Min epsilon | 0.01 | 0.1 | 0.1 | 0.01 | 0.01 | min_epsilon
Decay ratio | 0.4 | 0.4 | 0.12 | 0.1 | 0.1 | decay_ratio
Optimizer Settings |
Learning rate | $5e^{-5}$ | $5e^{-5}$ | $5e^{-5}$ | $1e^{-4}$ | $1e^{-4}$ | lr
PER Specific Settings |
Prioritization Exponent | $0.4$ | $0.6$ | $0.8$ | $0.4$ | $0.4$ | $\alpha$
Bias Annealing Parameter | $0.6$ | $0.6$ | $0.8$ | $0.6$ | $0.6$ | $\beta$
IER Specific Settings |
Use Hindsight for storing states | $-$ | $\checkmark$ | $-$ | $-$ | $\checkmark$ | use_hindsight
Mixing Factor (p) | $0$ | $0$ | $0$ | $0$ | $0$ | p

Table 7: Hyperparameters used for training DDPG on Pendulum environment.

Description | Pendulum | argument_name
---|---|---
General Settings |
Discount | $0.95$ | discount
Batch size | $256$ | batch_size
Number of epochs | $50$ | n_epochs
Steps per epochs | $50$ | steps_per_epoch
Number of train steps | $40$ | num_train_steps
Target update Tau | $0.01$ | target_update_frequency
Replay Buffer size | $1e^{6}$ | buffer_size
Algorithm Settings |
Policy hidden sizes (MLP) | $(400,300)$ | pol_hidden_sizes
QF hidden sizes (MLP) | $(400,300)$ | qf_hidden_sizes
Buffer batch size | $256$ | batch_size
Exploration Settings |
Exploration Policy | Ornstein Uhlenbeck Noise | exp_policy
Sigma | 0.2 | sigma
Optimizer Settings |
Policy Learning rate | $1e^{-4}$ | pol_lr
QF Learning rate | $1e^{-3}$ | qf_lr
PER Specific Settings |
Prioritization Exponent | $0.4$ | $\alpha$
Bias Annealing Parameter | $0.8$ | $\beta$
IER Specific Settings |
Use Hindsight for storing states | $-$ | use_hindsight
Mixing Factor (p) | $0$ | p

Table 8: Hyperparameters used for training TD3 on various environments.

Description | Ant | Reacher | Walker | Double-Pendulum | HalfCheetah | Hopper | FetchReach | argument_name
---|---|---|---|---|---|---|---|---
General Settings
Discount | $0.99$ | $0.99$ | $0.99$ | $0.99$ | $0.99$ | $0.99$ | $0.95$ | discount
Batch size | $250$ | $250$ | $250$ | $100$ | $100$ | $100$ | $256$ | batch_size
Number of epochs | $500$ | $500$ | $500$ | $750$ | $500$ | $500$ | $100$ | n_epochs
Steps per epochs | $40$ | $40$ | $40$ | $40$ | $20$ | $40$ | $50$ | steps_per_epoch
Number of train steps | $50$ | $50$ | $50$ | $1$ | $50$ | $100$ | $100$ | num_train_steps
Replay Buffer size | $1e^{6}$ | $1e^{6}$ | $1e^{6}$ | $1e^{6}$ | $1e^{6}$ | $1e^{6}$ | $1e^{6}$ | buffer_size
Algorithm Settings
Policy hidden sizes (MLP) | $(256,256)$ | $(256,256)$ | $(256,256)$ | $(256,256)$ | $(256,256)$ | $(256,256)$ | $(256,256)$ | pol_hidden_sizes
Policy noise clip | $0.5$ | $0.5$ | $0.5$ | $0.5$ | $0.5$ | $0.5$ | $0.5$ | pol_noise_clip
Policy noise | $0.2$ | $0.2$ | $0.2$ | $0.2$ | $0.2$ | $0.2$ | $0.2$ | pol_noise
Target update tau | $0.005$ | $0.005$ | $0.005$ | $0.005$ | $0.005$ | $0.005$ | $0.01$ | tau
Buffer batch size | $100$ | $100$ | $100$ | $100$ | $100$ | $100$ | $256$ | batch_size
Gaussian noise Exploration Settings
Max sigma | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | max_sigma
Min sigma | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | min_sigma
Optimizer Settings |  |
Policy Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-4}$ | $3e^{-4}$ | $1e^{-3}$ | $3e^{-4}$ | $1e^{-3}$ | pol_lr
QF Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | qf_lr
PER Specific Settings
Prioritization Exponent | $0.4$ | $0.4$ | $0.4$ | $0.8$ | $0.4$ | $0.4$ | $0.4$ | $\alpha$
Bias Annealing Parameter | $0.6$ | $0.6$ | $0.6$ | $0.6$ | $0.6$ | $0.2$ | $0.6$ | $\beta$
IER Specific Settings
Use Hindsight for storing states | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | use_hindsight
Mixing Factor (p) | $0$ | $0$ | $0.4$ | $0$ | $0.3$ | $0.8$ | $0$ | p

Additionally, we use Tabular MDPs for learning a policy in our toy example.
Since the environment is fairly simpler, and has very few states, function
approximation is unnecessary. For all the agents trained on GridWorld, we use
a common setting as described in Table 9.

Table 9: Hyperparameters used for training Tabular MDP on GridWorld-1D
environment.

Description | GridWorld | argument_name
---|---|---
Discount | $0.99$ | discount
Batch size | $1$ | batch_size
Number of epochs | $100$ | n_epochs
Replay Buffer size | $3e^{4}$ | buffer_size
Buffer batch size | $64$ | batch_size
Exploration factor | 0.3 | max_epsilon
Learning rate | $0.1$ | lr

##  Appendix C Additional Results

###  C.1 Analysis of Fault Tolerance in Reinforcement Learning

Selecting an appropriate metric for comparison in RL is crucial due to the
high variance in results attributed to the environment stochasticity and the
stochasticity in the learning process. As such, using the average performance
metric with small number of seeds can give statistically different
distributions in results. This has been highlighted by the works of (Henderson
et al., 2018). Reliable reinforcement learning still remains an open problem.
As such, using the average over all seeds might not give us a stable
comparison of the algorithms considered, unless we can increase our number of
seeds to large numbers such as > 20\. This is computationally very expensive
and such high number of trials are seldom used in practice. Instead, other
approaches use the maximum of N runs to report performance. This however is
not ideal since it overestimates your performance by a large amount. We can
see this with the PER hyper-parameter search where we deploy the best
performing hyper-parameter from the hyper-parameter grid search into an
independent experiment with Top-K out of n metric. We notice that this under-
performs the grid search result by a large margin. Thus, the top-1 metric is
not desirable.

Instead, we use the Top-K performance metric which we show below to be robust
to noise and is closer to providing the right analysis in comparison to
average metric approach. Similar to work such as (Mnih et al., 2016) which
used a top-5/50 experiments, we use a top-3/5 experiments to limit the
overestimation of results further.

To explain our decision choice of Top-K metric, we consider a fault-based
model for RL algorithm output and show via a simulation of this in a simple
toy example why Top-K seeds can give us a better way of inferring the best
algorithm than the average, especially when the number of independent trials
is low. RL algorithms usually perform well sometimes, and some other times
they fail badly. This has been noted in the literature (Henderson et al.,
2018) and can also be seen in our experiments. Since the deviations are so
large between seeds, this can be modeled as a ‘fault’ rather than a mean +
gaussian noise. To give a better understanding, let us elaborate with a
simplified scenario: Consider the following setting where we have 20 different
environments. There are two algorithms: A and B. Algorithm A gives a moving
average return of 0.9 50% of the time and a moving average return of 0 the
remaining times (on all 20 environments). Algorithm B, on the other hand,
gives a moving average return of 1 50% of the time and a moving average return
of 0 the remaining times (on all 20 environments). In reality, Algorithm B
performs better than Algorithm A in all 20 environments. We can conclude this
with the empirical average if we extend the number of experiments to very
large numbers, such as 50 seeds per environment or larger. We further extend
our above analysis by adding another Algorithm C that gives a moving average
return of 0.8 50% of the time and a moving average return of 0 the remaining
times (on all 20 environments).

In this simplified model, we test how well the Top-K metric and the average
metric perform in recovering the ground truth (i.e, Algorithm B) via Monte-
Carlo simulation with 500 trials. In each trial, we generate the return of
each algorithm over each of the 20 environments from the model described above
with 10 different random seeds. For each environment, we check the best
algorithm with the average metric and the best algorithm with the Top-K
metric. We compare this with the ground truth (i.e, algorithm B being the best
in all 20 environments). Figure 4(a) depicts the comparison between the
average metric, and Top-K metric with respect to the ground truth. As
illustrated, the Top-K metric is more robust to faults and is closer to the
ground truth than the other metrics. We further add gaussian noise of mean 0,
and standard deviation 0.2, keeping all other parameters constant. This noise
is added to fault runs and standard ones. We note little difference in our
results and depict the results averaged over 500 runs as depicted in Figure
4(b) Therefore, our model suggests that the Top-K metric is more robust to
faulty runs and can help facilitate the comparison of various learning
algorithms with high volatility.

(a) Without Gaussian Noise

(b) With Gaussian Noise ($\sigma=0.2$)

Figure 4: Analysis of Fault Tolerance Metrics.

###  C.2 Grid-Search for tuning PER hyperparameters

In this section we present the results from our grid search experiment for
tuning the hyperparameters for PER. We tune the bias annealing parameter
($\beta$), and the prioritization exponent ($\alpha$). We perform a robust
grid search across all environments we have experimented with with the range
of the priortization exponent and beta as $[0.2,0.4,0.6,0.8,1.0]$. Figure
LABEL:per_grid_search illustrates the performance of PER with varying
hyperparameter

We briefly summarize the findings from our grid-search experiment below:

  * •

On environments such as CartPole , Reacher, we notice only a marginal
differences between default hyper parameters ($\alpha=0.4$, $\beta=0.6$) and
the best hyper parameters tuned for the model.

  * •

For other environments such as HalfCheetah, Ant, FetchReach, and Walker, we
notice a significant gap between the performance of PER and IER even after a
thorough tuning of PER hyperparameters. For instance, IERoutperforms PER by
more that 8000 in terms of average return on HalfCheetah environment.
Furthemore, IER outperforms PER by almost 48 in terms of average return on
FetchReach. Finally, IER outperforms PER by more than 6200 in terms of average
return on Ant. On Walker, IER outperforms PER by almost 1500 in terms of
average return. For these environments, we report the results with the default
setting, i.e. $\alpha=0.4$ and $\beta=0.6$.

  * •

Some environments such as Acrobot, Pendulum, LunarLander, Hopper, and Double-
Inverted-Pendulum did show significant improvements when tuned for the
prioritization exponent, and bias annealing factor. We take the best
hyperparameters from the grid-search experiment, and re-run the code to
compute Top-K metrics (i.e, we pick top 3 out of 5 runs). However, from our
experiments, we show that even the select hyperparameter is not robust across
seeds, and overall performs worse than our proposed approach of IER across all
environments: Acrobot, Pendulum, LunarLander, Hopper, and Double-Inverted-
Pendulum.

  * •

For Atari environments such as Pong and Enduro, we use the default parameters
recommended by (Schaul et al., 2015), and do not perform a grid-search
experiment.

(a) CartPole-v0

(b) Acrobot-v1

(c) Pendulum-v0

(d) LunarLander-v2

(e) HalfCheetah-v2

(f) Ant-v2

(g) Reacher-v2

(h) Walker-v2

(i) Hopper-v2

(j) InvertedDoublePendulum-v2

(k) FetchReach-v1

Figure 5: Grid-Search of Prioritization exponent ($\alpha$), and Bias
Annealing parameter ($\beta$) respectively.

###  C.3 Performance of IER with Low-Dimensional State Space

This section briefly discusses our results on environments with a low-
dimensional state space, such as classic control environments (CartPole,
Acrobot, and Pendulum) and Box-2D environments (LunarLander). Figure 6 depicts
the learning curves of our DQN/DDPG agents in these environments.

(a) CartPole-v0

(b) Acrobot-v1

(c) Pendulum-v0

(d) LunarLander-v2

Figure 6: Learning curves of DQN/DDPG agents on Classic Control and Box-2D
environments.

We note that our proposed methodology can significantly outperform other
baselines for the classic control algorithms. Furthermore, Figure 6(a) shows
excellent promise as we achieve a near-perfect score across all seeds in one-
tenth of the time it took to train PER.

###  C.4 Performance in Multiple Joint Dynamics Simulation and Robotics
Environments

Multiple joint dynamic simulation environments (mujoco physics environments)
and robotics environments such as HalfCheetah, Ant, Inverted Double-Pendulum,
and FetchReach ((Todorov et al., 2012; Plappert et al., 2018)) are more
complex and enable us to study whether the agent can understand the physical
phenomenon of real-world environments. Figure 7 depicts the learning curves of
our TD3 agents in these environments. Again, our proposed methodology
outperforms all other baselines significantly in most of the environments
studied in this section. Additionally, it is essential to point out that our
proposed method shows an impressive speedup of convergence in Inverted Double
Pendulum and convergence to a much better policy in Ant.

(a) HalfCheetah-v2

(b) Ant-v2

(c) Reacher-v2

(d) Walker-v2

(e) Hopper-v2

(f) InvertedDoublePendulum-v2

(g) FetchReach-v1

Figure 7: Learning curves of TD3 agents on Mujoco and Robotics environments.

###  C.5 Performance of IER in Human Challenging Environments

This section briefly discusses our results on human-challenging environments
such as Atari environments (Pong and Enduro). These environments are highly
complex, and our algorithms take millions of steps to converge to a locally
optimal policy. Figure 8 depicts the learning curves of our DQN agents in
these environments. We note that our proposed methodology can perform
favorably when compared to other baselines for the Atari environments and can
reach large reward policies significantly faster than UER.

(a) Pong-v0

(b) Enduro-v0

Figure 8: Learning curves of DQN agents on Atari environments.

###  C.6 Whole vs. Component Parts

This section briefly presents the learning curves of our models on three
different sampling schemes: IER, OER and RER.

(a) CartPole-v0

(b) Acrobot-v1

(c) Pendulum-v0

(d) LunarLander-v2

(e) HalfCheetah-v2

(f) Ant-v2

(g) Reacher-v2

(h) Walker-v2

(i) Hopper-v2

(j) InvertedDoublePendulum-v2

(k) FetchReach-v1

(l) Pong-v0

(m) Enduro-v0

Figure 9: Ablation study of OER, RER and IER.

###  C.7 Buffer Batch size sensitivity of IER

This section briefly presents the sensitivity to the buffer batch size
hyperparameter for our proposed approach (IER). To analyze this, we run our
experiments on the CartPole environment with varying batch size of the range
2-256. Table 10 and Figure 10 depict the buffer batch size sensitivity results
from our proposed sampler.

Table 10: Buffer Batch size sensitivity of IER on the CartPole environment.

Buffer Batch Size | Average Reward
---|---
2 | 126.69 $\pm$ 41.42
4 | 192.33 $\pm$ 13.29
8 | 181.27 $\pm$ 32.13
16 | 199.24 $\pm$ 1.32
32 |  199.99 $\pm$ 0.001
64 | 199.83 $\pm$ 0.31
128 | 193.23 $\pm$ 10.08
256 | 179.95 $\pm$ 18.94

Figure 10: Buffer batch size sensitivity of IER sampler on the CartPole
Environment

###  C.8 How important is sampling pivots?

This section briefly presents the ablation study to analyze the importance of
sampling “surprising” states as pivots. As a baseline, we build a experience
replay where these pivots are randomly sampled from the buffer. The "looking
back" approach is used to create batches of data. For nomenclature, we refer
to our proposed approach (IER) to use the “TD Metric” sampling of pivots, and
the baseline that uses “Uniform” sampling of pivots. Table 11 and Figure 11
depict the buffer batch size sensitivity results from our proposed sampler.

Table 11: Importance of sampling pivots in our proposed approach (IER) on the
CartPole environment.

Sampling Scheme | Average Reward
---|---
TD Metric (IER) |  199.83 $\pm$ 0.31
Uniform (IER) | 136.71 $\pm$ 19.59

Figure 11: Ablation study of Importance Sampling of IER sampler on the
CartPole Environment. Here “Uniform Sampling” denotes the uniformly random
sampling of pivots, and “TD Metric Sampling” denotes our proposed approach
(IER).

###  C.9 How important is “looking back”?

This section briefly presents the ablation study to analyze the importance of
“looking back” after sampling pivots. As a baseline, we build a experience
replay where we sample uniformly instead of looking back. For nomenclature, we
refer to our proposed approach (IER) to use the “Looking Back” approach
(similar to IER), and the baseline that uses “Uniform” approach. We refer to
these two approaches as possible filling schemes, i.e. fill the buffer with
states once the pivot state is sampled.

Table 12 and Figure 12 depict the buffer batch size sensitivity results from
our proposed sampler.

Table 12: Importance of looking back in our proposed approach (IER) on the
CartPole environment.

Filling Scheme | Average Reward
---|---
Looking Back (IER) |  199.83 $\pm$ 0.31
Uniform (IER) | 182.5 $\pm$ 23.49

Figure 12: Ablation study of Filling Scheme of IER sampler on the CartPole
Environment. Here “Uniform Filling” denotes the uniformly random sampling of
states to fill after sampling the pivot state, and “Looking Back Filling”
denotes our proposed approach (IER).

##  Appendix D Ablation study of Temporal effects

This section studies the ablation effects of going temporally forward and
backward once we choose a pivot/surprise point. Furthermore, Figure 13 depicts
the learning curves of the two proposed methodologies. The forward sampling
scheme is worse in most environments compared to the reverse sampling scheme.

(a) CartPole-v0

(b) Acrobot-v1

(c) Pendulum-v0

(d) LunarLander-v2

(e) HalfCheetah-v2

(f) Ant-v2

(g) Reacher-v2

(h) Walker-v2

(i) Hopper-v2

(j) InvertedDoublePendulum-v2

(k) FetchReach-v1

(l) Pong-v0

(m) Enduro-v0

Figure 13: Ablation study of the effects of the temporal structure on the
performance of the agent.

##  Appendix E Sparsity and Rewards of Surprising States

###  E.1 Surprising States Have Large Rewards

In this section, we study the “learning from sparse reward” intuition provided
in Section 4 – i.e., we want to check if the states corresponding to large TD
error correspond to states with large (positive or negative) rewards. To test
the hypothesis, we consider a sampled buffer and plot the TD error of these
points in the buffer against the respective reward. Figure 3 shows the
distribution of TD error against reward for the sampled buffers in the Ant
environment. We see that high reward states (positive or negative) also have
higher TD errors. Therefore, our algorithm picks large reward states as
endpoints to learn in such environments.

###  E.2 Surprising States are Sparse and Isolated

Figure 14 and Figure 15 depict the distribution of “surprise”/TD error in a
sampled batch for CartPole and Ant environments respectively. These two
figures help show that the states with a large “surprise” factor are few and
that even though the pivot of a buffer has a large TD error, the rest of the
buffer typically does not.

Figure 14(d) and Figure 15(d) show a magnified view of Figure 14(c) and Figure
15(d) where the pivot point selected is dropped. This helps with a uniform
comparison with the remaining timesteps within the sampled buffer. Again, we
notice little correlation between the timesteps within the sampled buffer.

(a) UER

(b) RER

(c) IER

(d) IER Magnified

Figure 14: Normalized TD Error ("Surprise factor") of each timestep over three
different sampled buffers on the CartPole environment. Best viewed when
zoomed.

(a) UER

(b) RER

(c) IER

(d) IER Magnified

Figure 15: Normalized TD Error ("Surprise factor") of each timestep over three
different sampled buffers on the Ant environment. Best viewed when zoomed.

##  Appendix F Reverse Experience Replay (RER)

This section discusses our implementation of Reverse Experience Replay (RER),
which served as a motivation for our proposed approach. The summary of the RER
approach is shown in Figure 16. Furthermore, an overview of our implemented
approach to RER is described briefly in Algorithm 2.

Figure 16: An illustration of Reverse Experience Replay (RER) when selecting
$k$ batches from the Replay Buffer. Algorithm 2 Reverse Experience Replay

Input: Data collection mechanism $\mathbb{T}$, Data buffer $\mathcal{H}$,
Batch size $B$, grad steps per Epoch $G$, number of episodes $N$, learning
procedure $\mathbb{A}$

$n\leftarrow N$ $P\leftarrow\mathsf{len}(\mathcal{H})$ ;

// Set index to last element of Buffer $\mathcal{H}$ while  _$n <N$_ do

$n\leftarrow n+1$ $\mathcal{H}\leftarrow\mathbb{T}(\mathcal{H})$ ;

// Add a new episode to the buffer $g\leftarrow 0$ while  _$g <G$_ do

if  _$P-B <0$_ then

$P\leftarrow\mathsf{len}(\mathcal{H})$ ;

// Set index to last element of Buffer $\mathcal{H}$

else

$P\leftarrow P-B$

end if

$D\leftarrow\mathcal{H}[P-B,P]$ ;

// Load batch of previous $B$ samples from index $P$ $g\leftarrow g+1$
$\mathbb{A}(D)$;

// Run the learning algorithm with batch data $D$

end while

end while


GPT-4ostory |  76.0(19/25) |  64.3(9/14) |  66.7(8/12)
GPT-4-Turbo |  36.0(9/25) |  35.7(5/14) |  50.0(6/12)
GPT-4-Turbostory |  60.0(15/25) |  42.9(6/14) |  58.3(7/12)
Claude-3-Opus |  38.2(13/34) |  31.6(6/19) |  43.8(7/16)
Claude-3-Opusstory |  73.5(25/34) |  68.4(13/19) |  37.5(6/16)
Claude-3.5-Sonnet |  38.2(13/34) |  31.6(6/19) |  50.0(8/16)
Claude-3.5-Sonnetstory |  61.8(21/34) |  47.4(9/19) |  31.2(5/16)
Gemini Pro 1.5 |  52.2(12/23) |  70.0(7/10) |  40.0(4/10)
Gemini Pro 1.5story |  56.5(13/23) |  70.0(7/10) |  40.0(4/10)
Gemini Flash 1.5 |  39.1(9/23) |  27.3(3/11) |  60.0(6/10)
Gemini Flash 1.5story |  34.8(8/23) |  63.6(7/11) |  30.0(3/10)
Command R |  20.0(5/25) |  21.4(3/14) |  33.3(4/12)
Command Rstory |  36.0(9/25) |  35.7(5/14) |  33.3(4/12)
Command R+ |  20.0(5/25) |  0.0(0/14) |  8.3(1/12)
Command R+story |  48.0(12/25) |  50.0(7/14) |  41.7(5/12)
Phi-3-mini |  6.2(1/16) |  8.3(1/12) |  14.3(1/7)
Phi-3-ministory |  18.8(3/16) |  33.3(4/12) |  0.0(0/7)
Phi-3-minisimple |  12.5(3/24) |  15.4(2/13) |  0.0(0/12)
Phi-3-minisimple-story |  37.5(9/24) |  30.8(4/13) |  25.0(3/12)
Mixtral-8x22Bstory |  61.8(21/34) |  52.6(10/19) |  50.0(8/16)
Qwen-2-72Bstory |  61.8(21/34) |  63.2(12/19) |  56.2(9/16)
Table 30: Models’ performance on stories at different depths of the story
collection. Each collection is divided into three parts based on the number of
tokens: (1) beginning (first third), (2) middle (second third), and (3) end
(last third). The subscript "story" (marked with ) refers to outputs obtained
by prompting with the story (rather than the collection) as the context, for
comparison. We also provide results for Mixtral-8x22B and Qwen-2-72B for
comparison.

####  Claude-3-Opus cites evidence more often than other models:

We employ a simple heuristic by looking for quotation marks in the evidence to
identify how often models cite excerpts from the source. Qualitatively, we
observed that Claude-3-Opus cites the source more often than other closed-
source models, and the citations we verified were always present in the
source. This quantitative analysis confirms our observation, as 49.6% of
Claude-3-Opus’s responses contain quotation marks, followed by 18.6% for both
GPT-4o and GPT-4-Turbo, 11.5% for Gemini Pro 1.5, and 10.8% for Gemini Flash
1.5. We also examined whether responses containing citations are more likely
to be correct but did not find such a relation (see Table 31).

| Quotations (%)
---|---
Model | All responses | Correct responses | Incorrect responses
GPT-4o | 18.4% 227/1234 | 18.8% 178/946 | 17.0% 49/288
GPT-4-Turbo | 18.8% 232/1234 | 19.5% 164/839 | 17.2% 68/395
Claude-3-Opus | 49.8% 934/1874 | 49.8% 685/1376 | 50.0% 249/498
Claude-3.5-Sonnet | 59.6% 1116/1874 | 62.2% 808/1300 | 53.7% 308/574
Gemini Pro 1.5 | 11.6% 119/1029 | 11.4% 84/738 | 12.0% 35/291
Gemini Flash 1.5 | 10.8% 111/1030 | 10.5% 70/668 | 11.3% 41/362

Table 31: Percentage of responses with identified quotations by model.
Separate values are reported for claims labeled correctly and incorrectly.
Note that the percentages are reported by claim, as each explanation is
generated at the claim level. The counts (quotations/total) are provided in
subscript. We do not report these numbers for open-weights models as the
generations often do not follow the requested output format.

#### Effect of genre:

We classify the books into three genres: historical, contemporary, and
speculative. Table 32 presents results by genre for all models.

|  Genre
---|---
Model | Historical | Contemporary | Speculative
GPT-4o | 70.3(26/37) | 59.0(229/388) | 44.2(72/163)
GPT-4-Turbo | 70.3(26/37) | 39.9(155/388) | 34.4(56/163)
Claude-3-Opus | 63.5(33/52) | 51.1(290/567) | 42.9(124/289)
Claude-3.5-Sonnet | 42.3(22/52) | 42.2(239/567) | 38.1(110/289)
Gemini Pro 1.5 | 53.8(28/52) | 49.2(127/258) | 44.4(84/189)
Gemini Flash 1.5 | 46.2(24/52) | 37.2(96/258) | 27.4(52/190)
Command R | 30.3(10/33) | 17.8(46/259) | 20.1(28/139)
Command Rsimple | 21.2(7/33) | 22.8(59/259) | 21.6(30/139)
Command R+ | 18.2(6/33) | 21.6(56/259) | 10.1(14/139)
Command R+simple | 15.2(5/33) | 13.1(34/259) | 15.1(21/139)
Phi-3-mini | 10.0(1/10) | 9.5(14/147) | 9.0(8/89)
Phi-3-minisimple | 20.0(3/15) | 15.8(29/183) | 12.6(15/119)
Gemma-10M | 1.5(1/67) | 3.2(18/567) | 4.4(15/338)
Gemma-10Msimple | 4.5(3/67) | 7.2(41/567) | 7.7(26/338)
LongLLaMAsimple | 0.0(0/52) | 5.1(29/567) | 4.2(12/289)
BM25+GPT-4o (k=5) | 35.8(24/67) | 28.6(162/567) | 26.0(88/338)
BM25+GPT-4o (k=25) | 50.7(34/67) | 42.9(243/567) | 44.4(150/338)
BM25+GPT-4o (k=50) | 56.7(38/67) | 50.8(288/567) | 45.9(155/338)
Table 32: Model performance by genre: historical (pre-WWII), contemporary
(post-WWII), and speculative (fantasy/SF/ghosts).

#### Length of the justifications:

Figure 12 provides lengths of the justification provided by each model for the
main prompt template (Table 3). We report the lengths in words by accuracy for
each model. Note that Gemma-10M never produces real explanation, rather
sometimes repeats part of the original prompt, i.e., <explanation>YOUR
ANSWER</explanation>.

Figure 12: Boxplots of the length of justification (in words) provided for
each model for correct (green) and incorrect (purple) predictions.

#### Length of the retrieved chunks:

Figure 13 shows the average percentage of the book retrieved by BM25 for
varying values of k, i.e., for top 5, 25, and 50 chunks.

Figure 13: Average percentage of the book retrieved by BM25 for varying values
of top k. For example, k = 5 means that the top 5 ranked excerpts according to
BM25 were fed to GPT-4o as context for a claim.

#### Performance by book:

We report the performance of each model by the book title in Figure 14. Empty
cells indicate cases where the model did not process the book Gemini Pro 1.5
and Gemini Flash 1.5) or could not process the book because of its context
window. See Table 12 for the number of claim pairs written for each book.

Figure 14: Heatmap of model performance by book title. Empty cells indicate
cases where the model did not process the book (Gemini Pro 1.5 and Gemini
Flash 1.5) or could not process the book because of its context window.

#### Statistical analysis:

We conducted statistical analysis by fitting generalized linear mixed-effects
models using the glmer() function in R Bates et al. (2015).474747All models
were fitted using the closed-source models data only, as all open-weights
models performed below random. The response variable was pairwise accuracy, a
binary categorical variable ("correct" or "incorrect"). Pair IDs and
annotators were modeled as random effects, with various predictors (fixed
effects) included in different models:

  1. 1.

Model \- a 6-level categorical variable (GPT-4o, GPT-4-Turbo, Claude-3-Opus,
Claude-3.5-Sonnet, Gemini Pro 1.5, Gemini Flash 1.5), with analysis restricted
to pairs processed by all models. GPT-4o was set as the reference level
(intercept). See Table 33 and Table 34 for the results.;

  2. 2.

Length group \- a 4-level categorical variable ("below 75k", "75k-127k",
"127k-180k", "above 180k"). "Below 75k" category was set as the reference
level (intercept). See Table 35 and Table 36 for the results;

  3. 3.

Year \- a 3-level categorical variable ("classics", "2023", "2024") with
"classics" set as the reference level. See Table 37 for the results;

  4. 4.

Genre \- a 3-level categorical variable ("historical", "contemporary",
"speculative") with "speculative" set as the reference level. See Table 38 and
Table 39 for the results;

  5. 5.

Scope \- a 3-level categorical variable ("sentence", "passage", "global") with
"sentence" set as the reference level. See Table 40 and Table 41 for the
results.

All models were fitted using the bobyqa optimizer with a binomial link
function. We chose mixed-effects models for two main reasons: (1) to account
for repeated measures, as each model validates multiple pairs and each pair is
validated by multiple models, and (2) to flexibly model pair IDs and
annotators as random variables, partially controlling for the inherent
difficulty of the pairs unrelated to the predictors.

For these mixed-effects models, we report two types of R2 values:

  * •

Marginal R2, which indicates the proportion of variance explained by the fixed
effects (predictors) alone.

  * •

Conditional R2, which represents the proportion of variance explained by both
the fixed and random effects Nakagawa et al. (2017).

We further conducted a post-hoc analysis using the emmeans package Lenth
(2023) in R with Tukey adjustments for multiple comparisons. To obtain
probabilities, we first converted log-odds to odds ratios by exponentiating
the estimates, and then converted the odds ratios to probabilities which are
reported in the post-hoc tables.

model <\- glmer(accuracy $\sim$ model + (1|id) + (1|annotator),
common_set_data)
---
Predictors | Odds Ratios | CI (95%) | p-value |
(Intercept) | 1.73 | 1.18 – 2.55 | 0.005 | **
GPT-4-Turbo | 0.33 | 0.23 – 0.48 | <0.001 | ***
Claude-3-Opus | 0.64 | 0.45 – 0.91 | 0.014 | *
Claude-3.5-Sonnet | 0.34 | 0.24 – 0.50 | <0.001 | ***
Gemini Pro 1.5 | 0.55 | 0.38 – 0.78 | 0.001 | ***
Gemini Flash 1.5 | 0.24 | 0.17 – 0.35 | <0.001 | ***
Random Effects |  |  |  |
$\sigma^{2}$ (residual variance) | 3.29 |  |  |
$\tau_{00}$ (id) (variance of random intercepts) | 2.54 |  |  |
$\tau_{00}$ (annotator) (variance of random intercepts) | 0.19 |  |  |
ICC | 0.45 |  |  |
N (id) | 354 |  |  |
N (annotator) | 15 |  |  |
Observations | 2124 |  |  |
R2 (marginal) | 0.036 |  |  |
R2 (conditional) | 0.473 |  |  |
Table 33: Summary of generalized linear mixed model with model as the
predictor of accuracy: model <\- glmer(accuracy $\sim$ model + (1|id) +
(1|annotator), data). GPT-4o was set as the reference level (intercept). Note
that while we observe significant differences between the models’ performance,
the marginal R2 is low, suggesting that model type alone does not explain the
majority of the variance in the data. See Table 34 for post-hoc analysis.

Contrast | Estimate | SE | Odds Ratio | Probability | p-value |
---|---|---|---|---|---|---
GPT-4o \- Claude-3-Opus | 0.4480 | 0.1820 | 1.5652 | 0.6102 | 0.1355 |
GPT-4o \- Claude-3.5-Sonnet | 1.0654 | 0.1849 | 2.9021 | 0.7437 | <0.001 | ***
GPT-4o \- Gemini Flash 1.5 | 1.4229 | 0.1888 | 4.1492 | 0.8058 | <0.001 | ***
GPT-4o \- Gemini Pro 1.5 | 0.6013 | 0.1822 | 1.8244 | 0.6459 | 0.0124 | *
GPT-4o \- GPT-4-Turbo | 1.1005 | 0.1852 | 3.0056 | 0.7503 | <0.001 | ***
Claude-3-Opus \- Claude-3-Opus | 0.6174 | 0.1820 | 1.8541 | 0.6496 | 0.0090 | **
Claude-3-Opus \- Gemini Flash 1.5 | 0.9749 | 0.1855 | 2.6509 | 0.7261 | <0.001 | ***
Claude-3-Opus \- Gemini Pro 1.5 | 0.1533 | 0.1801 | 1.1656 | 0.5382 | 0.9578 |
Claude-3-Opus \- GPT-4-Turbo | 0.6524 | 0.1822 | 1.9202 | 0.6576 | 0.0046 | **
Claude-3.5-Sonnet \- Gemini Flash 1.5 | 0.3575 | 0.1852 | 1.4297 | 0.5884 | 0.3833 |
Claude-3.5-Sonnet \- Gemini Pro 1.5 | -0.4642 | 0.1816 | 0.6287 | 0.3860 | 0.1084 |
Claude-3.5-Sonnet \- GPT-4-Turbo | 0.0350 | 0.1826 | 1.0356 | 0.5088 | 1.0000 |
Gemini Flash 1.5 \- Gemini Pro 1.5 | -0.8216 | 0.1849 | 0.4397 | 0.3054 | <0.001 | ***
Gemini Flash 1.5 \- GPT-4-Turbo | -0.3224 | 0.1853 | 0.7244 | 0.4201 | 0.5050 |
Gemini Pro 1.5 \- GPT-4-Turbo | 0.4992 | 0.1819 | 1.6474 | 0.6223 | 0.0667 | .

Table 34: Post-hoc comparisons of models for accuracy (Table 33) using Tukey
adjustments for multiple comparisons. The probability values refer to the
likelihood that the first model in each contrast is more accurate than the
second model (i.e., a value of 0.5 suggests that both models are comparable in
terms of accuracy). model <\- glmer(accuracy $\sim$ length_group + (1|pairID)
+ (1|annotator), data)
---
Predictors | Odds Ratios | CI (95%) | p-value |
(Intercept) | 0.56 | 0.34 – 0.94 | 0.028 | *
75k–127k | 1.45 | 0.87 – 2.43 | 0.154 |
127k–180k | 1.26 | 0.74 – 2.14 | 0.391 |
above 180k | 0.75 | 0.28 – 1.96 | 0.551 |
Random Effects |  |  |  |
$\sigma^{2}$ (residual variance) | 3.29 |  |  |
$\tau_{00}$ (id) (variance of random intercepts) | 2.15 |  |  |
$\tau_{00}$ (annotator) (variance of random intercepts) | 0.24 |  |  |
ICC | 0.42 |  |  |
N (id) | 967 |  |  |
N (annotator) | 22 |  |  |
Observations | 4137 |  |  |
R2 (marginal) | 0.004 |  |  |
R2 (conditional) | 0.423 |  |  |
Table 35: Summary of generalized linear mixed model with length group as the
predictor of accuracy: model <\- glmer(accuracy $\sim$ length_group +
(1|pairID) + (1|annotator), data). The "below 75k" group was set as the
reference level (intercept). Note that the marginal R2 is very low, indicating
that length group alone does not explain the majority of the variance in the
data. See Table 36 for post-hoc analysis.

Contrast | Estimate | SE | Odds Ratio | Probability | p-value |
---|---|---|---|---|---|---
Below 75k - 127k-180k | -0.2312 | 0.2698 | 0.7936 | 0.4424 | 0.8267 |
Below 75k - 75k-127k | -0.3739 | 0.2620 | 0.6880 | 0.4076 | 0.4823 |
Below 75k - Above 180k | 0.2938 | 0.4925 | 1.3416 | 0.5729 | 0.9331 |
127k-180k - 75k-127k | -0.1427 | 0.1652 | 0.8670 | 0.4644 | 0.8235 |
127k-180k - Above 180k | 0.5251 | 0.4461 | 1.6906 | 0.6283 | 0.6415 |
75k-127k - Above 180k | 0.6677 | 0.4464 | 1.9498 | 0.6610 | 0.4399 |

Table 36: Post-hoc comparisons of length groups for accuracy (Table 35) using
Tukey adjustments for multiple comparisons. The probability values refer to
the likelihood that the verification for the first group in each contrast is
more accurate than the verification for the second group. model <\-
glmer(accuracy $\sim$ year + (1|pairID) + (1|annotator), data)
---
Predictors | Odds Ratios | CI (95%) | p-value |
(Intercept) | 0.65 | 0.35 – 1.20 | 0.170 |
2023 | 1.17 | 0.64 – 2.15 | 0.321 |
2024 | 1.17 | 0.63 – 2.16 | 0.344 |
Random Effects |  |  |  |
$\sigma^{2}$ (residual variance) | 3.29 |  |  |
$\tau_{00}$ (id) (variance of random intercepts) | 2.16 |  |  |
$\tau_{00}$ (annotator) (variance of random intercepts) | 0.22 |  |  |
ICC | 0.42 |  |  |
N (id) | 967 |  |  |
N (annotator) | 22 |  |  |
Observations | 4137 |  |  |
R2 (marginal) | 0.000 |  |  |
R2 (conditional) | 0.420 |  |  |
Table 37: Summary of generalized linear mixed model with year as the predictor
of accuracy: model <\- glmer(accuracy $\sim$ year + (1|pairID) +
(1|annotator), data). The "classics" year group was set as the reference level
(intercept). Note that the marginal R2 is 0.000, indicating that year alone
does not explain the majority of the variance in the data. model <\-
glmer(accuracy $\sim$ genre + (1|id) + (1|annotator), data = filtered_data)
---
Predictors | Odds Ratios | CI (95%) | p-value |
(Intercept) | 0.55 | 0.40 – 0.74 | <0.001 | ***
Contemporary | 1.53 | 1.11 – 2.11 | 0.010 | *
Historical | 2.18 | 1.20 – 3.96 | 0.010 | *
Random Effects |  |  |  |
$\sigma^{2}$ (residual variance) | 3.29 |  |  |
$\tau_{00}$ (id) (variance of random intercepts) | 2.10 |  |  |
$\tau_{00}$ (annotator) (variance of random intercepts) | 0.16 |  |  |
ICC | 0.41 |  |  |
N (id) | 938 |  |  |
N (annotator) | 22 |  |  |
Observations | 3991 |  |  |
R2 (marginal) | 0.010 |  |  |
R2 (conditional) | 0.412 |  |  |
Table 38: Summary of generalized linear mixed model with genre as the
predictor of accuracy: model <\- glmer(accuracy $\sim$ genre + (1|id) +
(1|annotator), data = filtered_data). The "speculative" genre was set as the
reference level (intercept). We also excluded two books which felt into
"historical and contemporary" and "essays" categories. Note that the marginal
R2 is low, indicating that genre alone does not explain the majority of the
variance in the data. The post-hoc analysis for this model is presented in
Table 39.

Contrast | Estimate | SE | Odds Ratio | Probability | p-value |
---|---|---|---|---|---|---
Speculative \- Contemporary | -0.4234 | 0.1648 | 0.6548 | 0.3957 | 0.0275 | *
Speculative \- Historical | -0.7802 | 0.3042 | 0.4583 | 0.3143 | 0.0278 | *
Contemporary \- Historical | -0.3568 | 0.2874 | 0.6999 | 0.4117 | 0.4287 |

Table 39: Post-hoc comparisons of genres for accuracy (Table 38) using Tukey
adjustments for multiple comparisons. The probability values refer to the
likelihood that the validation of books of the first genre in each contrast is
more accurate than the validation of books of the second genre (i.e., a value
of 0.5 suggests that both genres are comparable in terms of accuracy). model
<\- glmer(accuracy $\sim$ scope + (1|id) + (1|annotator), data =
filtered_data)
---
Predictors | Odds Ratios | CI (95%) | p-value |
(Intercept) | 2.12 | 0.74 – 6.11 | 0.163 |
Passage | 0.35 | 0.14 – 0.87 | 0.024 | *
Global | 0.32 | 0.13 – 0.78 | 0.012 | *
Random Effects |  |  |  |
$\sigma^{2}$ (residual variance) | 3.29 |  |  |
$\tau_{00}$ (id) (variance of random intercepts) | 1.19 |  |  |
$\tau_{00}$ (annotator) (variance of random intercepts) | 0.53 |  |  |
ICC | 0.34 |  |  |
N (id) | 121 |  |  |
N (annotator) | 4 |  |  |
Observations | 544 |  |  |
R2 (marginal) | 0.030 |  |  |
R2 (conditional) | 0.363 |  |  |
Table 40: Summary of generalized linear mixed model with scope as the
predictor of accuracy: model <\- glmer(accuracy $\sim$ scope + (1|id) +
(1|annotator), data = filtered_data). The "sentence" scope was set as the
reference level (intercept). Note that the marginal R2 is low, indicating that
scope alone does not explain the majority of the variance in the data. See
Table 41 for the post-hoc analysis.

Contrast | Estimate | SE | Odds Ratio | Probability | p-value |
---|---|---|---|---|---|---
Sentence \- Global | 1.1322 | 0.4506 | 3.1026 | 0.7563 | 0.0321 | *
Sentence \- Passage | 1.0488 | 0.4642 | 2.8542 | 0.7405 | 0.0617 | .
Global \- Passage | -0.0834 | 0.3139 | 0.9199 | 0.4792 | 0.9618 |

Table 41: Post-hoc comparisons of scopes for accuracy (Table 40) using Tukey
adjustments for multiple comparisons. The probability values refer to the
likelihood that the validation of claim pairs in the first scope in each
contrast is more accurate than the validation of claim pairs in the second
scope (i.e., a value of 0.5 suggests that both scopes are comparable in terms
of accuracy).