# Efficient Scheduling of Data Augmentation
for Deep Reinforcement Learning

Byungchan Ko
NALBI
<EMAIL_ADDRESS>&Jungseul Ok
GSAI, POSTECH
<EMAIL_ADDRESS>This work was done while Byungchan Ko studied in GSAI,
POSTECH.

###### Abstract

In deep reinforcement learning (RL), data augmentation is widely considered as
a tool to induce a set of useful priors about semantic consistency and to
improve sample efficiency and generalization performance. However, even when
the prior is useful for generalization, distilling it to RL agent often
interferes with RL training and degenerates sample efficiency. Meanwhile, the
agent is forgetful of the prior due to the non-stationary nature of RL. These
observations suggest two extreme schedules of distillation: (i) over the
entire training; or (ii) only at the end. Hence, we devise a stand-alone
network distillation method to inject the consistency prior at any time (even
after RL), and a simple yet efficient framework to automatically schedule the
distillation. Specifically, the proposed framework first focuses on mastering
train environments regardless of generalization by adaptively deciding which
or no augmentation to be used for the training. After this, we add the
distillation to extract the remaining benefits for generalization from all the
augmentations, which requires no additional new samples. In our experiments,
we demonstrate the utility of the proposed framework, in particular, that
considers postponing the augmentation to the end of RL training.
https://github.com/kbc-6723/es-da

##  1 Introduction

Deep reinforcement learning (RL) aims at finding a neural network to represent
policy or value functions taking raw observation as input, of which the most
common form in practice is visual data or images of high-dimensionality, e.g.,
video games [24], board games [31, 30], and robot controls [34, 19]. RL
handling high-dimensional input often suffers from poor sample efficiency and
generalization capability, mainly due to the curse of dimensionality [4, 16].
To overcome these issues, it has been widely considered to augment data based
on prior knowledge that a set of transformations preserve the meaning or
context of input observations, e.g., cropping out unimportant parts of images,
and changing colors [21, 20, 22, 15]. On one hand, RL agent can be directly
fed with the original and augmented data so that it implicitly learns a
representation with the prior and improves the sample efficiency and
generalization [21]. On the other hand, the prior knowledge in data
augmentation can be explicitly distilled via a self-supervised learning, which
introduces additional regularization to ensure consistency between responses
to original and augmented inputs [26, 15].

However, data augmentation shows highly task-dependent effect in RL, and it is
prone to generate severe interference with the training even when it truly
conveys a useful prior to train and test environments [21, 26, 15]. We address
the problem of alleviating the interference between data augmentation and RL
training to improve sample efficiency for acquainting train tasks, and
generalization capability for unseen test environments. This problem in
(online) RL is more critical and challenging than that in supervised learning
(SL) since the objective function and data distribution are time-varying in
RL, while they are not in SL. Indeed, according to [1, 11], it is sufficient
to partly apply data augmentation just for a short period of SL at any time.
Meanwhile, we empirically observe that the prior from data augmentation can be
easily forgotten in RL of the non-stationary nature (see Section 5.3), i.e.,
the effect of augmentation is time-sensitive.

Based on our observation about the interference and the time-sensitivity, we
propose two simple yet effective methods according to timings of data
augmentation : Intra Distillation with Augmented observations (InDA) and Extra
Distillation with Augmented observations (ExDA). Data augmentation beneficial
for the sample efficiency needs to be applied over the entire RL training,
i.e., InDA. Conversely, data augmentation useful only for the generalization
should be postponed to the end of RL training, i.e., ExDA, so that we can
minimize the interference in the training, while enjoying the benefit in the
testing. InDA and ExDA are equipped with Distillation with Augmented
observation (DA). DA is a stand-alone self-supervised learning which enables
us to induce the prior after RL training, and shows a relatively small
interference with RL training by explicitly preserving the response of RL
agent to the original input.

The best timing (InDA or ExDA) depends on traits of train task and
augmentation. We hence suggest a framework of adaptive scheduling, named UCB-
ExDA, that (i) first aims at maximizing the sample efficiency by adaptively
selecting which or no augmentation to be used during RL training; and then,
(ii) distill the priors from all the augmentations after RL training.
Specifically, inspired by [26], we devise UCB-InDA for the adaptive selection
in the first part by modifying the upper confidence bound (UCB) algorithm [3]
for multi-armed bandit, where differently from [26], the set of arms includes
all the augmentations and the option to not augment. In summary, UCB-ExDA is
nothing but executing UCB-InDA followed by ExDA. Our experiment demonstrates
the utility of the proposed framework.

Our contributions are summarized as follows:

  1. (i)

We devise DA (Section 4.1) which explicitly preserves the knowledge of RL
agent so that enables distilling the consistency prior of augmentation into RL
agent not only during but also after RL training, while other methods [21, 26,
15] need to be applied concurrently with RL training and thus show relatively
strong interference in our experiments (Section 5.2).

  2. (ii)

We identify the simple yet effective timings of data augmentation: either InDA
or ExDA (Section 4.2, Section 4.3), based upon the discovery of the time-
sensitivity (Section 5.3) that has not been observed in SL [11], i.e., the
proposed timings are effective particularly for RL.

  3. (iii)

We finally establish UCB-ExDA which automatically decides the best timing of
augmentation, and demonstrate its superiority compared to others (Section
5.4). The advantage of UCB-ExDA is particularly substantial when the best
strategy is ExDA postponing augmentation to the end of RL training.

##  2 Related Works

Augmented experience in RL. To solve the problem of poor generalization and
sparse data, a popular approach is to fabricate virtual experiences and train
the RL agent to learn with them. Domain randomization is a technique to
produce such experiences from a simulator of a targeted system, [34, 25, 27].
Accurate simulators of practical systems are difficult to obtain, and it has
limited applicability. However, visual augmentation has no such limit because
the method uses simple image transformations such as cropping, tilting, and
color jitter, although applications require a careful understanding of the
targeted system to guide the design of an appropriate image transformer. A
method of curriculum learning for domain randomization, in which the
difficulty is gradually increasing [27] provided insights that coincide with
some of our findings. However, we provide a further understanding of the types
of visual augmentation that should be early or late during training.

Regularization from augmented data in vision-based RL has been implemented in
various learning frameworks, including but not limited to representation [15,
33], self-supervised [26], and contrast [32]. One proposed algorithm [26]
applies the UCB algorithm [3] to automatically select the most effective
augmentations over RL training, where each augmentation is considered as an
arm and then evaluate the effectiveness of augmentation by using a sliding
window average. The idea of adapting augmentation concurs with our main
message regarding the timing of augmentation. In [26], ’not augmenting’ is not
an option, whereas our findings indicate that it should be. In addition, [26]
does not consider post augmentation followed by RL training, as in ExDA.

Other time-sensitivity in deep learning. During deep learning, the early stage
of training often has a significant effect [7, 1]. Therefore, we devised time-
sensitive methods that adapt to the progress of training, such as learning
rate decay [37] and curriculum learning [35]. Golatkar et al. [11] studied
such a time-sensitivity of regularization techniques for SL, where the effect
of data augmentation in different time does not change much. We find that the
time-sensitivity of augmentation can be significant in RL. This contrast may
occur because of the non-stationary nature of RL, which SL does not have.
Although a set of techniques originally developed for SL such as convolutional
neural network, weight decay, batch normalization, dropout and self-supervised
learning improve deep RL [17, 6, 23, 9, 32, 36, 14], a thorough study should
be conducted before introducing a method from different learning frameworks,
because we find the contrasting time-sensitivities of data augmentation. This
spirit is also shared with an application [18] of implicit bias in SL [13, 2,
10] to RL.

##  3 Background

#### Notation.

We consider a standard agent-environment interface of vision-based
reinforcement learning in a discrete Markov decision process of state space
$\mathcal{S}$, action space $\mathcal{A}$, and kernel
$P=P(s_{t+1},r_{t}|s_{t},a_{t})$ which determines the state transition and
reward distribution. The goal of the RL agent is to find a policy that
maximizes the expectation of cumulative reward
$\sum_{t=0}^{t^{\prime}-1}\gamma^{t}r_{t}$, where $t^{\prime}$ is terminating
time and $\gamma\in[0,1]$ is discount factor. At each timestep $t$, the agent
selects an action $a_{t}\in\mathcal{A}$ and receives a reward $r_{t}$ and an
image $o_{t+1}=O(s_{t+1})\in\mathbb{R}^{k\times k^{\prime}}$ as an observation
of the next state $s_{t+1}$. Data augmentation can be described by an image
transformation function $\phi:\mathbb{R}^{k\times
k^{\prime}}\mapsto\mathbb{R}^{k\times k^{\prime}}$ of which output is presumed
to have the same or similar semantics of the input.

#### Baseline RL algorithm.

Throughout this paper, we use Proximal Policy Optimization (PPO) [28] as a
baseline, although we believe our findings and methods can be easily adjusted
to others. PPO is a representative on-policy RL algorithm to learn policy
$\pi_{\theta}(a\mid o)$ and value function $V_{\theta}$ of neural agent
parameterized by $\theta$. Storing a set of recent transitions
$\tau_{t}:=(o_{t},a_{t},r_{t},o_{t+1})$ in experience buffer $\mathcal{D}$,
the agent is updated to minimize the following loss function:

$\displaystyle L_{\text{PPO}}(\theta)=-L_{\pi}(\theta)+\alpha
L_{V}(\theta)\;,$ (1)

where $\alpha$ is a hyperparameter and canonical regularization terms are
omitted. The clipped policy objective function $L_{\pi}$ and value loss
function $L_{V}$ are defined as:

$\displaystyle\\!\\!\\!\\!\\!L_{\pi}(\theta)$
$\displaystyle=\hat{\mathbb{E}}\\!\left[\text{min}(\rho_{t}(\theta)\hat{A}_{t},\text{clip}(\rho_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\right]\\!\\!\\!$
(2) $\displaystyle\\!\\!\\!\\!\\!L_{V}(\theta)$
$\displaystyle=\hat{\mathbb{E}}\\!\left[\left(V_{\theta}(o_{t})-V_{t}^{\text{targ}}\right)^{2}\right]\;,$
(3)

where the expectation $\hat{\mathbb{E}}$ is taken with respect to
$\tau_{t}\sim\mathcal{D}$, we denote by $\theta_{\text{old}}$ the parameter
before the update, $\rho_{t}(\theta)$ is the importance ratio
$\frac{\pi_{\theta}(a_{t}|o_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|o_{t})}$,
and $\hat{A}_{t}$ is the generalized advantage estimation [28].

##  4 Method

In what follows, we present our methods: Distillation with Augmented
observations (DA; Section 4.1), Intra DA (InDA; Section 4.2), Extra DA (ExDA;
Section 4.3), and then the adaptive scheduling frameworks based on UCB (UCB-
InDA and UCB-ExDA; Section 4.4). DA is a stand-alone knowledge distillation
method, which can be used at any time to instill the underlying prior of
augmentation into a given RL agent. InDA and ExDA conduct either DA or PPO in
each epoch but have different schedules (Figure 2), where InDA interleaves PPO
and DA, whereas ExDA performs PPO first then DA. UCB-InDA adaptively decides
which or no augmentation to be used in each DA epoch of InDA based on the UCB
of estimated gain from each option. UCB-ExDA performs ExDA preceded by UCB-
InDA.

Figure 1: An illustration of InDA and ExDA

Figure 2: An illustration of distillation losses.

###  4.1 Distillation with augmented observations (DA)

The underlying prior of augmentation can be infused by minimizing a measure of
inconsistency between the agent’s responses to original and augmented inputs
(resp. $o_{t}$ and $\phi(o_{t})$). For instance, with PPO agent learning
policy $\pi_{\theta}$ and value $V_{\theta}$, Raileanu et al. [26] proposes
the following measure:

$\displaystyle
L_{\text{dis}}(\theta,\phi):=\hat{\mathbb{E}}_{o_{t}\sim\mathcal{D}}\left[\text{KL}[\pi_{\theta}(\cdot|o_{t}),\pi_{\theta}(\cdot|\phi(o_{t}))]\right]+\hat{\mathbb{E}}_{o_{t}\sim\mathcal{D}}\left[(V_{\theta}(o_{t})-V_{\theta}(\phi(o_{t})))^{2}\right]\;,$
(4)

which uses Kullback-Leibler divergence (first term) and mean squared deviation
(second term) for policy and value inconsistencies, respectively. Noting that
$L_{\text{dis}}(\theta;\phi)$ can be minimized to be zero by a constant
response to all inputs, the distillation with Eq (4) can distort the RL agent,
in particular, when applying it outside of RL training.

We hence devise a network distillation technique (DA) which explicitly
preserves the RL agent’s response to original input and thus is applicable
even after RL training. DA distills the knowledge of RL agent
$\theta_{\text{old}}$ into $\theta$ using not only original but also augmented
observations. More formally, the loss of DA is written as:

$\displaystyle
L_{\text{DA}{}}(\theta,\phi;\theta_{\text{old}}):=L_{\text{dis}}(\theta,\mathbb{I};\theta_{\text{old}})+L_{\text{dis}}(\theta,\phi;\theta_{\text{old}})\;.$
(5)

We here denote the identity transformation by $\mathbb{I}$ such that
$\mathbb{I}(o)=o$ for all $o$, and extend the definition of $L_{\text{dis}}$
in Eq (4) as follows:

$\displaystyle L_{\text{dis}}(\theta,\phi;\theta_{\text{old}})$
$\displaystyle=\hat{\mathbb{E}}_{o_{t}\sim\mathcal{D}}\left[\text{KL}[\pi_{\theta_{\text{old}}}(\cdot|o_{t}),\pi_{\theta}(\cdot|\phi(o_{t}))]\right]+\hat{\mathbb{E}}_{o_{t}\sim\mathcal{D}}\left[(V_{\theta_{\text{old}}}(o_{t})-V_{\theta}(\phi(o_{t})))^{2}\right]\;.$
(6)

In Eq (5), the first term ensures that $\theta$ and $\theta_{\text{old}}$
behave identically for the original inputs, and the second one infuses the
consistency prior. In Figure 2,
$L_{\text{dis}}(\theta,\mathbb{I};\theta_{\text{old}})$,
$L_{\text{dis}}(\theta,\phi;\theta_{\text{old}})$, and
$L_{\text{dis}}(\theta,\phi;\theta)$ graphically correspond to (A), (B), and
(C), respectively. From this, it follows that minimizing
$L_{\text{\text{DA}{}}}$ in Eq (5) eventually reduces
$L_{\text{dis}}(\theta,\phi;\theta)$ in Eq (4). In addition, the minimization
of $L_{\text{DA}{}}$ secures the responses of $\theta$ to the original inputs
(which can be pre-computed to reduce computation cost) to the those of
$\theta_{\text{old}}$, while the alternatives (e.g., (A)+(C):
$L_{\text{dis}}(\theta,\mathbb{I};\theta_{\text{old}})+L_{\text{dis}}(\theta,\phi;\theta)$)
does not and thus may generate interference with RL training. Our experiments
(Section 5.2; Figure 4) justifies the design of DA by showing substantial
advantage compared to the other alternatives such as DrAC [26] using
$L_{\text{dis}}(\theta,\phi;\theta)$ in Eq (4). We note that this advantage
becomes much clearer when a wrong augmentation is given, c.f., the
supplementary material.

###  4.2 Intra distillation with augmented observations

InDA (Algorithm 1) alternates between minimizing $L_{\text{PPO}}$ and
$L_{\text{DA}{}}$, i.e., PPO and DA are explicitly separated, whereas they are
often executed simultaneously in other methods [26]. Such a clear separation
reduces the interference [15]. We can control the frequency and timing of
applying distillation with hyperparameters $I$, $S$ and $T$, where we perform
DA after each $I$ rounds of RL training only if the current epoch $n$ is in
the interval of $[S,T]$, i.e., $S$ and $T$ are the epochs to begin and
terminate DA, respectively. We vary $S$ and $T$ to study the time-dependency
of data augmentation. We denote InDA$[S,T]$ to indicate the period to apply
DA, while we omit the indication when DA is applied over the entire period. We
provide further details on InDA in the supplementary material.

Algorithm 1 InDA

0: $N,I,\phi$, $S$, $T$

1: Initialize $\theta$ close to origin.

2: for $n=1,2,\ldots,N$ do

3: // RL training

4: Store sampled transitions to $\mathcal{D}$;

5: Optimize RL objective ${L}_{\text{PPO}}(\theta)$ with $\mathcal{D}$;

6: // Distillation

7: if $n\in[S,T]$ and $\text{mod}(n-1,I)=0$ then

8: Store $\theta_{\text{old}}\leftarrow\theta$;

9: Minimize $L_{\text{DA}{}}(\theta)$ for $\mathcal{D}$, $\theta_{\text{old}}$
and $\phi$;

10: end if

11: end for

Algorithm 2 ExDA

0: $N$, $M$, $\phi$

1: Initialize ${\theta}$ close to origin.

2: //Pre-training phase with RL algorithm

3: for $n=1,2,\ldots,N$ do

4: Store sampled transitions to $\mathcal{D}$;

5: Optimize RL objective ${L}_{\text{PPO}}(\theta)$ with $\mathcal{D}$;

6: end for

7:

8: Store $\theta_{\text{old}}\leftarrow\theta$;

9: // Distillation at the end of RL training

10: for $m=1,2,\ldots,M$ do

11: Minimize $L_{\text{DA}{}}(\theta)$ for $\mathcal{D}$,
$\theta_{\text{old}{}}$ and $\phi$;

12: end for

###  4.3 Extra distillation with augmented observations

ExDA (Algorithm 2) performs the distillation after the end of RL training,
where the lengths of DA and RL training are parameterized by $M$ and $N$,
respectively. We note that computational cost can be reduced by removing the
value inconsistency measure
$\hat{\mathbb{E}}_{o_{t}\sim\mathcal{D}}\left[(V_{\theta_{\text{old}}}(o_{t})-V_{\theta}(\phi(o_{t})))^{2}\right]$
from $L_{\text{dis}}$ in Eq (6) because the value consistency is not necessary
for RL training nor distillation in the actor-critic framework and including
it has a potential risk of generating additional interference. In the
supplementary material, it is empirically verified that this reduction does
not degrade RL performance. We leave more interesting details in the
supplementary. For instance, one can consider re-initializing $\theta$ before
starting DA as a part of exploiting the implicit bias [13, 18] to improve
generalization. However, test performance is often dropped. This is mainly
because the dataset to distill $\pi_{\theta_{old}}$ has much less diversity
than that observed along the trajectory. Thus, we use no re-initialization for
the experiments in the main paper.

###  4.4  UCB-based adaptive scheduling frameworks

The training benefit by augmentation differs depending on the task. This
dependency complicates the choice of whether to use InDA or ExDA for
augmentation. Hence, we devise an auto-augmentation method, called UCB-InDA,
inspired by UCB-DrAC [26], where each augmentation is corresponded to an arm
in multi-armed bandit (MAB) problem and assessed its potential gain in
training with upper confidence bound (UCB) [3]. More formally, in UCB-InDA,
the set of arms is the set of augmentations,
$\Phi=\\{\phi_{1},\dots,\phi_{K}\\}$, which must include the identity function
$\mathbb{I}$, i.e., the option not to augment. The inclusion of identity
function is small but makes crucial difference than UCB-DrAC [26] since we
observe that using augmentation sometimes needs to be postponed after RL
training for the sake of better sampling complexity and test performance.

A round of MAB corresponds to every $I$ epoch of InDA, where we let
$\phi_{k(s)}\in\Phi$ be the augmentation selected at the $s$-th round of DA.
Let $G(s)$ be the average return, the sum of estimated advantage $\hat{A}$ and
predicted value $V_{\theta}$, over $(I-1)$ epochs of PPO followed by the
$s$-th DA. UCB algorithm assumes that each arm generates reward independently
drawn from a fixed distribution, and estimates the empirical mean over the
entire sampling process. However, in RL, the return $G(s)$ is non-stationary,
so UCB-InDA computes moving average gain $\bar{G}_{k}(s)$, instead, taken over
a certain number (chosen to be 3 in our experiment) of most recent rounds
selecting $\phi_{k}$ as Raileanu et al. [26] proposed. Then, inspired by UCB1
algorithm [3], UCB-InDA selects action $k(s)$ at round $s$ as follows:

$k(s)=\operatorname*{arg\,max}_{k\in\\{1,...,K\\}}\bar{G}_{k}(s)+c\sqrt{\frac{\log(s)}{N_{k}(s)}}$
(7)

where $c$ is the UCB exploration coefficient, and $N_{k}(s)$ is the number of
times selecting $\phi_{k}$ up to round $s$. We refer to the supplementary
material for the hyperparameter choice. We remark that compared to UCB-DrAC
[26], the proposed UCB-InDA has subtle but important differences, summarized
in two folds: (i) the inclusion of identity transformation (i.e., no
augmentation) and (ii) the distillation with augmentation via InDA. The gain
of each component is numerically studied in Section 5. Finally, we note that
UCB-ExDA is nothing but UCB-InDA followed by ExDA.

###  4.5 PAGrad

Inspired by [38], we devise an alternative approach to reduce the interference
by interpreting RL training with data augmentation as a multi-task learning,
where $L_{\text{PPO}}$ and $L_{\text{dis}}$ correspond to the main and
auxiliary task losses, respectively. In [38], the degree of conflict from task
A to task B is estimated by the inner product of the gradient of task A and
the negative gradient of task B, c.f., Figure 3. From this, we propose PAGrad
(Projecting Auxiliary Gradient) to compute a modified gradient of
$L_{\text{PPO}}$ excluding the conflict from the auxiliary gradient $\nabla
L_{\text{dis}}$ to the main one $\nabla L_{\text{PPO}}$. Formally, PAGrad
computes the gradient given as follows:

$\displaystyle\nabla L_{\text{PPO}}+\left(\nabla
L_{\text{dis}}-\tfrac{\min\\{0,\langle\nabla L_{\text{dis}},\nabla
L_{\text{PPO}}\rangle\\}}{\|\nabla L_{\text{PPO}}\|^{2}}\nabla
L_{\text{PPO}}\right)\;,$ (8)

where $\tfrac{\min\\{0,\langle\nabla L_{\text{dis}},\nabla
L_{\text{PPO}}\rangle\\}}{\|\nabla L_{\text{PPO}}\|^{2}}\nabla L_{\text{PPO}}$
is the components of $\nabla L_{\text{dis}}$ opposite to $\nabla
L_{\text{PPO}}$ which may disturb optimizing the main objective
$L_{\text{PPO}}$. Based on this, we devise DrAC+PAGrad that updates the model
parameter toward the negative direction of (8). This is an alternative of
InDA, while it concurrently optimizes $L_{\text{PPO}}$ and $L_{\text{dis}}$
differently from InDA adopting the time-separation of optimizing
$L_{\text{PPO}}$ and $L_{\text{dis}}$. We also note that it differs from the
original method proposed in [38] alternating the main and auxiliary tasks to
accomplish every task at equal priority, while we have a clear priority on RL
task.

(a) non-conflicting: $\langle g_{\text{aux}},g_{\text{main}}\rangle\geq 0$

(b) conflicting: $\langle g_{\text{aux}},g_{\text{main}}\rangle<0$

Figure 3:  An illustration of gradient conflicting and PAGrad. We here let
$g_{\text{main}}$ and $g_{\text{aux}}$ denote the gradients of main (shown in
red; e.g., $\nabla L_{\text{PPO}}$) and auxiliary (shown in blue; e.g.,
$\nabla L_{\text{dis}}$) losses, respectively. In the computation of PAGrad
(shown in black), the component of $g_{\text{aux}}$ conflicting to
$g_{\text{main}}$ only when it exists (i.e., $\langle
g_{\text{aux}},g_{\text{main}}\rangle<0$).

##  5 Experiment

###  5.1 Setups

#### Train and test tasks.

We use the OpenAI Procgen benchmark of 16 video games [5], where a main
character tries to achieve a specific goal, e.g., finding exit (Maze) or
collecting coins (Coinrun), while avoiding enemies given a 2D map. At each
time $t$, visual observation $o_{t}$ is given as an image of size $64\times
64$. A train or test task is to achieve a high score on a set of environments
configured by game and mode, where a mode describes predefined sets of levels
(e.g., complexity of map) and backgrounds. Cobbe et al. [5] provide easy mode
for each game, consisting of $200$ levels and a certain set of backgrounds.

We simplify easy mode and train agents in easybg mode, of which the only
difference from easy mode [5] is showing only a single background. This is
useful to investigate the case that using visual augmentation is helpful for
testing but not for training. We denote the task configuration by
game_name(mode), e.g., Coinrun(easybg). Then, we evaluate generalization
capabilities using two modes: test-bg and test-lv, which contain unseen
backgrounds and levels, respectively, in addition to easybg mode that we use
for training.

#### Types of augmentation.

For clarity, we mainly focus on two types of visual augmentation, each of
which conveys distinguishing inductive bias:

  1. (a)

Random color transforms an image by passing through either color jitter layer
or random convolutional layer proposed in [22]. From this, we can impose the
consistency to color changes, which may provide a strong generalization to
diverse backgrounds of test-bg mode.

  2. (b)

Random crop leaves a randomly-selected rectangle and pads zeros to the rest of
the image [26]. This augmentation is particularly useful in fully-observable
games (e.g., Chaser and Heist), because it imposes an efficient attention
mechanism.

We also report the result with other augmentations including color jitter,
random convolution, gray, and cutout color in the supplementary material, from
which the same messages can be interpreted.

#### Baseline methods for RL with data augmentation.

We mainly compare the proposed methods (InDA and ExDA) to the following
baselines:

  1. (a)

RAD [21] simply feeds PPO with experiences of original and augmented
observations.

  2. (b)

DrAC [26] is a method to simultaneously minimizing $L_{\text{PPO}}$ in Eq (1)
and $L_{\text{dis}}$ in Eq (4).

  3. (c)

DrAC+PAGrad is a variant of DrAC, which we devise to investigate another
mechanism to relieve the interference between RL training and augmentation in
Section 4.5.

The supplementary material presents further details and experiments, which we
omitted for simplicity. All results in the main paper are averaged over five
runs.

Figure 4:  Benefit of separating distillation from RL training. We compare
InDA, ExDA, DrAC, DrAC+PAGrad and ExDrAC when we start to apply each of them
to distill the prior of random crop after 20M timesteps of PPO. ExDrAC is a
variant of DrAC without RL training, i.e., minimizing only $L_{\text{dis}}$ in
Eq (4).

###  5.2 Benefit of separating distillation from RL training.

In Figure 4, after training PPO agent up to 20M timesteps, we start to
suddenly apply one of the different distillation methods with random crop. We
report averaged scores over 6 environments (Bigfish, Dodgeball, Plunder,
Chaser, Heist, Maze) after normalized by the highest train score of PPO on
each environment. After 20M timesteps, ExDA and ExDrAC have no RL training but
minimize $L_{\text{DA}{}}$ in Eq (5) and $L_{\text{dis}}$ in Eq (4),
respectively. The substantial gap between ExDA and ExDrAC justifies the design
of DA explicitly preserving the knowledge from RL when distilling the prior.
More importantly, it is remarkable that ExDA promptly learns the
generalization ability once it starts to distill the prior. This validates the
idea of postponing the distillation after RL training.

We now compare the distillation methods (InDA, DrAC, and DrAC+PAGrad)
concurrently optimizing the RL objective and distilling the prior in Figure 4.
Each method has performance degeneration due to the interference generated by
distillation. However, InDA and DrAC+PAGrad have clearly smaller degeneration
than DrAC which is the only one without any separation of optimizing the RL
objective and distillation loss. We note that DrAC+PAGrad has more
interference than InDA, and it seems to fail to impose the prior since there
is not much difference to PPO in testing. Hence, this verifies the superiority
of InDA which enables distilling the prior while alleviating the interference.

(a) train (easybg)

(b) test-lv

(c) Policy Distance

(d) train (easybg)

(e) test-lv

(f) Policy Distance

Figure 5: Time-sensitivity of applying augmentation. We compare train and test
performance of InDA$[S,T]$ with random crop, where the timing of applying DA
is governed by starting time $S$ and terminating time $T$, and we evaluate
four different pairs of $[S,T]=[0,0],[0,5],[20,25],[0,25]$ up to 25M
timesteps. Furthermore, we show the change of distance between two policies on
an original observation and an augmented observation. Note that InDA with
$[S,T]=[0,0]$ means RL training with no augmentation, i.e., vanilla PPO. We
focus on Chaser and Heist since they exemplify two representative time-
dependencies. Each train task uses easybg mode. We present further details and
results with other games in the supplementary material.

###  5.3 Effective timings of distillation

In what follows, we aim to identifying effective timings of distillation. To
this end, in Figure 5, we test several schedules of applying DA on two
representative environments of distinguishing traits. The supplementary
material presents the result on more environments, which is similar one of the
representatives.

#### An effective timing: InDA.

Chaser with random crop (Figure 5(a) and 5(b)) represents the case when
augmentation improves the sample efficiency of training and thus the
generalization ability in training. To compare the generalization ability, we
measure policy distance between original and augmented observations using
Jenson-Shannon divergence (Figure 5(c)). InDA[0, 5]’s policy distance is
increased after it stops using DA. Thus, the generalization ability is
degraded, if we do not continue to use DA. In this case, it is clear that InDA
should be applied during the entire RL training, as InDA[0, 25] shows the best
performance in both training and testing. In addition, it is also important to
apply DA as soon as possible since the effect of InDA[0,5] applying DA in the
beginning is relatively prompt and significant comparing to that of
InDA[20,25]. This suggests that the automatic framework needs to explore more
in the early stage.

#### An effective timing: ExDA.

Heist with random crop (Figure 5(d) and 5(e)) shows the opposite use of data
augmentation to what Chaser case suggests, i.e., postponing data augmentation
as much as possible. Random crop generates a slight interference, although it
immediately improves the generalization ability. We remark that the inductive
bias from the random crop is easily forgotten, as the test performance of
InDA[0, 5] is saturated right after stopping the distillation. This can be
explained by the time-varying nature of sample distribution and objective in
RL. Interestingly, it is in contrast to the data augmentation in SL, where an
early distillation is sufficient to impose the prior [11]. On the other hand,
the test performance curve of InDA[20, 25] soars right after DA. Furthermore,
Figure 5(f) shows that InDA[20, 25] narrows the gap between two policies on
the original and augmented observation. Recalling the interference between RL
training and distillation, this suggests postponing the distillation at the
end of RL training, and motivates our ExDA.

#### Performance benchmark on InDA and ExDA.

In Table 1, we summarize the train and test performances of InDA and ExDA on a
set of games with different augmentations and modes. ExDA outperforms other
baselines with random color on test-bg. Moreover, we note that ExDA consumes
only 0.5M timesteps to inject the prior at the end of RL training, whereas the
others use all the training data. ExDA in both sample efficiency and
generalization ability with random crop on test-lv. It is worth noting that
DrAC+PAGrad is slightly better than DrAC, while there is a substantial gap
between InDA and each DrAC-based algorithm. This again confirms the benefit
from the separated distillation of InDA observed in Figure 4. These results
demonstrate that each combination of environment and augmentation has a
suitable time at which to apply augmentation, and the gain from using the
right distillation timing, i.e., online (InDA, DrAC, or DrAC+PAGrad) or
offline (ExDA), is rigid regardless of the choice of algorithms.

Augmentation | Task | PPO | RAD | DrAC | DrAC+PAGrad | InDA | ExDA
---|---|---|---|---|---|---|---
Rand color | Rand conv | Train | 1.00 | 0.98 | 0.88 | 0.89 | 0.88 | 0.98
Test-bg | 1.00 | 1.08 | 1.86 | 1.88 | 1.92 | 2.11
Test-lv | 1.00 | 0.81 | 0.84 | 0.84 | 0.7 | 0.87
Color jitter | Train | 1.00 | 0.94 | 0.95 | 0.95 | 0.96 | 0.98
Test-bg | 1.00 | 1.37 | 1.44 | 1.44 | 1.43 | 1.48
Test-lv | 1.00 | 0.83 | 0.86 | 0.86 | 0.84 | 0.88
Rand crop | Train | 1.00 | 0.28 | 1.08 | 1.09 | 1.25 | 0.91
Test-bg | 1.00 | 0.64 | 0.87 | 0.94 | 0.94 | 0.95
Test-lv | 1.00 | 0.46 | 1.52 | 1.53 | 1.80 | 1.09

Table 1: Benchmark of InDA and ExDA. We report normalized train and test
scores of InDA, ExDA and DrAC with PAGrad on Open AI Procgen, compared to
baselines PPO, DrAC [26], RAD [21]. Boldface indicates the best performance.
We average the score among several environments [(Rand color: coinrun, ninja,
climber, fruitbot, jumper, heist, maze), (Rand crop: Bigfish, Dodgeball,
Plunder, Heist, Chaser, Maze)] after normalization considering PPO score to be
$1$. Every method is trained on 200 levels, using easybg mode. We evaluate
test performance on both test-bg and test-lv. The results can be interpreted
as an upper bound of potential gain from using data augmentation at perfect
timing.

###  5.4 Adaptive scheduling methods

(a) Chaser

(b) Heist

Figure 6: Exploration & exploitation to find the most beneficial augmentation.
We show that selected augmentation during training with UCB-InDA for each
Chaser and Heist. UCB-InDA automatically selects the augmentation among three
options, random color, identity and random crop.

Env | Method | PPO | DrAC | UCB-DrAC | InDA | ExDA | UCB-InDA | UCB-ExDA
---|---|---|---|---|---|---|---|---
Heist | Train |  9.2 $\pm$ 0.46 |  3.53 $\pm$0.3 |  7.41$\pm$ 2.09 |  4.9 $\pm$ 0.79 |  9.14 $\pm$ 0.56 | 9.67$\pm$ 0.23 |  9.45 $\pm$ 0.29
Test-bg |  5.18 $\pm$ 1.53 |  3.58 $\pm$ 0.31 |  3.76$\pm$ 0.54 |  4.9 $\pm$ 0.87 |  7.05 $\pm$ 1.29 |  6.23$\pm$ 1.29 | 7.86$\pm$ 0.83
Test-lv |  4.13 $\pm$ 1.39 |  3.07 $\pm$ 0.33 |  3.49$\pm$0.48 |  1.47$\pm$ 0.77 |  5.05 $\pm$ 0.98 |  4.8$\pm$ 1.24 | 5.74$\pm$ 0.67
Chaser | Train |  5.63$\pm$ 1.12 |  0.16$\pm$0.02 |  4.6$\pm$ 1.22 |  5.69$\pm$ 1.42 |  5.58$\pm$ 1.33 | 7.49$\pm$1.27 |  7.23$\pm$ 1.15
Test-bg |  0.87$\pm$ 0.06 |  0.1$\pm$ 0.02 |  0.57$\pm$0.12 |  3.51$\pm$ 1.33 |  1.02$\pm$ 0.04 |  1.08$\pm$ 0.08 | 3\. 18$\pm$ 0.79
Test-lv |  4.83$\pm$ 0.88 |  0.14$\pm$0.01 |  4.14$\pm$1.01 |  4.96$\pm$ 1.21 |  5.11$\pm$ 1.05 | 6.45$\pm$ 0.8 |  6.43$\pm$ 1.28

Table 2: Full exploitation of augmentation to improve generalization on both
test-bg and test-lv. We compare InDA, ExDA, DrAC, UCB-DrAC, UCB-InDA, UCB-ExDA
and PPO about train, test-bg and test-lv. Boldface indicates the best method.
InDA and DrAC use both random color and random crop during RL training. ExDA
use both augmentation after RL training. UCB-InDA and UCB-DrAC are trained as
automatically selecting the augmentation during training. UCB-ExDA trains ExDA
after UCB-InDA with both augmentation.

#### Adaptive selecting of timings: UCB-InDA and UCB-ExDA.

It is hard to know in advance whether a certain augmentation helps the
training or not [26]. We hence employ UCB-InDA which estimates the gain or
damage from each augmentation from trial and error and identifies the one with
most help. Recall Table 2 where PPO (without augmentation) shows much better
training performance than InDA in Heist. As shown in Figure 6(b), UCB-InDA is
able to identify that no augmentation is best for training in Heist. This
implies that ExDA is more appropriate than InDA. Conversely, random crop is
selected on Chaser (Figure 6(a)). As the result, we can automatically select
InDA or ExDA appropriately for each task.

#### Fully exploitation of augmentation

In Table 2, when both random color and random crop are used to improve
generalization on both test-bg and test-lv, we report numerical evaluation of
UCB-InDA and UCB-ExDA with other baselines on train and test tasks. Decreased
train performance of DrAC and InDA compare to PPO show the difficulty of
simultaneous training with several augmentations. Train performance of UCB-
InDA and UCB-DrAC are improved by adaptive selecting, especially, UCB-InDA is
better than UCB-DrAC. The gap is made due to the robustness about the change
of augmentation during training. In terms of generalization, UCB-ExDA clearly
surpasses UCB-InDA thanks to ExDA to extract all the priors from the complete
set of data augmentations at the end of RL training.

##  6 Discussion

We have identified two most effective yet simple timings (InDA and ExDA) of
data augmentation for RL, and proposed UCB-ExDA framework to adaptively select
the best scheduling augmentations. We note that the effectiveness of this
framework is restricted but specialized for RL with the unique non-stationary
nature. Indeed, in SL without shift of data distribution and objective, it is
sufficient to apply data augmentation at the beginning [11]. Our framework
employs the most basic multi-armed bandit algorithm with a finite set of data
augmentation. It is interesting to investigate a room to improve by further
optimizing continuous parameters of data augmentation for RL, c.f., an auto-
augmentation technique to optimize continuous augmentation parameter per
sample for SL [12]. Another promising direction is to accelerate the
distillation process of DA by data condensation with augmentation [39]. This
is possible with our framework clearly separating between RL training and
distillation, and may be particularly useful to train distributed RL agents
since a condensed data for an agent’s distillation is usable for the other.

## Acknowledgments and Disclosure of Funding

We thank Kimin Lee for helpful discussions. This work was supported by
Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government (MSIT) (No.2019-0-01906,
Artificial Intelligence Graduate School Program (POSTECH)) and Institute of
Information & communications Technology Planning & Evaluation (IITP) grant
funded by the Korea government (MSIT) (No.2021-0-02068, Artificial
Intelligence Innovation Hub) and the National Research Foundation of Korea
(NRF) grant funded by the Korea government (MSIT) (No. 2021M3E5D2A01023887).

## References

  * [1] Alessandro Achille, Matteo Rovere, and Stefano Soatto.  Critical learning periods in deep networks.  In International Conference on Learning Representations, 2018.
  * [2] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo.  Implicit regularization in deep matrix factorization.  In Advances in Neural Information Processing Systems, pages 7411–7422, 2019.
  * [3] Peter Auer.  Using confidence bounds for exploitation-exploration trade-offs.  Journal of Machine Learning Research, 3(Nov):397–422, 2002.
  * [4] Richard E Bellman.  Adaptive control processes.  Princeton university press, 2015.
  * [5] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  In International conference on machine learning, pages 2048–2056. PMLR, 2020.
  * [6] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman.  Quantifying generalization in reinforcement learning, 2019.
  * [7] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent.  Why does unsupervised pre-training help deep learning?  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201–208. JMLR Workshop and Conference Proceedings, 2010.
  * [8] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  arXiv preprint arXiv:1802.01561, 2018.
  * [9] Jesse Farebrother, Marlos C. Machado, and Michael Bowling.  Generalization and regularization in dqn, 2020.
  * [10] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien.  Implicit regularization of discrete gradient dynamics in linear neural networks.  In Advances in Neural Information Processing Systems, pages 3196–3206, 2019.
  * [11] Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto.  Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence.  In Advances in Neural Information Processing Systems, pages 10678–10688, 2019.
  * [12] Denis Gudovskiy, Luca Rigazio, Shun Ishizaka, Kazuki Kozuka, and Sotaro Tsukizawa.  Autodo: Robust autoaugment for biased data with label noise via scalable probabilistic implicit differentiation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16601–16610, 2021.
  * [13] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro.  Implicit regularization in matrix factorization.  In Advances in Neural Information Processing Systems, pages 6151–6159, 2017.
  * [14] Nicklas Hansen, Yu Sun, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang.  Self-supervised policy adaptation during deployment.  arXiv preprint arXiv:2007.04309, 2020.
  * [15] Nicklas Hansen and Xiaolong Wang.  Generalization in reinforcement learning by soft data augmentation.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13611–13617. IEEE, 2021.
  * [16] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters, 2019.
  * [17] Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner.  Darla: Improving zero-shot transfer in reinforcement learning, 2018.
  * [18] Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson.  The impact of non-stationarity on generalisation in deep reinforcement learning.  arXiv preprint arXiv:2006.05826, 2020.
  * [19] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  arXiv preprint arXiv:1806.10293, 2018.
  * [20] Ilya Kostrikov, Denis Yarats, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  arXiv preprint arXiv:2004.13649, 2020.
  * [21] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas.  Reinforcement learning with augmented data.  arXiv preprint arXiv:2004.14990, 2020.
  * [22] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee.  Network randomization: A simple technique for generalization in deep reinforcement learning.  arXiv, pages arXiv–1910, 2019.
  * [23] Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell.  Regularization matters in policy optimization – an empirical study on continuous control, 2020.
  * [24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * [25] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel.  Asymmetric actor critic for image-based robot learning.  arXiv preprint arXiv:1710.06542, 2017.
  * [26] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus.  Automatic data augmentation for generalization in deep reinforcement learning.  arXiv preprint arXiv:2006.12862, 2020.
  * [27] Sharath Chandra Raparthy, Bhairav Mehta, Florian Golemo, and Liam Paull.  Generating automatic curricula via self-supervised active domain randomization, 2020.
  * [28] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  arXiv preprint arXiv:1506.02438, 2015.
  * [29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [30] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [31] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [32] Aravind Srinivas, Michael Laskin, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
  * [33] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin.  Decoupling representation learning from reinforcement learning.  arXiv preprint arXiv:2009.08319, 2020.
  * [34] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30. IEEE, 2017.
  * [35] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur.  When do curricula work?, 2021.
  * [36] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus.  Improving sample efficiency in model-free reinforcement learning from images, 2020.
  * [37] Kaichao You, Mingsheng Long, Jianmin Wang, and Michael I. Jordan.  How does learning rate decay help modern neural networks?, 2019.
  * [38] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  arXiv preprint arXiv:2001.06782, 2020.
  * [39] Bo Zhao and Hakan Bilen.  Dataset condensation with differentiable siamese augmentation.  In International Conference on Machine Learning, pages 12674–12685. PMLR, 2021.

##  Appendix A Modified Procgen Environments

(a) train (partial)

(b) test-bg (partial)

(c) test-lv (partial)

(d) train (fully)

(e) test-bg (fully)

(f) test-lv (fully)

Figure 7: An example set of training and testing environments in Procgen
benchmark: (upper row) an example of partially observable environment with
Coinrun; (lower row) an example of fully observable environment with Heist;
(left column) train: a set of levels and backgrounds for training; (center
column) test-bg: the same training levels on unseen backgrounds; (right
column) test-lv: a set of unseen levels on the same training backgrounds

This section explains Modified Procgen Environments, which is designed to
verify different types of generalization, backgrounds, and levels. Open AI
Procgen environments [5] share background themes such as space_backgrounds,
platform_backgrounds, topdown_backgrounds, water_backgrounds,
water_surface_backgrounds. We create new difficulties as Easybg, Easybg-test,
Easy-test. Easybg generates environments which contain only one for each
background, wall and agent theme. Easybg-test and Easy-test are for test about
background change after trained on Easybg and Easy. Wall theme in (Climber,
Coinrun, Jumper, Ninja) and Agent theme in (Climber, Coinrun) also compose
with only one image resource in Easybg. Figure 7 presents an example set of
modes that we use in evaluation. Furthermore, We fix the exit_wall_choice and
enemy theme in Dodgeball. We describe the usage themes in each environment,
which are grouped by backgrounds theme as below:

  * •

space_backgrounds (Bossfight, Starpilot)
Background: "space_backgrounds/deep_space_01.png"

  * •

platform_backgrounds (Caveflyer, Climber, Coinrun, Jumper, Miner, Ninja)
Background:"platform_backgrounds/alien_bg.png", Coinrun (Agent color: Beige,
Wall themes: Dirt), Climber (Agent color: Blue, Wall themes: tileBlue), Jumper
(Wall theme: tileBlue), Ninja (Wall theme: bricksGrey)

  * •

topdown_backgrounds (Chaser, Dodgeball, Fruitbot, Heist, Leaper, Maze)
Background:"topdown_backgrounds/floortiles.png", Dodgeball (Enemy theme:
"misc_assets/character1.png", Exit_wall_choice: 0)

  * •

water_backgrounds (Bigfish)
Background:"water_backgrounds/water1.png"

  * •

water_surface_backgrounds (Plunder)
Background:"water_backgrounds/water1.png"

Easybg-test uses backgrounds in each background group, except the one used in
Easybg. Easy-test is only defined for Climber, Jumper, Ninja, and they compose
with topdown_backgrounds.

##  Appendix B Implementation details

In this section, we explain about InDA, ExDA, UCB-InDA and other baselines. We
train the agent with IMPALA-CNN [8] in every experiment.

###  B.1 InDA

We use PPO [29] as a base RL algorithm, For data efficiency, we store the
observations during RL training in buffer $\mathcal{D}_{O}$. Before DA phase,
we also make policy buffer $\mathcal{D}_{\Pi}$, value function buffer
$\mathcal{D}_{V}$ and augmented observation buffer $\mathcal{D}_{\phi}$ for
distillation, because we only use one network model. We randomly sample pairs
of $(o,\pi,V)$ from buffer, and minimize loss function
$L_{\text{DA}{}}(\theta)$. We reuse the sample three times like PPO, it can be
controlled by $\\#$ Epochs of DA. We did a greed searches for learning rate of
DA $l_{DA}\in[1\times 10^{-3},5\times 10^{-4},2\times 10^{-4},1\times
10^{-4},5\times 10^{-5}]$ and interval $I\in[1,5,10]$ and found the best
combination $l_{\text{DA}{}}=10^{-4}$ and interval $I=5$. We fix the buffer
size $\mathcal{D}_{O}=40960$, because we collect the observations during five
RL phases $(5\times 256\times 32)$. We describe the every hyperparameter as
below:

Hyperparamter | Value
---|---
$\gamma$ | 0.999
$\lambda$ | 0.95
# Timesteps per rollout | 256
# Epochs per rollout | 3
# Minibatches per epoch | 8
Reward Normalization | Yes
# Workers | 1
# Environments per worker | 32
Total timesteps | 25M
LSTM | No
Frame Stack | No
Optimizer | Adam optimizer
Entropy bonus | 0.01
PPO clip range | 0.2
Learning rate | $5\times 10^{-4}$
Interval $I$ | 5
Size of $\mathcal{D}_{O}$ | 40960
# Epochs of DA | 3
Learning rate of DA $l_{\text{DA}{}}$ | $1\times 10^{-4}$
Image transformation $\phi$ | Any augmentation

###  B.2 ExDA

In ExDA, we generate and store $(o,\pi,V)$ using $f_{\theta_{old}}$ in buffer
$\mathcal{D}$. The optimal buffer size depends on the episode length of each
environment. However, we standardize the buffer size as 0.5M in every
environment. We augment the observation with three epochs intervals when using
randomized augmentation methods. We did greed searches for # minibatches
$[1024,2048,4096]$ and learning rate $[5\times 10^{-4},1\times 10^{-3},2\times
10^{-3}]$. As a result, we select $\\#$ of minibatches $4096$ and a learning
rate $1e-3$.We describe every hyperparameter as below:

Hyperparameter | Value
---|---
Size of $\mathcal{D}_{O}$ | 0.5M
# Epoch | 30
# Minibatches per epoch | 4096
Learning rate | $1\times 10^{-3}$
# Workers | 1
Optimizer | Adam optimizer
Image transformation $\phi$ | Any augmentation

###  B.3 UCB-InDA

We use UCB-InDA as a discriminator to determine the necessity of augmentation
during the training. The gain of an augmentation is a mean of return during
interval I, $G(s)=\frac{1}{I}\sum_{i=0}^{j-1}R(s+j)$. The return is computed
by the sum of estimated advantage and predicted value, which are expected
value of the agent trajectory,
$R(s)=\hat{\mathbb{E}}_{(o_{t},a_{t})\sim\pi_{\theta}}[\hat{A}_{t}+V_{\theta}(o_{t})]$.
The $\hat{A}_{t}$ is advantage from Generalized Advantage Estimator [28].
Thus, we can evaluate how augmentation affects the return on the agent
trajectory. However, the distribution of return is non-stationary, as the
agent policy is changed. Therefore, we use the window average gain
$\bar{G_{\phi}}$ rather than the whole gain from the past evaluation.
Furthermore, the drastic change of return causes the gap of gain between the
augmentation according to sampling time at the transient time of training and
leads to poor exploration about some augmentation methods. For stable
exploration, we fix the minimum exploration frequency and use forced
exploration method after the minimum exploration as below:

$\bar{G}_{\phi_{max}}(s)+c\sqrt{\frac{\log(s)}{N_{\phi_{max}}(s)}}\leq\bar{G}_{\phi_{min}}(s)+c\sqrt{\frac{\log(s)}{N_{\phi_{min}}(s)}}$
(9)
$c=\frac{\bar{G}_{\phi_{max}}-\bar{G}_{\phi_{min}}+\epsilon}{\sqrt{\log(s)}\times\max(\frac{1}{\sqrt{N_{\phi_{min}}(s)}}-\frac{1}{\sqrt{N_{\phi_{max}}(s)}},\frac{1}{\sqrt{W-1}}-\frac{1}{\sqrt{W}})}$
(10)

where
$\phi_{max}=\operatorname*{arg\,max}_{\phi\in\Phi}\bar{G}_{\phi},\phi_{min}=\operatorname*{arg\,min}_{\phi\in\Phi}\bar{G}_{\phi}$.
We set the hyperparameter as below table:

Hyperparameter | Value
---|---
Window size of gain W | 3
Minimum exploration frequency | 15

###  B.4 Baselines

We compare ExDA and InDA with PPO [5], DrAC [26], Rand-FM [22], RAD [21].
Every baseline is based on PPO [5] and we adopt the implementation of PPO in
[5].

  * •

DrAC [26] regularizes both policy and value function as self-supervised
learning. Regularization term have hyperparameter $\alpha_{r}$ for ratio with
PPO objective. We use the hyperparameter recommended by the author.

  * •

Rand-FM [22] is composed with random convolution networks and feature
matching. They also need hyperparameter $\beta$ for ratio between feature
matching and PPO objective. We use same $\beta$ with author.

  * •

RAD [21] naively use augmented observations in state distribution. Thus, there
are no additional hyperparameters.

We describe the hyperparameter of baselines in below table:

Hyperparameter | Value
---|---
$\gamma$ | 0.999
$\lambda$ | 0.95
# of timesteps per rollout | 256
# of epochs per rollout | 3
# of Minibatches per epoch | 8
Reward Normalization | Yes
# of Workers | 1
# of environments per worker | 64
Total timesteps | 25M
LSTM | No
Frame Stack | No
Optimizer | Adam optimizer
Entropy bonus | 0.01
PPO clip range | 0.2
Learning rate | $5\times 10^{-4}$
$\alpha_{r}$ (DrAC) | 0.1
$\beta$ (Rand-FM) | 0.002

##  Appendix C Data augmentation

In our experiments, we use five augmentation methods: crop, grayscale, cutout
color, random convolution and color jitter. We refer the implementation of
augmentations from Lee et al. [22] (random convolution), Laskin et al. [21]
(cutout color, color jitter) and Raileanu et al. [26] (grayscale, crop). We
expect the generalization about background change from random convolution,
color jitter, gray, cutout color. About the change of levels, we use crop and
cutout color for generalization. Examples of data augmentation are represented
below:

(a) Original (b) Rand Conv (c) Crop

(d) Color jitter (e) Gray (f) Cutout color

Figure 8: Examples of visual augmentations Figure 9: Interference from
increased complexity of learning. We present train performance when training
agent on Heist(easybg-$B$), where easybg-$B$ is a variant of easybg mode with
$B$ backgrounds. We plot the train performance curves of PPO and InDA with
random convolution over training epochs in absolute score.

##  Appendix D Interference from increased complexity of learning

Even if the policy for the original observation is fixed and augmentation is
used, learning may still be hindered. In Figure 9, we report training curves
of the agents on train tasks of different degree of background diversity. InDA
with random convolution is anticipated to lead the prior on color consistency,
which seems helpful to handle background diversity. However, PPO outperforms
InDA in terms of the sample complexity to master train tasks, while the gap is
decreasing as the background diversity increases.This implies that diverse
backgrounds make hard to train by increased complexity of learning, and also
data augmentation can cause similar difficulty even with right prior, when
using it during RL training. We further remark that ExDA using random
convolution after PPO trained on Heist(easybg-1) achieves 8.15 on
Heist(easybg-15), which is much higher than PPO’s 6.53 trained on
Heist(easybg-15). This suggests the importance of simplifying the train task
and the utility of ExDA which completely separates RL training and
distillation with augmentation.

##  Appendix E Robustness in loss function change

In ExDA, we transfer the policy after training 20M time steps with PPO. Thus,
we explain why other augmentations are not used after pre-training. We compare
the results of training and test performance with Drac [26], Rand-FM [22], Rad
[21] when we train each method for 5M after training PPO for 20M time steps.
We use random convolution and crop as data augmentation methods, and we do not
compare with RAD when we use crop in Figure 12 and Figure 13. The crop method
used in our paper do not work well in RAD, because they use a different crop
method with [26] in their paper [21]. InDA is more stable than others in
training, and it affects generalization performance.

Every training curves decline immediately after starting training with
augmented observations at 20M time steps. The objective function is changed to
each baseline, and augmented data is newly added to data distribution. Thus,
the optimizer should find a new optimal point for new objective function and
data. During find the new optimal points, the agent learns along with the
different directions from the optimization direction in pure PPO. Thus,
performance can be degraded because the learning direction on loss landscape
is different from maximizing rewards on non-augmented data in PPO. In spite of
using self-supervised learning or representation learning, the policy is
changed because they update the same network’s parameter for matching policy
or latent features, such as DrAC [26] and Rand-FM [22]. However, InDA is more
stable than the others because we distill the fixed policy and value using DA.
It does the stable training through conserving the policy on non-augmented
observations during optimizing for augmented data.

Figure 10: Comparison of the training performance when random convolution is
applied after 20M timesteps with various augmentation methods.  Figure 11:
Comparison of the test performance when random convolution is applied after
20M timesteps with various augmentation methods.  Figure 12: Comparison of the
training performance when crop is applied after 20M timesteps with InDA and
Drac.  Figure 13: Comparison of the test performance when crop is applied
after 20M timesteps with InDA and Drac.

##  Appendix F Robustness against wrong augmentation

In this section, we verify the robustness of ExDA from an obvious wrong
augmentation: just making an black image, denoted by black. Using this clearly
interferes with RL training. We compare PPO, InDA, ExDA, UCB-InDA, UCB-ExDA
and DrAC. The hyperparameter is almost the same with Section 5.4 except we use
black instead of random color and random crop. For UCB-based methods, we use 4
arms: black, random color, random crop, and no aug. As expected, ExDA
preserves the PPO score, but InDA and DrAC degrade the score. Furthermore,
UCB-InDA improves the performance in Chaser, and also it almost preserves the
score in Heist. Lastly, UCB-ExDA also maintains the score from UCB-InDA.

Env | PPO | DrAC | InDA | ExDA | UCB-InDA | UCB-ExDA
---|---|---|---|---|---|---
Heist | 9.2 $\pm$ 0.46 |  7.35 $\pm$ 0.684 |  6.72 $\pm$0.419 |  8.6 $\pm$ 0.189 |  8.84 $\pm$ 0.307 |  8.64$\pm$ 0.264
Chaser |  5.63$\pm$ 1.12 |  1.49$\pm$0.036 |  5.43$\pm$ 0.633 |  5.1$\pm$0.331 | 6.74$\pm$0.588 |  6.28$\pm$ 0.436

Table 3: Robustness from the wrong augmentation

##  Appendix G Ablation study of ExDA

###  G.1 Initialization and regularization term

In this section, we do an ablation study about the factor of ExDA. We mention
the loss function and re-initialization issue in subsection 4.3. ExDA does not
have to minimize $L_{VD}$ because the value function is useless after RL
training. The below results show that $L_{VD}$ cannot give any benefit in
ExDA. Thus, we only use $L_{PD}$ for computational complexity. Furthermore, we
also compare to verify the effect of non-stationarity with a re-initialized
agent before distillation. Igl et al. [18] argued that the non-stationarity
causes the reduction of generalization. However, the re-initialization is not
critical in test performance, as shown in Figure 15. Moreover, sometimes re-
initialization makes it difficult to distill training performance such as
Fruitbot and Ninja in Figure 14. We use random convolution  as an augmentation
method in here.

Figure 14: Training performance of ExDA with re-initialization or
regularization with value funtion. Figure 15: Test performance of ExDA with
re-initialization or regularization with value funtion on unseen backgrounds.

###  G.2 ExDA after InDA with various backgrounds

When augmentation helps the training, ExDA struggle to follow the training
performance of InDA because ExDA’s training performance is limited by pre-
trained agent’s policy. Thus, we use InDA for ExDA’s pre-training , and call
it as ExDA (InDA). As shown in Table 5, ExDA (InDA) is comparable to InDA, but
not beyond. Thus, unless there is a meaningful difference in training
performance, ExDA has no better generalization than InDA. However, in
computational complexity, ExDA is more efficient than others such as InDA and
DrAC when they have a similar performance. In the following section, we
discuss computational complexity.

Easy | PPO | InDA | ExDA (PPO) | ExDA (PPO) + reinit | ExDA (InDA) | ExDA (InDA) + reinit
---|---|---|---|---|---|---
Jumper | 8.55 | 8.94 | 8.5 | 8.6 | 8.83 | 8.83
$\pm$0.17 |  $\pm$0.09 |  $\pm$0.183 |  $\pm$0.156 |  $\pm$0.215 |  $\pm$0.126
Ninja | 7.49 | 8.88 | 7.03 | 7.23 | 8.71 | 8.56
$\pm$0.42 |  $\pm$0.34 |  $\pm$0.058 |  $\pm$0.159 |  $\pm$0.344 |  $\pm$0.394
Climber | 8.63 | 8.5 | 8.1 | 8.09 | 8.16 | 7.99
$\pm$0.46 |  $\pm$0.29 |  $\pm$0.268 |  $\pm$0.268 |  $\pm$0.441 |  $\pm$0.383
Table 4: The comparison with diverse agents which are trained with ExDA Easy | PPO | InDA | ExDA (PPO) | ExDA (PPO) + reinit | ExDA (InDA) | ExDA (InDA) + reinit
---|---|---|---|---|---|---
Jumper | 6.85 | 7.94 | 7.54 | 7.48 | 7.98 | 7.67
$\pm$0.19 |  $\pm$0.19 |  $\pm$0.158 |  $\pm$0.154 |  $\pm$0.148 |  $\pm$0.155
Ninja | 6.29 | 6.5 | 5.56 | 5.73 | 6.27 | 5.94
$\pm$0.19 |  $\pm$0.19 |  $\pm$0.158 |  $\pm$0.154 |  $\pm$0.148 |  $\pm$0.155
Climber | 6.96 | 7.28 | 7.06 | 6.89 | 6.8 | 5.45
$\pm$0.65 |  $\pm$0.35 |  $\pm$0.541 |  $\pm$0.237 |  $\pm$0.441 |  $\pm$0.383
Table 5: Test performance of agents, which is trained on easy mode with random
convolution.

###  G.3 Computational complexity

We compare the computational complexity with ExDA and InDA. InDA do DA for
every 25M observations during training and reuse the sample in three times.
However, ExDA only use 0.5M for DA during 30 epochs. Thus, ExDA is almost 5
times more efficient than InDA by rough calculation. Furthermore, the ExDA
saves the time for augmentation compared to InDA. When we train with same
computational setting (GPU: GeForce RTX 2080 TI), ExDA only consumes 5 hours +
2 hours (PPO) when using random convolution, but, InDA consumes 18 hours.
Thus, we recommend ExDA when InDA cannot give a meaningful gain in training
performance.

##  Appendix H Comparison between InDA and ExDA using the same steps of RL
training

In every experiment in the paper, we set the evaluation setup to be somewhat
unfavorable to ExDA (using 20M time steps of RL training followed by
additional 0.5M steps of distillation; denoted by ExDA(20M)) compared to InDA
or other baselines (25M time steps of RL training) to clearly avoid potential
complaint about the extra 0.5M steps for ExDA, It is obvious that the
performance of ExDA is improved if we put more time steps for RL training, and
thus the benefit of ExDA compared to InDA becomes more conspicuous if ExDA is
the effective timing. Thus, we do an additional experiment evaluating ExDA
(25M) using 25M RL time steps and 0.5M distillation time steps as below table:

Easybg |  | PPO | DrAC | RAD | InDA |  ExDA(20M) |  ExDA(25M)
---|---|---|---|---|---|---|---
Heist | Train | 9 | 5.95 | 7.94 | 5.15 | 8.72 | 8.93
Test-bg | 5.18 | 5.47 | 4.78 | 4.96 | 8.15 | 8.26
Test-lv | 4.13 | 5.4 | 3.81 | 5.91 | 5.35 | 5.41
Table 6:  Additional RL training before ExDA

The result of ExDA(25M) reinforces our main message: postponing data
augmentation when it generates a severe interference with RL training in Table
6. ExDA(25M) has a slight drop in train score compared to PPO, but it could be
eventually removed if we put (slightly) more steps for distillation.

##  Appendix I UCB with a large set of augmentation

In Figure 6, we only use three arms such as random color, random crop and no
augmentation. Thus, we try to select the usuful augmentations at each time
among a large set of augmentation, which contains gray, cutout color, random
convolution, color jitter, random crop and no augmentation. In Figure 16(a),
we show the result of arm selection with UCB-InDA. The identity function is
most frequently used in the same as Figure 6. On the other hand, we ablate the
neccesity of identity function in UCB-InDA, it shows that the identity
function is needed.

(a) Augmentation selection

(b) Performance

Figure 16: Figure 16(a) show the selected number of augmentations by UCB on
Heist. We compare two UCB-InDAs, w/ and w/o identity function with PPO, InDA,
ExDA, UCB-ExDA on Heist easybg in Figure 16(b). UCB-InDA is trained after UCB-
InDA w/ identity, we use random convolution as a data augmentation in InDA,
ExDA. Solid line: train performance; dotted line: test performance. ExDA
achieves larger test performance than InDA by preserving train performance.
Moreover, UCB-InDA w/ identity outperforms UCB-InDA w/o identity in the
training.

##  Appendix J Time matter in training

This section shows every result of Figure 5 about time dependency with InDA.
We experiment with random convolution, crop, color jitter and evaluate the
test on unseen backgrounds (random convolution, color jitter) and levels
(random crop). However, the effect of generalization is hard to recognize in
most cases, as shown in Appendix K. Thus, we mainly discuss the most effective
augmentation, such as random convolution and crop in the main paper, and only
represent some environments that have helped the generalization by color
jitter. easybg mode is used as default mode with three easy mode (Climber,
Jumper, Ninja) in our experiments. The shaded regions and solid line represent
the standard deviation and mean, across five runs.

###  J.1 Random convolution

Figure 17: Comparison of training performance according to usage period of
augmentation with InDA (random convolution): The easybg is disturbed by random
convolution, but, easy mode is improved training performance by random
convolution. Figure 18: Comparison of generalization on unseen backgrounds
according to usage period of augmentation with InDA (random convolution): Most
cases’ tendencies are coincidence with the jumper, which is mentioned in the
main paper.

###  J.2 Crop

Figure 19: Comparison of training performance according to usage period of
augmentation with InDA (crop): Crop improve the training performance in
Bigfish, Chaser, Dodgeball, Plunder. Furthermore, interrupted augmentation is
also improved similarly with (0, 25). Figure 20: Comparison of generalization
on unseen levels according to usage period of augmentation with InDA (crop):
The generalization is improved by crop, and it is conserved after interrupted
in Heist and Maze. Bigfish, Chaser, Dodgeball, and Plunder have similar curves
with training.

###  J.3 Color jitter

Figure 21: Comparison of training performance according to usage period of
augmentation with InDA (color jitter): Color jitter does not impede the
training as much as random convolution in most environments. However, color
jitter helps the training in easy mode.  Figure 22: Comparison of
generalization on unseen backgrounds according to usage period of augmentation
with InDA (color jitter): Test performance is influenced by color jitter as
the trend, which is similar to random convolution.

##  Appendix K Benchmark on Modified Open AI Procgen

We compare the training and test performance on various environments with each
augmentation. We also use DrAC [26], RAD [21], DrAC+PAGrad as baselines. In
every result, we train the agent for 25M timesteps, except the ExDA. ExDA is
trained with 0.5M after training 20M with PPO. We also compare the average
score after normalized by PPO’s score and indicate the best score as bold.
Mean and standard deviation is calculated after five runs. We show the result
about random conv, color jitter, random crop in Table 1. Additionally, we
evaluate benchmark with gray and cutout color in Table 7. For benchmark, we
classify Procgen environments with each characteristic as Figure 23.
Furthermore, we attach detail results on each environments with Oracle and
Rand-FM [22]. Red one is the Oracle score, which is trained on test
environments such as easybg-test, easy-test. For your information, RAD does
not work well when using crop, because we use [26]’s crop method which is
different with [21].

Figure 23: Group of OpenAI Procgen environments: We classify environments according to characteristics about a type of observation window and the importance of color. For Group1, we do not use random crop because they have agent centered window, thus the random crop can make hard to find agent. For Group3, we do not use random color, because they have important meaning in the color of objects, thus the transformation of color would make hard to learn information in color. Group2 can apply both of the transformations.  Augmentation | Task | PPO | RAD | DrAC | DrAC+PAGrad | InDA | ExDA
---|---|---|---|---|---|---|---
Grayscale | Train | 1.00 | 0.94 | 0.93 | 0.94 | 0.95 | 0.99
Test-bg | 1.00 | 1.04 | 1.03 | 1.04 | 0.97 | 1.13
Test-lv | 1.00 | 0.81 | 0.84 | 0.84 | 0.84 | 0.86
Cutout color | Train | 1.00 | 0.72 | 0.82 | 0.84 | 0.76 | 0.94
Test-bg | 1.00 | 1.33 | 1.27 | 1.29 | 1.19 | 1.53
Test-lv | 1.00 | 0.69 | 0.83 | 0.83 | 0.69 | 0.93
Table 7: Train and test score of InDA and ExDA on Open AI Procgen, compared to
baselines PPO, Drac [26], RAD [21], DrAC+PAGrad. Boldface indicates the best
method.

###  K.1 Random convolution

Easy | PPO | DrAC | Rand_FM | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 8.63 | 8.33 | 8.27 | 7.93 | 8.5 | 8.1
$\pm$0.462 |  $\pm$0.407 |  $\pm$0.187 |  $\pm$0.37 |  $\pm$0.291 |  $\pm$0.268
Jumper | 8.55 | 8.62 | 8.47 | 8.51 | 8.94 | 8.5
$\pm$0.168 |  $\pm$0.075 |  $\pm$0.13 |  $\pm$0.102 |  $\pm$0.09 |  $\pm$0.183
Ninja | 7.49 | 8.57 | 7.69 | 7.9 | 8.88 | 7.03
$\pm$0.421 |  $\pm$0.069 |  $\pm$0.529 |  $\pm$0.652 |  $\pm$0.343 |  $\pm$0.058
Avg | 1.00 | 1.04 | 0.99 | 0.99 | 1.07 | 0.96
Table 8: Training performance benchmark on easy with random convolution.  Easy | PPO | DrAC | Rand_FM | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 6.96 | 7.21 | 6.63 | 6.08 | 7.28 | 7.06
$\pm$0.651 |  $\pm$0.447 |  $\pm$0.39 |  $\pm$0.264 |  $\pm$0.341 |  $\pm$0.541
Jumper | 6.85 | 7.97 | 6.7 | 6.74 | 7.94 | 7.54
$\pm$0.192 |  $\pm$0.128 |  $\pm$0.167 |  $\pm$0.299 |  $\pm$0.185 |  $\pm$0.158
Ninja | 6.29 | 6.18 | 6.22 | 6.22 | 6.5 | 5.56
$\pm$0.529 |  $\pm$0.193 |  $\pm$0.57 |  $\pm$0.324 |  $\pm$0.191 |  $\pm$0.158
Avg | 1.00 | 1.06 | 0.97 | 0.95 | 1.08 | 1
Table 9: Test performance benchmark on unseen backgrounds (easy, random convolution).  Easybg | PPO | Oracle | DrAC | Rand-FM | RAD | InDA | ExDA
---|---|---|---|---|---|---|---
Climber | 12.35 | 9.78 | 11.23 | 12.2 | 12.15 | 10.89 | 12.07
$\pm$0.083 |  $\pm$0.306 |  $\pm$0.353 |  $\pm$0.128 |  $\pm$0.09 |  $\pm$0.162 |  $\pm$0.073
Coinrun | 9.64 | 7.11 | 9.17 | 9.57 | 9.56 | 8.81 | 9.44
$\pm$0.07 |  $\pm$0.205 |  $\pm$0.161 |  $\pm$0.126 |  $\pm$0.107 |  $\pm$0.992 |  $\pm$0.149
Fruitbot | 29.78 | 29.74 | 26.07 | 30.19 | 29.92 | 26.17 | 28.76
$\pm$0.899 |  $\pm$0.443 |  $\pm$0.658 |  $\pm$0.512 |  $\pm$0.623 |  $\pm$0.575 |  $\pm$0.79
Heist | 9 | 7.21 | 5.95 | 7.7 | 7.94 | 5.15 | 8.72
$\pm$0.513 |  $\pm$0.27 |  $\pm$0.343 |  $\pm$0.6 |  $\pm$0.919 |  $\pm$0.614 |  $\pm$0.533
Jumper | 8.95 | 8.72 | 8.86 | 8.91 | 9.04 | 8.78 | 8.94
$\pm$0.066 |  $\pm$0.119 |  $\pm$0.088 |  $\pm$0.13 |  $\pm$0.135 |  $\pm$0.172 |  $\pm$0.048
Maze | 9.75 | 8.56 | 8.1 | 9.61 | 9.51 | 9.12 | 9.73
$\pm$0.513 |  $\pm$0.27 |  $\pm$0.343 |  $\pm$0.6 |  $\pm$0.919 |  $\pm$0.614 |  $\pm$0.533
Ninja | 9.75 | 7.81 | 9.43 | 9.75 | 9.78 | 9.53 | 9.7
$\pm$0.073 |  $\pm$0.422 |  $\pm$0.109 |  $\pm$0.084 |  $\pm$0.03 |  $\pm$0.113 |  $\pm$0.062
Avg | 1.00 | 0.85 | 0.98 | 0.98 | 0.88 | 0.88 | 0.98
Table 10: Training performance benchmark on easybg with random convolution.  Easybg | PPO | Oracle | DrAC | Rand-FM | RAD | InDA | ExDA
---|---|---|---|---|---|---|---
Climber | 1.97 | 9.78 | 7.13 | 2 | 2.34 | 7.36 | 8.11
$\pm$0.51 |  $\pm$0.306 |  $\pm$0.419 |  $\pm$0.59 |  $\pm$1.258 |  $\pm$0.273 |  $\pm$0.457
Coinrun | 5.48 | 7.11 | 7.54 | 5.65 | 5.48 | 7.14 | 7.81
$\pm$0.583 |  $\pm$0.205 |  $\pm$0.188 |  $\pm$0.216 |  $\pm$0.542 |  $\pm$0.479 |  $\pm$0.388
Fruitbot | 10.83 | 29.74 | 19.77 | 15.19 | 11.61 | 21.93 | 23.57
$\pm$1.908 |  $\pm$0.443 |  $\pm$0.77 |  $\pm$3.363 |  $\pm$4.615 |  $\pm$0.664 |  $\pm$0.745
Heist | 5.18 | 7.21 | 5.47 | 5.03 | 4.78 | 4.96 | 8.15
$\pm$0.838 |  $\pm$0.27 |  $\pm$0.326 |  $\pm$0.6 |  $\pm$0.785 |  $\pm$0.777 |  $\pm$0.633
Jumper | 3.38 | 8.72 | 8.14 | 4.12 | 3.77 | 8.16 | 7.87
$\pm$0.368 |  $\pm$0.119 |  $\pm$0.17 |  $\pm$0.514 |  $\pm$0.435 |  $\pm$0.231 |  $\pm$0.485
Maze | 6.48 | 8.56 | 6.4 | 6.6 | 6.29 | 8.41 | 8.92
$\pm$0.523 |  $\pm$0.665 |  $\pm$0.419 |  $\pm$0.494 |  $\pm$0.466 |  $\pm$0.436 |  $\pm$0.155
Ninja | 3.83 | 7.81 | 6.8 | 3.36 | 3.98 | 6.61 | 6.85
$\pm$0.462 |  $\pm$0.422 |  $\pm$0.243 |  $\pm$0.505 |  $\pm$0.44 |  $\pm$0.327 |  $\pm$0.25
Avg | 1.00 | 2.33 | 1.86 | 1.08 | 1.04 | 1.92 | 2.11
Table 11: Test performance benchmark on unseen backgrounds (easybg, random
convolution).

###  K.2 Crop

Easybg | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Bigfish | 14.08 | 15.92 | 5.05 | 19.35 | 11.07
$\pm$2.229 |  $\pm$1.535 |  $\pm$3.718 |  $\pm$2.792 |  $\pm$3.683
Chaser | 5.63 | 3.97 | 1.24 | 6.52 | 4.81
$\pm$0.467 |  $\pm$0.642 |  $\pm$0.253 |  $\pm$0.825 |  $\pm$0.325
Dodgeball | 7.71 | 10.74 | 1.23 | 12.74 | 6.74
$\pm$0.678 |  $\pm$0.711 |  $\pm$0.944 |  $\pm$1.729 |  $\pm$0.815
Heist | 9 | 7.58 | 4.53 | 8.15 | 8.79
$\pm$0.513 |  $\pm$0.11 |  $\pm$0.266 |  $\pm$0.57 |  $\pm$0.424
Maze | 9.75 | 9.03 | 3.95 | 9.63 | 9.72
$\pm$0.033 |  $\pm$0.348 |  $\pm$3.418 |  $\pm$0.143 |  $\pm$0.026
Plunder | 7.18 | 10.73 | 0 | 10.29 | 6.59
$\pm$0.73 |  $\pm$1 |  $\pm$0 |  $\pm$0.285 |  $\pm$1.108
Avg | 1.00 | 1.08 | 0.28 | 1.25 | 0.91
Table 12: Training performance benchmark on easybg with crop. Easybg | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Bigfish | 7.43 | 13.63 | 4.93 | 15.19 | 6.35
$\pm$1.65 |  $\pm$1.504 |  $\pm$3.696 |  $\pm$2.724 |  $\pm$2.466
Chaser | 4.83 | 3.59 | 1.2 | 5.86 | 4.48
$\pm$0.56 |  $\pm$0.519 |  $\pm$0.259 |  $\pm$0.745 |  $\pm$0.379
Dodgeball | 3.78 | 9.26 | 1.11 | 11.92 | 3.79
$\pm$0.659 |  $\pm$0.685 |  $\pm$0.831 |  $\pm$1.556 |  $\pm$0.748
Heist | 4.13 | 5.4 | 3.81 | 5.91 | 5.35
$\pm$0.146 |  $\pm$0.448 |  $\pm$0.412 |  $\pm$0.516 |  $\pm$0.22
Maze | 6.79 | 7.77 | 3.9 | 8.01 | 7.74
$\pm$0.158 |  $\pm$0.328 |  $\pm$3.377 |  $\pm$0.288 |  $\pm$0.054
Plunder | 5.94 | 9.49 | 0 | 8.98 | 5.98
$\pm$0.698 |  $\pm$0.605 |  $\pm$0 |  $\pm$0.369 |  $\pm$0.944
Avg | 1.00 | 1.519 | 0.459 | 1.798 | 1.094
Table 13: Test performance benchmark on unseen levels (easybg, crop).

###  K.3 Color jitter

Easy | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Climber | 8.5 | 9.33 | 8.64 | 9.43 | 8.18
$\pm$0.575 |  $\pm$0.212 |  $\pm$0.156 |  $\pm$0.21 |  $\pm$0.45
Jumper | 8.54 | 8.64 | 8.63 | 8.92 | 8.44
$\pm$0.22 |  $\pm$0.135 |  $\pm$0.17 |  $\pm$0.174 |  $\pm$0.185
Ninja | 7.48 | 8.69 | 8.24 | 9.23 | 7.37
$\pm$0.324 |  $\pm$0.331 |  $\pm$0.251 |  $\pm$0.081 |  $\pm$0.212
Avg | 1.00 | 1.09 | 1.04 | 1.13 | 0.98
Table 14: Training performance benchmark on easy with color jitter.  Easy | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Climber | 6.92 | 8.53 | 8.37 | 8.66 | 8.14
$\pm$0.761 |  $\pm$0.422 |  $\pm$0.023 |  $\pm$0.24 |  $\pm$0.477
Jumper | 6.89 | 7.58 | 7.86 | 7.97 | 7.25
$\pm$0.223 |  $\pm$0.053 |  $\pm$0.297 |  $\pm$0.292 |  $\pm$0.131
Ninja | 6.39 | 6.79 | 7.31 | 7.57 | 6.2
$\pm$0.585 |  $\pm$0.32 |  $\pm$0.613 |  $\pm$0.555 |  $\pm$0.085
Avg | 1.00 | 1.13 | 1.16 | 1.2 | 1.07
Table 15: Test performance benchmark on unseen backgrounds (easy, color jitter).  Easybg | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 12.31 | 9.85 | 11.84 | 12 | 11.94 | 12.04
$\pm$0.092 |  $\pm$0.298 |  $\pm$0.223 |  $\pm$0.256 |  $\pm$0.071 |  $\pm$0.152
Coinrun | 9.61 | 7.2 | 8.94 | 8.62 | 9.74 | 9.45
$\pm$0.074 |  $\pm$0.195 |  $\pm$0.285 |  $\pm$0.091 |  $\pm$0.05 |  $\pm$0.09
Fruitbot | 30.2 | 29.69 | 30.05 | 29.48 | 26.87 | 29
$\pm$0.691 |  $\pm$0.619 |  $\pm$0.611 |  $\pm$0.507 |  $\pm$0.912 |  $\pm$0.878
Heist | 8.82 | 7.33 | 7.22 | 6.89 | 7.63 | 8.53
$\pm$0.523 |  $\pm$0.308 |  $\pm$0.76 |  $\pm$0.348 |  $\pm$0.338 |  $\pm$0.307
Jumper | 8.97 | 8.67 | 8.9 | 8.94 | 9.03 | 9.03
$\pm$0.075 |  $\pm$0.132 |  $\pm$0.05 |  $\pm$0.029 |  $\pm$0.123 |  $\pm$0.086
Maze | 9.75 | 8.08 | 9.46 | 9.46 | 9.3 | 9.67
$\pm$0.035 |  $\pm$0.101 |  $\pm$0.404 |  $\pm$0.184 |  $\pm$0.379 |  $\pm$0.111
Ninja | 9.74 | 7.56 | 9.52 | 9.65 | 9.75 | 9.54
$\pm$0.087 |  $\pm$0.286 |  $\pm$0.393 |  $\pm$0.112 |  $\pm$0.046 |  $\pm$0.171
Avg | 1.00 | 0.85 | 0.95 | 0.94 | 0.96 | 0.98
Table 16: Training performance benchmark on easbg with color jitter.  Easybg | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 1.82 | 9.85 | 5.05 | 4.31 | 4.25 | 4.34
$\pm$0.605 |  $\pm$0.298 |  $\pm$0.407 |  $\pm$0.492 |  $\pm$0.338 |  $\pm$0.856
Coinrun | 5.42 | 7.2 | 6.46 | 6.47 | 7.13 | 6.53
$\pm$0.744 |  $\pm$0.195 | 0$\pm$.526 |  $\pm$0.194 |  $\pm$0.372 |  $\pm$0.375
Fruitbot | 11.78 | 29.69 | 9.49 | 8.51 | 10.88 | 18
$\pm$1.949 |  $\pm$0.619 |  $\pm$8.098 |  $\pm$1.941 |  $\pm$2.263 |  $\pm$7.442
Heist | 4.79 | 7.33 | 5.65 | 5.39 | 5.66 | 5.43
$\pm$0.323 |  $\pm$0.308 |  $\pm$0.984 |  $\pm$0.745 |  $\pm$0.271 |  $\pm$0.508
Jumper | 3.3 | 8.6 | 5.65 | 5.67 | 5.81 | 5.31
$\pm$0.467 |  $\pm$0.132 |  $\pm$0.09 |  $\pm$0.953 |  $\pm$0.369 |  $\pm$0.351
Maze | 6.52 | 8.08 | 8.22 | 8.26 | 8.35 | 8.65
$\pm$0.304 |  $\pm$0.101 |  $\pm$0.455 |  $\pm$0.175 |  $\pm$0.238 |  $\pm$0.017
Ninja | 3.56 | 7.56 | 4.22 | 4.18 | 4.34 | 4.07
$\pm$0.363 |  $\pm$0.286 |  $\pm$0.487 |  $\pm$0.475 |  $\pm$0.345 |  $\pm$0.332
Avg | 1.00 | 2.4 | 1.44 | 1.37 | 1.43 | 1.48
Table 17: Test performance benchmark on unseen backgrounds (easybg, color
jitter).

###  K.4 Gray

Easybg | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 12.31 | 9.85 | 11.12 | 11.84 | 11.9 | 12.06
$\pm$0.092 |  $\pm$0.298 |  $\pm$0.26 |  $\pm$0.505 |  $\pm$0.115 |  $\pm$0.03
Coinrun | 9.61 | 7.2 | 9.53 | 9.49 | 9.74 | 9.48
$\pm$0.074 |  $\pm$0.195 |  $\pm$0.135 |  $\pm$0.188 |  $\pm$0.046 |  $\pm$0.08
Fruitbot | 30.2 | 29.69 | 30.01 | 29.6 | 28.03 | 29.32
$\pm$0.691 |  $\pm$0.619 |  $\pm$0.572 |  $\pm$0.27 |  $\pm$0.994 |  $\pm$0.937
Heist | 8.82 | 7.33 | 6.24 | 6.53 | 5.51 | 8.51
$\pm$0.523 |  $\pm$0.308 |  $\pm$0.214 |  $\pm$0.474 |  $\pm$0.146 |  $\pm$0.225
Jumper | 8.54 | 8.67 | 8.91 | 8.93 | 9.18 | 8.95
$\pm$0.22 |  $\pm$0.132 |  $\pm$0.19 |  $\pm$.247 |  $\pm$0.18 |  $\pm$0.075
Maze | 9.75 | 8.08 | 9.46 | 9.48 | 9.2 | 9.75
$\pm$0.035 |  $\pm$0.101 |  $\pm$0.192 |  $\pm$0.08 |  $\pm$0.367 |  $\pm$0.087
Ninja | 9.74 | 7.56 | 9.73 | 9.61 | 9.6 | 9.72
$\pm$0.087 |  $\pm$0.286 |  $\pm$0.045 |  $\pm$0.096 |  $\pm$0.081 |  $\pm$0.021
Avg | 1 | 0.85 | 0.93 | 0.94 | 0.95 | 0.99
Table 18: Training performance benchmark on easybg with gray.  Easybg | PPO | Oracle | RAD | DrAC | InDA | ExDA
---|---|---|---|---|---|---
Climber | 1.82 | 9.85 | 1.75 | 1.81 | 1.24 | 2.45
$\pm$0.605 |  $\pm$0.298 |  $\pm$0.654 |  $\pm$0.211 |  $\pm$0.502 |  $\pm$0.727
Coinrun | 5.42 | 7.2 | 5.34 | 5.31 | 6.05 | 5.79
$\pm$0.744 |  $\pm$0.195 |  $\pm$0.751 |  $\pm$0.501 |  $\pm$0.465 |  $\pm$0.061
Fruitbot | 11.78 | 29.69 | 17.57 | 15.47 | 15.12 | 15.81
$\pm$1.949 |  $\pm$0.619 |  $\pm$0.191 |  $\pm$1.449 |  $\pm$0.958 |  $\pm$0.11
Heist | 4.79 | 7.33 | 5.43 | 5.15 | 4.32 | 5.1
$\pm$0.323 |  $\pm$0.308 |  $\pm$0.18 |  $\pm$0.172 |  $\pm$0.112 |  $\pm$0.504
Jumper | 6.89 | 8.67 | 2.7 | 4.07 | 3.55 | 4.47
$\pm$0.223 |  $\pm$0.132 |  $\pm$0.894 |  $\pm$0.46 |  $\pm$0.992 |  $\pm$0.415
Maze | 6.52 | 8.08 | 7.77 | 7.93 | 7.67 | 8.33
$\pm$0.304 |  $\pm$0.101 |  $\pm$0.611 |  $\pm$0.104 |  $\pm$0.312 |  $\pm$0.119
Ninja | 3.56 | 7.56 | 3.72 | 3.91 | 4.02 | 4.03
$\pm$0.363 |  $\pm$0.286 |  $\pm$0.131 |  $\pm$0.62 |  $\pm$0.666 |  $\pm$0.071
Avg | 1 | 2.2 | 1.03 | 1.04 | 0.97 | 1.13
Table 19: Test performance benchmark on unseen backgrounds (easybg, gray).  Easy | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Climber | 8.5 | 6.95 | 7.55 | 7.22 | 8.05
$\pm$0.575 |  $\pm$0.547 |  $\pm$0.256 |  $\pm$0.312 |  $\pm$0.461
Jumper | 8.54 | 8.4 | 8.58 | 8.85 | 8.5
$\pm$0.22 |  $\pm$0.224 |  $\pm$0.199 |  $\pm$0.015 |  $\pm$0.224
Ninja | 7.48 | 6.67 | 7.1 | 8.91 | 7.05
$\pm$0.324 |  $\pm$0.435 |  $\pm$0.718 |  $\pm$0.165 |  $\pm$0.24
Avg | 1 | 0.9 | 0.95 | 1.026 | 0.96
Table 20: Training performance benchmark on easybg with gray.  Easy | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Climber | 6.92 | 4.49 | 5.57 | 5.11 | 7.24
$\pm$0.761 |  $\pm$0.332 |  $\pm$0.307 |  $\pm$0.483 |  $\pm$0.721
Jumper | 6.89 | 5.38 | 6.59 | 6.35 | 6.87
$\pm$0.223 |  $\pm$0.215 |  $\pm$0.055 |  $\pm$0.234 |  $\pm$0.182
Ninja | 6.39 | 5.67 | 5.14 | 6.84 | 6.01
$\pm$0.585 |  $\pm$0.318 |  $\pm$0.628 |  $\pm$0.206 |  $\pm$0.651
Avg | 1 | 0.77 | 0.86 | 0.91 | 0.99
Table 21: Test performance benchmark on unseen backgrounds (easy, gray).

###  K.5 Cutout color

Easybg | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 12.31 | 9.85 | 11.92 | 8.26 | 11.76 | 12.07
$\pm$0.092 |  $\pm$0.298 |  $\pm$0.158 |  $\pm$0.663 |  $\pm$0.027 |  $\pm$0.127
Coinrun | 9.61 | 7.2 | 9.23 | 8.07 | 9.7 | 9.39
$\pm$0.074 |  $\pm$0.195 |  $\pm$0.323 |  $\pm$0.645 |  $\pm$0.084 |  $\pm$0.012
Fruitbot | 30.2 | 29.69 | 29.73 | 29.2 | 27.18 | 28.95
$\pm$0.691 |  $\pm$0.619 |  $\pm$0.898 |  $\pm$0.64 |  $\pm$1.302 |  $\pm$0.907
Heist | 8.82 | 7.33 | 8.47 | 6.25 | 6.1 | 8.65
$\pm$0.523 |  $\pm$0.308 |  $\pm$0.397 |  $\pm$0.704 |  $\pm$0.693 |  $\pm$0.21
Jumper | 8.97 | 8.67 | 8.87 | 8.75 | 9.1 | 8.91
$\pm$0.075 |  $\pm$0.132 |  $\pm$0.123 |  $\pm$0.131 |  $\pm$0.081 |  $\pm$0.053
Maze | 9.75 | 8.08 | 9.41 | 9.17 | 9.27 | 9.74
$\pm$0.035 |  $\pm$0.101 |  $\pm$0.134 |  $\pm$0.118 |  $\pm$0.125 |  $\pm$0.133
Ninja | 9.74 | 7.56 | 9.65 | 7.17 | 9.72 | 9.7
$\pm$0.087 |  $\pm$0.286 |  $\pm$0.138 |  $\pm$1.993 |  $\pm$0.02 |  $\pm$0.02
Bigfish | 13.89 | 13.22 | 2.54 | 5.19 | 1.95 | 11.22
$\pm$3.127 |  $\pm$1.488 |  $\pm$0.13 |  $\pm$3.658 |  $\pm$0.311 |  $\pm$3.66
Chaser | 5.49 | 3.04 | 2.88 | 1.98 | 3.34 | 5
$\pm$0.562 |  $\pm$0.183 |  $\pm$0.699 |  $\pm$0.112 |  $\pm$0.755 |  $\pm$0.187
Dodgeball | 7.76 | 5.74 | 5.71 | 5.98 | 2.79 | 6.57
$\pm$0.859 |  $\pm$1.118 |  $\pm$1.008 |  $\pm$0.103 |  $\pm$1.612 |  $\pm$0.693
Plunder | 7.15 | 6.05 | 5.43 | 4.34 | 4.92 | 6.87
$\pm$0.95 |  $\pm$0.58 |  $\pm$0.082 |  $\pm$0.24 |  $\pm$0.625 |  $\pm$1.255
Avg | 1.00 | 0.82 | 0.82 | 0.72 | 0.76 | 0.94
Table 22: Training performance benchmark on easybg with cutout color.  Easy | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 8.5 | 9.85 | 7.69 | 6.67 | 9.02 | 8.02
$\pm$0.575 |  $\pm$0.298 |  $\pm$0.237 |  $\pm$0.381 |  $\pm$0.473 |  $\pm$0.506
Jumper | 7.48 | 7.56 | 6.28 | 5.6 | 8.57 | 7.41
$\pm$0.324 |  $\pm$0.286 |  $\pm$0.257 |  $\pm$0.276 |  $\pm$0.122 |  $\pm$0.125
Ninja | 8.54 | 8.67 | 8.45 | 8.32 | 8.93 | 8.53
$\pm$0.22 |  $\pm$0.132 |  $\pm$0.183 |  $\pm$0.051 |  $\pm$0.166 |  $\pm$0.095
Avg | 1.00 | 1.06 | 0.91 | 0.84 | 1.08 | 0.98
Table 23: Training performance benchmark on easy with cutout color.  Easybg | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 1.82 | 9.85 | 3.54 | 3.97 | 3.4 | 4.29
$\pm$0.605 |  $\pm$0.298 |  $\pm$0.164 |  $\pm$0.999 |  $\pm$0.645 |  $\pm$0.154
Coinrun | 5.42 | 7.2 | 5.87 | 5.93 | 6.2 | 6.41
$\pm$0.744 |  $\pm$0.195 |  $\pm$.251 |  $\pm$0.061 |  $\pm$0.357 |  $\pm$0.131
Fruitbot | 11.78 | 29.69 | 18.18 | 19.24 | 17.69 | 17.7
$\pm$1.949 |  $\pm$0.619 |  $\pm$3.744 |  $\pm$3.385 |  $\pm$4.026 |  $\pm$0.888
Heist | 4.79 | 7.33 | 6.6 | 5.76 | 4.97 | 7.51
$\pm$0.323 |  $\pm$0.308 |  $\pm$.092 |  $\pm$0.551 |  $\pm$0.33 |  $\pm$0.119
Jumper | 3.3 | 8.67 | 4.99 | 5.48 | 5.43 | 6.02
$\pm$0.467 |  $\pm$0.132 |  $\pm$0.114 |  $\pm$0.28 |  $\pm$1.116 |  $\pm$0.235
Maze | 6.52 | 8.08 | 7.33 | 7.66 | 7.01 | 7.83
$\pm$0.304 |  $\pm$0.101 |  $\pm$0.223 |  $\pm$.243 |  $\pm$0.17 |  $\pm$0.22
Ninja | 3.56 | 7.56 | 4.29 | 3.96 | 3.75 | 3.76
$\pm$0.363 |  $\pm$0.286 |  $\pm$0.245 |  $\pm$0.152 |  $\pm$0.333 |  $\pm$0.348
Bigfish | 3.4 | 13.22 | 1.29 | 2.5 | 1.29 | 4.49
$\pm$0.487 |  $\pm$1.488 |  $\pm$0.08 |  $\pm$2.331 |  $\pm$0.152 |  $\pm$0.776
Chaser | 0.91 | 3.04 | 1.08 | 1.13 | 1.68 | 1.73
$\pm$0.061 |  $\pm$0.183 |  $\pm$0.038 |  $\pm$0.157 |  $\pm$0.305 |  $\pm$0.698
Dodgeball | 2.17 | 5.74 | 3.92 | 4.02 | 1.97 | 4.37
$\pm$0.53 |  $\pm$1.118 |  $\pm$0.53 |  $\pm$0.345 |  $\pm$1.098 |  $\pm$0.527
Plunder | 6.87 | 6.05 | 5.27 | 4.77 | 4.71 | 6.45
$\pm$0.933 |  $\pm$0.58 |  $\pm$0.208 |  $\pm$0.612 |  $\pm$0.622 |  $\pm$1.232
Avg | 1.00 | 2.51 | 1.27 | 1.33 | 1.19 | 1.53
Table 24: Test performance benchmark on unseen backgrounds (easybg, cutout color).  Easy | PPO | Oracle | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---|---
Climber | 6.92 | 9.85 | 6.54 | 5.24 | 7.61 | 7.25
$\pm$0.761 |  $\pm$0.298 |  $\pm$0.213 |  $\pm$0.417 |  $\pm$0.486 |  $\pm$0.325
Jumper | 6.39 | 7.56 | 5.06 | 4.9 | 6.71 | 5.78
$\pm$0.585 |  $\pm$0.286 |  $\pm$0.137 |  $\pm$0.382 |  $\pm$0.352 |  $\pm$0.488
Ninja | 6.89 | 8.67 | 6.88 | 6.79 | 6.81 | 6.92
$\pm$0.223 |  $\pm$0.132 |  $\pm$0.083 |  $\pm$0.278 |  $\pm$0.355 |  $\pm$0.212
Avg | 1.00 | 1.29 | 0.91 | 0.84 | 1.05 | 0.99
Table 25: Test performance benchmark on unseen backgrounds (easy, cutout color).  Easybg | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Climber | 11.14 | 10.77 | 7.26 | 9.45 | 10.75
$\pm$0.077 |  $\pm$0.279 |  $\pm$0.843 |  $\pm$0.193 |  $\pm$0.114
Coinrun | 8.64 | 8.36 | 6.89 | 7.76 | 8.32
$\pm$0.05 |  $\pm$0.348 |  $\pm$0.503 |  $\pm$0.096 |  $\pm$0.224
Fruitbot | 28.26 | 26.88 | 26.22 | 23.79 | 26.33
$\pm$0.461 |  $\pm$1.276 |  $\pm$1.258 |  $\pm$0.971 |  $\pm$0.894
Heist | 4.07 | 3.92 | 2.27 | 2.15 | 3.93
$\pm$0.07 |  $\pm$0.276 |  $\pm$0.448 |  $\pm$0.553 |  $\pm$0.184
Jumper | 7.38 | 7.32 | 6.98 | 6.68 | 7.25
$\pm$0.15 |  $\pm$0.195 |  $\pm$0.199 |  $\pm$0.24 |  $\pm$0.117
Maze | 6.8 | 6.84 | 6.04 | 5.91 | 6.17
$\pm$0.2 |  $\pm$0.137 |  $\pm$0.258 |  $\pm$0.03 |  $\pm$0.162
Ninja | 8.56 | 8.63 | 6.28 | 7.81 | 8.34
$\pm$0.061 |  $\pm$0.132 |  $\pm$1.866 |  $\pm$0.21 |  $\pm$0.119
Bigfish | 7.16 | 0.91 | 2.29 | 0.95 | 6.04
$\pm$2.263 |  $\pm$0.037 |  $\pm$2.306 |  $\pm$0.06 |  $\pm$2.783
Chaser | 4.54 | 2.61 | 1.8 | 2.47 | 4.22
$\pm$0.503 |  $\pm$0.509 |  $\pm$0.12 |  $\pm$0.506 |  $\pm$0.331
Dodgeball | 3.78 | 2.71 | 2.53 | 1.26 | 2.82
$\pm$0.823 |  $\pm$0.362 |  $\pm$0.135 |  $\pm$0.48 |  $\pm$0.593
Plunder | 5.99 | 5.08 | 4.07 | 4.55 | 5.83
$\pm$0.814 |  $\pm$0.305 |  $\pm$0.479 |  $\pm$0.393 |  $\pm$1.061
Avg | 1.00 | 0.83 | 0.69 | 0.69 | 0.93
Table 26: Test performance benchmark on unseen levels (easybg, cutout color). Easy | PPO | DrAC | RAD | InDA | ExDA
---|---|---|---|---|---
Climber | 5.45 | 5.9 | 5.3 | 4.26 | 5.71
$\pm$0.77 |  $\pm$0.352 |  $\pm$0.307 |  $\pm$0.122 |  $\pm$0.303
Jumper | 5.81 | 6.01 | 4.93 | 4.56 | 5.43
$\pm$0.227 |  $\pm$0.389 |  $\pm$0.08 |  $\pm$0.161 |  $\pm$0.333
Ninja | 5.77 | 5.67 | 5.8 | 5.65 | 5.87
$\pm$0.09 |  $\pm$0.023 |  $\pm$0.071 |  $\pm$0.166 |  $\pm$0.079
Avg | 1.00 | 1.03 | 0.94 | 0.85 | 1
Table 27: Test performance benchmark on unseen levels (easy, cutout color).

##  Appendix L Primitive evaluation on DeepMind Control Suite with SAC

We experiment on DeepMind Control Suite (DMC) with a preliminary experiment
which provides justification for our proposed method ExDA due to the limited
time and computation resources. We first note that [26] has demonstrated that
UCB-DrAC, which is similar to UCB-InDA, can accelerate RL training in DMC.
Hence, in order to justify our proposed method UCB-ExDA, which combines UCB-
InDA and ExDA, it would be sufficient to show the existence of the cases
showing a benefit of ExDA in DMC. Furthermore, we use SAC as the base RL
algorithm instead of PPO to show the versatility of our method ExDA. SAC is
off-policy actor-critic RL algorithm which trains actor and critic networks
separately with a replay buffer. Thus, we can use stored data in the replay
buffer after RL training, and also we can only distill the output of the
actor-network, because the actor-network determines the policy $\pi$ from
observations, while the critic-network computes the value function $V$ from
observations. We denote the actor-network parameters using $\theta$ same as
the notation of PPO in Section 4. We ablate the effect of ExDA to compare w/
and w/o inconsistency loss of policy on original observations. Thus ExDA is
trained with Eq 5 after SAC, but ablated ExDA is trained with Eq 11.

$\displaystyle
L_{abl-\text{ExDA}{}}(\theta,\phi;\theta_{\text{old}}):=L_{\text{dis}}(\theta,\phi;\theta_{\text{old}})\;.$
(11)

Both methods are trained 10 epochs after 10K time-steps SAC training and
detailed hyper-parameters are described below the table. We compare the mean
reward of each method SAC, ExDA, abl-ExDA on three environments Walker-Walk,
Finger-Spin, Cartpole-Swingup and 5 seeds. We evaluate each reward from 50
evaluation episodes in train environments. As below Figure 24, ExDA almost
maintains the mean reward of SAC, abl-ExDA degrades the performance of SAC by
distilling policy only to augmented observations except the original.

Figure 24: Ablation of ExDA on DMcontrol

Hyperparameter | Value
---|---
# of action repeat | 8
# of frame stack | 3
Data augmentation | Random Convolution
Learning rate of DA | $10^{-2}$
Batch size | 128
# of train steps | 100000
# of distill epochs | 10
Replay buffer capacity | 100000
Init steps | 1000
Learning rate of critic | $10^{-3}$
$\beta$ of critic | 0.9
$\tau$ of critic | 0.01
Target update frequency of critic | 2
Learning rate of actor | $10^{-3}$
$\beta$ of actor | 0.9
Log std min of actor | -10
Log std max of actor | 2
Update frequency of actor | 2
Encoder type | pixel
Feature dimension of encoder | 50
Learning rate of encoder | $10^{-3}$
$\tau$ of encoder | 0.05
# of layers | 4
# of filters | 32
Latent dimension | 128
Discount factor | 0.99
Learning rate of $\alpha$ | $10^{-4}$
$\beta$ of $\alpha$ | 0.5