# Investigating the Properties of Neural Network Representations in
Reinforcement Learning

Han Wang, Erfan Miahi, Martha White, Marlos C. Machado,
Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White

###### Abstract

In this paper we investigate the properties of representations learned by deep
reinforcement learning systems. Much of the early work on representations for
reinforcement learning focused on designing fixed-basis architectures to
achieve properties thought to be desirable, such as orthogonality and
sparsity. In contrast, the idea behind deep reinforcement learning methods is
that the agent designer should not encode representational properties, but
rather that the data stream should determine the properties of the
representation—good representations emerge under appropriate training schemes.
In this paper we bring these two perspectives together, empirically
investigating the properties of representations that support transfer in
reinforcement learning. We introduce and measure six representational
properties over more than 25 thousand agent-task settings. We consider Deep
Q-learning agents with different auxiliary losses in a pixel-based navigation
environment, with source and transfer tasks corresponding to different goal
locations. We develop a method to better understand _why_ some representations
work better for transfer, through a systematic approach varying task
similarity and measuring and correlating representation properties with
transfer performance. We demonstrate the generality of the methodology by
investigating representations learned by a Rainbow agent that successfully
transfer across games modes in Atari 2600.

###### Index Terms:

Representation Learning, Reinforcement Learning, Neural Networks,
Representation Transfer, Auxiliary Tasks.

##  1 Good Representations for RL

In reinforcement learning an agent interacts with their environment, receiving
observations and taking actions based on those observations, with the goal of
maximizing the sum of a special numerical signal, the reward. The function
that converts these observations is known as _representation_ , its elements
are known as _features_ , and the process of learning such function is known
as _representation learning_. Ultimately, many other subproblems depend on the
agent’s representation. Bad representations hinder predictions and diminish
the effectiveness of planning and learning algorithms [1, 2, 3, 4]. Good
representations can lead to better sample efficiency [5, 6]. Therefore, the
key question motivating this research is: _what are good representations and
how can the agent find them?_

Good representations are classically defined in service to other tasks. Often,
good representations are said to be those that improve agents in some
dimension, such as learning efficiency [7, 8], performance in unseen tasks [9,
10, 11, 12, 13, 14, 15, 16], the accuracy of learned models for planning [17,
18, 19, 20, 21, 22], and in the agent’s ability to represent the world the way
humans do [23, 24]. In this context, there are two main approaches for
obtaining good representations: using a fixed, expert-designed transformations
of the agent’s observations, or learning such transformations from data.

Fixed transformations of the agent’s observations, which lead to fixed-basis
architectures, have been extensively explored in reinforcement learning. They
allow us to enforce specific properties that are thought to be beneficial. For
example, many approaches either use or search for orthogonal or decorrelated
features, such as orthogonal matching pursuit [25], Bellman-error basis
functions [26], Fourier basis [27], tile coding [28], and proto-value
functions [29, 30]. Prototypical input matching methods have been explored, as
in kernel methods, radial basis functions [31], cascade correlation networks
[32], and Kanerva coding [33]. Some of these approaches produce high-
dimensional, sparse representations, which are more likely to be orthogonal.
Moreover, by activating only a small subset of features, a sparse
representation reduces computation and increases scalability, such as in tile
coding [28] and in sparse distributed memories [34]. However, fixed-basis
architectures are not adaptive and are difficult to scale to high-dimensional
inputs.

Recent developments in representation learning for reinforcement learning
explore a different perspective: we should avoid optimizing specific
properties111There is, of course, work in reinforcement learning exploring how
to encode specific properties on the network (e.g., sparse activations [35,
36], disentangled features [12], and orthogonality constraints [37, 21]). and
instead use gradient descent to let the training data dictate the properties
of the representation. This is achieved with specific training regimes,
including multi-task (parallel) training [38, 39, 40], auxiliary losses [8,
41], and training on a distribution of problems (à la meta-learning) [10, 17,
42, 43, 44]. The underlying idea is that good representations will emerge if
the problem setting is complex enough, where goodness is often measured by
success on some held-out test task.

There are many different ways to evaluate and understand these emergent
representations. Recent work has explored this question in roughly two ways:
what do good representations look like, and what capabilities do good and bad
representations allow. The most common approach is to visualize the learned
representations [45, 7, 18, 46, 47, 23, 48, 49, 20, 19, 50, 21, 41]. This
approach has been used, for example, to provide evidence for the emergence of
abstraction and compositionality in supervised learning [51, 52]. However, in
reinforcement learning, the impact of delayed consequences and temporally
correlated data make it difficult to import these analysis techniques from
other fields, and recent work highlighted how popular approaches like saliency
maps may not always be appropriate [53].

This paper explores the properties of representations learned by deep
reinforcement learning systems: specifically (1) DQN [45]—Q-learning with
neural network function approximation—combined with different auxiliary tasks,
and (2) Rainbow [54]. We investigate properties grouped in three categories:
_capacity_ (complexity reduction, dynamics awareness, and diversity),
_redundancy_ (orthogonality and sparsity), and _robustness_ (non-
interference). Our property set consists of both a subset of properties
discussed in the literature, and properties newly introduced in this paper. We
focus on the fully observable setting; our goal is to understand how the
representation transforms the current input. Other properties and experiments
would be suitable to understand (recurrent) representations that summarize
histories for the partially observable setting, but this is left to future
work.

We focus on a representation transfer setting: a training phase to learn a
representation in one source task, followed by testing in a related transfer
task where the agent learns using that fixed, pre-learned representation. Our
primary goal is to identify representations that are useful for future
learning. Ideally, it should be useful for future learning in the same task.
Additionally, the pre-learned representation should also be useful later when
learning about other related tasks. Such a representation should enable faster
learning, compared with re-learning everything from scratch, even if we
prevent the agent from changing the representation after pre-training—perhaps
the clearest and sternest evaluation of the usefulness of prior learning. If
fine-tuning or transfer of both the representation and the value function are
required to outperform learning from scratch, then one might fairly wonder if
any representation is learned at all.

Representation transfer, as defined above, relies on a set of related tasks to
evaluate future learning. We start with a simple image-based maze environment,
that is computationally efficient and so allows exhaustive experiments, as
well as provides a natural notion of task similarity (goal location). We then
test across different game modes in Atari 2600 games [55]. Recent work has
demonstrated that transfer between games modes is possible with a variant of
the Rainbow agent, [56], however transfer is clearly nontrivial as prior
investigations with DQN reported failure [57].

Our first study in the image-based maze environment uses nine auxiliary tasks,
resulting in 150 representations for 173 target tasks. We investigated two
activations: the widely used ReLU which produces a relatively compact, dense
representation, and a new activation function called FTA [36] that produces
high-dimensional, sparse representations. The key insights are as follows.

  1. 1.

Auxiliary tasks can facilitate emergence of representations effective for
transfer, however with ReLU networks many auxiliary tasks do not outperform
learning from scratch (do not transfer).

  2. 2.

Using sparse activations (FTA) was a significant factor in improving transfer.
The FTA-based representations transferred consistently, with or without
auxiliary tasks.

  3. 3.

ReLU-based representations transfered well to very similar tasks (better than
FTA), but significantly worse than FTA to less similar tasks in our setup.

  4. 4.

Transfer was not possible with linear value functions: performance was
significantly better when the representation was inputted into a nonlinear
value function.

  5. 5.

The representations that transferred best had high levels of complexity
reduction, medium-high levels of dynamics awareness and diversity, and medium
levels of orthogonality and sparsity.

A key contribution of this work is providing a systematic approach to
investigate representations and their properties. The empirical design took
many iterations, including 1) developing the transfer setup and a novel way of
ranking task similarity using successor features so that we could
systematically vary the level of difficulty in transfer, 2) developing the set
of properties to measure, 3) appropriately sweeping hyperparameters to obtain
reasonably performing agents but still avoiding over-tuning, and 4) providing
several mechanisms to aggregate and visualize the mountain of data produced
across representations. For example, initial results had little consistency
because the agents themselves were not effectively trained; it turns out
analyzing poorly performing agents results in unclear conclusions.

Using these insights, we applied our methodology to understand representation
transfer across different Atari 2600 game modes [55, 58]. We trained a Rainbow
agent on the default game mode, and showed that the learned representation
facilitated transfer to other modes in three different games. We found similar
outcomes in terms of the properties and transfer performance, which is both
somewhat surprising and an indicator that the properties and methodology we
propose here are meaningful. More specifically, we similarly found that the
Rainbow representation had

  1. 1.

high complexity reduction—increasing this level significantly from its random
initialization

  2. 2.

high orthogonality and sparsity, even more so than the representations in the
maze environment

  3. 3.

medium diversity

The results in this work complement the growing literature on representation
transfer in reinforcement learning by providing a quantitative approach to
understand the learned representations. We start the paper by more explicitly
defining representations (Section 2) and what it means to be a good
representation (Section 3). We then explain the auxiliary losses (Section 4)
that result in the many different representations we analyze in the results.
We then explain the experimental setup (Section 5) and provide our first key
result showing transfer performance, both successful and failed, for these
many representations (Section 6). We then introduce the properties (Section 7)
and then analyze these properties (Section 8) showing correlations to transfer
performance (using 150 representations), the evolution of property values over
time and finally a contrast between final property values for three
representations that provided the bulleted conclusions above. We conclude with
results in Atari (Section 9).

##  2 Problem Formulation and Notation

We formalize the agent’s interaction as a finite Markov Decision Process (MDP)
with a finite state space $\mathcal{S}$, finite action space $\mathcal{A}$,
transition function
$P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$, and bounded
reward function
$R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$. On
each time step, $t=1,2,...$, the agent takes action $A_{t}$ in state $S_{t}$
and the environment transitions to state $S_{t+1}\sim P(\cdot|S_{t},A_{t})$
and emits a reward $R_{t+1}$. The agent’s objective is to find a policy,
$\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ that maximizes the expected
discounted sum of future rewards, the _return_ , $G_{t}\doteq
R_{t+1}+\gamma_{t+1}G_{t+1}$, where $\gamma_{t+1}\in[0,1]$ denotes a discount
that depends on the transition $(S_{t},A_{t},S_{t+1})$ [59]. In episodic
problems, which we study here, the discount might be 1 during the episode, and
it becomes zero when $S_{t},A_{t}$ lead to termination.

For much of this work we study the representations learned by DQN [45], a
widely used value-based method in deep reinforcement learning. (In Section 9,
we study Rainbow [54], and leave the description to then.) The approximate
value function is parameterized by a set of weights $\theta$:
$Q_{\theta}(s,a)\approx q_{\pi}(s,a)$. The word _deep_ , in deep reinforcement
learning, stems from the use of neural networks to approximate $q_{\pi}$. In
particular, DQN iteratively updates its action-value estimates by training the
parameters of a neural network, $\theta$, with stochastic gradient descent:
$\Delta\theta\propto\big{(}R_{t+1}+\gamma_{t+1}\max\bar{Q}(S_{t+1},a)-Q_{\theta}(S_{t},A_{t})\big{)}\nabla_{\theta}Q_{\theta}(S_{t},A_{t})$.
The target network, $\bar{Q}$, is not updated on every step, but only
periodically set equal to the current $Q_{\theta}$. Actions are selected
according to an $\epsilon$-greedy policy, where
$A_{t}\doteq\arg\max_{a\in\mathcal{A}}Q_{\theta}(S_{t},a)$ with probability
$1-\epsilon$ or a random action with probability $\epsilon$. As is typical, we
use mini-batch updates from an experience replay buffer [60].

Figure 1: We experiment with agents using this network architecture, with
different auxiliary losses. The representation network, $\Phi_{\theta_{R}}$,
learns a mapping from input-state $s_{t}$ to the agent-state (representation
of $s_{t}$). The representation network is learned to improve two objectives:
performance on a main task and on an auxiliary task. The diagram depicts the
auxiliary tasks we use in this work, described in Section 4. Our agents only
use one auxiliary task at a time.

We use image inputs, a convolutional network and a fully connected layer with
a lower dimension $d$. We consider two activations on this fully connected
layer: rectified linear units (ReLU) [61] and the fuzzy tiling activation
(FTA) [36]. FTA is a one-to-many activation that leads to a larger number of
units in the representation layer: $k\times d$ instead of $d$, but with only a
small number of active features at once. This allows us to investigate more
compact, lower-dimensional representations produced by the ReLU and higher-
dimensional, sparse representations produced by FTA. We call these
representations the _representation layer_.

We make extensive use of auxiliary tasks [8] to both induce better
representations and to study them. Auxiliary tasks are additional prediction
tasks given to agents to incentivize the network to learn about properties of
the environment which are, in principle, not directly related to reward
maximization. Examples include predicting pixel changes [8] and the next state
given an action [62]. These tasks are posed as additional loss functions and
the agent is tasked to balance between them and return maximization.

We use a unified architecture to explore representations induced by a variety
of different auxiliary tasks, as shown in Figure 1. The first layers,
parameterized by $\theta_{R}$, produce the representation
$\phi_{t}=\Phi_{\theta_{R}}(s_{t})$. The last layers, which are parameterized
with $\theta_{V}$, use the representation to estimate the action-values.
Auxiliary tasks are encoded with additional layers and separate heads (with
parameters $\theta_{A}$), further impacting the updates to $\theta_{R}$ via
gradient descent: $\Phi_{\theta_{R}}(s)$ must be adjusted to be useful for
both estimating action-values and reducing the auxiliary losses.

Given this setup, a natural (and basic) question is: _what is the
representation in a deep RL agent?_ The answer relies on the role of the
representation, which is primarily to promote _future learning_. We learn a
transformation on the inputs—features—to facilitate downstream learning. The
features should 1) be _reusable_ or generally useful for multiple predictions,
2) improve _sample efficiency_ for the given online algorithm (e.g., SGD), and
3) be _computationally efficient_.222Another criteria that has been considered
for partially observable settings, especially where the true state is low-
dimensional and compact, is how well the representation captures the true
state, with related ideas about finding a compact set of disentangled features
or causal features; see [63] for a nice overview. Such representations could
facilitate all three of these goals—reusability, sample efficiency and
computational efficiency—but may not be necessary to achieve them.

For example, an agent may want to pickup objects in a room, with new objects
continually added. The features could describe the objects, so that new
objects _reuse_ previously learned concepts (e.g., red, cup) that are a
succinct (efficient) description of the object. If the agent uses online
updating, from temporally correlated samples, we may prefer sparsely activated
features that only change a subset of the weights, to reduce interference and
promote _sample efficiency_.

Of course, this is only a hypothetical example; we do not truly know the
semantics of what is learned, nor what features would improve learning. In
this paper, we attempt to systematically measure representation properties and
relationships to transfer performance (future learning), to gain more insight
into the representations of deep RL agents.

##  3 Good representations for Transfer

We aim to understand the properties of representations that emerge in deep
reinforcement learning, but it is critical to do so for both good and bad
representations; which, again, begs the question: _what is a good
representation_? We use a simple definition: a good representation is one that
transfers. More precisely, if features learned during an initial learning
phase allow for faster learning on future data, then those features transfer.
Good representations that reliably achieve good transfer may exhibit
properties and attributes different than representations that result in poor
or negative transfer.

We seek to empirically relate representation properties and performance, which
requires an environment where transfer is possible. We investigate how the
complexity of the value function interacts with transfer in a simple pixel-
based navigation environment with obstacles. This environment, depicted in
Figure 2(a), can be readily used to generate numerous related tasks. The agent
must learn to navigate to a given goal state in as few steps as possible. The
problem is episodic, with $\gamma=0.99$, a reward of $+1$ when reaching the
goal and 0 otherwise. The input state consists of an RGB input of a $15\times
15$ grid (size $15\times 15\times 3$), encoding the agent’s current location
(but not the goal). The actions correspond to the four cardinal directions,
and transitions the agent deterministically by one pixel, or not at all if the
action is into a wall. To simplify exploration, the agent starts in a uniform
random state and episodes are cut-off at $100$ steps; the agent is then
teleported to a new random state and this transition is discarded.

(a) Maze problem

(b) Relative Performance

Figure 2: Representation transfer is possible in the Maze navigation task and
auxiliary tasks improve transfer when using non-linear function approximation.
(a) The position of the  walls (dark grey), the  goal (green) in the training,
and two  transfer tasks (purple) are shown. (b) Performance relative to the
baseline (from scratch) agent: above $1$ represents an improvement and below
$1$ denotes negative transfer. Error bars show a 95% confidence interval.
Performance is reported for the best auxiliary task for each activation:
VirtualVF5 for ReLU-based representations, and SF for FTA, explained in
Section 4. The dotted lines labeled with a $1$ are at different locations, to
indicate the relative performance between FTA and ReLU from scratch. In non-
linear, FTA from scratch was better; in linear, ReLU from scratch was better.

We use different navigation tasks (different goal locations) to define
training and transfer in our environment. The agent is first trained to go to
the goal at a specific location (e.g., $[9,9]$, depicted in Figure 2(a)).
Next, we create a new agent with the parameters of the network up to the
representation layer copied over from the training agent and we freeze them to
prevent further adaption—a transfer agent. In this transfer phase of the
experiment, the new agent is trained to navigate to a nearby but different
goal location. We compare this to a baseline agent where all parameters of the
network are not pre-trained but instead randomly initialized and updated
during the transfer phase. This baseline agent learns the task from scratch
because it does not benefit in any way from prior learning. If the transfer
agent learns more quickly than the baseline scratch agent, then we say the
representation learned in the first phase of the experiment facilitates
transfer. Because of the multiple possible goal locations, there are many
possible training and transfer tasks in this environment.

At this point we have the major ideas in place to present a simple but
foundational result: _representation transfer is possible in our navigation
task and auxiliary tasks improve transfer_. Figure 2(b) summarizes this result
upfront; we give details later about the experiment as we proceed to more
complex and nuanced results. Representations pre-trained on the training task
significantly outperformed representations learned from scratch on the
transfer task. In addition, we found auxiliary tasks were important for
transfer, at least for ReLU. Pre-trained representations using ReLU exhibited
negative transfer, whereas ReLU-based representations combined with well-
designed auxiliary tasks did transfer.

These results, though, produced some surprises. First, even in this simple
environment, transfer with ReLU and auxiliary tasks was difficult. This
reaffirms some of the previous anecdotal and documented [57, 16] issues with
transferring representations. In fact, switching to a different (sparse)
activation, FTA, had a bigger impact than any auxiliary task. This suggest
some issues with the representations learned under ReLU.

Second, _we were unable to obtain successful transfer with linear value
functions_. This outcome was not for a lack of effort. Figure 2(b) strongly
suggests that, in our navigation environment, non-linear value functions
significantly improve transfer, even in a relatively simple environment. We
see that when the value function was linear in the features, neither a pre-
trained representation (labelled ReLU and FTA) nor any other representations
trained with auxiliary tasks improved over training a fresh representation
from scratch. This suggests representations may emerge in earlier layers of
the network, and that it may be more feasible to learn re-usable features when
they can be nonlinearly combined, even if only with a simple shallow network.
We highlight this important point here; for the remainder of this paper, we
restrict our scope only to non-linear value functions.

We presented these overall results upfront, before diving deeper and
understanding _why_ we see these outcomes. In the next sections, we describe
the auxiliary tasks we used to find good representations; then how we evaluate
the representations for different transfer tasks; and finally give insights
into failures in transfer under ReLU due to transfer task similarity. Later,
we dive even deeper, measuring and correlating representation properties to
transfer performance.

##  4 The Auxiliary Losses

We focus on the representations that emerge under different auxiliary losses
for two reasons: 1) Adding or changing auxiliary tasks does not change the
functional capacity and size of the learned representations; the consistency
on the capacity makes comparisons more interpretable. 2) The role and impact
of auxiliary losses in reinforcement learning remains poorly understood, and
constitutes an important area of study. We introduce the general idea behind
each auxiliary task in text, while their formal definition is presented in
Table I in Appendix B.1.

Input Reconstruction (IR): This auxiliary task tries to reconstruct the
network’s input, as in an autoencoder. This extraction is achieved by using a
bottleneck layer: a low-dimensional layer that forces only the most important
information to be retained and the remainder, including the noise, to be
discarded. We include this auxiliary task as a classic and simple choice.

Next Agent State Prediction (NAS): Another common choice is to predict the
next agent state [8, 64, 65, 66, 21, 22, 19, 42, 43]. This loss encourages the
representation to capture the transition dynamics. The agent predicts
$\phi_{t+1}$ using $\phi_{t}$ and action $a_{t}$. Predicting the next agent
state might give vacuous solutions when it is the only training signal;
jointly training with the main task, however, prevents this from happening.
The combination of this auxiliary loss with the main task encourages the
representation to both be useful for action-value estimation, as well as
capable of anticipating features on the next step. Several papers have
highlighted that the ability to predict the next state is related to the
ability to predict action-values [41, 67, 68].

Successor Feature Prediction (SF): NAS can be taken one step further, with the
target including not just the next agent-state but many future agent states
[69]. _Successor features_ (SFs) provide just such a target. SFs are defined
with respect to a particular policy $\pi$ as
$\psi^{\pi}_{t}=\mathbb{E}\big{[}\sum_{i=0}^{\infty}\gamma^{i}\phi_{t+i}\big{]}$.
They have been used in the transfer setting because they can be used to
quickly infer value estimates for new reward functions that are a linear
function $\phi_{t}$ [11]. In the tabular case, SFs correspond to successor
representations [70], which have an equivalence to proto-value functions [66].
We opt to use the greedy policy according to the action values for the main
task, which means the SFs are tracking a changing policy.

Reward Prediction (Reward): Another auxiliary task we consider is predicting
the immediate reward in the future based on the current state and action [8].
The prediction requires the agent to encode the reward information that it can
obtain in a short term in the representation function.

Expert Target Prediction (XY): Another auxiliary task is the prediction of
expert-designed targets. It is based on the idea that a good representation
should be able to predict key artifacts of an environment. This requires
domain knowledge and is not always possible. Here we consider the coordinates
of the agent in the environment as the target predictions.

Virtual Value Function Learning (VirtualVF): This auxiliary task is based on
the tasks the agent will face in transfer. We consider one auxiliary loss that
uses a goal location at the center of the maze (VirtualVF-1), and another that
uses five goals at the four corners and the center of the maze (VirtualVF-5).
These are virtual tasks, because the agent imagines achieving these goals,
even though they are not the training goal. We use VirtualVF-1 and VirtualVF-5
to assess the utility of having a larger set of virtual goals. We learn these
auxiliary value functions with DQN.

Augmented Temporal Contrast (ATC): The contrastive loss encourages the network
to learn similar representations for input-states that are temporally close to
each other [71, 72]. This auxiliary task led to the first successful pre-
training of a deep reinforcement learning agent, meaning it led to
representations that could be generally reused for other tasks. ATC also
includes other augmentations, like data augmentation. We test it with these
additions, to report performance of the originally proposed approach, even
though it goes beyond strictly only adding an auxiliary loss.

##  5 Experiment Design

Our experiments consist of two stages: a representation learning stage in a
training task, and a transfer stage using this learned representation in a
transfer task. In this section, we outline the details for these two stages in
our experiments, and outline the agents and how we evaluate them.

###  5.1 Transfer Setup

The first stage is to train the representation. All representations are
trained with a DQN agent in the training task, with goal location depicted in
Figure 2(a). To prevent overfitting, we employ an _early-saving_ strategy to
save the representation function, $\Phi_{\theta_{R}}$, as soon as the agent is
able to finish 100 consecutive episodes in 100 steps or less. Each
representation corresponds to a choice of activation function and auxiliary
loss—including choosing not to use an auxiliary loss.

In the second stage, we learn with the representation from the first stage, in
a new transfer task. Specifically, we 1) load and freeze the learned
representation, 2) re-initialize the value function and 3) learn the value
function for the transfer task with DQN, with the fixed representation. No
auxiliary tasks are used in transfer, and only the 64$\times$64 value function
network is learned with DQN. Learning with a re-initialized value function
rather than fine-tuning prevents negative effects from the old value function
during transfer, especially to less similar transfer tasks. Further, re-
initializing the value function ensures that the difference between transfer
learning and learning from scratch is due to the learned representation. The
agent learns in this new task for 100 thousand steps.

We consider 173 transfer tasks – all possible goal locations, including the
training goal state. To sort performance amongst these tasks, we provide a
novel method to measure their similarity to the training task. In this way, we
can ask questions about transfer to more or less similar transfer tasks. The
key idea is to first obtain successor representations for each state, and then
compute similarity in this new space. The successor representation encodes
similarity based on transition dynamics, meaning that states are considered
nearby due to ability to reach them rather than due to other distances, such
as Euclidean distances which does not respect the walls in the Maze. For
specific details, see Appendix B.3.

###  5.2 Network Choices and Activation Functions

To obtain representations with different properties, we use two different
activation functions: Rectified Linear Unit (ReLU) [61] and Fuzzy Tiling
Activation (FTA) [36]. ReLU is a standard activation function, defined as
$\max(z,0)$ for input $z$, where $z$ is a linear weighting on the previous
layer.

FTA is a newly introduced activation, designed to generate sparse outputs.
Essentially, it bins the scalar input into $k$ bins, with some smoothing to
ensure non-zero gradients through the activation. The smoothness and bin width
is controlled by a parameter $\eta>0$. The interval is from
$[-\tfrac{\eta}{2}k,\tfrac{\eta}{2}k]$, with $k$ equally sized bins of size
$\eta$. Assume the input $z$ is in bin $i$, namely
$-\tfrac{\eta}{2}k+(i-1)\eta\leq z\leq-\tfrac{\eta}{2}k+i\eta$ where
$i\in\\{1,2,\ldots,k\\}$. The $k$-dimensional output vector $\mathbf{h}(z)$
given by FTA on $z$ has entries $h_{j}(z)\in[0,1]$ defined as

$\displaystyle h_{j}(z)=\begin{cases}1&\text{if }j=i,\\\
1+\eta(j-\tfrac{k}{2})-z&\text{if }j<i\text{ and }z<\eta(1+j-\tfrac{k}{2}),\\\
1+z\\!-\\!\eta(j\\!-\\!1-\tfrac{k}{2})&\text{if
}j>i,z>\eta(j\\!-\\!1-\tfrac{k}{2}),\\\ 0&\text{else}.\end{cases}$

Larger $\eta$ activates more entries in $\mathbf{h}(z)$, and smaller $\eta$
results in more sparsity. This formulation removes a hyperparameter, by using
the suggested default choice of $\eta=\delta$.

For our experiments, the representation function consists of two convolutional
layers, one linear transformation, and a choice of activation function. The
linear layer projects the output of the convolutional layer to a
32-dimensional space. When using ReLU, the representation layer has $d=32$
features. If FTA is used, it has 640 features since FTA projects each scalar
to a short, sparse vector with 20 bins. Note that FTA still uses the same
number of learned parameters to produce this 640 features, as ReLU uses to
produce the 32 features, because binning occurs after the linear weighting.
However, the outputted number of features is higher, and so the value function
and auxiliary tasks all have more parameters, at least in their first layer.
We therefore also evaluate ReLU(L)—L for large—which uses 640 features.
ReLU(L) uses significantly more parameters to produce these 640 features than
FTA.

The structure for the value function and auxiliary tasks is given in Figure 1.
We use two hidden layers with 64 nodes each for the value function, and one
hidden layer with 64 nodes for the auxiliary task. We use a simpler network
for the auxiliary task to force the representation to learn as much as
possible. We use a slightly larger network for the value function, to avoid
overly constraining it and so confounding transfer performance.

###  5.3 Agent Specification

We use standard choices for DQN, including the use of $\epsilon$-greedy
exploration, an experience replay buffer, target networks, and the Adam
optimizer [73]. In total there are 9 choices for auxiliary tasks: No-aux, ATC,
IR, NAS, SF, Reward, XY, VirtualVF-1, and VirtualVF-5. There are 3
activations: FTA, ReLU, and ReLU(L). When using FTA with auxiliary tasks, we
set the number of bins $k=20$ and $\eta=0.2$. This implicitly specifies the
range for binning to $[-2,2]$. For the No-aux task agents, we test
$\eta=0.2,0.4,0.6,$ and $0.8$ and report performance for each, not the best
one. This gives a total of 30 agent specifications.

We consider three baseline agents, which we call Random, Input, and Scratch.
They allow us to falsify different hypotheses about the role of the learned
representation. Random uses a randomly initialized network as the
representation, without any learning. The agents start with a random network,
so this baselines checks whether learning actually improved the
representation. Input omits the representation network and directly inputs the
agent’s observation to the value function component. It is meant to check if
the learned representations play any (useful) role, and if learning from
scratch in the transfer task might just have been faster with smaller
networks. Finally, Scratch is a DQN agent that starts learning from randomly
initialized weights in the transfer task. The purpose of learning the
representation is to learn faster than learning from scratch in the transfer
task. This is the most important baseline, as it defines whether a learned
representation was successful for transfer—facilitated learning faster than
Scratch—or not—was comparable to or worse than Scratch.

###  5.4 Reporting Performance and Hyperparameters

To report performance, we have to consider how to measure performance and how
to set hyperparameters. In both the training and transfer tasks, every 10
thousand steps we record the average return of the last 100 episodes. To
summarize performance across the 300k steps, we take the sum of these recorded
values, also called the Area Under Curve (AUC). The AUC is used to select
hyperparameters.

The only hyperparameter common across all agents is the stepsize; we therefore
only sweep this hyperparameter. We separately pick the stepsize in Stage 1, in
the training task, and in Stage 2, when just learning the value function. We
use the average performance over 5 runs to select the step-sizes. Namely, in
Stage 1 we run each of the 30 agent specifications with different step-sizes,
for 5 runs. We select the best step-size according to training AUC, and use
the representations produced under those step-sizes. Then in Stage 2, we
evaluate each stepsize only for those representations, and pick the best step-
size for an agent specification for each transfer task by using averaging
performance across the 5 runs. We sweep the stepsize to ensure we are
evaluating reasonably well-optimized agents. Additional hyperparameter
details, including the selected values, are available in Appendix B.

When we report performance across agents, we do not average across these 5
runs. Instead, each run produces a different representation and we report
performance for each one as an independent data point. When showing aggregate
performance, we aggregate from this larger pool of 5 runs for each 30 agents
specifications, namely over 150 representations. We do so because each
representation has different properties; when correlating agent properties and
performance, we may not care which auxiliary task was used, but rather only
care about its emergent properties. Averaging across runs compares methods
(agent specification), rather than representations.

Finally, we obtain transfer performance in 173 transfer tasks. This means we
get 173 transfer performance samples for each of the 150 representations. In
total, when aggregating across transfer tasks or agent specifications, we
obtain a significant number of samples to estimate aggregate performance, even
though each agent specification only has 5 runs in the training task. For
example, in Figure 2(b), each bar in the plot is for one agent specification
and uses $173\times 5=865$ performance samples to estimate medians, means and
standard deviations. In total, we generate $173\times 150=25,950$ agent-task
combinations.

##  6 Good, bad and ugly Representations

We expect some agent specifications to result in representations that aid
transfer, and others to impede transfer. Unreal [8], the first large-scale
deep reinforcement learning system to highlight the utility of auxiliary
tasks, showed that although auxiliary tasks like pixel prediction improved
performance substantially, other tasks such as feature control had a much
smaller impact. Other work has highlighted that it can be difficult to obtain
any transfer in reinforcement learning [57, 16]. It seems the design and
deployment of auxiliary tasks remains largely an art.

In this section, we provide some clarity on these discrepancies by showing
that 1) there is large variability in performance across auxiliary tasks, and
2) transfer performance can degrade significantly as tasks become more
dissimilar.

Transfer TaskTotal reward in transfer, averaged over 5 runs Figure 3:
Transfer performance of 105 different representations (ReLU and FTA) on 173
transfer tasks. The tasks on the x-axis are arranged by similarity to training
tasks: on the left (small x-values) being most similar and on the right (large
x-values) being most dissimilar. The black line shows the performance when
learning in each transfer task from scratch. Lines completely above the black
line indicate a representation yielded successful transfer in all tasks. Lines
that start above the black line but fall below as we move left to right
indicate a representation that transfers to similar tasks but not dissimilar
tasks. The Input and Random baselines are not competitive; for completeness,
we still report their performance, but with ligther lines. Overall, many
representations achieve transfer and generally FTA-based representation are
better on this problem. Details on how task similarity was computed and how
this plot was generated can be found in the Appendix. The transfer performance
of ReLU(L) is shown in Figure 13; it exhibits the same pattern as ReLU,
transferring well to similar tasks and not as well in less similar tasks.

Figure 3 summarizes the transfer performance of many different representations
corresponding to different auxiliary tasks and activation functions. The plot
has task similarity on the x-axis, and each point on the plot summarizes the
performance of one representation on one particular transfer task. The lines
show how much transfer performance degrades as tasks become more
dissimilar.333Figure 2(a) shows two transfer tasks as purple squares. The one
beside the training goal is most similar according to our ranking, and the
other is least similar to the training goal. The bold black line shows
performance in the transfer task if the representation and value function were
trained from scratch—no transfer. Any point above the bold black line
indicates a representation that achieved better performance than training from
scratch on that task—successful transfer. Any line completely above its
corresponding black line indicates a representation that achieved successful
transfer for all goal states.

The most important conclusions from Figure 3 are that 1) several
representations achieve successful transfer across all tasks, and 2) a great
variety of representations emerge with transfer performance ranging from good
to significantly worse than scratch. Looking more closely, some
representations achieve successful transfer in dozens of tasks which are most
similar to the training tasks, but for tasks less similar to the training one,
performance is poor as seen by the step down in many of the lines.

Generally, we found that FTA-based representations yield better
representations for transfer compared to ReLU. Almost all FTA representations
outperformed Scratch (FTA), and transfer to less similar tasks was effectively
the same as on similar tasks to training, as evidenced by the nearly flat
lines in Figure 3. Interestingly, many ReLU-based representations achieved
very good performance in transfer to similar tasks but performed significantly
worse than Scratch (ReLU)—nearly as bad as the input baseline—on less similar
tasks. ReLU (L) performed similarly to ReLU; this result is in Appendix C.2.
Another note of interest is that Scratch (FTA) performs better than Scratch
(ReLU), and yet FTA representations were still better able to transfer than
ReLU-based ones: using FTA improved on using ReLU and then training the FTA
representation first in a training task improved performance in the transfer
task even more.

Digging a little deeper, Figure 4 depicts the transfer performance of each
auxiliary task. We again see that FTA-based representations achieve higher
performance overall and higher performance across auxiliary tasks—the worst
performing representation never used FTA, always ReLU. Inspecting each
auxiliary task, the FTA-based representations exhibited lower variance across
runs. Larger ReLU representations, ReLU(L), did improve performance over the
smaller ReLU representations, but not uniformly. The IR auxiliary task
representation, for example, improves with large ReLU networks, but ATC
performs worse—though not significantly in either case.

At the auxiliary task level, there are no obvious trends (except for the fact
IR and Reward are generally not useful). For example, the successor-feature
auxiliary task (labelled SF) is among the best performing FTA representations
and among the worst performing ReLU representations. The subgoal-navigation
auxiliary tasks (VirtualVF) result in the best performance with ReLU
representations. These subgoals can be thought of as way-points placed at
strategic locations in the environment; perhaps these tasks force the network
to represent how to navigate to these waypoints which then speeds learning
when navigating to other nearby goals in transfer. Perplexingly, these are not
the best performing representations when combined with FTA activation
functions. Perhaps FTA networks already extract a general and transferable
representation (as evidenced by the performance of ‘No Aux’), and thus the
subgoal auxiliary tasks simply do not help much. It is difficult to know
looking at performance only; in the following sections we look at different
properties of the learned representations as a lens to understand such
mysteries.

We included ATC, a recent representation learning strategy, to calibrate the
quality of transfer performance. This approach uses multiple networks to
compute a contrastive loss, while also using data augmentation of the input
images. ATC worked well, but it did not significantly outperform the best ReLU
and FTA representations. In addition, we found that the ReLU network combined
with data augmentation (random shifting of the input imagines) and no
contrastive setup achieved similar performance to ReLU ATC.

ptTotal reward, averaged over transfer tasks

ptReLU

ptReLU(L)

ptFTA

Figure 4: Transfer performance depends on the activation function,
representation size, and auxiliary tasks. Overall, FTA-based representations
achieved the best performance and exhibited the least variation in performance
across auxiliary tasks. The orange lines depict the median, the upper and
lower edges of the box show the $25_{th}$ and the $75_{th}$ percentiles, while
the whiskers show 1.5 times the inter-quartile range. These results are
computed over $173\times 5=865$ samples, and so the standard errors are quite
small (as you can see in Figure 14 in Appendix C.3).

##  7 Representational Properties, Old & New

Before the emergence of deep reinforcement learning, the study of
representations in reinforcement learning and their effects on learning was
focused on fixed bases. The problem with this is not that a fixed basis cannot
capture complex non-linear relationships (e.g., see the work by Liang et al.
[74]), but rather that the representation is fixed—the features are not
adapted to the task. In some sense, this is good because it forces the agent
designer to consider what are desirable representation properties—a level of
analysis complementary to the design of good algorithms. Over the years,
researchers have proposed and debated numerous properties. We leverage these
discussions to analyze our learned representations.

We characterize a representation into three main axes: _capacity_ ,
_efficiency_ , and _robustness_. Capacity reflects whether a representation
can represent a given function. Efficiency captures the lack of redundancy of
the features and the computational cost of using them. Robustness captures the
idea that interference is undesirable and that representations should avoid
it; a more complete name is _update robustness_. We define six metrics that
capture these three axes, and we use them to evaluate the representations
learned by our agents. Our goal is to develop a systematic methodology for
assessing learned representations, based on a diverse set of properties. This
evaluation list does not suggest that a property is necessary; rather, it
provides some quantitative measures to supplement more qualitative evaluation
like visualization. Such a list is necessarily incomplete; we attempt only to
start with a reasonably broad set of properties.

We assume we have a dataset of 1000 transitions to measure the properties,
$\mathcal{D}_{\text{test}}=\\{(s_{1},a_{1},s_{1}^{\prime},r_{1}),(s_{2},a_{2},s_{2}^{\prime},r_{2}),$
$\ldots,(s_{N},a_{N},s_{N}^{\prime},r_{N})\\}$, where $s_{i}^{\prime}$ is the
outcome state from $s_{i}$. This dataset is obtained by running the random
policy for $N$ episodes, and then randomly subsampling $N$ transitions, to
ensure we cover the state space. We store the transition because some of the
properties rely on consecutive states or the entire transition. The symbol
$\phi_{i}$ refers to the representation of $s_{i}$;
$V(\phi)=\max_{a}Q(\phi_{i},a)$ is the value learned given that
representation. We compute distances both according to the representation and
according to action-values,

$\displaystyle d_{v,i,j}$
$\displaystyle\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}|V(\phi_{i})-V(\phi_{j})|,$ (1) $\displaystyle d_{s,i,j}$
$\displaystyle\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\|\phi_{i}-\phi_{j}\|_{2}.$

###  7.1 Capacity: Retaining Relevant Information and
Nonlinear Transformations

The first property to consider for a representation is its capacity: can it
represent the functions we want to learn? The value function network should be
a simple function of these features, such as a simple neural network. To
measure capacity, we use one direct measure, _complexity reduction_ , and two
indirect measures, _dynamics-awareness_ and _diversity_.

Complexity reduction reflects how much the representation facilitates
simplicity of the learned value function on top of those features. If
complexity is small, the features encode much of the non-linearity needed. A
well-known result is that the composition of a Lipschitz function $V$ with
another function $\phi$ has (Rademacher) complexity: complexity($V\circ\phi$)
$\leq L$ complexity($\phi$), for $L$ the Lipschitz constant of $V$. The
Lipschitz constant $L$ is one where $\tfrac{d_{v,i,j}}{d_{s,i,j}}\leq L$ for
any $\phi_{i},\phi_{j}$. For a smaller $L$, the representation $\phi$ handles
most of the complexity, which is good because we have a longer initial
learning phase to obtain $\phi$. $V$ on top of $\phi$ should be fast to learn.
Lipschitz value functions have also been motivated for value transfer [75] and
learning models [76].

Let us turn this intuition into a measure of how much the representation helps
reduce the required complexity in the value function. We can measure
$\tfrac{d_{v,i,j}}{d_{s,i,j}}$ for all pairs $(i,j)$ in our dataset. An
estimate of $L$ is the maximum over these slopes. However, empirically we find
that the average of these slopes results in bigger differences between
representations with better correlation to transfer performance. The max loses
significant precision, possibly making very different $Q$ have similar
measures; the average summarizes the whole surface. For example, one
representation might reduce complexity for most of the space, but due to one
small part, have a similarly high maximum slope to a representation that
barely reduces complexity in any part of the space. We call these averaged
ratios $L_{\text{rep}}$, giving

Complexity Reduction
$\displaystyle\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}1-\frac{L_{\text{rep}}}{L_{\text{max}}}$ (2)
$\displaystyle\text{where }\quad L_{\text{rep}}$
$\displaystyle\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\tfrac{2}{N(N-1)}{\textstyle\sum^{N}_{i,j,i<j}}\tfrac{d_{v,i,j}}{d_{s,i,j}}.$

When this ratio is computed on given time step $t$—either during learning or
on the last time step before the representation is frozen for transfer—we use
the current action-values. We normalize $L_{\text{rep}}$ between 0 and 1 using
$L_{\text{max}}$, computed as the maximum $L_{\text{rep}}$ over all
representations and across time steps. This is subtracted from 1 to ensure
higher values refer to higher reduction in complexity.

We can also indirectly measure complexity—that is without specifying a set of
value functions—by testing if the representation is dynamics-aware. This means
that pairs of states where one is a successor to the other should have similar
representations, and states further apart in terms of reachability should have
a low similarity. This measure is in fact related to the Laplacian used for
proto-value functions and successor features [77, 66]. For every state in the
dataset, we take its successor state and a random state. If the distance, in
representation space, between the successor state is smaller than the distance
to a random state, then the representation has high dynamics awareness.

$\\!\\!\text{Dynamics
Awareness}\\!\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\\!\tfrac{\sum^{N}_{i}||\phi_{i}-\phi_{j\sim
U(1,N)}||-\sum^{N}_{i}||\phi_{i}-\phi^{\prime}_{i}||}{\sum^{N}_{i}||\phi_{i}-\phi_{j\sim
U(1,N)}||}.$ (3)

In addition, we can measure the diversity of a representation, which is the
opposite of _specialization_. If a representation is specialized to one value
function, then it likely uses a small subspace of the larger Euclidean space
and likely does not produce a diversity of possible feature vectors. This
specialization may be problematic, as it means the representation is unlikely
to perform well when it is transferred to learn another value function.

To define diversity, we use a ratio between state and value differences. Given
two states $s_{i}$ and $s_{j}$, we can compare the distance between their
representations ($d_{s,i,j}$) and the distance between their values
($d_{v,i,j}$). If the value distance is high—the two state values are very
different—then the representation distance is also likely to be high to allow
this. The interesting case is when the value distance is low. The
representation distance can be high or low, and still allow two states to have
similar values, because we project from a higher-dimensional feature vector to
a scalar value. A representation with high diversity would have high
representation distance when possible, allowing two states to be distinguished
even when they have similar values. A representation with low diversity would
simply map these two states with similar values to similar representations,
specializing to this value function. The measure is

$\text{Diversity}\\!\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\\!1\\!-\\!\tfrac{1}{N^{2}}\\!\\!\sum^{N}_{i,j}\min{\\!\left(\frac{d_{v,i,j}/{\scriptstyle\max_{i,j}d_{v,i,j}}}{d_{s,i,j}/{\scriptstyle\max_{i,j}d_{s,i,j}}\\!+\\!10^{-2}},1\right)}.$
(4)

We normalize by the maximum distances, to be invariant to value and
representation scales. Diversity can be seen as 1-specialization. The
specialization is lower when $d_{v,i,j}$ is small and $d_{s,i,j}$ is large,
causing this ratio to be closer to zero. The specialization is higher when the
ratio between $d_{v,i,j}$ and $d_{s,i,j}$ is nearly one. Diversity allows us
to indirectly measure capacity, as we can check the level of specialization
for a given function without needing to have access to the larger set of
possible functions.

###  7.2 Efficiency: Feature Redundancy

Many function classes can satisfy these capacity properties and so we consider
other functional properties of the features. Reducing redundancy in the
representation, finding _linearly independent_ features, is a basic
requirement. Orthogonality satisfies this requirement and additionally
provides distributed features as well as minimal interference. For example,
factor analysis finds a dense set of orthogonal (latent) factors to explain
the data. This representation is highly distributed, as each feature is used
to describe many different inputs. At the same time, interference is reduced:
the interference for two states with orthogonal feature vectors is zero under
linear updating. As before, we normalize magnitudes and ensure higher
orthogonality means that more feature vectors $\phi_{i}$ and $\phi_{j}$ are
orthogonal to each other.

$\text{Orthogonality}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}1-\tfrac{2}{N(N-1)}\sum^{N}_{i,j,i<j}\frac{|\langle\phi_{i},\phi_{j}\rangle|}{\|\phi_{i}\|_{2}\|\phi_{j}\|_{2}}.$
(5)

Note that there is an equivalence between orthogonal feature
vectors—orthogonal representations—and orthogonal features: the sum over all
states $i,j$ of $\langle\phi_{i},\phi_{j}\rangle^{2}$ is equal to the sum over
all pairs of features of the dot product between the vector of those feature
values across states (see Appendix A.1). Additionally, for centered features,
orthogonality is also equivalent to decorrelation (see Appendix A.3).

One idea related to orthogonality is sparsity. If only a small number of
features are active for an input, then the features are sparse—with typically
the additional condition that each feature is active for some inputs (no dead
features). For non-negative features, maximizing sparsity corresponds to
finding orthogonal features: dot products can only be zero when features are
non-overlapping for two inputs. Sparsity has the additional benefit, though,
of improving efficiency for querying and updating the function, because only a
small number of features are active. To measure sparsity, we calculate the
percentage of inactive features on average across states in the dataset.

$\text{Sparsity}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\frac{1}{dN}\sum_{i=1}^{N}\sum^{d}_{j=1}\mathds{1}(\phi_{i,j}=0),$
(6)

where the representation $\phi_{i}$ for state $s_{i}$ is $d$ dimensional.
Equality with zero is tested within a tolerance of $10^{-10}$.

###  7.3 Robustness: Interference Reduction

More recent work in neural networks has also focused on robustness, both to
interference and noise. Interference reflects how much updates in one state
reduce accuracy in other states. We use a recent measure developed for
reinforcement learning [78], which uses the difference in temporal difference
errors before and after an update. We do the comparison each time the target
network is synchronized, which occurs every 64 steps, for a total of $T$ times
during learning. For every $t=1,\ldots,T$, we compare the error between
$\theta_{t}$ and the parameters after 64 updates, $\theta_{t+1}$; note $t$
here references the synchronization iterator rather than time.

$\displaystyle\text{Non-
interference}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}1-\frac{\text{Interference}}{\text{MaxInterference}},$ (7)
$\displaystyle\text{Interference}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\\{\text{Update Interference}_{t,\geq
Percentile_{0.9}}\\}_{t=1}^{T},$ $\displaystyle\text{Update
Interference}_{t}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\frac{1}{N}\sum_{i=1}^{N}\text{err}_{t,i}(\theta_{t+1})-\text{err}_{t,i}(\theta_{t}),$
$\displaystyle\text{err}_{t,i}(\theta)\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\left(r_{i+1}+\gamma_{t}\max_{a}Q_{{\theta}_{t}}(s_{i}^{\prime},a)-Q_{\theta}(s_{i},a_{i})\right)^{2}.$

The maximal Interference is across all representations.

ptTotal reward, averaged over transfer tasks  ptFTA ReLU

Figure 5: Performance averaged over transfer tasks versus representation
property values. Each dot in a plot corresponds to one representation, at the
(x,y) point corresponding to its property value and average transfer
performance. The histograms under the plots visualize the density of points at
each property value. The green stars correspond to the three best performing
representations. We separate out FTA-base and ReLU-based representation, which
exhibit notably different behavior.

ptProperty value

ptTime Step ($10^{4}$, logscale) Figure 6:  Plotting the representation
properties over time, with one subplot per property. Each curve shows the
property of one agent specification (activation and auxiliary task pair),
averaged over the 5 runs in the training task. The curve changes color, to
black, at the time point where we took the representation and fixed it; this
point was chosen based on when the return for the agent stopped changing. Line
colors vary from light gray to black based on how much their value fluctuate
from the property value after their return converges, with darker lines
denoting lower variation. In these plots, we allowed the representation to
keep learning to understand if properties significantly change afterwards. Our
primary focus is to show the general trend that properties converge over time,
and that they converge approximately when the return does; therefore we use
the same color for all agent specifications.

##  8 The Properties of Good Representations

We can now return to the main question of this work: do good representations
that facilitate transfer exhibit particular properties? In this section, we
investigate how the properties defined in the previous section relate to
transfer performance.

Figure 5 contains the main result; for now let us focus on only the FTA
representations in the top row to better understand this figure. Each subplot
shows the transfer performance of every single FTA-representation averaged
over all transfer tasks. The representations in each subplot are ordered based
on a single measured property. For example, the first subplot in the first row
plots the transfer performance of FTA representations and the dots are ordered
by complexity reduction on the x-axis. Representations with high complexity
reduction and good transfer performance would appear as a dot in the top right
of the subplot. Representations with low complexity reduction and good
transfer performance would appear as a dot in the top left of the subplot, and
so on.

At the highest level we see FTA (top row) and ReLU (bottom row) exhibit
different properties across representations. FTA-based representations by
large exhibit high complexity reduction and high diversity, whereas ReLU
representations range widely from low to medium on the same two measures. In
fact, the lowest observed complexity reduction and high diversity of any FTA
representation was greater than the highest observed complexity reduction and
high diversity for ReLU. ReLU representations could be sparse and have low or
high orthogonality, whereas FTA representations are mostly sparse.
Interestingly, the top representations in terms of sparsity were ReLU. ReLU
representations with similar property values can achieve very different
transfer performance (visible as points stacked vertically). There appears to
be no clear relationship between sparsity and performance for ReLU
representations.

Now consider the properties of the top performing representations. Again, let
us focus our attention on the FTA representations in the top row of Figure 5.
The green stars in each subplot correspond to the top performing
representations (in terms of transfer). First notice the stars are typically
close together in x and y indicating all three achieve similar performance
with similar property values; this is true for ReLU representations as well.
In general, we see that the best performing representations are not at the
extremes of any property (high or low). Given that FTA representations by
large exhibit high complexity reduction, diversity and sparsity, it is notable
that the best performing representations are the lowest of those three
properties.

_In general, the best representations for both FTA and ReLU exhibit fairly
similar properties_ (relative to other representations with the same
activation function): high complexity reduction, medium-high dynamic
awareness, and medium orthogonality and sparsity. Of particular note is the
clear pattern in complexity reduction and diversity for ReLU: both needed to
be higher, and performance clearly drops for lower values. FTA seems to more
naturally produce representations that are higher on these measures; we
hypothesize that this is the main explanation for why FTA representations work
well across the board for transfer.

The property values depicted in Figure 5 were computed from the
representations when frozen for transfer, but one might wonder what are the
dynamics of the properties over time. Recall that we froze and transferred
each representation after 100 episodes were completed in the training phase.
This choice balances the need for reasonable performance without having to
select somewhat arbitrary steps budgets or performance criteria. However, our
choice does mean that each representation could receive different amounts of
experience, and thus begs the natural question: would the properties reported
in Figure 5 be very different with more or less training. Figure 6 provides
the answers.

Generally, across all auxiliary tasks and activation functions, the
representation property values remained similar after initial transients in
early learning. Each subplot of Figure 6 shows a particular property value for
every single representation tested over an extended training period. We
intentionally do not distinguish between activation functions and auxiliary
tasks in this plot. The change in color indicates when the representation was
frozen for transfer, in terms of training time. Note, many representations
were frozen after the same number of training steps. Orthogonality, dynamic
awareness, and sparsity of a small number of representations slowly increases
with more training and complexity reduction of a few representations slowly
decreases. Overall, the properties for the most part converge, and do so just
before the representations were frozen for transfer. Training the
representations longer would not have resulted in significant changes to the
property values.

Finally, we investigated why VirtualVF5 was helpful for ReLU and harmful for
FTA through the lens of our representation properties. We plotted properties
for these representations in Figure 7, and found that the addition of this
auxiliary loss significantly decreased dynamics awareness for FTA—to the
detriment of performance—but increased it for ReLU. Additionally, it caused
the FTA-based representation to have much higher orthogonality, likely
increasing it to one of the extremes that performed more poorly. For ReLU, the
increase in orthogonality was to an interim level, from a value that was very
small. It is yet unclear why this auxiliary loss caused these effects, but
this clear and systematic change in the properties helps explain this outcome.

Figure 7: The VirtualVF5 task produces bad FTA representations but improves
ReLU representations. Each subplot shows a property value achieved by four
different representations: FTA and ReLU with VirtualVF5 and FTA and ReLU with
no auxiliary tasks. It is clear this auxiliary task changes the properties of
the representations; particularly Dynamics Awareness and Orthogonality. We did
not include non-interference as VirtualVF5 had no impact on it.

##  9 Transfer across Atari 2600 Game Modes

In this section we investigate our methodology and properties on a larger
environment, with a substantially more complicated and high-performance agent.
So far, in order to perform a thorough empirical evaluation, our analysis
focused on a small, controllable pixel-based navigation environment. To
demonstrate the generality of our approach, we apply our methodology to
understand representations learned for transfer across different Atari 2600
game modes [55, 58] (see Figure 8). We chose this setting because transfer in
Atari is an active research question, with some recent promising results [56]
when transferring policies learned by a state-of-the-art agent called Rainbow
[54]. We used the same three Atari games: Freeway, Space Invaders and
Breakout.

Rainbow can be seen as an auxiliary task learning agent, with auxiliary heads
produced by two value heads (duelling networks [79]) and estimates of the
distribution of returns (C51 [80]). More agent details are given in Appendix
D.

The experimental protocol is very similar to before. We still learn in one
source task (the default game mode 0), where the agent learns for 200 million
frames, with the learned policy evaluated every iteration for 500,000 steps.
In the target task (a different game mode), the representation is frozen and
the agent learns the values from scratch for 10 million frames. We contrast
the representations and transfer performance of Rainbow to a simple baseline:
a random representation. We take the exact same Rainbow agent, with the same
network architecture, and fix the representation to its random initialization.
The value function is still updated in the same way, using dueling networks,
distributional updates, prioritized replay and so on.

Previous transfer results [56] investigated transferring both the learned
values and the representation and allowed fine-tuning. Rusu et al. [56] found
Rainbow policies can transfer. We are asking a different question: can the
learned representation transfer and what are its properties?

We first look at the transfer performance of Rainbow and Rainbow with a fixed
random representation, in comparison to learning from scratch. In Figure 9, we
can see two clear outcomes. In general, we see successful transfer: the
initial learning phase in the source task resulted in improved performance for
Rainbow, as compared to learning from scratch (if the blue line is above the
black line positive representation transfer is observed). It was not entirely
obvious that this positive representation transfer would be possible, as prior
work transferred the entire network and utilized fine-tuning. Second, the
random representation generally does not allow the agent to learn in the
target task, indicating learning features in the source mode facilitated
transfer. This result is surprisingly similar to our results with DQN in the
maze (compare Figure 3 and Figure 9).

Figure 8: Three different modes of the Atari game Freeway. Atari 2600
cartridges allowed players to choose between game variations that modified
sprites, velocities, etc. These different modes are an excellent platform for
investigating transfer. They are free from experimenter’s bias and, because of
hardware limitations, the different modes of a game were often small
variations of the default mode, thus are not too different from one another.
In this particular example, in Freeway, the default mode (left is m0) consists
of a thin traffic in which there are only cars and these cars move at a
relatively slow speed. The mode in the center (m1) has trucks, traffic is
heavier and moves much faster. The mode on the right (m3) has trucks in all
lanes with varying speeds. Figure 9:  Transfer in three Atari games, to game
modes of increasing difficulty. The episodic return (computed over 500,000
steps) of the greedy policy is obtained every 1 million training steps, then
averaged into a single performance number (AUC), plotted on the y-axis. Shaded
regions correspond to standard 95% confidence intervals. The source task,
default game mode zero, is plotted first. The remaining modes are ordered by
the AUC achieved by training rainbow from scratch. In some cases, the default
game mode appears to be more difficult, making performance not monotonically
decreasing. Some of the gaps between Rainbow and Scratch look small, but they
are actually substantial. They look smaller only because Random baseline does
so poorly, skewing the scale of the plot. Notice that in Freeway, the
representation even transferred to quite different modes (e.g., m1 and m3
visualized in Figure 8), with heavier traffic, different vehicle speeds and
vehicle types (trucks).

Figure 10: Properties for the representations learned by Rainbow in Atari.

Next, we consider the properties of the representation produced by Rainbow in
the source game mode. We contrast these properties to the representation at
initialization, to see how much Rainbow changed the representation properties
after training. In the radar plots in Figure 10, we see that in general
Rainbow increased all the properties, except for dynamics awareness. The most
stark change was in complexity reduction, which was significantly increased
(see Table III in Appendix E for unnormalized complexity–L_rep–results). This
matches our previous results in the maze, where high complexity reduction was
key. We also see that Rainbow significantly increases sparsity and
orthogonality. Recall that one of our hypotheses for the success of FTA was
the fact that it was more naturally able to obtain higher sparsity and
orthogonality. We see that diversity only slightly increases, similar to our
results under FTA, though different from our previous ReLU network. Both FTA
and Rainbow produced more sparse and orthogonal representations than the ReLU
network, which may explain why diversity increased for them both but not for
ReLU.

Finally, dynamics awareness was brought to a medium-low value for all
environments (increased in Freeway and significantly decreased in the other
two). This outcome is different from what we observed in the maze experiments,
where dynamics awareness was significantly increased to a relatively high
value. One difference is that the random representations for Space Invaders
and Breakout start with relatively high dynamics awareness: a random
initialization makes nearby points look more similar and more distant points
more dissimilar. Decreasing the dynamics awareness could happen either due to
making temporally distant points have more similar representations or to
making representations for successor states very different. We can speculate
further why this happens. In the Maze and in Freeway the learned
representations exhibited higher dynamics awareness compared to a random
representation. In both environments the dynamics are regular throughout
training: the location of walls and the movement of cars are independent of
the agent’s actions. Breakout and Space Invaders are very different. The games
start with many objects on the screen and a successful agent learns to destroy
all of them. A well trained representation might produce a smoothly
generalizing function with mostly similar states along the trajectory of a
well played game. Regardless of the explanation, our result suggests high
dynamics awareness is environment specific and perhaps not necessary for good
performance.

##  10 Conclusions

The goal of this work is to make progress towards an answer to a classic
question: how do the properties of representations—that emerge under standard
architectures used in reinforcement learning—relate to the transfer
performance? We introduced a method of measuring the similarity between
training and transfer tasks and designed experiments to assess learned
representations. All tasks are similar, in that they involve navigating to
locations in the same Maze. Intuitively, transfer should be possible, even to
locations that are quite far from the goal in training. We found that 1) ReLU-
based representations transferred only to very similar tasks 2) some auxiliary
tasks improved transfer of ReLU-based representations, but none facilitated
transfer to less similar tasks, 3) the FTA activation significantly improved
transfer, suggesting it might be a promising activation to use going forward,
and 4) transfer was not possible with a linear value function, even in this
seemingly simple environment.

We extensively and systematically investigated the properties of all of these
(good and bad) representations attempting to better understand what causes the
improvement in transferability. We defined diversity, complexity reduction,
and dynamics awareness, as well as used measures of orthogonality, sparsity
and non-interference from the literature. In general, interim values for
properties were better: representations at the very extremes were never the
best. Further, we found that the best representations maintained high
complexity reduction, medium-high dynamics awareness, medium diversity and
medium orthogonality and sparsity. These conclusions do not mean
representations should have medium orthogonality, for example, but rather
representations that emerge under training with auxiliary losses tend to do
more poorly if orthogonality is higher or if it is very small (at the
extreme), in our transfer setting.

One message from this work is that it was important to use a simpler
environment to develop a systematic methodology and obtain comprehensive
results. Even in just this setting, there was a mountain of data to analyze.
Further, results in this simple environment were already informative and
changed our perspective on these representations. A priori, one might have
thought that transfer would be very easy in this environment; after all, we
are not learning small networks here! Yet, repeatedly we hit roadblocks. The
specific conclusions about network architectures and activations, auxiliary
losses, and even properties, may be different in other environments, but the
higher-level conclusions about the relevance of these properties, the
interactions between components, and the need for a careful methodology to
understand these nuances extends.

We demonstrated the generality of the approach by using the same methodology
to analyze the representation of a Rainbow agent that transfers across Atari
modes. We found that—similarly to the high-performing representations in the
maze environment—the representation had high complexity reduction, higher
levels of orthogonality and sparsity and interim diversity. A key difference
was that the dynamics awareness was medium-low, rather than medium-high. This
difference might indicate that this property is useful for certain
environments, like cost-to-goal environments, and not a useful property for
representations to have in other environments, like Atari games.

There are many possible next steps to use this methodology to improve our
understanding of representations in reinforcement learning. There has been
substantial effort to characterize transfer, generalization, and overfitting
in deep reinforcement learning, primarily in terms of performance [81, 82,
83]. A natural next step is to build on this work: repeat their experiments
and measure the properties of the learned representations. Another important
next step is to consider correlation between properties and transfer when
fine-tuning the representation. In this work, we tested fixed features, to ask
if initial learning produced useful features for future learning. It is also
useful to ask if initial learning produced a useful starting point or
initialization for future learning. We may find that different representation
properties are useful in this fine-tuning regime.

## Acknowledgments

We thank Vlad Mnih for reviewing an earlier version of this paper. This work
was generously funded by the CIFAR Canada AI Chair program, Amii and NSERC.

## References

  * [1] E. Talvitie, “Self-Correcting Models for Model-Based Reinforcement Learning,” in _AAAI Conference on Artificial Intelligence_ , 2017.
  * [2] G. Z. Holland, E. J. Talvitie, and M. Bowling, “The Effect of Planning Shape on Dyna-Style Planning in High-Dimensional State Spaces,” _arXiv preprint arXiv:1806.01825_ , 2018.
  * [3] Y. Wan, M. Zaheer, A. White, M. White, and R. S. Sutton, “Planning with Expectation Models,” in _International Joint Conference on Artificial Intelligence_ , 2019.
  * [4] S. S. Du, S. M. Kakade, R. Wang, and L. F. Yang, “Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?” in _International Conference on Learning Representations_ , 2019.
  * [5] D. Russo and B. Van Roy, “Eluder Dimension and the Sample Complexity of Optimistic Exploration,” in _Advances in Neural Information Processing Systems_ , 2013.
  * [6] T. Lattimore, C. Szepesvari, and G. Weisz, “Learning With Good Feature Representations in Bandits and in RL with a Generative Model,” in _International Conference on Machine Learning_ , 2020.
  * [7] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller, “Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images,” in _Advances in Neural Information Processing Systems_ , 2015.
  * [8] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu, “Reinforcement Learning with Unsupervised Auxiliary Tasks,” in _International Conference on Learning Representations_ , 2017\.
  * [9] A. A. Rusu, M. Večerík, T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell, “Sim-to-Real Robot Learning from Pixels with Progressive Nets,” in _Conference on Robot Learning_ , 2017.
  * [10] C. Finn, P. Abbeel, and S. Levine, “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks,” in _International Conference on Machine Learning_ , 2017.
  * [11] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver, “Successor Features for Transfer in Reinforcement Learning,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [12] I. Higgins, A. Pal, A. Rusu, L. Matthey, C. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner, “DARLA: Improving Zero-Shot Transfer in Reinforcement Learning,” in _International Conference on Machine Learning_ , 2017.
  * [13] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,” in _IEEE International Conference on Robotics and Automation_ , 2018.
  * [14] A. Zhang, N. Ballas, and J. Pineau, “A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning,” _arXiv preprint arXiv:1806.07937_ , 2018.
  * [15] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual Reinforcement Learning with Imagined Goals,” in _Advances in Neural Information Processing Systems_ , 2018.
  * [16] J. Tyo and Z. Lipton, “How Transferable Are the Representations Learned by Deep Q Agents?” _arXiv preprint arXiv:2002.10021_ , 2020.
  * [17] D. Silver, H. van Hasselt, M. Hessel, T. Schaul, A. Guez, T. Harley, G. Dulac-Arnold, D. Reichert, N. Rabinowitz, A. Barreto, and T. Degris, “The Predictron: End-to-End Learning and Planning,” in _International Conference on Machine Learning_ , 2017.
  * [18] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, “Cognitive Mapping and Planning for Visual Navigation,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [19] A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn, “Universal Planning Networks: Learning Generalizable Representations for Visuomotor Control,” in _International Conference on Machine Learning_ , 2018.
  * [20] T. Kurutach, A. Tamar, G. Yang, S. J. Russell, and P. Abbeel, “Learning Plannable Representations with Causal Infogan,” in _Advances in Neural Information Processing Systems_ , 2018.
  * [21] V. François-Lavet, Y. Bengio, D. Precup, and J. Pineau, “Combined Reinforcement Learning via Abstract Representations,” in _AAAI Conference on Artificial Intelligence_ , 2019.
  * [22] G. Yang, A. Zhang, A. Morcos, J. Pineau, P. Abbeel, and R. Calandra, “Plan2Vec: Unsupervised Representation Learning by Latent Plans,” in _Conference on Learning for Dynamics and Control_ , 2020.
  * [23] C. Finn, X. Y. T. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel, “Deep Spatial Autoencoders for Visuomotor Learning,” in _IEEE International Conference on Robotics and Automation_ , 2016.
  * [24] I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner, “Towards a Definition of Disentangled Representations,” _arXiv preprint arXiv:1812.02230_ , 2018.
  * [25] C. Painter-Wakefield and R. Parr, “Greedy Algorithms for Sparse Reinforcement Learning,” in _International Conference on Machine Learning_ , 2012.
  * [26] R. Parr, C. Painter-Wakefield, L. Li, and M. Littman, “Analyzing Feature Generation for Value-Function Approximation,” in _International Conference on Machine Learning_ , 2007.
  * [27] G. Konidaris, S. Osentoski, and P. Thomas, “Value Function Approximation in Reinforcement Learning Using the Fourier Basis,” in _AAAI Conference on Artificial Intelligence_ , 2011.
  * [28] R. S. Sutton, “Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding,” in _Advances in Neural Information Processing Systems_ , 1996.
  * [29] S. Mahadevan and M. Maggioni, “Proto-Value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes,” _Journal of Machine Learning Research_ , 2007.
  * [30] V. R. Kompella, M. Luciw, and J. Schmidhuber, “Incremental Slow Feature Analysis: Adaptive Low-Complexity Slow Feature Updating from High-Dimensional Input Streams,” _Neural Computation_ , 2012.
  * [31] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT press, 2018.
  * [32] S. E. Fahlman and C. Lebiere, “The Cascade-Correlation Learning Architecture,” in _Advances in Neural Information Processing Systems_ , 1990\.
  * [33] P. Kanerva, _Sparse Distributed Memory_. MIT press, 1988.
  * [34] B. Ratitch and D. Precup, “Sparse Distributed Memories for On-Line Value-Based Reinforcement Learning,” in _European Conference on Machine Learning_ , 2004.
  * [35] V. Liu, R. Kumaraswamy, L. Le, and M. White, “The Utility of Sparse Representations for Control in Reinforcement Learning,” in _AAAI Conference on Artificial Intelligence_ , 2019.
  * [36] Y. Pan, K. Banman, and M. White, “Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online,” in _International Conference on Learning Representations_ , 2021.
  * [37] Y. Wu, G. Tucker, and O. Nachum, “The Laplacian in RL: Learning Representations with Efficient Approximations,” in _International Conference on Learning Representations_ , 2019.
  * [38] R. M. French, “Catastrophic Forgetting in Connectionist Networks,” _Trends in Cognitive Sciences_ , 1999.
  * [39] R. Caruana, “Multitask Learning,” _Machine learning_ , 1997.
  * [40] S. Thrun, “Is Learning the n-th Thing any Easier than Learning the First?” in _Advances in Neural Information Processing Systems_ , 1996.
  * [41] M. Bellemare, W. Dabney, R. Dadashi, A. A. Taiga, P. S. Castro, N. Le Roux, D. Schuurmans, T. Lattimore, and C. Lyle, “A Geometric Perspective on Optimal Representations for Reinforcement Learning,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [42] J. Oh, S. Singh, and H. Lee, “Value Prediction Network,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [43] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel _et al._ , “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model,” _Nature_ , 2020.
  * [44] K. Javed and M. White, “Meta-Learning Representations for Continual Learning,” in _Advances in Neural Information Processing Systems_ , 2019\.
  * [45] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-Level Control through Deep Reinforcement Learning,” _Nature_ , 2015\.
  * [46] C. Rupprecht, C. Ibrahim, and C. J. Pal, “Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents,” in _International Conference on Learning Representations_ , 2020.
  * [47] T. Zahavy, N. Ben-Zrihem, and S. Mannor, “Graying the Black Box: Understanding DQNs,” in _International Conference on Machine Learning_ , 2016.
  * [48] S. Greydanus, A. Koul, J. Dodge, and A. Fern, “Visualizing and Understanding Atari Agents,” in _International Conference on Machine Learning_ , 2018\.
  * [49] F. P. Such, V. Madhavan, R. Liu, R. Wang, P. S. Castro, Y. Li, J. Zhi, L. Schubert, M. G. Bellemare, J. Clune, and J. Lehman, “An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents,” in _International Joint Conference on Artificial Intelligence_ , 2019.
  * [50] T. Dai, K. Arulkumaran, S. Tukra, F. Behbahani, and A. A. Bharath, “Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation,” _arXiv preprint arXiv:1912.08324_ , 2019.
  * [51] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson, “Understanding Neural Networks through Deep Visualization,” in _Deep Learning Workshop_ , 2015.
  * [52] A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski, “Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [53] A. Atrey, K. Clary, and D. Jensen, “Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning,” in _International Conference on Learning Representations_ , 2020.
  * [54] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver, “Rainbow: Combining Improvements in Deep Reinforcement Learning,” in _AAAI Conference on Artificial Intelligence_ , 2018.
  * [55] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The Arcade Learning Environment: An Evaluation Platform for General Agents,” _Journal of Artificial Intelligence Research_ , vol. 47, pp. 253–279, 2013.
  * [56] A. A. Rusu, S. Flennerhag, D. Rao, R. Pascanu, and R. Hadsell, “Probing Transfer in Deep Reinforcement Learning without Task Engineering,” in _Conference on Lifelong Learning Agents_ , 2022.
  * [57] J. Farebrother, M. C. Machado, and M. Bowling, “Generalization and Regularization in DQN,” _arXiv preprint arXiv:1810.00123_ , 2018.
  * [58] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. J. Hausknecht, and M. Bowling, “Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents,” _Journal of Artificial Intelligence Research_ , vol. 61, pp. 523–562, 2018.
  * [59] M. White, “Unifying Task Specification in Reinforcement Learning,” in _International Conference on Machine Learning_ , 2017.
  * [60] L. J. Lin, “Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching,” _Machine Learning_ , 1992.
  * [61] V. Nair and G. E. Hinton, “Rectified Linear Units Improve Restricted Boltzmann Machines,” in _International Conference on Machine Learning_ , 2010.
  * [62] J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh, “Action-Conditional Video Prediction Using Deep Networks in Atari Games,” in _Advances in Neural Information Processing Systems_ , 2015.
  * [63] T. Lesort, N. Díaz-Rodríguez, J.-F. Goudou, and D. Filliat, “State representation learning for control: An overview,” _Neural Networks_ , vol. 108, pp. 379–392, 2018.
  * [64] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-Driven Exploration by Self-Supervised Prediction,” in _International Conference on Machine Learning_ , 2017.
  * [65] W. Chung, S. Nath, A. Joseph, and M. White, “Two-Timescale Networks for Nonlinear Value Function Approximation,” in _International Conference on Learning Representations_ , 2019.
  * [66] M. C. Machado, C. Rosenbaum, X. Guo, M. Liu, G. Tesauro, and M. Campbell, “Eigenoption Discovery through the Deep Successor Representation,” in _International Conference on Learning Representations_ , 2018.
  * [67] R. Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. L. Littman, “An Analysis of Linear Models, Linear Value-Function Approximation, and Feature Selection for Reinforcement Learning,” in _International Conference on Machine Learning_ , 2008.
  * [68] C. Szepesvári, _Algorithms for Reinforcement Learning_. Morgan and Claypool, 2010.
  * [69] M. C. Machado, M. G. Bellemare, and M. Bowling, “Count-Based Exploration with the Successor Representation,” in _AAAI Conference on Artificial Intelligence_ , 2020.
  * [70] P. Dayan, “Improving Generalization for Temporal Difference Learning: The Successor Representation,” _Neural Computation_ , 1993.
  * [71] A. Stooke, K. Lee, P. Abbeel, and M. Laskin, “Decoupling Representation Learning from Reinforcement Learning,” in _International Conference on Machine Learning_ , 2021.
  * [72] R. Agarwal, M. C. Machado, P. S. Castro, and M. G. Bellemare, “Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning,” in _International Conference on Learning Representations_ , 2021\.
  * [73] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in _International Conference on Learning Representations_ , 2015.
  * [74] Y. Liang, M. C. Machado, E. Talvitie, and M. H. Bowling, “State of the Art Control of Atari Games Using Shallow Reinforcement Learning,” in _International Conference on Autonomous Agents & Multiagent Systems_, 2016\.
  * [75] E. Lecarpentier, D. Abel, K. Asadi, Y. Jinnai, E. Rachelson, and M. L. Littman, “Lipschitz Lifelong Reinforcement Learning,” in _AAAI Conference on Artificial Intelligence_ , 2021.
  * [76] A.-m. Farahmand, A. Barreto, and D. Nikovski, “Value-Aware Loss Function for Model-Based Reinforcement Learning,” in _Artificial Intelligence and Statistics_ , 2017.
  * [77] K. L. Stachenfeld, M. Botvinick, and S. J. Gershman, “Design Principles of the Hippocampal Cognitive Map,” in _Advances in Neural Information Processing Systems_ , 2014.
  * [78] V. Liu, A. White, H. Yao, and M. White, “Towards a Practical Measure of Interference for Reinforcement Learning,” _arXiv preprint arXiv:2007.03807_ , 2020.
  * [79] Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling Network Architectures for Deep Reinforcement Learning,” in _International Conference on Machine Learning_ , 2016.
  * [80] M. G. Bellemare, W. Dabney, and R. Munos, “A Distributional Perspective on Reinforcement Learning,” in _International Conference on Machine Learning_ , 2017.
  * [81] C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song, “Assessing Generalization in Deep Reinforcement Learning,” _arXiv preprint arXiv:1810.12282_ , 2018.
  * [82] A. Rajeswaran, K. Lowrey, E. V. Todorov, and S. M. Kakade, “Towards Generalization and Simplicity in Continuous Control,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [83] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman, “Quantifying Generalization in Reinforcement Learning,” in _International Conference on Machine Learning_ , 2019.
  * [84] H. van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning with Double Q-Learning,” in _AAAI Conference on Artificial Intelligence_ , 2016\.
  * [85] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized Experience Replay,” in _International Conference on Learning Representations_ , 2016\.
  * [86] M. Fortunato, M. G. Azar, B. Piot, J. Menick, M. Hessel, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg, “Noisy Networks For Exploration,” in _International Conference on Learning Representations_ , 2018.
  * [87] W. Dabney, M. Rowland, M. Bellemare, and R. Munos, “Distributional Reinforcement Learning with Quantile Regression,” in _AAAI Conference on Artificial Intelligence_ , 2018.

|  Han Wang is a PhD student at the University of Alberta. She received her
MSc in Computing Science from the University of Alberta, in 2020. Her research
area of interest is reinforcement learning.
---|---
|  Erfan Miahi is an MSc student at the University of Alberta. He received his
BSc in Computer Engineering from the University of Guilan in 2020. Erfan has
diverse research experience in the field of artificial intelligence, ranging
from developing deep learning algorithms for medical image analysis to
fundamental research in reinforcement learning and evolutionary optimization.
---|---
|  Martha White is an Associate Professor of Computing Science at the
University of Alberta and a PI of Amii (the Alberta Machine Intelligence
Institute). She holds a Canada CIFAR AI Chair and received IEEE’s “AIs 10 to
Watch: The Future of AI" award in 2020. She has authored more than 50 papers
in top journals and conferences and is an associate editor for TPAMI.
---|---
|  Marlos C. Machado is a senior research scientist at DeepMind and holds a
Canada CIFAR AI Chair at the Alberta Machine Intelligence Institute (Amii).
Marlos is also an adjunct professor at the University of Alberta, an Amii
Fellow, and a principal investigator of the Reinforcement Learning and
Artificial Intelligence Laboratory (RLAI).
---|---
|  Zaheer Abbas is a Research Engineer at Deep Mind. He received his MSc in
computing science from the University of Alberta in 2019. His research focus
is reinforcement learning and planning.
---|---
|  Raksha Kumaraswamy completed this work as PhD student in Computing Science
at the University of Alberta. She is now a Research Scientist in Noah’s Ark
Labs in Huawei.
---|---
|  Vincent Liu is a PhD student at the University of Alberta. He received his
MSc in Computing Science from the University of Alberta, in 2020. His research
focus is sequential decision making in the real world, particularly
theoretically-sound methods for offline reinforcement learning.
---|---
|  Adam White is an Assistant Professor of Computing Science at the University
of Alberta and a Staff Research Scientist at DeepMind Alberta. He holds a
Canada CIFAR AI Chair, is an Amii Fellow, and a Principle Investigator of the
Reinforcement Learning and Artificial Intelligence Laboratory (RLAI).
---|---

##  Appendix A More on Some Representation Properties

Interestingly, orthogonality in the representations, as measured in the paper,
has a strong relationship with three other properties. Specifically,
orthogonality in the representations can be the equivalent to orthogonality
between features, which implies that each feature captures distinct
information from the input states. Orthogonality can also result in
interference, where interference measures how much an update on one state
interferes with the updates on other states. Lastly, orthogonal features can
be indicators of linearly uncorrelated features.

In this section, we show the aformentioned relationships between orthogonal
representations, orthogonal features, interference, and uncorrelated features.
To do so, we make the assumption of having a finite set of input-states
$\\{s_{1},s_{2},\dots,s_{n}\\}$ represented in a $d$-dimensional space, where
the corresponding representations are
$\phi(s_{i})=[f_{1}(s_{i}),f_{2}(s_{i}),\dots,f_{d}(s_{i})]^{\top}$ ($f_{l}$
being a function to produce the $l$-th feature dimension), and features are
$\psi(f_{i})=[f_{i}(s_{1}),f_{i}(s_{2}),\dots,f_{i}(s_{n})]^{\top}$.

###  A.1 Relationship between Orthogonal Representations and Orthogonal
Features

Here, we show that there is an equivalence relationship between orthogonality
in the representation and between features. Below we show that
$\sum_{i,j}^{n}(\phi(s_{i})^{\top}\phi(s_{j}))^{2}=\sum_{k,l}^{d}(\psi(f_{k})^{\top}\psi(f_{l}))^{2}$.

$\displaystyle\sum_{i,j}^{n}(\phi(s_{i})^{\top}\phi(s_{j}))^{2}=$
$\displaystyle\sum_{i,j}^{n}(\phi(s_{i})^{\top}\phi(s_{j}))(\phi(s_{i})^{\top}\phi(s_{j}))$
$\displaystyle=$
$\displaystyle\sum_{i,j}^{n}\sum_{k=1}^{d}f_{k}(s_{i})f_{k}(s_{j})\sum_{l=1}^{d}f_{l}(s_{i})f_{l}(s_{j})$
$\displaystyle=$
$\displaystyle\sum_{i,j}^{n}\sum_{k,l}^{d}f_{k}(s_{i})f_{l}(s_{i})f_{k}(s_{j})f_{l}(s_{j})$
$\displaystyle=$
$\displaystyle\sum_{k,l}^{d}\sum_{i,j}^{n}f_{k}(s_{i})f_{l}(s_{i})f_{k}(s_{j})f_{l}(s_{j})$
$\displaystyle=$
$\displaystyle\sum_{k,l}^{d}\sum_{i=1}^{n}f_{k}(s_{i})f_{l}(s_{i})\sum_{j=1}^{n}f_{k}(s_{j})f_{l}(s_{j})$
$\displaystyle=$
$\displaystyle\sum_{k,l}^{d}(\psi(f_{k})^{\top}\psi(f_{l}))(\psi(f_{k})^{\top}\psi(f_{l}))$
$\displaystyle=$
$\displaystyle\sum_{k,l}^{d}(\psi(f_{k})^{\top}\psi(f_{l}))^{2}$

Therefore, when the sample-space is not enumerable, that is $f_{i}$ is
infinite-dimensional, orthogonality of representations may be used as a
surrogate for measuring the orthogonality of features.

###  A.2 Relationship between Orthogonal Representations and Interference

Under linear updating, orthogonal representations would reduce interference.
This is because, if two states $s_{1}$ and $s_{2}$ have
$\phi(s_{1})^{\top}\phi(s_{2})=0$, then performing an update in $s_{1}$,
$\tilde{w}=w+\alpha\delta\phi(s_{1})$ has no impact on the prediction in
$s_{2}$:
$\phi(s_{2})^{\top}\tilde{w}=\phi(s_{2})^{\top}(w+\alpha\delta\phi(s_{1}))=\phi(s_{2})^{\top}w+\alpha\delta\phi(s_{2})^{\top}\phi(s_{1})=\phi(s_{2})^{\top}w$.

###  A.3 Relationship between Orthogonal and Uncorrelated Features

Here, we show the relationship between orthogonal features and uncorrelated
features. Let
$\bar{\psi}(f_{i})=[\bar{f}_{i},\bar{f}_{i},\dots,\bar{f}_{i}]^{\top}$, where
$\bar{f}_{i}=\frac{1}{n}\sum_{j=1}^{n}[f_{i}(s_{j})]$, denote the expected
value of feature $i$ over the set of input-states. If all features are
centered, that is, $\bar{f}_{i}=0$ for all $i$, then it is trivial to see that

$\displaystyle\frac{1}{n^{2}}\sum_{k,l}^{d}(\psi(f_{k})-\bar{\psi}(f_{k}))^{\top}(\psi(f_{l})-\bar{\psi}(f_{l}))$
$\displaystyle=\frac{1}{n^{2}}\sum_{k,l}^{d}\psi(f_{k})^{\top}\psi(f_{l}).$

The LHS is a measure of correlation and the RHS is a measure of orthogonality.

##  Appendix B Empirical Details

###  B.1 Auxiliary Tasks

TABLE I: Auxiliary losses used in this paper, defined with respect to transitions in $\mathcal{D}$. ATC |  $\mathcal{L}_{ATC}(\mathcal{D})=-\mathbb{E}_{\mathcal{S}}\left[\log\frac{\exp l_{i,i+k}}{\sum_{s_{j}\in\mathcal{S}}\exp l_{i;j+k}}\right],\hskip 2.84544pt\text{where $l_{t;k}$ are logits computed bilinearly such that $l_{i,j+k}=p_{i}Wc_{j+k}$ }$ $\text{and}\hskip 2.84544ptp_{t}=F_{\theta_{C}}(F_{\theta_{A}}(\Phi_{\theta_{R}}(\text{AUG}(s_{t}))))+F_{\theta_{A}}(\Phi_{\theta_{R}}(\text{AUG}(s_{t}))),\hskip 2.84544ptc_{t+k}=F_{\breve{\theta}_{A}}(\phi_{\breve{\theta}_{R}}(\text{AUG}(s_{t+k})))$
---|---
IR |  $\mathcal{L}_{\text{IR}}\left(\mathcal{D}\right)=\underset{s\sim\mathcal{D}}{\mathbb{E}}\left[\|F_{\theta_{A}}(\Phi_{\theta_{R}}(s))-s\|_{2}^{2}\right]$
NAS |  $\begin{gathered}\scalebox{0.9}{ $\mathcal{L}_{\text{NAS}}\left(\mathcal{D}\right)=\underset{\begin{subarray}{c}(s_{t},a_{t},s_{t+1})\sim\mathcal{D}\\\ (s_{k\neq t},a_{k\neq t})\sim\mathcal{D}\end{subarray}}{\mathbb{E}}\Big{[}\|F_{\theta_{A}}(\Phi_{\theta_{R}}(s_{t}),a_{t})-(\Phi_{\theta_{R}}(s_{t+1})-\Phi_{\theta_{R}}(s_{t}))\|_{2}^{2}+\max\left(0,1-\|F_{\theta_{A}}(\Phi_{\theta_{R}}(s_{k}),s_{k})-(\Phi_{\theta_{R}}(s_{t+1})-\Phi_{\theta_{R}}(s_{t}))\|_{2}^{2}\right)\Big{]}$}\end{gathered}$
Reward |  $\begin{gathered}\mathcal{L}_{\text{Reward}}\left(\mathcal{D}\right)=\underset{(s_{t},a_{t},r_{t+1})\sim\mathcal{D}}{\mathbb{E}}\left[\|F_{\theta_{A}}(\Phi_{\theta_{R}}(s_{t}),a_{t})-r_{t+1}\|_{2}^{2}\right]\end{gathered}$
SF |  $\mathcal{L}_{\text{SF}}\left(\mathcal{D}\right)=\underset{(s_{t},a_{t},s_{t+1},a_{t+1})\sim\mathcal{D}}{\mathbb{E}}\left[\|F_{\theta_{A}}(\Phi_{\theta_{R}}(s_{t}),a_{t})-(\Phi_{\theta_{R}}(s_{t})+\gamma F_{\hat{\theta}_{A}}(\Phi_{\hat{\theta}_{R}}(s_{t+1}),a_{t+1}))\|_{2}^{2}\right]$
VirtualVF |  $\mathcal{L}_{\text{VirtualVF}}\left(\mathcal{D}\right)=\sum_{g\in\mathcal{G}}\underset{(s_{t},a_{t},r_{t+1},s_{t+1},a_{t+1})\sim\mathcal{D}}{\mathbb{E}}\left[\|F^{g}_{\theta_{A}}(\Phi_{\theta_{R}}(s_{t}),a_{t})-(r^{g}_{t}+\gamma F^{g}_{\hat{\theta}_{A}}(\Phi_{\hat{\theta}_{R}}(s_{t+1}),a_{t+1})\|_{2}^{2}\right]$
XY |  $\mathcal{L}_{\text{XY}}\left(\mathcal{D}\right)=\underset{(s_{t},x_{t},y_{t})\sim\mathcal{D}}{\mathbb{E}}\left[\|F_{\theta_{A}}(\Phi_{\theta_{R}}(s_{t}))-(x_{t},y_{t})\|_{2}^{2}\right]$ $\mathcal{G}$ is a set of goal locations. The reward $r_{g}$ and representation function $\phi^{g}_{\theta_{A}}$ are associated with the goal $g$, sampled from this set.

In this section, we provide a more detailed explanation of each of the seven
auxiliary tasks used for helping with representation learning. All the losses
use the same samples taken from the replay buffer to update the value
function, except the ATC loss. The detailed formulas for the auxiliary losses
are presented in Table I.

Augmented Temporal Contrast (ATC) Loss: This contrastive loss encourages the
network to learn similar representations for an input state, $s_{t}$, with one
from a pre-determined, near-future time step input state, $s_{t+k}$, where
$k=3$. In contrast to other losses that we have employed in this work, this
loss uses more than a single auxiliary head to compute itself. To do so, it
first applies a data augmentation technique called random shift with a
probability of $0.1$ and padding of $4$ to both of these input states. Then,
it feeds the augmented version of $s_{t}$, $\text{AUG}(s_{t})$, through a set
of networks to compute $p_{t}$, where $F_{\theta_{A}}$ is a linear mapping of
representation into an embedding space with a size of $32$, and
$F_{\theta_{C}}$ is a single layer neural network with a hidden layer size of
$64$, and an output size of $32$. Then, $\text{AUG}(s_{t+k})$ is fed into a
momentum encoder of $\Phi_{\theta_{R}}$ ($\phi_{\breve{\theta}_{R}}$) and
$F_{\theta_{A}}$ ($F_{\breve{\theta}_{A}}$) to compute $c_{t+k}$. The output
of these networks, $p_{t}$ and $c_{t+k}$, are combined with each other through
a $32\times 32$ matrix called $W$ to compute logits, $l{i,j+k}$. At last,
these logits are used to compute the InfoNCE loss, $\mathcal{L}_{ATC}$. In
contrast to the original paper that uses a complex learning-rate scheduling
technique, we implement this loss in its simplest form by using a fixed
learning-rate. We sweep through values of
$[0.003,0.001,0.0003,0.0001,0.00003,0.00001]$ of this learning-rate. We update
the momentum encoder in every step using a $\tau$ of $0.01$. The batch size is
the same as the batch size used for computing the value function loss, and the
weight of this auxiliary loss is set to $1$.

Input Reconstruction (IR): This auxiliary task reconstructs the input image
from the representation through a deconvolutional network. On the auxiliary
head, the representation was firstly projected to a hidden layer with $1024$
nodes, then sent to a two layer deconvolutional neural network, with $4$
kernels, $32$ and $3$ channels, $2$ and $1$ strides, $2$ and $1$ pads on $2$
layers separately. The output had the same size as the input image (i.e.,
$15\times 15\times 3$). The weight of the auxiliary loss was set to $0.0001$.

Next Agent State Prediction (NAS): This task motivates the representation of
the current state to minimize its prediction error on the next state
representation. To do so, it uses a contrastive loss that minimizes the
prediction error on the next state while maximizes the prediction error on the
rest of the states. For this task, the auxiliary head, $F_{\theta_{A}}$,
consists of two fully connected layers with $64$ neurons on each. The weight
of this auxiliary loss was $0.001$.

Successor Feature Prediction (SF): Successor feature prediction task was
similar to next-agent-state prediction, though the target was constructed by
bootstrapping. Given the transition on latent space
$<\phi_{t},a_{t},\phi_{t+1},a_{t+1}>$, the auxiliary head learned to minimize
the difference between the prediction $F_{\theta_{A}}(\phi_{t},a_{t})$ and the
target $(1-\lambda)\phi_{t+1}+\lambda F_{\theta_{A}}(\phi_{t+1},a_{t+1})$,
where $\lambda$ is set to $0.99$. To satisfy the property of successor
features, we added an extra head to predict the reward linearly from the
representation $\phi_{t}$. This head for predicting the successor feature used
the same neural network architecture as NAS. The weight was set to $1$.

Reward Prediction (Reward): Another auxiliary prediction task was to predict
the reward independently given the representation $\phi_{t}$. This auxiliary
task used the same non-linear transformation structure as SF and the weight is
set to $1$.

Expert Target Prediction (XY): The last prediction task was expert-designed
targets prediction. The agent was asked to predict its current position given
the image. Since predicting the position was considered a regression task, we
used MSE loss with the same network structure as Reward. We set a low weight
of $0.0001$ for this auxiliary task.

Virtual Value Function Learning (VirtualVF): This auxiliary task learns a
different value function on the auxiliary head. There were $2$ settings in
Maze—learning a value function assuming the goal is on grid $[7,7]$, and
learning $5$ value functions when the goals are on grid $[0,0]$, $[0,14]$,
$[14,0]$, $[14,14]$, $[7,7]$ separately. The weight of this auxiliary task
remained $1$ but the discount rate on the auxiliary head was set to be lower,
$0.9$, so that the agent can focus on the main task. The auxiliary head
learned this task with the same network structure as XY.

###  B.2 General Hyperparameter Setting

We used the same neural network architecture across representations learned
with the various loss functions, for each domain. All hidden layers are
initialized with Xavier. Table II shows the number of nodes on the
representation function’s last hidden layer, and the number of features.

TABLE II: Representation size settings Label | Activation | Last Hidden Layer Nodes | Features
---|---|---|---
ReLU | ReLU | 32 | 32
ReLU(L) | ReLU | 640 | 640
FTA | FTA | 32 | 640

During training, the inputs are normalized to be in the range $[-1,1]$. We use
Adam optimizer to update weights, and we used the mean-squared error as the
loss. The batch size is set to be $32$. The buffer has length $10,000$. The
input image is normalized. For the representation function, we use a two layer
convolutional network with kernel size of $4$, stride of $1$, padding of $1$,
and $32$ channels for the first layer; kernel size of $4$, stride of $2$,
padding of $2$, and $16$ channels on the second layer. A target network was
used with the synchronization frequency set to $64$. The buffer’s memory size
was $100,000$ and there were $32$ samples randomly chosen at each step. The
agent learns for $300,000$ steps with $\epsilon$-greedy policy. During
transfer learning, all agents, including baselines, learned for $100,000$
steps only.

As for the FTA setting, we use $20$ bins with the higher and lower bounds
equal to $2$ and $-2$. We tested $\eta$ of $0.2,0.4,0.6$, and $0.8$ for the no
auxiliary task agent, and we fixed $\eta=0.2$ for agents trained with
auxiliary task.

The learning rate was swept for every representation learning architecture and
control task. The best setting was picked according to the averaged
performance over $5$ runs. Each run uses a different random seed. We tested
the learning rate in $[0.001,0.0003,0.0001,0.00003,0.00001]$ during the
representations training step and for learning from scratch baseline agents.
During transfer, we tested $[0.01,0.003,0.001,0.0003,0.0001]$ for
representations with $32$ features and $[0.001,0.0003,0.0001,0.00003,0.00001]$
for all representations with $640$ features, as there were more weights to
train, which required more careful learning.

In the non-linear value function case and transfer tasks with linear value
function, we use a fixed $\epsilon=0.1$. However, in representation learning
in the linear case, it turned out to be harder for the agent to converge when
keeping other settings as the same as in the non-linear value function. Thus,
we provided a better exploration in the early learning stage to speed up
learning by decreasing $\epsilon$, which decreases from $1$ to $0.1$ in the
first $100,000$ steps.

###  B.3 Using SFs to Measure Task Similarity

When checking how the difficulty of transfer affects the transfer performance,
we consider the similarity between each transfer task to the original task.
When a transfer task is similar to the task in which the representation is
trained, we consider the transfer to be easier.

We measure the similarity between tasks according to the successor
representations, $\psi$. Since successor representations encode the
trajectories of the agent, the difference between successor representations
generated by optimal policy can reflect the difference between optimal
policies that the agent learns in different tasks. If the optimal policies of
two tasks turn out to be dissimilar, the similarity between these tasks is
considered low.

To compute a highly accurate estimate of successor features, we solve the maze
by using a simple tabular algorithm. To do so, we define the state as the cell
in the maze that is occupied by the agent. Doing so results in having $173$
states in total. Taking this into account, we use value iteration to generate
an optimal policy, then calculate the successor representation of each state
based on the optimal policy.

The successor representations of all states in the same task are considered.
The successor representation of each task, $\Psi$, is obtained by
concatenating all successor features in the same task. The similarity is
defined as the dot product between $\Psi$’s in the transfer and the original
tasks. We choose the dot product to keep both angle and magnitude information
between concatenated successor representations. A higher dot product value
means the transfer task is more similar to the original task and vice versa.

$\displaystyle\psi(s,task_{x})$
$\displaystyle=\mathbb{E}_{\pi^{*}_{task_{x}}}\left[\sum^{T}_{t=0}\gamma^{t}f(S_{t})|S_{0}=s\right]$
$\displaystyle\Psi_{task_{x}}$
$\displaystyle=\left[\psi(s_{0})\;\psi(s_{1})\;\cdots\;\psi(s_{|\mathcal{S}|})\right]$
$\displaystyle similarity(task_{x},task_{y})$
$\displaystyle=\Psi_{task_{x}}\cdot\Psi_{task_{y}}$

Interestingly, the goal states that are more distant from each other become
more dissimilar by computing the similarity this way. This is more clear when
we take a look at the similarity rankings of the goal states, as depicted in
Figure 11. As shown in Figure 3, the representations have a hard time
transferring to higher-ranking goals, so there is a clear connection between
the ranks of the transfer tasks and the transfer performance. These findings
support the use of this approach for calculating task similarity and ranking.

Figure 11: Transfer tasks are ranked by the similarity between successor
features of each task. This figure shows the similarity ranking of different
tasks compared to the source task, where the source task is marked by O. Each
number in the cell indicates the similarity rank of the task when the goal is
moved to that specific position.

ptReturnperEpisode

ptTime Steps ($10^{4}$, logscale)

Figure 12: Return converges in the representation training step. The plot
shows the averaged return over the most recent 100 episodes at each
checkpoint. The x-axis is the number of time steps and the y-axis is the
average return. Each curve represents one agent specification (activation and
auxiliary task pair). As our main focus is not to compare the learning
efficiency during the representation learning step, and the difference between
learning curves is not large, we only show the general trend by plotting every
curve with the same color. The curve changes color to black, at the time point
where we took the representation and fixed it.

##  Appendix C Additional Results

###  C.1 Representation Training

We show the learning curve of all representation learning architectures in
Figure 12, to show that the early-saved representations have converged when
they are saved.

###  C.2 Larger ReLU Transfer

ptTotalreward intransfer,averagedover5 runs Transfer Task Figure 13:  Transfer
performance of larger ReLU representations (45 in total) on 173 transfer
tasks. The tasks on the x-axis are arranged by similarity to training tasks:
on the left (small x-values) being most similar and on the right (large
x-values) being most dissimilar. The black line shows the performance when
learning in each transfer task from scratch, with the same representation size
as ReLU(L). Lines completely above the black line indicate a representation
yielded successful transfer in all tasks. Lines that start above the black
line but fall below it as we move left to right indicate a representation that
transfers to similar tasks but not dissimilar tasks.

We show the transfer performance of ReLU(L) in Figure 13. The ReLU(L) setting
stays between ReLU and FTA representations: ReLU(L) keeps the same activation
function as ReLU representation, but increases the size of the representation
layer to 640, which is the same as the size of FTA representations. Therefore,
it maintains the same value function capacity as the FTA representations. The
pattern in the transfer performance of ReLU(L) is similar to ReLU (Figure 3).
As the transfer tasks become dissimilar, the transfer performance drops below
the Scratch agent. In general, when considering the total reward obtained by
the agent, the performance of ReLU(L) is better than ReLU and worse than FTA.

###  C.3 Transfer with Different Architectures

Figure 14 shows the 95% confidence interval of transfer performance with
different representation sizes, activation functions, and different auxiliary
tasks, with a non-linear value function.

ptTotal reward, averaged over transfer tasks pt

ptReLU(32)

ptReLU(640)

ptFTA

Figure 14: Transfer performance depends on the activation function,
representation size, and auxiliary tasks. This plot presents the same data as
Figure 4, but the error bar shows a 95% confidence interval. The bar shows the
mean value over 5 seeds $\times$ 173 transfer tasks.

###  C.4 Relationship between Properties

Figure 15: There exists strong positive correlation between diversity and
complexity reduction. Meanwhile, when diversity is high, the increment of
diversity comes with an increment of orthogonality. The above 2 subplots are
highlighted with orange color. Each subplot shows the relationship between a
pair of properties. Each scatter refers to one representation. The x and y
coordinates are property measures. The properties’ names are shown as labels
on each axis. The correlations (Cor) are reported above each subplt.

We also checked the relationship between properties. The result is shown in
Figure 15. Two subplots are highlighted with the orange color. We noticed
diversity and complexity reduction showed strong positive linear correlation.
This suggests that monitoring either diversity or complexity reduction should
be informative to predict the dissimilar task transfer performance in
practice. Furthermore, a threshold exists when looking at diversity and
orthogonality, as well as diversity and complexity reduction. For
representations with higher diversity (higher than 0.5, in this case), it also
showed higher orthogonality, while this pattern does not exist in low
diversity representations. Although there exists several outliers, this still
indicates the possibility that pursuing a representation with high
orthogonality may result in a relatively high diversity at the same time in
practice.

##  Appendix D Experimental Details for Atari

We used Rainbow-DQN with the default hyper-parameter setting provided by
Hessel et al. [54]. The rainbow agent combines DQN [45] double Q-learning
[84], prioritized experience replay with the proportional variant [85],
dueling networks [79], noisy networks for exploration [86], and distributional
RL [80]. The agent is similar to the Rainbow-IQN agent used in the recent work
showing transfer across modes [56] except that it uses C51 for distributional
RL [80] instead of IQN [87].

We interpret the convolutional layers of the network of the Rainbow agent as
the representation network and the successive fully connecting layers as the
value network. The architectural details are worth mentioning here. The
representation network consists of three convolution layers each followed by
ReLU activations, and produces a representation of 3136 features, as a
function of stacked input frames (see Hessel et al. [54] for the number of
kernels and stride configuration). The value-network consists of separate
fully-connected networks that estimate value and advantage [79]. The value
network takes as input the output of the representation network and the
outputs of the value network are combined to estimate the value distribution
[80]. The fully-connected networks that estimate value and advantage have a
single hidden layer of 512 units, with noisy linear networks [86] and ReLU
activations.

##  Appendix E Additional Complexity Results in Atari

TABLE III: Unnormalized complexity (L_rep) of the learned and random Rainbow representations in Freeway, Space Invaders, and Breakout (average of 5 representations) Game | Learned Representation | Random Representation
---|---|---
Freeway | 0.029 | 4.178
Space Invaders | 0.228 | 6.908
Breakout | 0.229 | 6.034