that this high number of dead units is due to the representation being too
general for the task of manipulating relatively homogenous objects. Another
drawback of using pretrained networks is that information might be encoded
that is unimportant for the task. This has the effect that our search method
is not guaranteed to output the correct solution in the latent space, as there
might be distracting pockets of local minima.

The random baseline has an average success rate of 3.5$\%$, which is very
clearly outperformed by our method. As 64 is the best dimensionality for the
representation on average, we continue with that number for our method in the
transfer learning experiment.

We conclude that the proposed method of training a representation for
predictability is feasible. So far we have evidence that 64 is the best
dimensionality of the representation’s latent space for our planning tasks.
However, it is not yet conclusive what the best restriction is to place on the
representation to avoid the trivial solution, in terms of planning
performance.

####  4.4.3 Comparison with other RL methods

Now we divert our attention to research question 5 (benchmarking), where we
compare our method to the literature. For the comparison with standard RL
methods, we use the default configurations of the model-free methods DQN and
PPO as defined in OpenAI Baselines (Dhariwal et al.,, 2017). Our model-based
comparison is chosen to be world models (Ha and Schmidhuber,, 2018) from
https://github.com/zacwellmer/WorldModels. We make sure that the compared RL
methods are similar to our system in terms of the number of parameters as well
as architecture layout and compare them with our method on the car viewpoint-
matching task.

The task setup is the same as before and is converted to an OpenAI gym
environment: a start observation and a goal observation are passed to the
agent. If the agent manages to reach it within 2 times the minimum number of
actions required (the minimum number is calculated by the environment), the
agent receives a reward and the task is considered a success. Otherwise, no
reward is given.

The results of the comparison can be viewed in Fig. 4.10. Each point in the
curve contains each method’s mean success rate: the average of the cumulative
reward from 100 test episodes from 5 different instantiations of the RL
learner, so it is the average reward over 500 episodes in total. The test
episodes are done on the same environment as is used for training, except that
the policy is maximally exploiting and minimally exploring.

(a)

(b)

(c)

Fig 4.10: Reinforcement learning comparison. The vertical dashed lines
indicate when the compared algorithm has processed the same number of
transitions as our method and the horizontal dotted line indicates the test
performance of our method. Each data point is the mean success rate of 100
test episodes after a varying number of training steps, averaged over 5
different seeds of each learner. The model-free methods in (a) and (b) train
the representation and the controller simultaneously by acting in the
environment and collecting new experiences. The representation in (c) is
trained on 25.6k transitions, which is the same number we use. The plot shows
the optimization curve for the controller, using a Covariance-Matrix
Adaptation Evolution Strategy, which hardly improves after 500 or so training
steps. The horizontal line starts at 0 for world models because the
representation has finished training on the observations before the controller
is optimized.

In our experiments, the DQN networks are much more sample inefficient than
PPO, which in turn is more sample inefficient than our method. However, our
method is more time-consuming during test time. We require a forward pass of
the predictor network for each node that is searched before we take the next
step, which can grow rapidly if the target is far away. In contrast, only a
single pass through the traditional RL networks is required to compute the
next action.

Our method reaches $93.9\%$ success rate on the train car (Table 4.4) using
25.6k samples, but the best PPO run only reaches $70.5\%$ after training on
the same number of samples. The best single PPO run needed 41.3k samples to
get higher than $93.9\%$ success rate, and the average performance is higher
than $93.9\%$ at around 55k samples. After that, some PPO learners declined
again in performance. The world models policy quickly reaches the same level
of performance as DQN got after 50k steps and PPO after approximately 8k
steps, but it doesn’t improve beyond that.

(a)

(b)

Fig 4.11: Re-training after placing obstacles in a checkerboard pattern. (a)
The task is the same as before, but nothing happens if the agent attempts to
move to a state containing a black rectangle. (b) After training the agents,
we re-tested them after we introduced the checkerboard pattern of obstacles.
Our method does not allow for re-training in the new environment.

We conclude that our method compares favorably to other methods from the RL
literature in terms of data-efficiency.

####  4.4.4 Modifying the environment

We now modify the environment to answer research question 6 (generalization).
To see how the methods compare when obstacles are introduced to the
environment, we repeat the trial on one of the car objects except that the
agent can no longer pass through states whose elevation values are divisible
by 10 and azimuth values are divisible by 40 (Fig. 4.11, (a)).

As before, the goal and start locations can have any azimuth-elevation pair,
but the agent cannot move into states with the properties indicated by the
black rectangle. Every action is available to the agent at all locations as
before, but the agent’s state is unchanged if it attempts to move to a state
with a black rectangle.

We trained LARP using the contrastive loss, PPO, and DQN agents until they
reached $80\%$ accuracy on our planning task and then tested them with the
added obstacles. Our method loses about $10\%$ performance, but PPO loses
$50\%$. Nevertheless, we can continue training PPO until it quickly reaches
top performance again (Fig. 4.11, (b)). Our method is not re-trained for the
new task, and DQN did not reach a good performance again in the time we
allotted for re-training.

Thus, we see that our method is quite flexible and generalizes well when
obstacles are introduced to the environment.

####  4.4.5 Transfer to dissimilar objects

We now consider research question 6 (generalization) further by investigating
how well our method transfers knowledge from one domain to another.

Selecting the best dimensionality for the representation from the previous set
of experiments, we investigate further their performance in harder situations
using unseen, non-car objects. The models are trained on the same car objects
as in the previous experiment, but they are tested on an array of different
plastic soldiers: a kneeling soldier holding a bazooka, a standing soldier
with a rifle, a Native American with a bow and spear and a cowboy with a rifle
(Fig. 4.12).

Fig 4.12:  Toys for transfer learning experiments. From left to right: Soldier
(Kneeling), Soldier (Standing), Native American with Bow and Cowboy with
Rifle.

##### Qualitative results

We offer a visualization of the learned representation of the kneeling soldier
toy using our method, a convolutional encoder, and VGG16 in Fig. 4.13. Each
embedding was reduced to 2 dimensions using t-SNE.

Every method structures the domain similarly. In the bottom row, we see that
the largest clusters for all methods are the ones with the highest (teal dots)
illumination settings, which is explained by the effect of the lighting on the
pixel value intensities. Within these clusters, we see clustering based on the
azimuth (middle row). Finally, within these clusters, there is a gradient
structure based on elevation (top row). This is due to the elevation changing
in smaller step-sizes, with 5 degree differences, than azimuth with 20 degree
differences.

(a) LARP elevation t-SNE

(b) CAE elevation t-SNE

(c) VGG16 elevation t-SNE

(d) LARP azimuth t-SNE

(e) CAE azimuth t-SNE

(f) VGG16 azimuth t-SNE

(g) LARP lighting t-SNE

(h) CAE lighting t-SNE

(i) VGG16 lighting t-SNE

Fig 4.13:  Latent space visualization. The clustering structure of a toy
soldier embedding after dimensionality reduction with t-SNE. The top row
colors the samples by elevation, the middle row by azimuth and the bottom one
by lighting. The left column is the embedding given by our method using the
contrastive loss term, the middle by a convolutional autoencoder (CAE), and
the right by a pretrained VGG16 network.

##### Quantitative results

Our experiments include the pretrained VGG16 network because we believe that a
flexible algorithm should rather be based on a generic multi-purpose-
representation and not on a specific representation. To our surprise, it is
outperformed by our approach on every object, even as the viewpoint-matching
task is done on objects that are different from the original car toys they are
trained on.

We also try transfer learning a VAE and a convolutional encoder using the same
network as our representation and a UMAP embedding, with 64 features each. We
display the results of the same viewpoint-matching tasks as before using these
different representations on the unseen army figures in Table 4.5.
Convolutional encoders have the lowest performance out of all the methods, but
the VAE representation has the most similar performance to our own
representations.

Table 4.5:  Transfer learning performance. The methods are trained on the data set of different car images, and then their performance on dissimilar toys is tested. Each number is the mean success rate of viewpoint matching out of 1000 trials. Representation | Soldier | Soldier | Native American | Cowboy |
---|---|---|---|---|---
| (Kneeling) | (Standing) | with Bow | with Rifle | Mean
LARP (Contrastive) | 22.7 | 11.1 | 12.1 | 20.7 | 16.7
LARP (Sphering) | 18.4 | 15.3 | 15.2 | 15.8 | 16.2
LARP (Decoder) | 14.3 | 16.6 | 14.7 | 15.6 | 15.3
VAE | 14.4 | 13.1 | 13.6 | 13.8 | 13.7
VGG16 | 12.3 | 12.9 | 11.8 | 13.6 | 12.7
UMAP | 8.6 | 10.8 | 9.7 | 11.1 | 10.1
Conv. Encoder | 4.9 | 14.3 | 11.5 | 6.4 | 9.3

Our representation is better than the others for this experiment. But we see
that the performance of our method drops significantly as we attempt
generalizing to different objects with changed input statistics.

Note that even though the representation clustering (Fig. 4.13) was more
clear-cut using convolutional encoders and VGG, compared to ours, they did not
perform as well in the trials. We attribute this to the fact that the main
loss term in our representation was the one of future state predictability,
which may not give as clear of a structure in two dimensions as a
representation that is trained only for static image reconstruction.

###  4.5 Conclusion

In this chapter, we present Latent Representation Prediction (LARP) networks
with applications to visual planning. We jointly learn a model to predict
transitions in Markov decision processes with a representation trained to be
maximally predictable. This allows us to accurately search the latent space
defined by the representation using a heuristic graph traversal algorithm. We
validate our method on a viewpoint-matching task derived from the NORB data
set, and we find that a representation that is optimized jointly with the
predictor network performs best in our experiments.

A common issue of unsupervised representation learning is one of trivial
solutions: a constant representation which optimally solves the unsupervised
optimization problem but brings across no information. To avoid the trivial
solution, we constrain the training by introducing a sphering layer or a loss
term that is either contrastive or reconstructive. Any of these approaches
will do the job of preventing the representations from collapsing to
constants, and none of them displays stronger performance than the others in
our experiments.

Our LARP representation is competitive with pretrained representations for
planning and compares favorably to other reinforcement learning (RL) methods.
Our approach is a sound solution for learning a useful representation that is
suitable for planning only from interactions.

Furthermore, we find that our method has better data-efficiency during
training than several reinforcement learning methods from the literature.
However, a disadvantage of our approach compared to standard RL methods is
that the execution time of our method scales worse with the size of the state
space, as a forward pass is calculated for each node during the latent space
search, potentially resulting in a combinatorial explosion.

Our approach is adaptable to changes in the tasks. For example, our search
would only be slightly hindered if some obstacles were placed in the
environment or some states were forbidden to traverse through. Furthermore,
our method is independent of specific rewards or discount rates, while
standard RL methods are usually restricted in their optimization problems.
Often, there is a choice between optimizing discounted or undiscounted
expected returns. Simulation/rollout-based planning methods are not restricted
in that sense: If reward trajectories can be predicted, one can optimize
arbitrary functions of these and regularize behavior. For example, a risk-
averse portfolio manager can prioritize smooth reward trajectories over
volatile ones.

Future lines of work should investigate further the effect of the different
constraints on the end-to-end learning of representations suited for a
predictive forward model, as well as considering novel ones. The search
algorithm can be improved and made faster, especially for higher-dimensional
action spaces or continuous ones. Our network could also in principle be used
to train an RL system, for instance, by encouraging it to produce similar
outputs as ours and thereby combining data-efficiency with fast performance
during inference time.

##  Chapter 5 Reward prediction for representation learning and reward shaping

The previous chapter introduced the LARP network and showed its usefulness in
view-point matching experiments, in comparison to reinforcement learning (RL)
methods, particularly the model-based ones. The network learns state features
that are tailor-made for a prediction module, which is then used for graph-
based search. The learning of the state representation and predictor is done
in a self-supervised manner, independently of a reward signal. The method is
data-efficient and is able to speed to learning for new tasks after pre-
training on a similar one.

However, the method requires a forward pass of the network for every node that
is considered during the goal search, which scales poorly with the size of the
state space compared to methods that only need to process the current state.
Depending on the branching factor of the environment and the details of the
latent representation graph search, this can result in a combinatorial
explosion.

In this chapter111This chapter is adapted from (Hlynsson and Wiskott,, 2021).,
we approach the problem of learning state representations for RL from another
angle: instead of predicting the results of actions, we learn a representation
that is predictive of the local distance from the goal in single-goal
environments. The representation is learned alongside a reward predictor that
learns to estimate either a raw or a smoothed version of the true reward
signal.

We augment the training of out-of-the-box RL agents by shaping the reward
using our reward predictor during policy learning. Using our representation
for preprocessing high-dimensional observations, as well as using the
predictor for reward shaping, is shown to significantly enhance Actor Critic
using Kronecker-factored Trust Region and Proximal Policy Optimization in
single-goal environments with visual inputs.

The remainder of the chapter has the following structure: We start with a
chapter introduction in Section 5.1, followed by an explanation of required
background knowledge for this chapter in Section 5.2. We go into related work
in Section 5.3, after which the details of our approach is outlined in Section
5.4. The methodology of our experiments is explained in Section 5.5. We
display and discuss the results of our experiments in Section 5.6. Finally, we
close the chapter with concluding remarks in Section 5.7.

###  5.1 Introduction

Even though the dominance of humans is being tested by RL agents on numerous
fronts, there are still great difficulties for the field to overcome. For
instance, the data that is required for algorithms to reach human performance
is on a far larger scale than that needed by humans. Furthermore, the general
intelligence of humans remains unchallenged. Even though an RL agent has
reached superhuman performance in one field, its performance is usually poor
when it is tested in new areas.

The study of methods to overcome the problem of data-efficiency and
transferability of RL agents in environments where the agent must reach a
single goal is the focal point of this work. We consider a simple way of
learning a state representation by predicting either a raw or a smoothed
version of a sparse reward yielded by an environment. The two objectives,
learning a state representation and predicting the reward, are directly
connected as we train a deep neural network for the prediction, and the hidden
layers of this network learn a reward-predictive state representation.

The reward signal is created by collecting data from a relatively low number
of initial episodes using a controller that acts randomly. The representation
is then extracted from an intermediate layer of the prediction model and re-
used as general preprocessing for RL agents, to reduce the dimensionality of
visual inputs. The agent processes inputs corresponding to its current state
as well as the desired end state, which is analogous to mentally visualizing a
goal before attempting to reach it. This general approach of relying on state
representations, that are learned to predict the reward rather than maximizing
it, has been motivated in the literature (Lehnert et al.,, 2020) and we show
that our representation is well-suited for single-goal environments. Our work
adds to the recently growing body of knowledge related to deep unsupervised
(Hlynsson and Wiskott,, 2019) or self-supervised (Schüler et al.,, 2019)
representation learning.

We also investigate the effectiveness of augmenting the reward for RL agents,
when the reward is sparse, with a novel problem-agnostic reward shaping
technique. The reward predictor, which is used to train our representation, is
not only used as a part of an auxiliary loss function to learn a
representation, but it is also used during training the RL system to encourage
the agent to move closer to a goal location. Similar to advantage functions in
the RL literature (Schulman et al.,, 2015), given the trained reward
predictor, the agent receives an additional reward signal if it moves from
states with a low predicted reward to states with a higher predicted reward.
We find this reward augmentation to be beneficial for our test environment
with the largest state-space.

###  5.2 Background

####  5.2.1 Reward shaping

Sparse rewards in environments is a common problem for reinforcement learning
agents. The agent’s objective is to learn how to associate its inputs with
actions that lead to high rewards, which can be a lengthy process if the agent
only rarely experiences positive or negative rewards.

Reward shaping (Mataric,, 1994; Ng et al.,, 1999; Brys et al.,, 2015) is a
popular method of modifying the reward function of an MDP to speed up
learning. It is useful for environments with sparse rewards to augment the
training of the agent, but skillful applications of reward shaping can in
principle aid the optimization for any environment – although the efficacy of
the reward shaping is highly dependent on the details of the implementation
(Amodei and Clark,, 2016). In the last few years, reward shaping has been
shown to be useful for complex video game environments, such as real-time
strategy games (Efthymiadis and Kudenko,, 2013) and platformers (Brys et al.,,
2014) and it has also been combined with deep neural networks to improve
agents in first-person shooter games (Lample and Chaplot,, 2017).

As an illustration, consider learning a policy for car racing. If the goal is
to train an agent to drive optimally, then supplying it with a positive reward
for reaching the finish line first is in theory sufficient. However, if it is
punished for actions that are never beneficial, for instance crashing into
walls, it prioritizes learning to avoid such situations, allowing it to
explore more promising parts of the state space.

Furthermore, just reaching the goal is insufficient if there is competition.
To make sure that we have a winning racer, a small negative reward can be
introduced at every time step to urge the agent to reach the finish line
quickly. Note that the details of the reward shaping in this example requires
domain knowledge from a designer who is familiar with the environment. It
would be more generally useful if the reward shaping would be autonomously
learned, just as the policy of the agent, as we propose to do in this work.

####  5.2.2 Reward-predictive vs. reward-maximizing representations

Lehnert et al., (2020) make the distinction between reward-maximizing
representations are reward-predictive representations. They argue how reward-
maximizing representations can transfer poorly to new environments, while
reward-predictive representations generalize successfully. Take the simple
grid world navigation environments in Fig. 5.1, for example. The agent starts
at a random tile in the grid and gets a reward of +1 by reaching the rightmost
column in Environment A or by reaching the middle column in Environment B. The
state space in Environment A can be compressed from the $3\times 3$ grid to a
vector of length 3, $[\phi_{1}^{p},\phi_{2}^{p},\phi_{3}^{p}]$ of reward-
predictive representations. To predict the discounted reward, it suffices to
describe the agent’s state with $\phi_{j}^{p}$ if it is in the $j$th row.

Fig 5.1: Reward-maximizing vs. reward-predictive representations. In this grid
world example, the agent starts the episode at a random location and can move
up, down, left, or right. The episode ends with a reward of 1 and terminates
when the agent reaches the rightmost column. Both the reward-predictive
representation and reward-maximizing representation $\phi^{p}$ and $\phi^{m}$,
respectively, are useful for learning the optimal policy in Environment A. The
reward-predictive representation $\phi^{p}$ collapses each column into a
single state to predict the discounted future reward. The reward-maximizing
representation $\phi^{m}$ makes no such distinction, as moving right is the
optimal action in any state. It is a different story if the representations
are transferred to Environment B, where reaching the middle column is now the
goal. The representation $\phi^{p}$ can be reused, and the optimal policy is
found if the agent now takes a step left in $\phi_{3}^{p}$. However, the
representation $\phi^{m}$ is unable to discriminate between the different
states and is useless for determining the optimal policy.

The reward-maximizing representation for Environment A is much simpler: the
whole state space can be collapsed to a single element $\phi^{m}$, with the
optimal policy of always moving to the right. If these representations are
kept, then the reward-predictive representation $\phi^{p}$ is informative
enough for a RL agent to learn how to solve Environment B. The reward-
maximizing representation $\phi^{m}$ has discarded too many details of the
environment to be useful for solving this new environment.

####  5.2.3 Successor features

The successor representation algorithm learns two functions: the expected
reward $R_{\pi}^{\text{SF}}$ received after transitioning into a state $s$, as
well as the matrix $M_{\pi}^{\text{SF}}$ of discounted expected future
occupancy of each state, assuming that the agent starts in a given state and
follows a particular policy $\pi$. Knowing the quantities
$R_{\pi}^{\text{SF}}$ and $M_{\pi}^{\text{SF}}$ allows us to rewrite the value
function:

$V_{\pi}(s)=\mathbb{E}_{s^{\prime}}\left[R_{\pi}^{\text{SF}}(s)M_{\pi}^{\text{SF}}(s,s^{\prime})\right]$
(5.1)

The motivation for this algorithm is that it combines the speed of model-free
methods, by enabling fast computations of the value function, with the
flexibility of model-based methods for environments with changing reward
contingencies.

This method is made for small, discrete environments, but it has been
generalized for continuous environments with so-called feature-based successor
representations, or successor features (SFs) (Barreto et al.,, 2016). The SF
algorithm similarly calculates the discounted expected representation of
future states, given the agent takes the action $a$ in the state $s$ and
follows a policy $\pi$:

$\psi_{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t-1}\phi_{t+1}|s_{t}=s,a_{t}=a\right]$
(5.2)

where $\phi$ is some state representation. Both the SF $\psi$ and the
representation $\phi$ can be deep neural networks.

###  5.3 Related work

####  5.3.1 Reward-predictive representations

Lehnert et al., (2020) compare successor features (SFs) to a nonparametric
Bayesian predictor that is trained to learn transition and reward tables for
the environment, either with a reward-maximizing or a reward-predictive loss
function. Lehnert and Littman, (2020) prove under what conditions successor
features (SFs) are either reward-predictive or reward-maximizing (see
distinction in Section 5.2.2). They also show that SFs work successfully for
transfer learning between environments with changing reward functions and
unchanged transition functions, but they generalize poorly between
environments where the transition function changes.

Our work is distinct from the reward-predictive methods that they compare, as
our representation does not need to calculate expected future state occupancy,
as is the case for SFs. Our method scales better for more complicated state-
spaces because we do not tabulate the states, as they do with their Bayesian
model, but learn arbitrary continuous features of high-dimensional input data.
In addition to that, learning our reward predictor is not only a "surrogate"
objective function, as we use it for reward shaping as well.

####  5.3.2 Reward shaping

The advantages of reward shaping are well understood in the literature
(Mataric,, 1994). A recent trend in RL research is the study of methods that
can learn the reward shaping function automatically, without the need of
(often faulty) human intervention.

Marashi et al., (2012) assume that the environment can be expressed as a graph
and that this graph formulation is known. Under these strong assumptions, they
perform graph analysis to extract a reward shaping function. More recently,
Zou et al., (2019) have proposed a meta-learning algorithm for potential-based
automatic reward shaping. Our approach is different from previous work as we
assume no knowledge about the environment and train a simple predictor to
approximate (potentially smoothed) rewards, which is then used to construct a
potential-based reward shaping function.

####  5.3.3 Goal-conditioned reinforcement learning

Kaelbling, (1993) studied environments with multiple goals and small state-
spaces. In their problem setting, the agent must reach a known but dynamically
changing goal in the fewest number of moves. The observation space is of a low
enough dimension for dynamic programming to be satisfactory in their case.
Schaul et al., (2015) introduce the Universal Value Function Approximators and
tackle environments of larger dimensions by learning a value function neural
network approximator that accepts both the current state and a goal state as
the inputs. In a similar vein, Pathak et al., (2018) learn a policy that is
given a current state and a goal state and outputs an action that bridges the
gap between them. Hlynsson et al., (2020) learn a predictable representation
that is paired with a representation predictor and combine it with graph
search to find a given goal location. In contrast to these approaches, we
learn a reward-predictive representation in a self-supervised manner, which is
used to preprocess raw inputs for RL policies.

###  5.4 Approach

In this section, we explain our approach mathematically. Intuitively, we train
a deep neural network to predict either a raw or a smoothed reward signal from
a single-goal environment. The output of an intermediate layer in the network
is then extracted as the representation – for example, by simply removing the
top layers of the network. The full reward predictor network is used for
reward shaping by rewarding the agent for moving from lower predicted values
toward higher predicted values of the network.

####  5.4.1 Learning the representation

Suppose that $f_{\theta}:\mathbb{R}^{c}\rightarrow[0,1]$ is a differentiable
function parameterized by $\theta$ and $c$ is a positive integer. We use
$f_{\theta}$ to approximate the discounted return in a POMDP with a sparse
reward: the agent receives a reward of 0 for each time step except when it
reaches a goal location, at which point it receives a positive reward and the
episode terminates.

Given an experience buffer $\mathcal{D}=$
$\\{(s_{t},a_{t},r_{t},s_{t+1})_{i}\\}$, we create a new data set
$\mathcal{D}^{*}=\\{\left(s_{t},a_{t},r^{*}_{t},s_{t+1}\right)_{i}\\}$. The
new rewards are calculated according to the equation

$r^{*}_{t}=\gamma^{m}r_{t+m}$ (5.3)

where $\gamma\in[0,1]$ is a discount factor and $M>m>0$ is the difference
between $t$ and the time step index of the final transition in that episode,
for some maximum time horizon $M$. Throughout our experiments, we keep the
value of the discount factor equal to $0.99$ and we train on $\mathcal{D}$ or
$\mathcal{D}^{*}$.

Assume that our differentiable representation function
$\phi:\mathbb{R}^{d}\rightarrow\mathbb{R}^{c}$ is parameterized by
$\theta^{\prime}$ and maps the $d$-dimensional raw observation of the POMDP to
the $c$-dimensional feature vector. We train the representation for the
discounted-reward prediction by minimizing the loss function

$\mathcal{L}(f_{\theta}[\phi_{\theta^{\prime}}(s_{t+1})],r^{*}_{t})=\left(r^{*}_{t}-f_{\theta}[\phi_{\theta^{\prime}}(s_{t+1})]\right)^{2}$
(5.4)

with respect to the parameters $\theta$ of $f$ and the parameters
$\theta^{\prime}$ of $\phi$ over the whole data set $\mathcal{D}^{*}$. See
Fig. 5.2 for a conceptual overview of our representation learning.

Fig 5.2: Learning and using the representation. Our representation and reward
predictor is trained with the elements highlighted in blue. The trained
representation is then used for dimensionality reduction for an RL agent, that
interacts with the environment, as indicated by the elements highlighted in
red.

####  5.4.2 Reward shaping

Ng et al., (1999) define a reward shaping function $F$ as potential-based if
there exists a function $f:\mathcal{S}\rightarrow\mathbb{R}$ such that for all
states $s,s^{\prime}\in\mathcal{S}$ the following equation holds:

$F(s,a,s^{\prime})=\gamma f(s^{\prime})-f(s)$ (5.5)

and $\gamma$ is the MDP’s discount factor. They prove for single-goal
environments that every optimal policy for the MDP
$M=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathbb{P}(s_{0}),\gamma)$
is also optimal for its reward shaped counterpart
$M^{\prime}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}+F,\mathbb{P}(s_{0}),\gamma)$,
and vice versa. They also show, for a given state space $\mathcal{S}$ and
action space $\mathcal{A}$, that if $F$ is not potential-based, then there
exist a transition function $\mathcal{P}$ and a reward function $\mathcal{R}$
such that no optimal policy in $M^{\prime}$ is optimal in $M$.

Deciding that the reward shaping function should be potential-based is just
the first step in its design. Now assume that we have an environment where an
agent is tasked with reaching a goal state $g$. That is, for a given distance
function $d:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R}^{+}$ the agent
receives a reward of 1 if it is close enough to the goal location,
$d(s,g)\leq\delta$, for some reward threshold $\delta\in\mathbb{R}^{+}$.
Otherwise, it receives a reward of 0.

The distance $d$ between the agent’s location new $s^{\prime}$ and the goal
location $g$ can be a useful value to calculate in the design of a reward
shaping function

$F(s,a,s^{\prime})=\begin{cases}1&\text{if }d(s^{\prime},g)\leq\delta\\\
-d(s^{\prime},g)&\text{otherwise}\end{cases}$ (5.6)

However, this depends on the environment, as the agent could get stuck in
local optima before coming close to the goal, i.e. if it would have to move
through a region with a large $d(s,g)$ before it can globally minimize it.
This could for example be the case in a maze environment if $d$ is the
Euclidean distance between the $(x,y)$ coordinates of the agent’s location and
the goal location and there is a wall between the agent and the goal. Trott et
al., (2019) propose to solve this by incorporating the potential local optima
in the reward shaping function as so-called "anti-goals" $\bar{g}$ to be
avoided

$F(s,a,s^{\prime})=\begin{cases}1&\text{if }d(s^{\prime},g)\leq\delta\\\
\min[0,-d(s^{\prime},g)+d(s,\bar{g})]&\text{otherwise}\end{cases}$ (5.7)

These states can be hand-picked by domain experts. However, adding anti-goals
like this could iteratively introduce even more local optima and a solution to
the original problem is not guaranteed.

It is generally not true that the distance function $d$ and all the variables
needed to calculate it, such as the coordinates of the agent and the goal in a
maze, are available to the agent. Even if $d$ were computable, using it
naively can bring about its own problems, as was alluded to above.

We argue that instead of using $d$ in Equation 5.6, it would be better to
measure the distance between the agent and the goal in terms of how many
actions the agent has to take until the goal is reached. This function is not
assumed to be given, but it can be estimated as the agent is being trained on
the environment, for instance by optimizing Equation 5.4.

Additionally, we would like our reward shaping function to be potential-based
(Equation 5.5) to reap the theoretical advantages. Thus, we propose a
potential-based reward shaping function based on the discounted reward
predictor

$\begin{split}F(s,a,s^{\prime})&=\left(\gamma
f_{\theta}(\phi_{\theta^{\prime}}[s^{\prime}])-f_{\theta}(\phi_{\theta^{\prime}}[s])\right)(H-I)/H\\\
&=\gamma\left(f_{\theta}(\phi_{\theta^{\prime}}[s^{\prime}])(H-I)/H\right)-f_{\theta}(\phi_{\theta^{\prime}}[s])(H-I)/H\\\
&=\gamma f^{*}(s^{\prime})-f^{*}(s)\end{split}$ (5.8)

where $f^{*}=f_{\theta}(\phi_{\theta^{\prime}}[s^{\prime}])(H-I)/H$,
$f_{\theta}$ is the reward predictor and $\phi_{\theta^{\prime}}$ is our
representation from the previous section. Note that both $f_{\theta}$ and
$\phi_{\theta^{\prime}}$ are assumed to be fully trained before the policy of
the agent is trained, for example using data gathered by a random policy, but
they can in principle also be updated as the policy is being learned. The
factor $(H-I)/H$ scales down the intensity of the reward shaping where
$I\in\mathbb{N}^{+}$ is the number of episodes that the agent has experienced
and $H\in\mathbb{N}^{+}$ is the maximum number of episodes where the agent is
trained using reward shaping. The strength of the reward shaping is the
highest in the beginning to counteract potentially adverse effects of errors
in the reward predictor. It is also more important to incentivize moving
toward the general direction of the goal in the early stages of learning,
after which the un-augmented reward signal of the environment is allowed to
"speak for itself" and guide the learning of the agent toward the goal
precisely.

###  5.5 Methodology and implementation

####  5.5.1 Environment

The method is tested on three different gridworld environments based on the
Minimalistic Gridworld Environment (MiniGrid) (Chevalier-Boisvert et al.,,
2018). Tiles can be empty, occupied by a wall or occupied by lava. The
structure of the environments fit naturally into our POMDP tuple template (Eq.
2.1):

  * $\mathcal{S}$:

The constituent states of $\mathcal{S}$ are determined by the agent’s location
and direction (facing north, west, south or east). See Fig. 5.3(a) for three
different world states in one of our environments.

  * $\mathcal{A}$:

The action space $\mathcal{A}$ consists of three actions: (1) turn left, (2)
turn right and (3) move forward.

  * $\mathcal{P}$:

The transition function is deterministic. The agent relocates to the tile it
faces if it moves forward and the tile is empty, and nothing happens if the
tile is occupied by a wall. The episode terminates if the tile is occupied by
lava or the goal. The agent rotates in place if it turns left or right.

  * $\mathcal{R}$:

Reaching the green tile goal gives a reward of
$1-0.9\cdot\frac{\\#\textrm{steps taken}}{\\#\textrm{max steps}}$, every other
action gives 0 points. The environment automatically times out after
$\\#\textrm{max steps}=100$ steps.

  * $\mathbb{P}(s_{0})$:

Differs between the three environments (see below).

  * $\Omega$:

All $7\times 7$ subset of tiles, represented by $28\times 28\times 3$ arrays,
from the point of view of an agent who cannot see through walls, see Fig.
5.3(b).

  * $\mathcal{O}$:

The point of view of the agent from its current viewpoint (Fig. 5.3(b)) and a
goal observation (Fig. 5.3(c)).

  * $\gamma$:

the discount factor is $0.99$.

(a) Full world states.

(b) Agent’s point of view.

(c) Goal observations.

Fig 5.3: Two-room environment. The red agent must reach the green goal in as
few steps as possible. The agent starts each episode between the two rooms,
facing a random direction (up, down, left or right). Each column corresponds
to a snapshot of one episode. The light tiles correspond to what the agent
sees, while the dark tiles are unseen by the agent. (a) Examples of the full
state (b) The observation from the agent’s current state (c) A goal
observation. This is the agent’s point of view from a state that separates the
agent from the goal by one action.

We consider the following three environments:

##### Two-room environment

The world is a $8\times 17$ grid of tiles, split into two rooms, where walls
are placed at different locations to facilitate discrimination between the
rooms from the agent’s point of view. (Fig. 5.3). The agent is placed between
the two rooms, facing a random direction. The goal is at one of three possible
locations. This is a modified version of the classical four-room environment
layout (Sutton et al.,, 1999).

##### Lava gap environment

In this environment, the agent is in a $4\times 4$ room with a column of lava
either one or two spaces in front of the agent (Fig. 5.4) with a gap in a
random row. The agent always starts in the upper left corner and the goal is
always in the lower right corner.

Fig 5.4:  The lava gap environment. The red agent must reach the green goal in
as few steps as possible while avoiding the orange lava tiles. Each episode is
randomly selected to be one of the eight pictured configurations. If the agent
tries to go through the orange lava tile, it experiences an immediate episode
termination with no reward. Note that the wall tiles that are lighter than the
others are presently in the agent’s field of vision.

##### Four-room environment

An expansion to the two-room environment with two additional rooms (Fig. 5.5).
In this setup, both the agent and the goal location are placed at random
locations within the $17\times 17$ gridworld.

Fig 5.5:  Four-room environment. The agent (red triangle) must reach the goal
(green square) in as few steps as possible, both are randomly placed in each
episode. The $7\times 7$ grid of highlighted tiles in front of the agent
indicates its observation.

####  5.5.2 Baselines

We combine our representations with two RL algorithms as implemented in Stable
Baselines (Hill et al.,, 2018) using the default hyperparameters:

  * •

(ACKTR) Actor Critic using Kronecker-Factored Trust Region (Wu et al.,, 2017),
which combines actor-critic methods, trust-region optimization, and
distributed Kronecker factorization to enhance data-efficiency.

  * •

(PPO2) A version of the Proximal Policy Optimization algorithm (Schulman et
al.,, 2017). It modifies the original algorithm by using clipped value
functions and a normalized advantage.

For both algorithms, six variations are compared:

  * •

(Deep RL) The RL algorithm learns the representation from scratch on raw
images

  * •

(SF) The input is preprocessed using successor features

  * •

(Ours 1r) The input is preprocessed using our representation, trained on raw
reward predictions

  * •

(Ours 1r + Shaping) The input is preprocessed using our representation and the
reward is shaped, trained on raw reward predictions

  * •

(Ours 64r) The input is preprocessed using our representation, trained on
smoothed reward predictions

  * •

(Ours 64r + Shaping) The input is preprocessed using our representation and
the reward is shaped, trained on smoothed reward predictions

Care has been taken to ensure that each variation has the same architecture
and the same number of parameters.

####  5.5.3 Model architectures

Every model is realized as a neural network using Keras (Chollet et al.,,
2015). Below, the representation and policy networks are used for our method
and the SF comparison, the reward prediction network is used only for our
method and the deep RL network is used only for the deep RL comparison, where
the RL algorithm also learns the representation.

The representation networks are two convolutional networks (Table 5.1) with a
$28\times 28\times 3$ input, taking either the agent’s current observation or
the goal observation.

Layer | Filters | Filter size | Stride | Padding | Output shape
---|---|---|---|---|---
Input tensor | - | - | - | - | $28\\!\times\\!28\\!\times\\!3$
Convolution | 8 | $3\\!\times\\!3$ | 3 | None | $9\\!\times\\!9\\!\times\\!8$
ReLU | - | - | - | - | $4\\!\times\\!4\\!\times\\!8$
2D max pooling | 8 | $2\\!\times\\!2$ | - | None | $4\\!\times\\!4\\!\times\\!8$
Convolution | 16 | $3\\!\times\\!3$ | 2 | None | $1\\!\times\\!1\\!\times\\!16$
ReLU | - | - | - | - | $1\\!\times\\!1\\!\times\\!16$
Flatten | - | - | - | - | 16
Dense | - | - | - | - | 16
Table 5.1: Representation network.

The first layer subsamples the input, keeping only every other column and row.
This is followed by 8 filters of size $3\times 3$ with a stride of 3. This is
followed with a ReLU activation and a $2\times 2$ max pooling layer with a
stride value of 2. The pooling layer’s output is passed to a layer with 16
convolutional filters of size $3\times 3$ and a stride of 2 and a ReLU
activation function. The output is then flattened and passed to a dense layer
with 16 units and a linear activation, defining the dimension of the
representation. No zero padding is applied in the convolutional layers or the
pooling layer.

The policy networks are three-layer fully-connected networks (Table 5.2)
accepting the concatenated output of the representation network for the
agent’s current point of view and the goal observation as an input. The first
two layers have 64 units and a ReLU activation, and the last layer has 3 units
and a linear activation function. The three units represent the three actions
left, right, and forward in a one hot encoding. Winner takes all is used to
decide on the action.

Layer |  Units | Output shape
---|---|---
Input tensor | - | 32
Dense | 64 | 64
ReLU | - | 64
Dense | 64 | 3
ReLU | - | 3
Dense | 3 | 3
Table 5.2: Policy network.

Our reward prediction network is a three-layer fully-connected network (Table
5.3) with the same input as the policy network: the concatenated
representation of the agent’s current view and the goal observation. The first
two layers have 256 units and a ReLU activation, but the last layer has 1 unit
and a logistic activation function.

Layer |  Units | Output shape
---|---|---
Input tensor | - | 32
Dense | 256 | 64
ReLU | - | 64
Dense | 256 | 3
ReLU | - | 3
Dense | 1 | 3
Logistic | - | 3
Table 5.3: Reward prediction network.

The deep RL network stacks the representation network and the policy network
on top of each other. The representation network accepts the input and outputs
the low-dimensional representation to the policy network that outputs the
action scores.

####  5.5.4 Training the representation and predictor networks

We collect a data set of $10$ thousand transitions by following a random
policy in the two-room environment. For this data collection, each episode has
a $50\%$ chance to have the goal location in the bottom room or on the left
side of the top room (see the left and middle pictures in Fig. 5.3(a)). The
reward predictor and the representation are trained in this manner for all
experiments, including the lava gap and the four-room environment. Thus, we
use a representation and reward predictor that have never seen lava. For the
experiments with smoothed rewards, the sparse reward associated with the
observations in the data set is augmented by associating a new reward to the
$64$ states leading to observations with a positive reward, according to
Equation 5.3, with a discount factor of $0.99$. Additionally, after the reward
has been (potentially) smoothed in this way, observations associated with a
positive reward are oversampled $10$ times to balance the data set, regardless
of whether the reward has been augmented or not.

###  5.6 Results and discussion

In the experiments, we compare RL agents that learn their representations from
scratch (Deep RL) to agents that preprocess their inputs with different
representations. We compare our representation, trained on raw reward
predictions – with (Ours 1r) or without reward shaping (Ours 1r + Shaping) –
to our method trained on smoothed reward prediction, also with (Ours 64r) or
without reward shaping (Ours 64r + Shaping). We use "64r" to denote that our
method was trained with reward shaping and "1r" to denote that our method was
trained without augmentation. As a baseline, we compare our representation to
a reward-predictive representation from the literature, Successor Features
(SFs).

####  5.6.1 Two-room environment

We start by visualizing the outputs of our reward predictor in the rooms,
depending on the goal location, in Fig. 5.6. Each square indicates the average
predicted reward for transitioning to the corresponding tile in the room.

The predicted reward spikes in a narrow region around the two goal locations
that were used to train the raw reward predictor (Fig. 5.6(a)), but the area
of states with high predicted rewards is wider around the test goal. This
difference is due to overfitting on the specific training paths that were more
frequently taken toward the respective goals, but this does not harm the
generalization capabilities of the network. The peakyness of the predictions
disappears when the predictor is trained on the smoothed rewards (Fig.
5.6(b)). However, higher predicted rewards in the corner of the other room
appear. Both scenarios, raw and smoothed reward prediction, show promise for
the application of reward shaping under our training scheme, as the agent
would benefit from finding neighborhoods with higher values of predicted
reward until it reaches the goal, instead of having to rely solely on a sparse
reward that is only given when the agent lands exactly on the goal state.

(a) Raw reward prediction.

(b) Smoothed reward prediction.

Fig 5.6: Predicted rewards, two-room environment. The predictor is trained on
the setups shown on the left and in the middle, and tested for the setup on
the right. The color becomes warmer for states where the reward is predicted
to be higher.

In Figure 5.7, we illustrate the variance of the mean reward (left side) and
the variance of the optimal performance (right side) of the different methods,
as a function of the time steps taken for training. We average over 10 runs
and in each run we perform 10 test rollouts, so each point is the aggregate of
100 episodes in total.222Note that the standard deviation of the mean reward
of all episodes, from all runs put together, is approximately $26\%$ higher
than the standard deviation of the mean of the means or the mean of the mins.
The error bands indicate two standard deviations. This methodology of
generating the plots also applies to Fig. 5.9, Fig. 5.10 and Fig. 5.12.

The learning curves of both ACKTR and PPO2 get close to the highest achievable
mean reward of $1$ the fastest using our representations. There no significant
benefit from using smoothed reward shaping for ACKTR, and the raw reward
shaping is in fact harmful in this case. For PPO2, the agent using our
representation that is trained on raw reward predictions learns the fastest.
Regular deep RL, where the representations are learned from scratch, is
clearly outperformed by the variants that use reward-predictive
representations. We believe that this is because RL agents can generally
benefit from the input being preprocessed, as the computational overhead for
learning the policy is reduced. This effect is enhanced when the preprocessing
is good, which is the case for our reward-predictive representation: it
abstracts away unnecessary information as it is trained to output features
that indicate the distance between the agent and the goal, when the goal is in
view.

The difference in aggregated mean rewards vs. aggregated minimum episode
lengths can be explained due to systematically different behaviors. For
example, an agent might have a weak long-term strategy of checking the
different rooms, giving it poor average mean rewards, but a strong short-term
tactic of taking the direct course to the goal when it sees it, giving it a
good average minimum episode length.

Fig 5.7:  Two-room environment. In these experiments, there are only two rooms
and the agent must reach a goal that is always at the same location. The agent
can traverse between the rooms and starts each episode between them, facing a
random direction. The left side shows the mean of every agent’s mean reward
and the right side shows the mean of every agent’s minimum episode length.

####  5.6.2 Lava gap environment

##### Learning from scratch

The heatmaps of average predicted rewards are visualized in Fig. 5.8. The
reward predictor was trained on the two-room environment. The tiles closest to
the goal have the highest values, with a particularly smooth gradient toward
the goal for the smoothed-reward predictor, which demonstrates that there is
potential gain from transferring the prediction-based reward shaping between
similar environments. The learning performance of the different methods can be
seen in Fig. 5.9. The decidedly fastest learning can be observed when the
actor-critic method is combined with our representation, trained on raw reward
predictions and without reward shaping. Regular deep RL is the second-best,
but with a very large variance on the performance. Our reward shaping
variations and the SFs are very close in performance, albeit significantly
worse than the other two. The poor performance of reward shaping can be
explained by the fact that there are very few states, which makes the reward
shaping unnecessary in such a simple environment. All the methods look more
similar when PPO2 optimization is applied, with respect to the mean rewards,
but our variant that is trained on smooth reward prediction and uses reward
shaping reaches the highest average performance in the last iterations.

Fig 5.8:  Predicted rewards, lava gap. Average predicted reward per state in
the lava gap environment.

Fig 5.9:  Lava gap experiment. All policies are randomly initialized and learn
to solve the lava gap environment from scratch. The representations in all
methods except for Deep RL are learned on the training goals in the two-room
environment (see Fig. 5.6). The left side shows the mean of every agent’s mean
reward, and the right side shows the mean of every agent’s minimum episode
length.

##### Transfer learning

To investigate how the methods compare for adapting to new environments, we
trained the policies for 8000 steps on the two-room environment before
learning to solve the lava gap environment, see Fig. 5.10. Our method, without
reward shaping, facilitates the fastest learning for ACKTR in this case. Deep
RL is the most severely affected by this change, which is probably due to the
method learning a reward-maximizing representation in one environment that
does not transfer well to another environment. Every PPO2 variation looks bad
for this scenario, but the smooth-reward prediction representation with reward
shaping has the highest mean reward and our raw-reward prediction
representation has the lowest average minimum episode length.

Fig 5.10:  Re-learning experiment The different methods are trained for eight
thousand training steps on the two-room environment before being trained on
the lava gap environment. The curves show the mean reward on the lava gap
environment. The left side shows the mean of every agent’s mean reward, and
the right side shows the mean of every agent’s minimum episode length.

We visualize trajectories of an agent that is trained on our representation
(Ours 1r) as it traverses the lava gaps environment (Fig. 5.11). For
inspection of cases where it fails, we choose an agent that has been trained
for $50$ thousand time steps only and has around $0.75$ mean reward.

(a) Six successful episodes.

(b) Six failed episodes.

Fig 5.11: Lava gap trajectories. A visualization of six successful and six
failed trajectories by an agent that was trained up to approximately $75\%$
success rate using our representation. The color gradient goes from the first
actions taken in the episode in blue to the last actions in red. Note that
rotations in-place are not visualized, but only transitions between tiles.

####  5.6.3 Four-room environment

In our final comparison, we add two additional rooms to the two-room
environment and randomize both the goal location and the starting position of
the agent, with the results shown in Fig. 5.12. Looking at the minimum episode
lengths, for the ACKTR learner, our raw-reward prediction representation with
reward shaping performs best and the one without reward shaping comes in
second. There is little discernible difference between the performance of SFs
and Deep RL, but they both perform significantly worse than our methods. The
scale of the mean reward is a great deal lower than in the previous
experiments, since the average distance between the starting tile of the agent
and the goal is much larger than in the previous two environments.

For this scenario, all the methods look similarly bad for the PPO2 policy,
except for our raw-reward representations, with reward shaping, which has the
lowest minimum episode length. The big advantage of reward shaping in this
environment compared to the two-room environment can be explained by the
increased complexity, making the reward shaping more helpful in guiding the
agent’s search. In the previous experiments, the agent and goal locations
start at fixed locations, allowing the agents to solve it by rote
memorization. The reward shaping function calculated by the raw-reward
predictor fares significantly better in this situation. We hypothesize that
this is due to the smoothed-reward predictor distracting the agent by pushing
it to corners, as the visualization in Fig. 5.6(b) would suggest.

The reward shaping given by the raw-reward predictor is more discriminative,
as we see in Fig. 5.6(a). The agent receives a positive reward as soon as the
goal reaches its point of view, which is any location up to six tiles in front
of it and no further than 3 tiles away from it to the left or to the right.
This allows the reward shaping function to guide the agent directly to the
goal, assuming that they are in the same room and that there is no wall
obstructing the agent’s field of vision.

Fig 5.12:  Full four-room environment. The agent and goal are placed at random
locations at the start of each episode. The left side shows the mean of every
agent’s mean reward, and the right side shows the mean of every agent’s
minimum episode length.

Three successful and three failed trajectories of an ACKTR agent that has been
trained, using our representation (Ours 1r) for a million time steps are
visualized in Fig. 5.13. We can see undesirable behavior in both the
successful and the failed trajectories, that the agent wastes effort re-
visiting tiles it has already been to.

(a) Three successful trajectories

(b) Three failed trajectories

Fig 5.13: Four-room trajectories. The agent was trained on our representations
with Actor Critic using Kronecker-Factored Trust Region for a million time
steps.

###  5.7 Conclusion

Processing high-dimensional inputs for reinforcement learning (RL) agents
remains a difficult problem, especially if the agent must rely on a sparse
reward signal to guide its representation learning. In this work, we put
forward a method to help alleviate this problem with a method of learning
representations that preprocesses visual inputs for RL methods. Our
contributions are (i) a reward-predictive representation that is trained
simultaneously with a reward predictor and (ii) a reward shaping technique
using this trained predictor. The predictor learns to approximate either the
raw reward signal or a smoothed version of it, and it is used for reward
shaping by encouraging the agent to transition to states with higher predicted
rewards.

We used a view of the goal as a second input for the methods in our
experiments, but this is in principle not necessary, as moving toward the
green tile as it becomes visible is sufficient. Removing the goal input might
encourage the agents to learn policies that scan all the rooms faster until
the goal reaches its field of vision.

We have shown the usefulness of our representation and our reward shaping
scheme in a series of gridworld experiments, where the agent receives a high-
dimensional observation of its goal as an input along with an observation of
its immediate surroundings. Preprocessing the input using this representation
speeds up the training of two out-of-the-box RL methods, Actor Critic using
Kronecker-Factored Trust Region and Proximal Policy Optimization, compared to
having these methods learn the representations from scratch. In our most
complicated experiment, combining our representation with our reward shaping
technique is shown to perform significantly better than the vanilla RL
methods, which hints at its potential for success, especially in more complex
RL scenarios.

##  Chapter 6 Comparison of our three methods

We compare in this chapter the algorithms that we developed over the course of
the PhD work: the gradient-based ICA representation (GrICA) from Chapter 3,
the latent representation prediction network (LARP) from Chapter 4 and the
reward-prediction representation from Chapter 5 (RewPred). We use the
representations calculated by these methods to preprocess inputs for deep
reinforcement learning (Deep RL) agents on four different tasks: a cart-pole
balancing environment, the two-room and four-room goal-finding environments
from the previous chapter and an obstacle avoidance task. The implementation
of our methods is the same as described in the experimental section of their
corresponding chapter, unless specified otherwise.

In Section 6.2 we introduce the new visual cart-pole environment and discuss
the network architectures. In Section 6.3 we describe and discuss the results
of the experiments.

###  6.1 Introduction

As each method was introduced over the course of the PhD work, it was only
compared to other similar methods in the literature. We now take the
opportunity to compare them against each other in different environments. The
cart-pole environment and the obstacle avoidance environment have the
commonality that they both require only reactive short-term planning to
maximize the reward, but the goal-finding environments require more long-term
planning to find and reach the goal.

Every one of our representations is used for preprocessing the visual
observations of each environment for RL algorithms. The RL agents are the same
ones used in the previous chapter: ACKTR and PPO2, with the default parameters
from the Stable Baselines package. As we were not able to solve the cart-pole
environment using ACKTR, only PPO2 is used for that task. We compare the
method from Chapter 4, LARP, in this manner, even though it is not trained for
preprocessing inputs for model-free agents but is developed to be paired with
a predictor for predictive state representation rollouts

###  6.2 Methods

####  6.2.1 Visual cart-pole

We start by comparing our methods as visual preprocessing for a PPO2 policy on
a variant of the classical cart-pole balancing task. A pole is attached to a
cart that moves along a frictionless track. The goal is to keep the pole
upright for as long as possible, but the agent must push the cart to the left
or to the right at every time step – choosing to remain in place is not an
option. The original environment supplies four scalar variables to the agent:
the cart position and velocity and the pole angle and angular velocity. The
agent receives a positive reward of $+1$ for each transition.

Instead of using these scalars, we adapt this task to our paradigm by using a
visualization of the environment, see Fig. 6.1. To begin with, we extract the
rendering of the environment that is meant for debugging and visualization
purposes. As we can approximate the cart position and velocity and the pole
angle and angular velocity using two adjacent frames, we use the current and
previous observations. To simplify the input for the agent, we take the
difference of the two frames, crop the image around the cart and binarize it.
The only information that is lost in this process, compared to the original
environment, is the position of the cart, which is not so important for
solving the task. In addition to giving the agent a positive reward of $+1$
for every time step it keeps the pole balanced, we also supply it with a
negative reward of $-1$ when the pole falls down.

Fig 6.1: cart-pole processing pipeline. We replace the original 4-dimensional
state space of the OpenAI gym cart-pole environment with the output of its
rendering function. The previous frame is subtracted from the current frame,
the non-zero pixel values are centered, cropped and the final image is
converted to binary.

Our methods are trained on 5000 transitions that are collected from a random
policy. Even though this is not a sparse-reward environment, we calculated new
rewards for the RewPred network according to Equation 5.3 with a discount
factor of $\gamma=0.9$ and a maximum horizon of $M=6$. This has the effect
that every time step is associated with a reward of $+1$, except for the six
transitions leading up to the end of the episode, which have the rewards
$-(0.9^{5}),-(0.9^{4}),-(0.9^{3}),-(0.9^{2}),-(0.9)$ and $-(1.0)$.

####  6.2.2 Room environments

We compare the performance of PPO2 and ACKTR policies as they use our methods
for preprocessing on the two-room and four-room goal tasks from the previous
chapter, see Section 5.5.1. The data gathering for the representations used
for preprocessing for the RL agents is unchanged from the previous chapter.

####  6.2.3 Obstacle avoidance

In this gridworld environment, the agent is placed in a room filled with
circular objects that move randomly in each step (Fig. 6.2).

Fig 6.2: Obstacle avoidance environment. The agent is rewarded for maximizing
the distance between itself and the closest circle. The left figure displays
the full world state and the right figure displays the corresponding
observation that the agent receives.

The agent receives visual inputs of dimension $56\times 56\times 3$ and must
stay as far away from the objects as possible, as it receives a reward that is
equal to the Euclidean distance between itself and the closest circle. The
episode ends with a reward of $0$ after $100$ time steps or when the agent
collides with a circle, which gives a reward of $-1$. The discount factor for
the environment is 0.9.

####  6.2.4 Architectures

All representations and policies have the same architecture as the in the
previous chapter, except that the output of the representation has been
lowered to a 16-dimensional vector for lower computational times, see Table
5.1 and Table 5.2. The RL models have the same architecture except that the
representation and policy modules are trained end-to-end. The prediction
module of the LARP network is the same as in Chapter 4, see Table 4.3. The
mutual information neural estimator network used for training GrICA is the
same as in Chapter 3, see Section 3.4.

###  6.3 Results and discussion

####  6.3.1 Visual cart-pole

The results from the visual cart-pole experiment can be seen in Fig. 6.3. The
deep RL agent achieves the highest mean reward by a significant margin,
probably because the environment requires the agent to take quick actions in
response to the fast-changing environment.

Fig 6.3: cart-pole results. Each point is the aggregate of 5 different
policies that are trained from scratch and tested for 20 episodes, each.

The learning curve is the worst when the agent is trained on the LARP
representations, which is to be expected as it is learned to be paired with a
representation predictor and graph search, which it is not used for this
scenario.

The RewPred and GrICA learning curves look similarly good, with the RewPred
representation achieving a slightly higher average reward. Both
representations should theoretically be useful here: learning a representation
that is predictive of how close the pole is from falling down (RewPred) ought
to guide the agent’s actions away from states where the pole is about to fall
down, and recovering the three statistically independent latent variables
generating the environment (GrICA), which would be a perfect compression of
the environment. In this scenario, however, they do not beat the
representation learning of the Deep RL algorithm.

####  6.3.2 Room goal-finding

The results for training on the smooth-reward prediction RewPred111Denoted
”Ours 64r” in the previous chapter. and Deep RL representations are kept in
the plots from the previous chapter, and we have added the results for the
GrICA and the LARP representations.

##### Two-room environment

The results for the goal-finding task, when the agent starts facing a random
direction between two rooms and must locate a static goal, can be seen in Fig.
6.4. We display here the version of the reward-predictive RewPred
representation from the previous chapter that is trained on smoothed rewards.

Fig 6.4: Gridworld comparison: two-room goal-finding results. The agent starts
in between two rooms and get a reward from reaching a static goal location
that is one of two rooms. Each point in the learning curve is aggregated from
10 initializations of policies that run 10 test episodes each.

For the ACKTR policy, both the LARP and the GrICA learning curves222See the
previous chapter for the RewPred results are considerably lower than those of
the other two, with the GrICA representation again looking slightly better out
of the two, both when the mean reward and the minimum episode lengths are
considered. This is probably also due to the fact that the advantage of the
LARP representation disappears when the predictor that it is trained with is
discarded and the representation is being used in a model-free setting. For
the PPO2 policy, neither the policies that train on the LARP nor the GrICA
representations show signs of learning to solve the environment.

##### Four-room environment

In this experiment, no combination of RL agent and representation shows
progress toward solving the environment, except for RewPred paired with ACKTR.
Both the LARP and GrICA variants barely display learning in the previous
experiment, but all signs of progress disappear when the environment is
sufficiently complex. The training results can be seen in Fig. 6.5.

Fig 6.5: Four-room goal-finding results. The agent starts at a random location
get a reward from reaching a dynamic goal location that is in one of four
rooms. Each point in the learning curve is aggregated from 10 initializations
of policies that run 10 test episodes each.

####  6.3.3 Obstacle avoidance

The algorithm’s learning curves for the obstacle avoidance experiment can be
seen in Fig. 6.6. For both PPO2 and ACKTR, we observe that preprocessing with
our methods is not beneficial as the environment is solved the fastest using
deep RL, and by a significant margin for ACKTR. This shows that our method is
outperformed by regular deep RL for environments where short-term, reactive
policies are needed – in contrast to the goal-finding tasks, where the agent
needs to execute a long-term plan.

Fig 6.6: Obstacle avoidance results. The agent gets a reward for keeping its
distance from objects that move randomly. Each point in the learning curve is
aggregated from 10 initializations of policies that run 20 test episodes each.

###  6.4 Conclusion

In this chapter, we investigated how the three methods, that we have developed
in this PhD work, compare when they are used for preprocessing visual inputs
for RL agents. The RL agents are tested in four different environments, two of
which require short-term decision-making and the other two require long-term
decision-making for success. Our representation that was developed in Chapter
3, GrICA, does not facilitate learning for any of the environments. The same
holds true for the Chapter 4 representation, LARP, although that one was not
developed for preprocessing inputs to RL agents but rather to be used in
conjunction with a prediction function for planning in a latent representation
space. Our reward-predictive representation from Chapter 5 is shown to speed
up learning for a deep RL agent on the long-term planning tasks.

##  Chapter 7 Summary and conclusion

Just as the complexity of the tasks that deep reinforcement learning (RL) is
being applied to increases, so does the apparent number of problems the field
has. The goal of this research was to discover useful representations that can
be used to alleviate two of modern deep RL’s problems, namely, those of
training instability and data inefficiency. Over the course of the PhD work,
we developed three representation learning methods and investigated their
suitability for fulfilling this goal:

  * GrICA:

A gradient-based ICA method for learning statistically independent features
(Chapter 3). We estimate the mutual information between one output component
of an encoder and all the others using a neural network. In a push-pull
fashion, the mutual information estimator and the encoder are trained,
resulting in a system that outputs statistically independent features of the
input. The output of GrICA is compare to the output of FastICA, an established
ICA problem-solving method in the literature, for noisy blind signal
separation.

  * LARP:

This system, which we call "Latent Representation Predictor" (Chapter 4),
learns a transition model of the environment. We evaluate our system by
combining it with graph search to manipulate toy objects to match a given
viewpoint. Our algorithm learns a state representation jointly with a one-step
lookahead predictor. We discuss and compare three different constraints that
can be placed on the system to prevent the solution from collapsing to a
constant function. Our approach outperforms deep RL in a low-data regime on
the viewpoint-matching task.

  * RewPred:

A reward-predictive representation, that is learned along with a jointly
learned reward predictor (Chapter 5). The reward predictor is employed for
reward shaping: The agent is rewarded for moving from states of low predicted
rewards to states of higher predicted rewards. The method is tested in several
grid world environments where the agent must reach a goal. The learning of
deep RL agents is sped up when their inputs are preprocessed by the RewPred
representation in our experiments and the reward shaping is helpful when the
environment is sufficiently complex.

Not every unsupervised learning technique yields a state representation that
is useful in the context of RL, as seen by the results across the board using
our GrICA algorithm. However, based on our results, we have found that the
training of deep RL methods can be augmented by learning appropriate
representations in a self-supervised manner in environments where the agent
must carry out long-term planning to reach a single goal state.

An important limitation of this work is the high cost of computational
resources required for carrying out RL research. To illustrate, reproducing
DeepMind’s 2017 Go paper (Silver et al.,, 2017) is estimated to cost 35
million dollars using Google’s cloud computing service (Huang,, 2018) – a
budget currently unavailable to PhD students.

Reproducing their results only involves maximizing the performance of a single
model, which does not take into account the work behind finding the final
model. RL methods have many potential settings to tune, and running
hyperparameter searches is costly. For this reason, we concentrated only on
using the default parameters of RL model implementations as offered by the
Stable Baselines package.

To make matters worse, the performance of RL techniques can be highly
sensitive to the hyperparameter settings, and bad luck can cause the
researcher to dismiss a method if unfortunate values are initially chosen. A
larger computational budget allows researchers to try more combinations of
parameters when new methods are tested, lessening the risk of promising
approaches being prematurely dismissed due to bad luck.

The methods were also tested for preprocessing the visual inputs of deep RL
algorithms in environments where there is a set of state that the agent must
avoid, namely, preventing a pole from falling off a cart or preventing the
agent from colliding with randomly moving objects. Preprocessing the inputs
using our methods is not beneficial in these cases, potentially indicating
that allowing deep RL algorithms to learn the state representations is
preferable for tasks that require immediate reactions to quickly changing
environments, particularly where inaction leads to an undesirable outcome.

Although our gradient-based ICA was not found advantageous in our goal of
augmenting deep RL training, it is successful in recovering independent, noisy
sources just as well as FastICA. A promising avenue of research is to take
advantage of the flexibility offered by our method. Our algorithm can be
paired with any differentiable function, such as convolutional neural networks
(convnets), to tackle more difficult problems, for example nonlinear ICA. Even
though general nonlinear ICA problems are ill-posed, regularization can be
applied to make them well-posed. This can in principle be done using our
method via the design of the neural network.

One aspect of representation learning that has become popular in recent years,
but was not investigated in this PhD work, is the application of memory
modules in neural networks. The role of memory can easily be integrated into
our methods by designing our function approximators as artificial recurrent
neural networks, for example by combining long short-term memory networks with
convnets.

Although the problem of learning representations in the context of RL remains
unsolved, the implication of our work is that taking advantage of the rich
unsupervised source of supervision that is hidden in RL environments can lead
to data-efficient, stable algorithms that are resilient to changes in the
environment.

A straightforward way of extending our work would be to take advantage of the
full data that is given in every transition in an RL environment: the current
state $s$, the action $a$ taken in the state $s$, the resulting next state
$s^{\prime}$ and the reward $r$ given to the agent by the environment. It
would be an interesting future line of work to do this by simply combining the
loss function of LARP, which trains on $(s,a,s^{\prime})$ tuples, and RewPred,
which trains on $(r,s^{\prime})$ tuples. This new algorithm would train the
representation to be simultaneously predictable for a representation
predictor, but also to be useful for a reward predictor. See Fig. 7.1 for a
Venn diagram that summarizes the sources of supervision that our proposed
extension would use in addition to the sources of supervision that are used by
our methods.

Fig 7.1: Supervision source Venn diagram. An illustration of the components of
the RL transition tuples $(s,a,r,s^{\prime})=(\text{state},\text{
action},\text{ reward},\text{ next state})$ that are used by our methods:
GrICA (Chapter 3), LARP (Chapter 4), RewPred (Chapter 5). These tuples consist
of all the information involved in a single transition in an RL environment.
The intersection of every component corresponds to the theoretical extension
of our work, combining elements of LARP and RewPred.

This dissertation adds to a rapidly growing body of knowledge that sits at the
intersection of deep learning, reinforcement learning and representation
learning. We contribute three novel methods of learning state representations
to the literature and experimentally evaluate their effectiveness in the
context of reinforcement learning. A long road to artificial intelligence that
matches our general and efficient problem-solving capability still lies ahead
of us.

## References

  * Abadi et al., (2016) Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016).  Tensorflow: A system for large-scale machine learning.  In 12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16), pages 265–283.
  * Alom et al., (2019) Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., Hasan, M., Van Essen, B. C., Awwal, A. A., and Asari, V. K. (2019).  A state-of-the-art survey on deep learning theory and architectures.  Electronics, 8(3):292.
  * Amari et al., (1996) Amari, S.-i., Cichocki, A., and Yang, H. H. (1996).  A new learning algorithm for blind signal separation.  In Advances in neural information processing systems, pages 757–763.
  * Amodei and Clark, (2016) Amodei, D. and Clark, J. (2016).  Faulty reward functions in the wild.  https://blog.openai.com/faulty-reward-functions/,2016.
  * Arora et al., (2018) Arora, S., Hu, W., and Kothari, P. K. (2018).  An analysis of the t-sne algorithm for data visualization.  In Conference on Learning Theory, pages 1455–1462. PMLR.
  * Baldi and Hornik, (1989) Baldi, P. and Hornik, K. (1989).  Neural networks and principal component analysis: Learning from examples without local minima.  Neural networks, 2(1):53–58.
  * Barreto et al., (2016) Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Van Hasselt, H., and Silver, D. (2016).  Successor features for transfer in reinforcement learning.  arXiv preprint arXiv:1606.05312.
  * Belghazi et al., (2018) Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. D. (2018).  MINE: Mutual Information Neural Estimation.  arXiv preprint arXiv:1801.04062.
  * Belkin and Niyogi, (2003) Belkin, M. and Niyogi, P. (2003).  Laplacian eigenmaps for dimensionality reduction and data representation.  Neural computation, 15(6):1373–1396.
  * Bell and Sejnowski, (1995) Bell, A. J. and Sejnowski, T. J. (1995).  A non-linear information maximisation algorithm that performs blind separation.  In Advances in neural information processing systems, pages 467–474.
  * Bellman et al., (1957) Bellman, R., Corporation, R., and Collection, K. M. R. (1957).  Dynamic Programming.  Rand Corporation research study. Princeton University Press.
  * Bengio et al., (2013) Bengio, Y., Courville, A., and Vincent, P. (2013).  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828.
  * Blaschke and Wiskott, (2004) Blaschke, T. and Wiskott, L. (2004).  CuBICA: Independent component analysis by simultaneous third-and fourth-order cumulant diagonalization.  IEEE Transactions on Signal Processing, 52(5):1250–1256.
  * Blaschke et al., (2007) Blaschke, T., Zito, T., and Wiskott, L. (2007).  Independent slow feature analysis and nonlinear blind source separation.  Neural computation, 19(4):994–1021.
  * Brakel and Bengio, (2017) Brakel, P. and Bengio, Y. (2017).  Learning independent features with adversarial nets for non-linear ICA.  arXiv preprint arXiv:1710.05050.
  * Brys et al., (2015) Brys, T., Harutyunyan, A., Taylor, M. E., and Nowé, A. (2015).  Policy transfer using reward shaping.  In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 181–188.
  * Brys et al., (2014) Brys, T., Harutyunyan, A., Vrancx, P., Taylor, M. E., Kudenko, D., and Nowé, A. (2014).  Multi-objectivization of reinforcement learning problems by reward shaping.  In 2014 international joint conference on neural networks (IJCNN), pages 2315–2322. IEEE.
  * Bucilua et al., (2006) Bucilua, C., Caruana, R., and Niculescu-Mizil, A. (2006).  Model compression.  In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541.
  * Buckman et al., (2018) Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. (2018).  Sample-efficient reinforcement learning with stochastic ensemble value expansion.  In Advances in Neural Information Processing Systems, pages 8224–8234.
  * Cardoso, (1997) Cardoso, J.-F. (1997).  InfoMax and maximum likelihood for blind source separation.  IEEE Signal processing letters, 4(4):112–114.
  * Chen et al., (2015) Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z. (2015).  Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems.  arXiv preprint arXiv:1512.01274.
  * Chen et al., (2016) Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016).  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  In Advances in neural information processing systems, pages 2172–2180.
  * Chevalier-Boisvert et al., (2018) Chevalier-Boisvert, M., Willems, L., and Pal, S. (2018).  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid.
  * Chollet et al., (2015) Chollet, F. et al. (2015).  Keras.  https://keras.io.
  * Chua et al., (2018) Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018).  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  In Advances in Neural Information Processing Systems, pages 4754–4765.
  * Clark, (2015) Clark, J. (2015).  Why 2015 was a breakthrough year in artificial intelligence.  Bloomberg Technology [online journal], 18(1).
  * Corneil et al., (2018) Corneil, D., Gerstner, W., and Brea, J. (2018).  Efficient model-based deep reinforcement learning with variational state tabulation.  arXiv preprint arXiv:1802.04325.
  * Cuccu et al., (2018) Cuccu, G., Togelius, J., and Cudré-Mauroux, P. (2018).  Playing atari with six neurons.  arXiv preprint arXiv:1806.01363.
  * Curry, (1944) Curry, H. B. (1944).  The method of steepest descent for non-linear minimization problems.  Quarterly of Applied Mathematics, 2(3):258–261.
  * Cybenko, (1989) Cybenko, G. (1989).  Approximation by superpositions of a sigmoidal function.  Mathematics of control, signals and systems, 2(4):303–314.
  * Darmois, (1953) Darmois, G. (1953).  Analyse générale des liaisons stochastiques: etude particulière de l’analyse factorielle linéaire.  Revue de l’Institut international de statistique, pages 2–8.
  * Deng et al., (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009).  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.
  * Denton et al., (2017) Denton, E. L. et al. (2017).  Unsupervised learning of disentangled representations from video.  In Advances in neural information processing systems, pages 4414–4423.
  * Dhariwal et al., (2017) Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., and Zhokhov, P. (2017).  Openai baselines.  https://github.com/openai/baselines.
  * Donsker and Varadhan, (1975) Donsker, M. D. and Varadhan, S. S. (1975).  Asymptotic evaluation of certain markov process expectations for large time, i.  Communications on Pure and Applied Mathematics, 28(1):1–47.
  * Dozat, (2016) Dozat, T. (2016).  Incorporating Nesterov momentum into ADAM.
  * Du et al., (2019) Du, S. S., Kakade, S. M., Wang, R., and Yang, L. F. (2019).  Is a good representation sufficient for sample efficient reinforcement learning?  arXiv preprint arXiv:1910.03016.
  * Efthymiadis and Kudenko, (2013) Efthymiadis, K. and Kudenko, D. (2013).  Using plan-based reward shaping to learn strategies in starcraft: Broodwar.  In 2013 IEEE Conference on Computational Inteligence in Games (CIG), pages 1–8. IEEE.
  * Escalante-B and Wiskott, (2013) Escalante-B, A. N. and Wiskott, L. (2013).  How to solve classification and regression problems on high-dimensional data with a supervised extension of slow feature analysis.  The Journal of Machine Learning Research, 14(1):3683–3719.
  * Gal et al., (2016) Gal, Y., McAllister, R., and Rasmussen, C. E. (2016).  Improving pilco with bayesian neural network dynamics models.  In Data-Efficient Machine Learning workshop, ICML, volume 4, page 34.
  * Garcia et al., (1989) Garcia, C. E., Prett, D. M., and Morari, M. (1989).  Model predictive control: theory and practice—a survey.  Automatica, 25(3):335–348.
  * Gelada et al., (2019) Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Bellemare, M. G. (2019).  Deepmdp: Learning continuous latent space models for representation learning.  arXiv preprint arXiv:1906.02736.
  * Gidaris et al., (2018) Gidaris, S., Singh, P., and Komodakis, N. (2018).  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728.
  * Gogna and Majumdar, (2016) Gogna, A. and Majumdar, A. (2016).  Semi supervised autoencoder.  In International Conference on Neural Information Processing, pages 82–89. Springer.
  * Goodfellow et al., (2016) Goodfellow, I., Bengio, Y., and Courville, A. (2016).  Deep Learning.  MIT Press.  http://www.deeplearningbook.org.
  * Goodfellow et al., (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680.
  * Goroshin et al., (2015) Goroshin, R., Mathieu, M. F., and LeCun, Y. (2015).  Learning to linearize under uncertainty.  In Advances in Neural Information Processing Systems, pages 1234–1242.
  * Ha and Schmidhuber, (2018) Ha, D. and Schmidhuber, J. (2018).  World models.  arXiv preprint arXiv:1803.10122.
  * Hadsell et al., (2006) Hadsell, R., Chopra, S., and LeCun, Y. (2006).  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE.
  * Hafner et al., (2019) Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019).  Learning latent dynamics for planning from pixels.  In International Conference on Machine Learning, pages 2555–2565. PMLR.
  * Hamilton et al., (2014) Hamilton, W., Fard, M. M., and Pineau, J. (2014).  Efficient learning and planning with compressed predictive states.  The Journal of Machine Learning Research, 15(1):3395–3439.
  * Henaff et al., (2017) Henaff, M., Whitney, W. F., and LeCun, Y. (2017).  Model-based planning with discrete and continuous actions.  arXiv preprint arXiv:1705.07177.
  * Hill et al., (2018) Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., and Wu, Y. (2018).  Stable baselines.  https://github.com/hill-a/stable-baselines.
  * Hinton et al., (2015) Hinton, G., Vinyals, O., and Dean, J. (2015).  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531.
  * Hjelm et al., (2018) Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Trischler, A., and Bengio, Y. (2018).  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:R1808.06670.
  * Hlynsson et al., (2019) Hlynsson, H., Escalante-B., A., and Wiskott, L. (2019).  Measuring the data efficiency of deep learning methods.  In Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods. SCITEPRESS - Science and Technology Publications.
  * Hlynsson et al., (2020) Hlynsson, H. D., Schüler, M., Schiewer, R., Glasmachers, T., and Wiskott, L. (2020).  Latent representation prediction networks.  arXiv preprint arXiv:2009.09439.
  * Hlynsson and Wiskott, (2019) Hlynsson, H. D. and Wiskott, L. (2019).  Learning gradient-based ICA by neurally estimating mutual information.  In Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), pages 182–187. Springer.
  * Hlynsson and Wiskott, (2021) Hlynsson, H. D. and Wiskott, L. (2021).  Reward prediction for representation learning and reward shaping.
  * Huang, (2018) Huang, D. (2018).  How much did alphago zero cost?  Internet: https://www.yuzeh.com/data/agz-cost.html.
  * Hyvärinen and Oja, (2000) Hyvärinen, A. and Oja, E. (2000).  Independent component analysis: algorithms and applications.  Neural networks, 13(4-5):411–430.
  * Hyvärinen and Pajunen, (1999) Hyvärinen, A. and Pajunen, P. (1999).  Nonlinear independent component analysis: Existence and uniqueness results.  Neural Networks, 12(3):429–439.
  * Ilin and Honkela, (2004) Ilin, A. and Honkela, A. (2004).  Post-nonlinear independent component analysis by variational Bayesian learning.  In International Conference on Independent Component Analysis and Signal Separation, pages 766–773. Springer.
  * Irpan, (2018) Irpan, A. (2018).  Deep reinforcement learning doesn’t work yet.  Internet: https://www. alexirpan. com/2018/02/14/rl-hard. html.
  * Jutten and Karhunen, (2003) Jutten, C. and Karhunen, J. (2003).  Advances in nonlinear blind source separation.  In Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003), pages 245–256.
  * Kaelbling, (1993) Kaelbling, L. P. (1993).  Learning to achieve goals.  In IJCAI, pages 1094–1099. Citeseer.
  * Khan et al., (2018) Khan, S., Rahmani, H., Shah, S. A. A., and Bennamoun, M. (2018).  A guide to convolutional neural networks for computer vision.  Synthesis Lectures on Computer Vision, 8(1):1–207.
  * Kingma and Ba, (2014) Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kingma and Welling, (2013) Kingma, D. P. and Welling, M. (2013).  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114.
  * Kiran et al., (2021) Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A. A., Yogamani, S., and Pérez, P. (2021).  Deep reinforcement learning for autonomous driving: A survey.  IEEE Transactions on Intelligent Transportation Systems.
  * Krizhevsky et al., (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012).  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105.
  * Kurutach et al., (2018) Kurutach, T., Tamar, A., Yang, G., Russell, S. J., and Abbeel, P. (2018).  Learning plannable representations with causal infogan.  In Advances in Neural Information Processing Systems, pages 8733–8744.
  * Lample and Chaplot, (2017) Lample, G. and Chaplot, D. S. (2017).  Playing fps games with deep reinforcement learning.  Proceedings of the AAAI Conference on Artificial Intelligence, 31(1).
  * LeCun et al., (2004) LeCun, Y., Huang, F. J., and Bottou, L. (2004).  Learning methods for generic object recognition with invariance to pose and lighting.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II–104. IEEE.
  * LeCun et al., (2012) LeCun, Y. A., Bottou, L., Orr, G. B., and Müller, K.-R. (2012).  Efficient backprop.  In Neural networks: Tricks of the trade, pages 9–48. Springer.
  * Lehnert and Littman, (2020) Lehnert, L. and Littman, M. L. (2020).  Successor features combine elements of model-free and model-based reinforcement learning.  Journal of Machine Learning Research, 21(196):1–53.
  * Lehnert et al., (2020) Lehnert, L., Littman, M. L., and Frank, M. J. (2020).  Reward-predictive representations generalize across tasks in reinforcement learning.  PLoS computational biology, 16(10):e1008317.
  * Li et al., (2019) Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., and Gao, X. (2019).  Deep learning in bioinformatics: Introduction, application, and perspective in the big data era.  Methods, 166:4–21.
  * Linnainmaa, (1970) Linnainmaa, S. (1970).  The representation of the cumulative rounding error of an algorithm as a taylor expansion of the local rounding errors.  Master’s Thesis (in Finnish), Univ. Helsinki, pages 6–7.
  * Liu et al., (2020) Liu, K., Kurutach, T., Tung, C., Abbeel, P., and Tamar, A. (2020).  Hallucinative topological memory for zero-shot visual planning.  arXiv preprint arXiv:2002.12336.
  * Maaten and Hinton, (2008) Maaten, L. v. d. and Hinton, G. (2008).  Visualizing data using t-sne.  Journal of machine learning research, 9(Nov):2579–2605.
  * Marashi et al., (2012) Marashi, M., Khalilian, A., and Shiri, M. E. (2012).  Automatic reward shaping in reinforcement learning using graph analysis.  In 2012 2nd International eConference on Computer and Knowledge Engineering (ICCKE), pages 111–116. IEEE.
  * Masci et al., (2011) Masci, J., Meier, U., Cireşan, D., and Schmidhuber, J. (2011).  Stacked convolutional auto-encoders for hierarchical feature extraction.  In International conference on artificial neural networks, pages 52–59. Springer.
  * Mataric, (1994) Mataric, M. J. (1994).  Reward functions for accelerated learning.  In Machine learning proceedings 1994, pages 181–189. Elsevier.
  * McCorduck, (1979) McCorduck, P. (1979).  Machines who think.
  * McInnes et al., (2018) McInnes, L., Healy, J., and Melville, J. (2018).  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426.
  * Mnih et al., (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013).  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602.
  * Mnih et al., (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529.
  * Ng et al., (1999) Ng, A. Y., Harada, D., and Russell, S. (1999).  Policy invariance under reward transformations: Theory and application to reward shaping.  In Icml, volume 99, pages 278–287.
  * Nwankpa et al., (2018) Nwankpa, C., Ijomah, W., Gachagan, A., and Marshall, S. (2018).  Activation functions: Comparison of trends in practice and research for deep learning.  arXiv preprint arXiv:1811.03378.
  * Oh et al., (2015) Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S. (2015).  Action-conditional video prediction using deep networks in atari games.  In Advances in neural information processing systems, pages 2863–2871.
  * Paszke et al., (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019).  Pytorch: An imperative style, high-performance deep learning library.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
  * Pathak et al., (2018) Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, T. (2018).  Zero-shot visual imitation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2050–2053.
  * Pearson, (1901) Pearson, K. (1901).  Liii. on lines and planes of closest fit to systems of points in space.  The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559–572.
  * Pedregosa et al., (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011).  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830.
  * Rhodios, (2008) Rhodios, A. (2008).  The Argonautika, volume 25.  Univ of California Press.
  * Richards, (2005) Richards, A. G. (2005).  Robust constrained model predictive control.  PhD thesis, Massachusetts Institute of Technology.
  * Richthofer and Wiskott, (2015) Richthofer, S. and Wiskott, L. (2015).  Predictable feature analysis.  In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 190–196. IEEE.
  * Rosenblatt, (1958) Rosenblatt, F. (1958).  The perceptron: a probabilistic model for information storage and organization in the brain.  Psychological review, 65(6):386.
  * Saphal et al., (2020) Saphal, R., Ravindran, B., Mudigere, D., Avancha, S., and Kaul, B. (2020).  Seerl: Sample efficient ensemble reinforcement learning.  arXiv preprint arXiv:2001.05209.
  * Savinov et al., (2018) Savinov, N., Dosovitskiy, A., and Koltun, V. (2018).  Semi-parametric topological memory for navigation.  arXiv preprint arXiv:1803.00653.
  * Schaul et al., (2015) Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015).  Universal value function approximators.  In International conference on machine learning, pages 1312–1320.
  * Schmidhuber, (1992) Schmidhuber, J. (1992).  Learning factorial codes by predictability minimization.  Neural Computation, 4(6):863–879.
  * Schmidhuber, (2018) Schmidhuber, J. (2018).  Unsupervised neural networks fight in a minimax game.  https://people.idsia.ch/~juergen/unsupervised-neural-nets-fight-minimax-game.html.
  * Schroff et al., (2015) Schroff, F., Kalenichenko, D., and Philbin, J. (2015).  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823.
  * Schüler et al., (2019) Schüler, M., Hlynsson, H. D., and Wiskott, L. (2019).  Gradient-based training of slow feature analysis by differentiable approximate whitening.  In Asian Conference on Machine Learning, pages 316–331. PMLR.
  * Schulman et al., (2015) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2015).  High-dimensional continuous control using generalized advantage estimation.  arXiv preprint arXiv:1506.02438.
  * Schulman et al., (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Scikit-learn, (2019) Scikit-learn (2019).  Blind source separation using FastICA.  https://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html.  Accessed: 2019-02-24.
  * Sejnowski, (2018) Sejnowski, T. J. (2018).  The deep learning revolution.  Mit Press.
  * Sharif Razavian et al., (2014) Sharif Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S. (2014).  Cnn features off-the-shelf: an astounding baseline for recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813.
  * Shlens, (2014) Shlens, J. (2014).  A tutorial on principal component analysis.  arXiv preprint arXiv:1404.1100.
  * Silver et al., (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017).  Mastering the game of go without human knowledge.  nature, 550(7676):354–359.
  * Simonyan and Zisserman, (2014) Simonyan, K. and Zisserman, A. (2014).  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556.
  * Sprekeler et al., (2014) Sprekeler, H., Zito, T., and Wiskott, L. (2014).  An extension of slow feature analysis for nonlinear blind source separation.  The Journal of Machine Learning Research, 15(1):921–947.
  * Srinivas et al., (2018) Srinivas, A., Jabri, A., Abbeel, P., Levine, S., and Finn, C. (2018).  Universal planning networks.  arXiv preprint arXiv:1804.00645.
  * Sun et al., (2014) Sun, Y., Chen, Y., Wang, X., and Tang, X. (2014).  Deep learning face representation by joint identification-verification.  In Advances in neural information processing systems, pages 1988–1996.
  * Sutton, (1988) Sutton, R. S. (1988).  Learning to predict by the methods of temporal differences.  Machine learning, 3(1):9–44.
  * Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018).  Reinforcement learning: An introduction.  MIT press.
  * Sutton et al., (1999) Sutton, R. S., Precup, D., and Singh, S. (1999).  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  Artificial intelligence, 112(1-2):181–211.
  * Taleb and Jutten, (1999) Taleb, A. and Jutten, C. (1999).  Source separation in post-nonlinear mixtures.  IEEE Transactions on signal Processing, 47(10):2807–2820.
  * Tamar et al., (2016) Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. (2016).  Value iteration networks.  In Advances in Neural Information Processing Systems, pages 2154–2162.
  * Tieleman and Hinton, (2012) Tieleman, T. and Hinton, G. (2012).  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  COURSERA: Neural networks for machine learning, 4(2):26–31.
  * Tobler, (1970) Tobler, W. R. (1970).  A computer movie simulating urban growth in the detroit region.  Economic geography, 46(sup1):234–240.
  * Trott et al., (2019) Trott, A., Zheng, S., Xiong, C., and Socher, R. (2019).  Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards.  arXiv preprint arXiv:1911.01417.
  * Veličković et al., (2018) Veličković, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., and Hjelm, R. D. (2018).  Deep graph InfoMax.  arXiv preprint arXiv:1809.10341.
  * Vincent et al., (2008) Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008).  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning, pages 1096–1103.
  * Vincent et al., (2010) Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., and Bottou, L. (2010).  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  Journal of machine learning research, 11(12).
  * Vondrick et al., (2016) Vondrick, C., Pirsiavash, H., and Torralba, A. (2016).  Anticipating visual representations from unlabeled video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 98–106.
  * Wang et al., (2019) Wang, A., Kurutach, T., Liu, K., Abbeel, P., and Tamar, A. (2019).  Learning robotic manipulation through visual planning and acting.  arXiv preprint arXiv:1905.04411.
  * Wang et al., (2018) Wang, X., Ma, T., Ainooson, J., Cha, S., Wang, X., Molla, A., and Kunda, M. (2018).  The toybox dataset of egocentric visual object transformations.  arXiv preprint arXiv:1806.06034.
  * Watkins and Dayan, (1992) Watkins, C. J. and Dayan, P. (1992).  Q-learning.  Machine learning, 8(3-4):279–292.
  * Winston, (2010) Winston, P. (2010).  6.034 artificial intelligence, fall 2010.
  * Wiskott and Sejnowski, (2002) Wiskott, L. and Sejnowski, T. J. (2002).  Slow feature analysis: Unsupervised learning of invariances.  Neural computation, 14(4):715–770.
  * Wolf et al., (2020) Wolf, T., Chaumond, J., Debut, L., Sanh, V., Delangue, C., Moi, A., Cistac, P., Funtowicz, M., Davison, J., Shleifer, S., et al. (2020).  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45.
  * Wu et al., (2017) Wu, Y., Mansimov, E., Grosse, R. B., Liao, S., and Ba, J. (2017).  Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation.  In Advances in neural information processing systems, pages 5279–5288.
  * Xu et al., (2019) Xu, D., Martín-Martín, R., Huang, D.-A., Zhu, Y., Savarese, S., and Fei-Fei, L. F. (2019).  Regression planning networks.  In Advances in Neural Information Processing Systems, pages 1319–1329.
  * Yao et al., (2019) Yao, H., Zhu, D.-l., Jiang, B., and Yu, P. (2019).  Negative log likelihood ratio loss for deep neural network classification.  In Proceedings of the Future Technologies Conference, pages 276–282. Springer.
  * Zeiler and Fergus, (2014) Zeiler, M. D. and Fergus, R. (2014).  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer.
  * Zhang et al., (2016) Zhang, R., Isola, P., and Efros, A. A. (2016).  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer.
  * Zhang et al., (2019) Zhang, S., Yao, L., Sun, A., and Tay, Y. (2019).  Deep learning based recommender system: A survey and new perspectives.  ACM Computing Surveys (CSUR), 52(1):1–38.
  * Ziehe et al., (2003) Ziehe, A., Kawanabe, M., Harmeling, S., and Müller, K.-R. (2003).  Blind separation of post-nonlinear mixtures using linearizing transformations and temporal decorrelation.  Journal of Machine Learning Research, 4(Dec):1319–1338.
  * Zou et al., (2019) Zou, H., Ren, T., Yan, D., Su, H., and Zhu, J. (2019).  Reward shaping via meta-learning.  arXiv preprint arXiv:1901.09330.

## Appendix A