# Learning to Generalize with Object-centric Agents
in the Open World Survival Game Crafter

Aleksandar Stanić† 1, Yujin Tang2, David Ha2, Jürgen Schmidhuber1 3 † The
majority of this work was carried out as a student researcher at Google Brain.
1 IDSIA, USI, SUPSI, Lugano, Switzerland, 2 Google Brain, 3 AI Initiative,
KAUST, Thuwal, Saudi Arabia.A shorter version of this paper appeared at the
Decision Awareness in Reinforcement Learning and Responsible Decision Making
in Dynamic Environments Workshops at ICML 2022.

###### Abstract

Reinforcement learning agents must generalize beyond their training
experience. Prior work has focused mostly on identical training and evaluation
environments. Starting from the recently introduced Crafter benchmark, a 2D
open world survival game, we introduce a new set of environments suitable for
evaluating some agent’s ability to generalize on previously unseen (numbers
of) objects and to adapt quickly (meta-learning). In Crafter, the agents are
evaluated by the number of unlocked achievements (such as collecting
resources) when trained for 1M steps. We show that current agents struggle to
generalize, and introduce novel object-centric agents that improve over strong
baselines. We also provide critical insights of general interest for future
work on Crafter through several experiments. We show that careful hyper-
parameter tuning improves the PPO baseline agent by a large margin and that
even feedforward agents can unlock almost all achievements by relying on the
inventory display. We achieve new state-of-the-art performance on the original
Crafter environment. Additionally, when trained beyond 1M steps, our tuned
agents can unlock almost all achievements. We show that the recurrent PPO
agents improve over feedforward ones, even with the inventory information
removed. We introduce CrafterOOD, a set of 15 new environments that evaluate
OOD generalization. On CrafterOOD, we show that the current agents fail to
generalize, whereas our novel object-centric agents achieve state-of-the-art
OOD generalization while also being interpretable. Our code is
public.111https://github.com/astanic/crafter-ood

###### Index Terms:

Crafter, Open World Survival Games, Generalization, Object-centric Agents, PPO

##  I Introduction

Common benchmarks and solid baselines are essential for developing new models
and correctly measuring progress in machine learning. Datasets for supervised
learning (such as ImageNet [1]) and game-based environments for reinforcement
learning (RL) (such as Atari [2], ProcGen [3] and MineRL [4, 5]) have played a
crucial role in developing new methods and improving the state of the art. To
judge the improvements offered by novel methods, it is critical to compare
them to solid yet simple baselines, especially in deep RL, where reproducing
existing work is often hard due to a plethora of difficulties [6].

Early benchmarks were important to show that certain deep RL methods can learn
control from high-dimensional images, e.g., neuroevolution via compressed
network encodings [7] on the TORCS benchmark [8], or the DQN agent [9] on the
Atari benchmark [2]. These benchmarks, however, focused on narrow task sets,
and conducted training and evaluation on in-distribution environments, where
the agents could (simply) memorize sequences of actions leading to high reward
without understanding the underlying mechanics of the world. Intelligent
agents should discover salient objects (entities) of the world, identify
relevant properties of objects (e.g., shape is important, color is not) and
ignore irrelevant parts of the environment. For better evaluation, the focus
has recently shifted to studying agent performance on carefully designed
benchmarks [10], with emphasis on agents that generalize to environments
beyond what they are trained on (e.g., ProcGen or Crafter [11]).

Figure 1: Crafter gameplay showing an agent collecting resources (wood) and
crafting a weapon (wooden pickaxe), and its attention. Top row: input images.
Bottom row: visualized attention. Episode steps are plotted horizontally.

Here we focus on Crafter [11], a recently introduced open world survival game
that allows tractable investigation of new agents and their generalization,
exploration, and long-term reasoning capabilities. Compared to other
benchmarks, Crafter has a set of advantages that make it suitable for RL
research: fast iteration speed (agents can be trained within a few hours on a
standard GPU and a single CPU core), model evaluation by inspecting
semantically meaningful achievements unlocked by the agents, and controllable
environmental objects that facilitate systematic studies. Our experiments on
the original Crafter environment offer important, previously unpublished
insights into baselines and environmental workings. In addition, we introduce
two sets of new environments, CrafterOODapp and CrafterOODnum, that test out-
of-distribution (OOD) generalization to unseen types and numbers of objects,
respectively. This also sets the stage for developing fast-adaptation
algorithms in the context of meta-learning.

Our main contributions are: (1) We show that careful hyper-parameter tuning
improves the PPO baseline agent by a large margin. We achieve new state-of-
the-art performance on the original Crafter environment. (2) We show that
feedforward agents can unlock almost all achievements by relying on the
inventory display (bottom part of the images in Figures 1 and 2a) – a type of
“scratchpad”). (3) We show that recurrent agents improve over feedforward
ones, even when the inventory information is removed. (4) We show that tuned
agents can unlock almost all achievements. (5) We introduce CrafterOOD, a set
of 15 new environments, to evaluate OOD generalization of agents. (6) We
introduce novel object-centric agents that achieve state-of-the-art OOD
generalization while being interpretable.

(a)

(b)

(c)

Figure 2: (a) Crafter gameplay. (b) Object-centric Self-Attention (OC-SA)
model with a CLS token and queries (Q), keys (K) and values (V). (c) Slot-
based Object-centric Cross-Attention (OC-CA) model. Pool operation averages
the slots to produce the final input vector fed into the actor and critic
networks.

##  II Environments

Here we briefly summarize essential properties of the Crafter environment and
the newly introduced CrafterOODapp and CrafterOODnum environments. For
additional details on Crafter we refer to [11].

Crafter. Crafter is an open world survival game for RL research whose game
dynamics are inspired by the popular game Minecraft. The benchmark is designed
to facilitate existing research challenges: procedural generation to test
generalization, exploration due to a deep technology tree of achievements
conditioned on one another, high-dimensional image observations requiring
representation learning, memory through partial observability, and finally,
sparse rewards that require long-term reasoning and credit assignment. It has
the right level of complexity that is challenging enough for the current
agents to unlock all achievements, yet not overly complex such that it
facilitates interpretability of the agent’s behavior and evaluation by
unlocking semantically meaningful achievements. In Crafter, a unique terrain
is generated for every episode with grasslands, lakes, and mountains that
contain forests, caves, ores, and lava. The world consists of $64\times 64$
grid cells, but the agent only observes $7\times 9$ grid cells making Crafter
a partially observed environment. At each step the agent receives a $64\times
64\times 3$ input image as in Figure 2a, and returns an action from a
categorical space of 17 actions. Of those actions, four are for moving in the
environment, one is for sleeping, and the rest are for crafting or interacting
with specific tools or resources. For an example of gameplay, see Figure 2a,
where you can see the player (agent) in the middle, a skeleton on its top
right, a zombie bottom-left from the agent, a cow, water, trees, stones, iron
and lava.

In the bottom-left part of Figure 2a the agent’s health status is shown. The
heart represents the number of lives it has left, and next to it, the levels
of food, water, and energy (that it recharges by sleeping). Next to this is
the inventory display that shows the resources (here wood, stone, coal, and
iron) and tools and weapons (here wooden, stone, and iron pickaxes and swords)
it has collected or crafted so far. The agent must collect resources such as
wood, stone, coal, and iron to craft tools. The agent needs to eat food, drink
water, sleep and defend against enemies to prevent its health from reaching
zero. All these agents and resources are represented as objects in the
environment. Eating food, collecting resources, defeating enemies, or crafting
new tools are all “achievements”. The first time the agent unlocks an
achievement in an episode it receives a reward of $1$. On the other hand, for
every lost health point the reward is $-0.1$, and for every regained health
point $+0.1$. When the health reaches zero or when stepping into lava the
agent dies, and the episode is over. Every time the agent unlocks an
achievement previously unlocked in the episode, it does not receive any
further reward. This should encourage the agent to explore and collect as many
achievements as possible instead of exploiting the ones it already knows.
Collecting the diamond is the final and the most difficult achievement. To
collect a diamond, the agent must build several tools that require even more
basic tools and resources to be collected. This results in a very deep
technology tree (see Figure 4 in the Crafter paper [11]).

In total, there are $22$ achievements and the main evaluation metric is the
so-called crafter score $S$. It is computed by averaging the unlocked
achievements in the log space (to account for differences in their
difficulties): $S=\exp(\frac{1}{N}\sum_{i=1}^{N}\ln(1+s_{i}))$, where
$s_{i}\in[0,100]$ is percentage of episodes in which the achievement $i$ was
unlocked and $N$ is the number of achievements (here $N=22$). This score is
invariant to the number of times an achievement was unlocked in an episode
(once it is achieved). Due to averaging in the log space, the score will
increase more for unlocking difficult achievements in a few episodes (e.g. a
diamond) than for unlocking easy ones more frequently (e.g. collecting wood).
The final evaluation metrics is the geometric mean of $S$ over all episodes.

CrafterOODapp. Building on top of Crafter, we introduce CrafterOODapp, where
we develop another axis of variation: object appearance based on color. We
introduce new variants of objects the agent most frequently interacts with:
trees, cows, zombies, stones, coal, and skeletons. For a detailed overview of
the newly introduced objects, see Figure 9. This evaluates generalization to
unseen or infrequently seen objects. The agent is trained on an environment
with one distribution of such objects’ appearances and then evaluated on an
environment with a different distribution, possibly one with objects never
seen during training. In this way, CrafterOODapp sets the stage for the
development of agents based on fast-adaptation or meta-learning.

In the default Crafter environment, the agent always sees only the first
variant of an object. On the other hand, in CrafterOODapp, the agent
encounters one of the four object variants with varying frequencies of
occurrence. For example, in the easiest scenario, the agent sees all four
object variants ($O_{1},O_{2},O_{3},O_{4}$) with equal ($25\%$) probability in
the training environment, and then in the evaluation environment it encounters
only the “last” three objects $O_{2},O_{3},O_{4}$ with equal (but increased,
$33.3\%$) probability. We then generate progressively more challenging
scenarios where the training environment contains more instances of object
variant $O_{1}$. Lastly, in the most extreme case of zero-shot generalization,
we train agents on environments containing only the first object $O_{1}$ and
then evaluate them on environments containing only the last three objects
$O_{2},O_{3},O_{4}$.

CrafterOODnum. Also, building atop Crafter, we introduce CrafterOODnum, where
the agent encounters different numbers of objects during the training and
evaluation phases. As in CrafterOODapp, we vary the objects the agent most
frequently interacts with: trees, coal, cows, zombies, and skeletons. To get a
clear picture of these environments, in Table VI we show the numbers of each
of these objects in the respective environment variants. Starting from default
object distributions in Crafter, we increase or decrease object numbers by
powers of two. These variants are particularly challenging as the agent might
face never-before-seen situations such as fighting against more enemies or
surviving in environments with resources scarcer compared to what it was
trained on. Importantly, changing the number of objects does not change the
maximum score the agent can obtain, as it is still possible to unlock all the
achievements, but with different difficulties. However, this changes the
reward function and makes the credit assignment problem more difficult because
reward is sparser with fewer objects.

##  III Methods

In this section, we describe the agents’ network architectures. We first
introduce CNN feedforward baselines, followed by LSTM-based recurrent agents,
and finally introduce object-centric agents. To optimize agents we use the PPO
[12] implementation in the stable baselines [13].

###  III-A Linear and Recurrent PPO

PPO learns to map images to actions via policy gradients. Two feedforward
variants are investigated that differ by the CNN policy they use. We introduce
a recurrent version based on an LSTM [14] to account for partial
observability. Note that the first application of policy gradients to LSTM
dates back to 2007 [15]. For architecture details see Appendix E.

PPO with NatureCNN (PPO-CNN). This baseline is architecturally identical to
the one used in [11], with the CNN policy from the DQN paper [9].

PPO with size-preserving CNN (PPO-SPCNN). Size-preserving CNN (SPCNN) differs
from the PPO-CNN baseline by the CNN architecture. We introduce it as a
baseline for object-centric agents that use SPCNN for “feature mixing,”
inspired by SlotAttention’s visual encoder [16]. SPCNN does not have pooling
layers, so the resulting output tensor is of the same height and width as the
input image. The policy input tensor is much larger (64x64x64 instead of
8x8x64 for CNN) making the policy also very large (134M vs. 1M parameters).

Recurrent PPO Agents. Crafter is a partially observable environment. The agent
observes only a part of the world (see Figure 2a). To perform well, it needs
to remember the locations of resources, e.g. food, mining materials, and where
it placed objects for crafting new tools, e.g. table or furnace. Additionally,
some achievements require the agent to perform a long chain of reasoning. For
example, to collect a diamond, the agent needs to have an iron pickaxe, for
which it needs to have crafted a furnace and collected coal, for which in turn
it needs to have previously collected wood, placed a table, made a wooden
pickaxe, and collected stone (in that particular order). For the complete
overview of the technology tree, see Figure 4 in [11]. For these reasons, we
introduce recurrent agents (LSTM-CNN and LSTM-SPCNN) where we use LSTMs as the
critic and the actor networks. These networks, in theory, entitle the agent to
have memories of the world map and its inventory and can help unlock
achievements.

###  III-B Object-centric Agents

Our experiments show that the baseline agents fail to generalize to OOD
environments. This is in line with the previous findings that neural networks
fall short in generalization. One hypothesis for this is due to their
inability to dynamically and flexibly bind information distributed throughout
the network [17], also known as the binding problem [18, 19]. To address this
issue, there has been an increased interest in designing object-centric neural
networks that learn (discrete) object representations from raw visual input,
which support efficient learning and generalization to novel scenarios and
behaviors. Object-centric methods have successfully been used for OOD
generalization in supervised and unsupervised learning [20, 21, 22, 23, 24,
16, 25, 26] and in RL [27, 28, 29, 30]. Since Crafter is composed of objects,
we expect these methods to facilitate learning and show stronger
generalization capabilities. In this vein, we designed two object-centric
agents and investigated them on CrafterOOD environments. They take as input
either patches extracted from the image through a size-preserving CNN (Figure
3) or learn their own representation of an object by attending over the whole
input tensor (Figure 7).

Object-centric Self-Attention agents (OC-SA) (Figure 2b and Table XI) learn a
policy via a dot-product self-attention between the input patches and a
learned “CLS” token (a vector initialized to unit Gaussian parameters and
optimized via backprop). This is similar to using the CLS token in BERT [31].
Let $(x_{1},...,x_{k},CLS)\in\mathbb{R}^{(k+1)\times d_{in}}$ be the input
sequence, where $k$ is the number of input patches and $d_{in}$ is their
dimensionality. Dot-product self-attention is defined as:

$Attention(Q,K,V)=softmax\Big{(}\frac{QK^{T}}{\sqrt{d}}\Big{)}V,$ (1)

where $Q\in\mathbb{R}^{(k+1)\times d}$, $K\in\mathbb{R}^{(k+1)\times d}$,
$V\in\mathbb{R}^{(k+1)\times d}$, are the query, key and value matrices,
resulting from a linear mapping of the input sequence onto a space of
dimension $d$. Absolute sinusoidal positional embeddings [32] are added to the
input, enabling the learning of relative patch positions, e.g. if an enemy is
nearby.

Object-centric Cross-Attention agents (OC-CA) (Figure 2c and Table XII) learn
a set of vectors, which we refer to as ’slots’, by attending either to a patch
grid or the whole input (this can be seen as an extreme case of the patch grid
with patch size of one pixel). This idea was introduced in SetTransformer [33]
and successfully used for object detection [34], unsupervised learning of
objects [16], learning permutation-invariant agents [35, 36] and general
perception modules [37, 38, 39]. OC-CA computes attention between queries,
keys and values as in self-attention Equation 1, but with the queries coming
from the $n$ learned slots: $Q\in\mathbb{R}^{n\times d}$,
$K\in\mathbb{R}^{k\times d}$, $V\in\mathbb{R}^{k\times d}$, where $k$ is the
number of input patches or image pixels for learned objects. The extreme patch
size allows for higher expressiveness as each slot can attend to variable-size
or more distant image regions. After cross-attention, slots are pooled by a
mean operation and fed to the policy.

TABLE I: Scores on Crafter for agents trained on 1M environmental steps. Reported are mean scores and standard deviations of 10 random seeds. *score from [11]. Method | Crafter Score
---|---
PPO* |  4.6 $\pm$ 0.3
DreamerV2* |  10.0 $\pm$ 0.2
PPO-CNN |  10.3 $\pm$ 0.6
PPO-SPCNN |  11.6 $\pm$ 0.6
LSTM-CNN |  10.4 $\pm$ 0.2
LSTM-SPCNN |  12.1 $\pm$ 0.8
OC-SA |  11.1 $\pm$ 0.7
OC-CA |  10.0 $\pm$ 0.4
PPO-CNN (no inventory) |  6.9 $\pm$ 0.4
LSTM-CNN (no inventory) |  7.7 $\pm$ 0.5

##  IV Experiments on the Crafter environment

Here we present our findings on the original Crafter environment. We first
investigate feedforward PPO baselines both with and without inventory display.
Recurrent agents are then investigated, and we show that they outperform
feedforward ones. Finally, we evaluate asymptotic performance and show that
the tuned agents can learn to unlock all but the last achievement.

###  IV-A Improved baselines and hyper-parameter analysis.

By tuning a few hyper-parameters, we find that the simple PPO-CNN baseline
outperforms the model-based DreamerV2 [40] (Table I). Surprisingly, even
though the PPO-SPCNN model has 134M parameters, PPO is not only able to train
it, but often outperforms other methods (Tables I and IV). This holds for both
feedforward (PPO-SPCNN) and the recurrent (LSTM-SPCNN) agents. PPO hyper-
parameters we searched over are shown in Table II. For most hyper-parameters,
in the experiments we used the default values (shown in italic) that were
tuned for Atari in previous work; the ones we changed are bolded. For a
detailed hyper-parameter analysis we refer to Appendix B.

The best performing agent in the Crafter Benchmark paper [11] was the
DreamerV2 agent [40]. Therefore, we also tried tuning its hyper-parameters:
the actor entropy scale, the discount factor, the transition, and the entropy
loss scales (recommended in Table B.1 of the DreamerV2 paper [40]), as well as
the batch size, KL loss scale, reward loss scale and the discount $\lambda$.
However, our hyper-parameter search did not improve on results in [11] (see
Table III).

TABLE II:  PPO hyper-parameters we searched over. Final used values are
bolded, unless default values are used (shown in italic).
Hyper-parameter | Sweep Values
---|---
Learning rate |  [0.001, 0.0005, 0.0003, 0.0001, 0.00005]
Batch size |  [64, 128, 256]
Number of rollouts |  [1024, 2048, 4096, 8196, 16384]
Number of epochs |  [3, 4, 6, 7, 10]
Discount factor |  [0.8, 0.9, 0.95, 0.97, 0.99]
GAE $\lambda$ |  [0.5, 0.65, 0.75, 0.85, 0.95]
Clip Range |  [0.1, 0.2, 0.3]
Max Gradient Norm |  [0.1, 0.3, 0.5, 1.0]
TABLE III: DreamerV2 hyper-parameters we searched over.
Hyper-parameter | Sweep Values
---|---
Batch size | $16,32,64$
Discount factor | $[0.9,0.95,0.99,0.999]$
Actor entropy scale | $[0.003,0.001,0.0003,0.0001,0.00003]$
KL loss scale | $[0.1,0.3,1,3]$
Reward loss scale | $[0.5,1,2]$
Discount loss scale | $[0.5,1,2]$
KL balance | $[0.5,0.8,1,2]$
Discount $\lambda$ | $[0.8,0.9,0.95]$

###  IV-B Agents use inventory display as a “scratchpad.”

Although purely feedforward, PPO-SPCNN can unlock almost all achievements
(19/22 for 1M steps and 21/22 achievements when trained for 20M steps, Section
IV-D). This requires the agent to remember unlocked achievements (e.g., I have
an iron pick-axe now, need to mine a diamond next). However, the fact that the
memory-less feedforward models unlocked most achievements leads us to
hypothesize that it uses the inventory display as a “scratchpad.” We evaluate
this by removing the inventory display, thus making the environment much less
observable. Here the agent really needs memory to know when to drink or
whether it has already crafted an item or not. The results in Table I support
our hypothesis, e.g., compare PPO-CNN with and without the inventory display
(‘PPO-CNN (no inventory)‘).

###  IV-C Recurrent improve over feedforward agents, even without inventory
information.

The agent’s reliance on the inventory display as a “scratchpad” led us to
investigate whether recurrent agents could improve over feedforward ones by
memorizing actions, maps, unlocked achievements and crafted tools in its
inventory. With the observable inventory, LSTM-CNN does not improve upon the
feedforward variant PPO-CNN (10.3 vs. 10.4 in Table I). However, when the
inventory is not observable, LSTM-CNN outperforms PPO-CNN (6.9 vs. 7.7). This
might indicate that the recurrent agents learn to store achievements in
memory, although not perfectly, as we observe a drop from the case with the
observable inventory. On the other hand, LSTM-SPCNN outperforms its
feedforward variant PPO-SPCNN in both in-distribution and OOD settings (Table
IV). Here the inventory is observed, so the largest benefit must arise from
the agent’s ability to memorize the partially observed map layout.

TABLE IV:  Scores on CrafterOODapp (mean and standard deviations over 10 random seeds) for agents trained for 1M environmental steps. Each setting has two rows, denoting training (e.g. $O_{1-4}:25\%$) and evaluation ($O_{1}:0\%,O_{2-4}:33.3\%$) scores.  Train/Eval Dist | PPO-CNN | PPO-SPCNN | LSTM-CNN | LSTM-SPCNN | OC-SA | OC-CA
---|---|---|---|---|---|---
Training: $O_{1}:100\%$ |  10.4 $\pm$ 0.6 |  11.4 $\pm$ 0.5 |  10.5 $\pm$ 0.5 |  12.3 $\pm$ 0.4 |  11.0 $\pm$ 0.5 |  10.1 $\pm$ 0.6
Evaluation: $O_{1}:100\%$ |  10.3 $\pm$ 0.6 |  11.6 $\pm$ 0.6 |  10.4 $\pm$ 0.2 |  12.1 $\pm$ 0.8 |  11.1 $\pm$ 0.7 |  10.0 $\pm$ 0.4
$O_{1-4}:25\%$ |  9.2 $\pm$ 0.5 |  10.7 $\pm$ 0.6 |  10.7 $\pm$ 0.6 |  11.5 $\pm$ 0.4 |  9.7 $\pm$ 1.1 |  9.9 $\pm$ 0.6
$O_{1}:0\%,\ \ O_{2-4}:33.3\%$ |  9.2 $\pm$ 0.7 |  11.0 $\pm$ 1.1 |  11.0 $\pm$ 1.0 |  11.6 $\pm$ 0.6 |  9.7 $\pm$ 1.2 |  9.2 $\pm$ 0.7
$O_{1}:52\%,O_{2-4}:16\%$ |  9.9 $\pm$ 0.5 |  11.1 $\pm$ 0.7 |  11.5 $\pm$ 1.4 |  11.1 $\pm$ 0.5 |  9.6 $\pm$ 0.8 |  9.4 $\pm$ 0.9
$O_{1}:0\%,\ \ O_{2-4}:33.3\%$ |  10.0 $\pm$ 0.7 |  11.2 $\pm$ 1.1 |  11.4 $\pm$ 1.6 |  11.0 $\pm$ 0.5 |  10.6 $\pm$ 0.9 |  9.9 $\pm$ 0.9
$O_{1}:76\%,O_{2-4}:8\%$ |  9.9 $\pm$ 0.4 |  11.5 $\pm$ 0.6 |  10.7 $\pm$ 1.0 |  11.6 $\pm$ 0.6 |  9.8 $\pm$ 0.8 |  11.3 $\pm$ 0.5
$O_{1}:0\%,\ \ O_{2-4}:33.3\%$ |  9.2 $\pm$ 0.6 |  10.5 $\pm$ 0.8 |  10.4 $\pm$ 1.0 |  10.7 $\pm$ 0.7 |  9.2 $\pm$ 0.8 |  10.5 $\pm$ 0.7
$O_{1}:88\%,O_{2-4}:4\%$ |  10.1 $\pm$ 0.6 |  12.2 $\pm$ 0.8 |  11.5 $\pm$ 1.4 |  11.3 $\pm$ 0.4 |  10.5 $\pm$ 1 |  11.2 $\pm$ 0.9
$O_{1}:0\%,\ \ O_{2-4}:33.3\%$ |  9.1 $\pm$ 0.7 |  10.2 $\pm$ 0.7 |  10.1 $\pm$ 1.3 |  9.8 $\pm$ 0.8 |  9.4 $\pm$ 1.3 |  9.4 $\pm$ 1.0
$O_{1}:94\%,O_{2-4}:2\%$ |  10.9 $\pm$ 0.7 |  12.0 $\pm$ 0.8 |  11.4 $\pm$ 1.2 |  11.7 $\pm$ 0.6 |  10.5 $\pm$ 0.6 |  10.8 $\pm$ 1.1
$O_{1}:0\%,\ \ O_{2-4}:33.3\%$ |  8.6 $\pm$ 0.7 |  9.2 $\pm$ 1.1 |  9.1 $\pm$ 1.4 |  9.8 $\pm$ 0.8 |  9.9 $\pm$ 0.8 |  8.8 $\pm$ 0.9
$O_{1}:97\%,O_{2-4}:1\%$ |  10.5 $\pm$ 0.6 |  11.8 $\pm$ 0.7 |  11.9 $\pm$ 1.4 |  12.0 $\pm$ 0.3 |  10.3 $\pm$ 1.1 |  10.8 $\pm$ 0.6
$O_{1}:0\%,\ \ O_{2-4}:33.3\%$ |  7.3 $\pm$ 0.5 |  7.7 $\pm$ 1.0 |  8.2 $\pm$ 1.0 |  8.6 $\pm$ 1.0 |  9.3 $\pm$ 0.8 |  8.8 $\pm$ 0.8
$O_{1}:100\%,O_{2-4}:0\%$ |  10.5 $\pm$ 0.6 |  11.8 $\pm$ 0.6 |  10.7 $\pm$ 0.2 |  11.9 $\pm$ 0.8 |  11.1 $\pm$ 1.3 |  10.7 $\pm$ 0.6
$O_{1}:0\%,\ \ \ \ O_{2-4}:33.3\%$ |  7.3 $\pm$ 0.5 |  7.7 $\pm$ 0.9 |  5.8 $\pm$ 0.2 |  6.8 $\pm$ 1.0 |  8.0 $\pm$ 1.2 |  7.6 $\pm$ 0.5

###  IV-D Asymptotic Performance

TABLE V:  Asymptotic performance of PPO-CNN and PPO-SPCNN on the Crafter environment up to 20M environmental interactions.  Method | 1M | 2M | 5M | 10M | 15M | 20M
---|---|---|---|---|---|---
PPO-CNN | 10.3 | 15.9 | 18.3 | 27.1 | 28.3 | 29.1
PPO-SPCNN | 11.6 | 16.5 | 19.4 | 27.8 | 29.6 | 30.5

As the Crafter benchmark is designed to accelerate RL research and facilitate
short research cycles, the default evaluation protocol is to train agents up
to 1M steps. To better understand the asymptotic performance, we train PPO-CNN
and PPO-SPCNN agents for 20M steps. The resulting mean Crafter scores over ten
seeds are shown in Table V. Other agents do not improve significantly over
these two in the asymptotic regime. For readability, we omit standard
deviations (they are all below 0.6, decreasing to 0.2 at 20M steps due to the
geometric mean averaging in the score). Firstly, PPO-SPCNN consistently
outperforms the PPO-CNN baseline. Moreover, the PPO-SPCNN agent learns to
unlock all but one achievement (the diamond) at 5M steps (though this is not a
regular event). We suspect that the difficulty in mining the diamond is its
rarity, implying that better exploration strategies are needed. Unlike humans,
the agents have no prior notion that a diamond is valuable. This challenge
seems ideal for investigating novel curiosity-based agents [41, 42, 43].

##  V OOD Generalization Experiments

In this section, we evaluate agents on new CrafterOODapp and CrafterOODnum
environments, showing limited generalization of baseline agents and improved
generalization of object-centric ones.

TABLE VI:  Scores on CrafterOODnum (mean and standard deviations over 10 random seeds) for agents trained for 1M environmental steps. Each setting has two rows, denoting scores on the training and evaluation environments.  Train/Eval Dist | PPO-CNN | PPO-SPCNN | LSTM-CNN | LSTM-SPCNN | OC-SA | OC-CA
---|---|---|---|---|---|---
Train: easy (x2) |  12.4 $\pm$ 2 |  13.7 $\pm$ 1.2 |  14.2 $\pm$ 0.8 |  13.3 $\pm$ 1.0 |  15.0 $\pm$ 1.8 |  13.7 $\pm$ 1.8
Eval: default |  10.4 $\pm$ 1.7 |  12.1 $\pm$ 1.0 |  11.5 $\pm$ 0.7 |  11.4 $\pm$ 0.9 |  13.0 $\pm$ 1.3 |  11.7 $\pm$ 1.7
easy (x4) |  13.1 $\pm$ 2.2 |  14.5 $\pm$ 1.8 |  15.1 $\pm$ 1.2 |  15.0 $\pm$ 0.7 |  18.6 $\pm$ 2.3 |  13.7 $\pm$ 1.9
default |  8.8 $\pm$ 1.0 |  9.8 $\pm$ 1.7 |  9.1 $\pm$ 0.9 |  9.8 $\pm$ 0.9 |  12.8 $\pm$ 1.7 |  8.8 $\pm$ 0.9
mix (x4) |  13.4 $\pm$ 1.7 |  13.9 $\pm$ 2.1 |  13.0 $\pm$ 0.7 |  14.7 $\pm$ 1.6 |  15.5 $\pm$ 2.0 |  14.7 $\pm$ 2.3
default |  9.2 $\pm$ 0.6 |  10.2 $\pm$ 1.7 |  9.0 $\pm$ 0.9 |  10.5 $\pm$ 0.9 |  10.6 $\pm$ 1.7 |  9.4 $\pm$ 0.9
default |  10.2 $\pm$ 0.4 |  11.5 $\pm$ 0.5 |  10.7 $\pm$ 0.6 |  12.1 $\pm$ 0.6 |  11.3 $\pm$ 0.4 |  10.2 $\pm$ 0.7
mix (x4) |  11.2 $\pm$ 1.1 |  12.3 $\pm$ 1.0 |  11.6 $\pm$ 0.9 |  12.2 $\pm$ 1.3 |  12.6 $\pm$ 1.0 |  10.3 $\pm$ 0.9
default |  10.3 $\pm$ 0.4 |  11.2 $\pm$ 0.6 |  10.9 $\pm$ 0.7 |  12.0 $\pm$ 0.4 |  11.1 $\pm$ 0.5 |  10.1 $\pm$ 0.6
easy (x2) |  10.9 $\pm$ 1.4 |  13.7 $\pm$ 1.3 |  12.8 $\pm$ 0.9 |  12.9 $\pm$ 1.4 |  11.4 $\pm$ 1.2 |  10.1 $\pm$ 0.7
default |  10.3 $\pm$ 0.7 |  11.2 $\pm$ 0.3 |  10.6 $\pm$ 0.3 |  11.8 $\pm$ 0.7 |  11.5 $\pm$ 0.7 |  10.1 $\pm$ 0.6
easy (x4) |  11.3 $\pm$ 0.8 |  12.5 $\pm$ 0.9 |  12.8 $\pm$ 1.3 |  11.9 $\pm$ 1.0 |  12.9 $\pm$ 0.8 |  9.9 $\pm$ 1.2
easy (x2) |  11.7 $\pm$ 1.3 |  13.3 $\pm$ 0.4 |  13.7 $\pm$ 1.4 |  13.4 $\pm$ 0.2 |  15.4 $\pm$ 1.2 |  12.8 $\pm$ 1.6
hard (x2) |  8.0 $\pm$ 0.5 |  8.1 $\pm$ 1.7 |  7.6 $\pm$ 0.7 |  9.2 $\pm$ 0.8 |  10.5 $\pm$ 1.5 |  7.1 $\pm$ 0.8
easy (x4) |  14.6 $\pm$ 2.2 |  15.4 $\pm$ 1.6 |  15.5 $\pm$ 1.2 |  15.3 $\pm$ 1.0 |  17.8 $\pm$ 1.5 |  15.2 $\pm$ 1.8
hard (x4) |  3.0 $\pm$ 0.4 |  3.3 $\pm$ 0.4 |  3.0 $\pm$ 0.3 |  3.4 $\pm$ 0.4 |  4.9 $\pm$ 0.6 |  4.2 $\pm$ 0.3
Train: average |  12.0 $\pm$ 1.4 |  13.1 $\pm$ 1.1 |  13.0 $\pm$ 0.8 |  13.4 $\pm$ 0.8 |  14.5 $\pm$ 1.3 |  12.6 $\pm$ 1.4
Eval: average |  9.1 $\pm$ 0.9 |  10.2 $\pm$ 1.1 |  9.7 $\pm$ 0.9 |  9.1 $\pm$ 1.0 |  11.1 $\pm$ 1.1 |  8.9 $\pm$ 0.9
TABLE VII:  CrafterOODnum environment object numbers.
Default is the original Crafter environment. Easy environments have (x2 or x4) more resources and (x2 or x4) fewer enemies. In hard environments we decrease the number of resources and increase the number of enemies. Mix environment contains four times more resources and enemies.  Env. | Tree | Coal | Cow | Zombie | Skeleton
---|---|---|---|---|---
Easy (x4) | 764 | 206 | 100 | 3 | 2.5
Easy (x2) | 380 | 102 | 46 | 6 | 4.5
Default | 189 | 50 | 26 | 15 | 9.5
Hard (x2) | 95 | 27 | 13 | 33 | 19
Hard (x4) | 52 | 12.5 | 6 | 60 | 38
Mix (all x4) | 764 | 206 | 100 | 60 | 38

###  V-A CrafterOODapp - out-of-distribution object appearance.

In the CrafterOODapp environment, we generate a collection of increasingly
complex adaptation scenarios. Environments contain up to four object variants
for trees, cows, iron, stones, zombies, and skeletons, the objects the agent
interacts with most frequently (Figure 9). The adaptation scenarios differ in
object distributions in training and evaluation environments. For example, the
in-distribution training and evaluation corresponds to observing only the
first object variant $O_{1}$ (first two rows in Table IV).

Starting from training with uniformly distributed objects ($O_{1-4}=25\%$) we
make generalization progressively more difficult by skewing the training
distribution towards the first object. In this way, we create environments
that contain the first object ($O_{1}$) in $52\%$ and all the others in $16\%$
of the time ($O_{1}:52\%,O_{2-4}:16\%$ case in Table IV), then environments
that contain the first object in $76\%$, then $88\%$ percent of the time, etc.
The evaluation environment always contains only the last three objects (we
refer to them as the “evaluation” objects) uniformly distributed (each
observed in $33.3\%$ of the time). Up to the point of observing the evaluation
objects in $16\%$ of the time the agents generalize fairly well (Table IV, but
see also Figure 10 in Appendix C). Decreasing the percentage of evaluation
objects in the training environment further, the performance of all agents
consistently drops. Finally, in the especially difficult zero-shot
generalization scenario, when trained only on the first object
($O_{1}=100\%$), the agent relies on pure chance or interacting with objects
that do not change (e.g., water, iron). The agent’s failure to generalize is
expected: to perform well on unseen evaluation objects, it must perform fast
adaptation and systematic generalization in terms of composing the previously
obtained knowledge about the observed objects and the inferred representations
of the newly introduced evaluation objects.

Motivated by this observation, we introduce object-centric agents, which
should (at least in principle) learn better representations by decomposing the
input images into sets of objects and their distances via positional
encodings. Our experiments show that the object-centric agents OC-SA and OC-CA
(the last two columns in Table IV) match the vanilla PPO agents on in-
distribution and easy OOD generalization. In the most difficult OOD
generalization cases though (the last three pairs of rows in Table IV), OC-SA
and OC-CA generalize better compared to PPO-CNN and PPO-SPCNN (9.9 vs 8.6 and
9.2 for $O_{1}=94\%$, 9.3 vs 7.3 and 7.7 for $O_{1}=97\%$, and 8.0 vs 7.3 and
7.7 for $O_{1}=100\%$). Additionally, object-centric agents are interpretable
and potentially easier to build on in future work. We visualize their
attention maps in Section VI-B. The observed generalization gap makes
environments with large differences in object distributions (requiring zero-
shot adaptation) fruitful for developing and evaluating novel fast-adaptation
[44, 45, 46, 47, 48] and meta-learning [49] agents, such as the ones based on
Fast Weight Programmers [50, 51, 52, 53, 54].

###  V-B CrafterOODnum - out-of-distribution object numbers.

To test generalization on different numbers of objects, we introduce the
CrafterOODnum environment. We vary the numbers of objects the agent interacts
with most frequently: trees, coal, cows, zombies, and skeletons. These
variants are particularly challenging as the agent might face never-before-
seen situations such as fighting against more enemies or surviving in
environments with resources scarcer compared to what it was trained on. In
Table VI we show the numbers of objects in the respective environments. In
Table VI, ‘Default‘ is the Crafter environment introduced in [11]. In ‘Easy‘
environments, we increase (double or quadruple) the number of objects
representing resources and decrease the number of enemies. ‘Hard‘ environments
consist of fewer resources and more enemies. In the ‘mix‘ environment, we
quadruple the numbers of all objects—making it on the one hand easier in terms
of resources, but on the other hand harder in terms of enemies.

For each pair of training and evaluation environments in Table VI the agents
with the best generalization score are bolded. The object-centric agent OC-SA
outperforms all models in all but one case. Note how the performance decreases
when transferring to harder environments (Easy to Default) and increases when
transferring to easier environments (Default to Easy), confirming the intended
design difficulty level. The performance drop is largest when transferring
from ‘Easy (x4)‘ to ‘Hard (x4)‘ environments where the evaluation environment
contains 16x fewer resources and 16x more enemies than the training
environment. Finally, the object-centric OC-SA agent achieves the best
generalization on average across all the environments (the last row in Table
VI).

Figure 3: An agent collecting resources. Top row: input images. Bottom row:
attention visualized by its intensity. Episode steps are plotted horizontally.

##  VI Object-centric Agents Analysis

In this section, we first perform an ablation study of object-centric agents,
inspecting various design choices. Afterwards, we visualize attention patterns
of object-centric agents, and find their policies to be interpretable and to
match our gameplay intuitions.

TABLE VIII: OC-CA Agents Network Ablation. Variant | Crafter Score
---|---
OC-CA | 10.0 $\pm$ 0.4
OC-CA + Residual MLP |  7.3 $\pm$ 0.4
OC-CA + LayerNorm |  4.1 $\pm$ 0.3
OC-CA + Residual MLP + LayerNorm |  3.1 $\pm$ 0.6
OC-CA + Slot Competition |  7.1 $\pm$ 0.3
OC-CA, Number of Slots=1 |  8.2 $\pm$ 0.9
OC-CA, Number of Slots=2 |  7.8 $\pm$ 0.4
OC-CA, Number of Slots=4 |  8.7 $\pm$ 0.7
CA, Number of Slots=8 | 10.0 $\pm$ 0.4
OC-CA, Number of Slots=16 |  9.1 $\pm$ 0.9
OC-CA, Number of Heads=1 |  6.6 $\pm$ 0.5
OC-CA, Number of Heads=2 |  7.3 $\pm$ 0.9
OC-CA, Number of Heads=4 |  8.0 $\pm$ 0.6
CA, Number of Heads=8 | 10.0 $\pm$ 0.4
CA, Number of Heads=16 |  7.2 $\pm$ 0.8
OC-CA, Patch Size=1, Stride=1 |  6.9 $\pm$ 0.9
OC-CA, Patch Size=8, Stride=8 |  7.2 $\pm$ 0.4
OC-CA, Patch Size=12, Stride=8 |  8.7 $\pm$ 0.7
OC-CA, Patch Size=12, Stride=12 |  8.9 $\pm$ 0.7
OC-CA, Patch Size=16, Stride=8 |  6.2 $\pm$ 0.5
OC-CA, Patch Size=16, Stride=16 | 10.0 $\pm$ 0.4
Figure 4: An agent building a table. Top row: input images. Bottom row:
attention visualized by its intensity. Episode steps are plotted horizontally.
Figure 5: An agent crafting weapons. Top row: input images. Bottom row:
attention visualized by its intensity. Episode steps are plotted horizontally.
Figure 6: An agent defending against enemies. Top row: input images. Bottom
row: attention visualized by its intensity. Episode steps are plotted
horizontally.  Figure 7: Learned attention over the whole input image instead
of a fixed patch grid. Top row: input images. Bottom row: attention visualized
by its intensity.

###  VI-A Object-centric Agents Ablation.

To the best of our knowledge, this is the first time cross-attention-based
methods (e.g. SlotAttention- and Perceiver-like methods) are used in an open
world survival game. In this section, we perform an ablation analysis, compare
our agent to the standard architectural choices from the literature and show
that these vanilla agents underperform.

The ablation study results are shown in Table VIII. Firstly, cross-attention-
based methods typically employ LayerNorm [55] and residual MLPs. In Table VIII
we can see that the variant using these “CA + Residual MLP + LayerNorm”
underperforms. Including any of these modules individually (“CA + Residual
MLP” and “CA + LayerNorm”) does not improve performance. Moreover, competition
over slots (a softmax over queries) “CA + Slot Competition” akin to
SlotAttention also hurts the downstream performance. This led us to converge
to an architecture that uses a single cross-attention over the input image,
with no latent self-attention, no layer normalization, no residual
connections, and no slot-wise competition.

We found the optimal number of heads and slots to be eight. We speculate that
the higher number of slots lets each attend to different objects in the input,
and the higher number of heads allows for operation specialization for each of
the heads. Finally, we observe that a larger patch size (with the appropriate
stride) improves performance. On the other hand, the agents with learned
object representations “CA, Patch Size=1, Stride=1” do not perform as well as
the attention with larger patches. Although in theory more powerful as it can
attend to varying object sizes, this variant needs first to learn which pixels
belong to each object. In Crafter, objects are of equal size, so the patches
can be chosen to correspond to those. We suspect the attention over pixels
would better generalize to objects of different sizes.

###  VI-B OC-SA Agents Visualization and Interpretability.

Here we visualize the CLS token attention in OC-SA. Visualizations in Figures
3, 4, 5 and 6 stem from a single agent and are representative of most episodes
of a trained agent. In Figure 3, we can see an agent collecting resources. In
the first frame (the leftmost figure) the agent notices a tree nearby and then
focuses consistently on it and collects it. In the final frame, the agent
attends to the newly collected wood resource.

The episode is continued in Figure 4, in which the agent builds a table. After
collecting one tree, the agent needs to collect another one to build a table.
It spots more trees in its vicinity and goes towards them, never losing sight
of the tree and the position where it wants to craft a table. It collects
another tree and attends again to the wood resources. Determining that it has
enough wood (two pieces), it builds a table to facilitate crafting of further
tools and weapons.

Once it has built the table, the agent can use it to craft tools (weapons),
but it first needs to collect more resources. In Figure 5 it collects the two
nearby trees and uses them to build a wooden sword and a wooden pickaxe (shown
in the inventory). The sequence is reminiscent of what a human player would
do: collect one tree, craft a sword, collect another tree and craft a pickaxe.
It can use pickaxe to collect stones or coal or to defend against enemies.

In Figure 6, an agent defends against a zombie during the night (the reason
why the frame color is darker). Initially, the agent does not attend to the
zombie, probably because it is still far away. As the zombie gets closer, it
attends more and more to it and finally defeats it.

###  VI-C OC-CA Agents Visualization and Interpretability.

In Figure 7 we visualize the learned attention patterns when each slot can
attend to any set of pixels in the input image, compared to only a predefined
set of patches from a fixed grid. We find that the learned attention attends
to the salient objects in the scene: zombies (1st, 3rd, and 5th columns),
trees (2nd, 3rd, and 4th columns), resources (water overall), and the
inventory. We observe more numerous but smaller attention patterns than the
patch-based attention that is typically focused on two to five patches.
Although in theory more powerful, in practice the learned attention models
underperform the patch-based ones. We suspect this is because the learned
attention agents first need to find out what an object actually is, whereas
patch-based ones have a more “guided” learning process. We speculate that
greater gains of learned attention would occur in scenarios with objects of
varying sizes, in which case a fixed grid would be a too rigid representation.

##  VII Related Work

RL benchmarks have played a crucial role in developing and evaluating novel
(deep) RL algorithms [2, 56, 57, 58, 59, 60]. An especially important role has
been played by video games including arcade games [2], racing environments
[8], first-person shooters [57], strategy games [61, 62], and open world games
[4, 5]. For an extensive survey of deep (reinforcement) learning for video
game playing see [63]. Open world games have recently received special
attention, e.g., Minecraft [64, 4, 5].

However, these environments come with their drawbacks. For example, Minecraft
is too complex to be solved from scratch by current methods [65]. The recently
published method for mining a diamond [66] uses human-annotated data to
bootstrap learning via behavioral cloning. It is also unclear by what metric
agents should be evaluated. On the other hand, Atari requires large amounts of
computation: training an agent with five random seeds on each game for 200M
steps requires over 2000 GPU days [67, 68, 11], which hinders fast iteration
cycles. Most importantly, many Atari games are nearly deterministic, so the
agents can approximately memorize action sequences and are not required to
generalize to new situations [69].

Environments like ProcGen [3] evaluate the generalization capabilities of
agents through procedural level generation, which requires substantial compute
[11]. Chan et al. [70] recently investigated how RL algorithms perform in
environments where feature distribution is not uniform but Zipfian, similar to
the one of objects encountered in the real world, or in our CrafterOODapp
environments. Generalization benchmarks based on DeepMind’s Control Suite [59,
71, 72, 73, 74] test the agent’s robustness to background variations but not
to variations of environmental object configurations, e.g., visual appearance,
number of objects, or object compositions. For a thorough recent survey on
generalization in RL, see [75].

The Crafter game addresses these shortcomings. Crafter is an open world
survival game for RL research whose dynamics are inspired by the popular game
Minecraft. The benchmark is designed to address existing research challenges,
such as strong generalization via procedural generation, deep exploration via
achievements conditioned on one another, learning from high-dimensional image
observations and sparse rewards that require long-term reasoning and credit
assignment. It facilitates evaluation by combining semantically meaningful
achievements and fast iteration speed. Our newly introduced CrafterOODapp and
CrafterOODnum inherit all these benefits and additionally test some agent’s
generalization ability and robustness in presence of unseen objects, setting
the stage for the development of fast-adaptation or meta-learning agents.

Research on RL neural network-based policies that generalize to OOD
environments is an important research area currently tackled from several
directions [75]. One potential explanation of the poor generalization of
neural networks is that they cannot dynamically and flexibly bind the
information distributed throughout the network [17]. This is also known as the
binding problem [18, 19]. Recent work addresses this through object-centric
neural networks that learn (discrete) object representations from raw visual
input, to support efficient learning and generalization to novel scenarios and
behaviors. Object-centric methods have successfully been used for OOD
generalization in both supervised and unsupervised learning [76, 77, 20, 21,
78, 22, 23, 79, 24, 80, 16, 25, 81, 26] and in RL [27, 28, 29, 30]. To the
best of our knowledge, this is the first time they are used in open world RL
survival games.

Architecturally closest to our work are methods that learn a set of vectors
(slots) by attending to the input via cross-attention. This idea was
introduced in SetTransformer [33] and successfully used for object detection
[34], unsupervised learning of objects [16, 26], learning permutation-
invariant agents [35, 36], and general perception modules [37, 38, 39]. Most
similar to our object-centric agents are the AttentionAgent [35] and the
SensoryNeuron [36]. The former uses neuroevolution to optimize an architecture
with a hard attention bottleneck, resulting in a network that only receives a
fraction of the visual input and generalizes to unseen backgrounds. The latter
further improves AttentionAgent’s robustness to permuted orderings of its
inputs. Our agents also share similarities with attention-based modular neural
networks [82, 83, 84, 30], learning a (modular) representation by attending
over the input. However, unlike our object-centric work, prior work did not
consider object-centric agents for OOD generalization in procedurally
generated open world games.

##  VIII Conclusion and Discussion

Challenging benchmarks are essential for research on new reinforcement
learning methods. However, they should be simple enough to allow for
convenient systematic analysis and fast iteration cycles. Tests against strong
baselines are crucial.

Our paper offers contributions to all of the above. We report important
observations on the Crafter environment. Our analysis shows that PPO-based
agents trained for 20M steps can unlock all but the last achievement. However,
this impressive score still falls short of the human score [11], indicating
that Crafter remains an open research challenge, especially for sample-
efficient RL agents.

We introduce CrafterOODapp and CrafterOODnum, new environments that evaluate
some agent’s robustness against varying object appearances and numbers of
objects. Baseline agents fail to adapt in evaluation environments containing
unseen objects, whereas our novel object-centric agents perform well. An
additional benefit of object-centric agents is their interpretability. In the
investigated environments, self-attention-based methods outperform cross-
attention methods. Future work will confirm or disconfirm similar superiority
in environments with varying object sizes.

Our OOD environments should help to study fast-adaptation and meta-learning
agents. We intend to further improve object-centric agents by incorporating
decoding objectives, to extract as much signal from the environment as
possible, and by exploring ways of training larger object-centric networks. We
also want to investigate whether object-centric inductive biases can enable
model-based RL agents to learn better world models. Crafter and our newly
introduced CrafterOOD are good candidates for exploring all of these
directions, offering control of environmental configurations, excellent visual
inspection, and fast iteration cycles. To facilitate future research in this
area, we will publish the code for our agents and environments.

## Acknowledgements

We thank Kazuki Irie, Danijar Hafner, Shixiang Shane Gu and Yingtao Tian for
their useful comments and suggestions on an earlier version of this paper.
This research was partially funded by the ERC Advanced grant no: 742870,
project AlgoRNN.

## References

  * [1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [2] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” _Journal of Artificial Intelligence Research_ , vol. 47, pp. 253–279, jun 2013.
  * [3] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, “Leveraging procedural generation to benchmark reinforcement learning,” in _International conference on machine learning_. PMLR, 2020, pp. 2048–2056.
  * [4] W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin _et al._ , “Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors,” _arXiv preprint arXiv:1904.10079_ , 2019.
  * [5] W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno, C. Loomis, S. Milani, S. Mohanty, K. Nakata, R. Salakhutdinov _et al._ , “The minerl 2020 competition on sample efficient reinforcement learning using human priors,” _arXiv preprint arXiv:2101.11071_ , 2021.
  * [6] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, 2018.
  * [7] J. Koutník, G. Cuccu, J. Schmidhuber, and F. Gomez, “Evolving large-scale neural networks for vision-based reinforcement learning,” in _Proceedings of the 15th annual conference on Genetic and evolutionary computation_ , 2013, pp. 1061–1068.
  * [8] B. Wymann, E. Espié, C. Guionneau, C. Dimitrakakis, R. Coulom, and A. Sumner, “Torcs, the open racing car simulator,” _Software available at http://torcs. sourceforge. net_ , vol. 4, no. 6, p. 2, 2000.
  * [9] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [10] I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepesvári, S. Singh, B. Van Roy, R. Sutton, D. Silver, and H. van Hasselt, “Behaviour suite for reinforcement learning,” in _International Conference on Learning Representations_ , 2020\. [Online]. Available: https://openreview.net/forum?id=rygf-kSYwH
  * [11] D. Hafner, “Benchmarking the spectrum of agent capabilities,” _arXiv preprint arXiv:2109.06780_ , 2021.
  * [12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [13] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” _Journal of Machine Learning Research_ , vol. 22, no. 268, pp. 1–8, 2021\. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html
  * [14] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [15] D. Wierstra, A. Foerster, J. Peters, and J. Schmidhuber, “Solving deep memory pomdps with recurrent policy gradients,” in _International conference on artificial neural networks_. Springer, 2007, pp. 697–706.
  * [16] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf, “Object-centric learning with slot attention,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 11 525–11 538, 2020.
  * [17] K. Greff, S. van Steenkiste, and J. Schmidhuber, “On the binding problem in artificial neural networks,” _arXiv preprint arXiv:2012.05208_ , 2020.
  * [18] C. Von Der Malsburg, “The correlation theory of brain function,” in _Models of neural networks_. Springer, 1994, pp. 95–119.
  * [19] A. L. Roskies, “The binding problem,” _Neuron_ , vol. 24, no. 1, pp. 7–9, 1999.
  * [20] K. Greff, S. van Steenkiste, and J. Schmidhuber, “Neural expectation maximization,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6691–6701.
  * [21] S. van Steenkiste, M. Chang, K. Greff, and J. Schmidhuber, “Relational neural expectation maximization: Unsupervised discovery of objects and their interactions,” in _International Conference on Learning Representations_ , 2018.
  * [22] A. R. Kosiorek, H. Kim, I. Posner, and Y. W. Teh, “Sequential attend, infer, repeat: Generative modelling of moving objects,” in _Proceedings of the 32Nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18. USA: Curran Associates Inc., 2018, pp. 8615–8625.
  * [23] A. Stanić and J. Schmidhuber, “R-sqair: Relational sequential attend, infer, repeat,” _NeurIPS PGR Workshop_ , 2019.
  * [24] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner, “Multi-object representation learning with iterative variational inference,” in _International Conference on Machine Learning_ , 2019, pp. 2424–2433.
  * [25] A. Stanić, S. van Steenkiste, and J. Schmidhuber, “Hierarchical relational inference,” _arXiv preprint arXiv:2010.03635_ , 2020.
  * [26] T. Kipf, G. F. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff, “Conditional object-centric learning from video,” _arXiv preprint arXiv:2111.12594_ , 2021.
  * [27] N. Watters, L. Matthey, M. Bosnjak, C. P. Burgess, and A. Lerchner, “Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration,” _ICML workshop on Generative Modeling and Model-Based Reasoning for Robotics and AI_ , 2019.
  * [28] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and S. Levine, “Entity abstraction in visual model-based reinforcement learning,” in _Conference on Robot Learning_. PMLR, 2020, pp. 1439–1456.
  * [29] T. Kipf, E. van der Pol, and M. Welling, “Contrastive learning of structured world models,” in _International Conference on Learning Representations_ , 2020.
  * [30] W. Carvalho, A. Lampinen, K. Nikiforou, F. Hill, and M. Shanahan, “Feature-attending recurrent modules for generalization in reinforcement learning,” _arXiv preprint arXiv:2112.08369_ , 2021.
  * [31] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [33] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3744–3753.
  * [34] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [35] Y. Tang, D. Nguyen, and D. Ha, “Neuroevolution of self-interpretable agents,” in _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , 2020, pp. 414–424.
  * [36] Y. Tang and D. Ha, “The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [37] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, “Perceiver: General perception with iterative attention,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 4651–4664.
  * [38] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer _et al._ , “Perceiver io: A general architecture for structured inputs & outputs,” _arXiv preprint arXiv:2107.14795_ , 2021.
  * [39] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _arXiv preprint arXiv:2204.14198_ , 2022.
  * [40] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” _arXiv preprint arXiv:2010.02193_ , 2020.
  * [41] J. Schmidhuber, “Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments,” _Institut für Informatik, Technische Universität München. Technical Report FKI-126_ , vol. 90, 1990\.
  * [42] ——, “Curious model-building control systems,” in _Proceedings of the International Joint Conference on Neural Networks, Singapore_ , vol. 2. IEEE press, 1991, pp. 1458–1463.
  * [43] ——, “Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem,” _Frontiers in psychology_ , vol. 4, p. 313, 2013.
  * [44] ——, “Steps towards “self-referential” learning,” Dept. of Comp. Sci., University of Colorado at Boulder, Tech. Rep. CU-CS-627-92, November 1992.
  * [45] ——, “An introspective network that can learn to run its own weight change algorithm,” in _Proc. IEE Int. Conf. on Artificial Neural Networks_ , Brighton, UK, May 1993, pp. 191–195.
  * [46] ——, “A self-referential weight matrix,” in _Proc. Int. Conf. on Artificial Neural Networks (ICANN)_ , Amsterdam, Netherlands, Sep. 1993, pp. 446–451.
  * [47] ——, “A neural network that embeds its own meta-levels,” in _Proc. IEEE Int. Conf. on Neural Networks (ICNN)_ , San Francisco, CA, USA, Mar. 1993.
  * [48] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber, “A modern self-referential weight matrix that learns to modify itself,” _arXiv preprint arXiv:2202.05780_ , 2022.
  * [49] J. Schmidhuber, “Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook,” Ph.D. dissertation, Technische Universität München, 1987.
  * [50] ——, “Learning to control fast-weight memories: An alternative to recurrent nets,” Institut für Informatik, Technische Universität München, Tech. Rep. FKI-147-91, March 1991.
  * [51] ——, “Learning to control fast-weight memories: An alternative to dynamic recurrent networks,” _Neural Computation_ , vol. 4, no. 1, pp. 131–139, 1992\.
  * [52] ——, “Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets,” in _International Conference on Artificial Neural Networks (ICANN)_ , Amsterdam, Netherlands, Sep. 1993, pp. 460–463.
  * [53] I. Schlag, K. Irie, and J. Schmidhuber, “Linear transformers are secretly fast weight programmers,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 9355–9366.
  * [54] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber, “Going beyond linear transformers with recurrent fast weight programmers,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [55] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [56] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [57] M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaśkowski, “Vizdoom: A doom-based ai research platform for visual reinforcement learning,” in _2016 IEEE conference on computational intelligence and games (CIG)_. IEEE, 2016, pp. 1–8.
  * [58] C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik _et al._ , “Deepmind lab,” _arXiv preprint arXiv:1612.03801_ , 2016.
  * [59] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq _et al._ , “Deepmind control suite,” _arXiv preprint arXiv:1801.00690_ , 2018.
  * [60] A. Juliani, V.-P. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar _et al._ , “Unity: A general platform for intelligent agents,” _arXiv preprint arXiv:1809.02627_ , 2018.
  * [61] G. Synnaeve, N. Nardelli, A. Auvolat, S. Chintala, T. Lacroix, Z. Lin, F. Richoux, and N. Usunier, “Torchcraft: a library for machine learning research on real-time strategy games,” _arXiv preprint arXiv:1611.00625_ , 2016.
  * [62] O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, J. Schrittwieser _et al._ , “Starcraft ii: A new challenge for reinforcement learning,” _arXiv preprint arXiv:1708.04782_ , 2017.
  * [63] N. Justesen, P. Bontrager, J. Togelius, and S. Risi, “Deep learning for video game playing,” _IEEE Transactions on Games_ , vol. 12, no. 1, pp. 1–20, 2019\.
  * [64] M. Johnson, K. Hofmann, T. Hutton, and D. Bignell, “The malmo platform for artificial intelligence experimentation.” in _IJCAI_. Citeseer, 2016, pp. 4246–4247.
  * [65] S. Milani, N. Topin, B. Houghton, W. H. Guss, S. P. Mohanty, K. Nakata, O. Vinyals, and N. S. Kuno, “Retrospective analysis of the 2019 minerl competition on sample efficient reinforcement learning,” in _NeurIPS 2019 Competition and Demonstration Track_. PMLR, 2020, pp. 203–214.
  * [66] B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune, “Video pretraining (vpt): Learning to act by watching unlabeled online videos,” 2022. [Online]. Available: https://arxiv.org/abs/2206.11795
  * [67] P. S. Castro, S. Moitra, C. Gelada, S. Kumar, and M. G. Bellemare, “Dopamine: A research framework for deep reinforcement learning,” _arXiv preprint arXiv:1812.06110_ , 2018.
  * [68] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in _Thirty-second AAAI conference on artificial intelligence_ , 2018.
  * [69] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling, “Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents,” _Journal of Artificial Intelligence Research_ , vol. 61, pp. 523–562, 2018.
  * [70] S. C. Chan, A. K. Lampinen, P. H. Richemond, and F. Hill, “Zipfian environments for reinforcement learning,” _arXiv preprint arXiv:2203.08222_ , 2022.
  * [71] A. Stone, O. Ramirez, K. Konolige, and R. Jonschkowski, “The distracting control suite–a challenging benchmark for reinforcement learning from pixels,” _arXiv preprint arXiv:2101.02722_ , 2021.
  * [72] N. Hansen and X. Wang, “Generalization in reinforcement learning by soft data augmentation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 611–13 617.
  * [73] J. Grigsby and Y. Qi, “Measuring visual generalization in continuous control from pixels,” _arXiv preprint arXiv:2010.06740_ , 2020.
  * [74] A. Zhang, Y. Wu, and J. Pineau, “Natural environment benchmarks for reinforcement learning,” _arXiv preprint arXiv:1811.06032_ , 2018.
  * [75] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel, “A survey of generalisation in deep reinforcement learning,” _arXiv preprint arXiv:2111.09794_ , 2021.
  * [76] K. Greff, R. K. Srivastava, and J. Schmidhuber, “Binding via reconstruction clustering,” _arXiv preprint arXiv:1511.06418_ , 2015.
  * [77] K. Greff, A. Rasmus, M. Berglund, T. Hao, H. Valpola, and J. Schmidhuber, “Tagger: Deep unsupervised perceptual grouping,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 4484–4492.
  * [78] S. A. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, G. E. Hinton _et al._ , “Attend, infer, repeat: Fast scene understanding with generative models,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 3225–3233.
  * [79] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner, “Monet: Unsupervised scene decomposition and representation,” _arXiv preprint arXiv:1901.11390_ , 2019.
  * [80] M. Engelcke, A. R. Kosiorek, O. P. Jones, and I. Posner, “Genesis: Generative scene inference and sampling with object-centric latent representations,” _arXiv preprint arXiv:1907.13052_ , 2019.
  * [81] A. Creswell, R. Kabra, C. Burgess, and M. Shanahan, “Unsupervised object-based transition models for 3d partially observable environments,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [82] A. Santoro, R. Faulkner, D. Raposo, J. Rae, M. Chrzanowski, T. Weber, D. Wierstra, O. Vinyals, R. Pascanu, and T. Lillicrap, “Relational recurrent neural networks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 7299–7310.
  * [83] A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf, “Recurrent independent mechanisms,” _arXiv preprint arXiv:1909.10893_ , 2019.
  * [84] A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, and D. Jimenez Rezende, “Towards interpretable reinforcement learning using attention augmented agents,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019\.
  * [85] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” _arXiv preprint arXiv:1506.02438_ , 2015.

##  Appendix A OOD environment Objects

Player Plant Cow Zombie Skeleton Arrow

Water Sand Grass Tree Path Stone

Coal Iron Diamond Lava Table Furnace

Figure 8: Original Crafter objects. Figure from [11].

TreeV1 CowV1 ZombieV1 StoneV1 CoalV1 SkeletonV1

TreeV2 CowV2 ZombieV2 StoneV2 CoalV2 SkeletonV2

TreeV3 CowV3 ZombieV3 StoneV3 CoalV3 SkeletonV3

TreeV4 CowV4 ZombieV4 StoneV4 CoalV4 SkeletonV4

Figure 9: In CrafterOODapp there are four variants of objects for trees, cows,
zombies, stone coal and skeletons.

##  Appendix B Network Configurations

TABLE IX: PPO-CNN: baseline agent with the CNN from DQN [9].
Feature Extractor
---
$8\times 8$ conv, $32$ ReLU units, stride $4$
$4\times 4$ conv, $64$ ReLU units, stride $2$
$3\times 3$ conv, $64$ ReLU units, stride $1$
Flatten
Linear, $512$ ReLU units.
Action Network
Linear, $17$ units.
Value Network
Linear, $1$ units.
TABLE X: PPO-SPCNN: agent with the size-preserving CNN.
Feature Extractor
---
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
Flatten
Linear, $512$ ReLU units.
Action Network
Linear, $17$ units.
Value Network
Linear, $1$ units.
TABLE XI: OC-SA: agent with the self-attention module.
Feature Extractor
---
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
Split into $8\times 8$ patches and flatten the patch grid
Self-Attention Network
Learned CLS token of $256$ size.
Slot-wise projection: Linear, $256$ units.
// Self-Attention layer 1:
Query map: Linear, $256$ units.
Key map: Linear, $256$ units.
Values map: Linear, $256$ units.
// Self-Attention layer 2:
Query map: Linear, $256$ units.
Key map: Linear, $256$ units.
Values map: Linear, $256$ units.
Action Network
Linear, $17$ units.
Value Network
Linear, $1$ units.
TABLE XII: OC-CA: agent with the cross-attention module.
Feature Extractor
---
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
Split into $16\times 16$ patches and flatten the patch grid
Self-Attention Network
Learned slots $8\times 256$ size.
Query map: Linear, $256$ units.
Key map: Linear, $256$ units.
Values map: Linear, $256$ units.
Action Network
Linear, $17$ units.
Value Network
Linear, $1$ units.
TABLE XIII: LSTM-CNN: recurrent agent with an LSTM and a CNN from DQN [9].
Feature Extractor
---
$8\times 8$ conv, $32$ ReLU units, stride $4$
$4\times 4$ conv, $64$ ReLU units, stride $2$
$3\times 3$ conv, $64$ ReLU units, stride $1$
Flatten
Linear, $512$ ReLU units.
Action Network
LSTM, $256$ units.
Linear, $17$ units.
Value Network
LSTM, $256$ units.
Linear, $1$ units.
TABLE XIV: LSTM-SPCNN: recurrent agent with an LSTM and a size-preserving CNN.
Feature Extractor
---
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
$5\times 5$ conv, $64$ ReLU units, stride $1$, padding $2$
Flatten
Linear, $512$ ReLU units.
Action Network
LSTM, $256$ units.
Linear, $17$ units.
Value Network
Linear, $256$ units.
Linear, $1$ units.

blabla

##  Appendix C CrafterOODapp performance

In Figure 10 we provide heatmap of the CrafterOODapp scores previously
reported in Table IV.

Figure 10: Scores on CrafterOODnum for agents trained for 1M environment
steps. Mean over 10 random seeds are reported. For standard deviations see
Table IV. Each setting has two rows, denoting scores in training (e.g.
$25,25,25,25$ for $O_{1-4}:25\%$) and evaluation ($0,33,33,33$, for
$O_{1}:0\%,O_{2-4}:33.3\%$) environments.

##  Appendix D CrafterOODnum performance

In Figure 11 we provide heatmap of the CrafterOODnum scores previously
reported in Table VI.

Figure 11: Scores on CrafterOODapp for agents trained for 1M environment
steps. Mean over 10 random seeds are reported. For standard deviations see
Table VI. Each setting has two rows, denoting scores in training and
evaluation environments.

##  Appendix E Hyper-parameter heatmaps

In Figures 12 and 13 we show the pairwise heatmaps of the hyper-parameters we
tuned for the PPO-CNN. Important to note here is that once we tuned the hyper-
parameters for PPO-CNN, we fixed them and used the same values for all other
models. Therefore, other models might have a slight disadvantage as we did not
specifically tune hyper-parameters for each model.

In Figure 12 we start from the default hyper-parameters from [13] that were
tuned for Atari: GAE lambda 0.95, number of epochs 10, gamma 0.99, batch size
64 and number of steps 1024. From here, we sweep over individual pairs of
hyper-parameters, e.g. GAE lambda and the number of epochs. Looking at the
heatmaps, it is clear how each hyper-parameters contributes to the improvement
(which becomes evident by looking at their combination in Figure 13). However,
some hyper-parameters contribute much more than others, e.g. GAE lambda and
the number of epochs the agent is trained on the collected rollouts. Given
these plots, we settled on the following values for hyper-parameters: number
of steps 4096, number of epochs 4, batch size 128, GAE lambda 0.65, and gamma
0.95. GAE-lambda can be interpreted as an extra discount factor applied after
performing reward shaping transformation on the MDP [85], and we found low
values to perform better. This also makes sense given that we found lower
discount factors also to give better performance (0.95 compared to the most
commonly used 0.99). On the other hand, we extended the horizon (number of
rollout steps collected for the update) from default 1024 to 4096) and
similarly increased batch size from 64 to 128, stabilizing training agents in
Crafter. In general, when there are frequent rewards within an episode, the
number of steps can be smaller. However, in Crafter, the rewards are sparse,
which is also reflected in empirically better performance when using longer
rollouts.

In Figure 12 we start from the best hyper-parameters we found: number of steps
4096, number of epochs 4, batch size 128, GAE lambda 0.65 and gamma 0.95. We
then ablate pairs of hyper-parameters by changing them over their respective
ranges. The agents are fairly robust to hyper-parameters changes. From the
heatmaps, we can see that the performance drops going away from the optimal
values, though not by a significant value.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 12: Crafter scores starting from the default hyper-parameters from [13]
that were tuned for Atari: GAE lambda 0.95, number of epochs 10, gamma 0.99,
batch size 64 and number of steps 1024, and then ablating individual hyper-
parameter pairs.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 13: Crafter scores starting from the best hyper-parameters we found:
GAE lambda 0.65, number of epochs 4, gamma 0.95, batch size 128 and number of
steps 4096, and then ablating individual hyper-parameter pairs.