# Dungeons and Data: A Large-Scale NetHack Dataset

Eric Hambro
Meta AI
&Roberta Raileanu
Meta AI
&Danielle Rothermel
Meta AI
&Vegard Mella
Meta AI
Tim Rocktäschel
University College London
&Heinrich Küttler
Inflection AI†&Naila Murray
Meta AI
Correspondence to<EMAIL_ADDRESS>done while at Meta AI.

###### Abstract

Recent breakthroughs in the development of agents to solve challenging
sequential decision making problems such as Go [50], StarCraft [58], or DOTA
[3], have relied on both simulated environments and large-scale datasets.
However, progress on this research has been hindered by the scarcity of open-
sourced datasets and the prohibitive computational cost to work with them.
Here we present the NetHack Learning Dataset (NLD), a large and highly-
scalable dataset of trajectories from the popular game of NetHack, which is
both extremely challenging for current methods and very fast to run [23]. NLD
consists of three parts: 10 billion state transitions from 1.5 million human
trajectories collected on the NAO public NetHack server from 2009 to 2020; 3
billion state-action-score transitions from 100,000 trajectories collected
from the symbolic bot winner of the NetHack Challenge 2021; and, accompanying
code for users to record, load and stream any collection of such trajectories
in a highly compressed form. We evaluate a wide range of existing algorithms
including online and offline RL, as well as learning from demonstrations,
showing that significant research advances are needed to fully leverage large-
scale datasets for challenging sequential decision making tasks.

##  1 Introduction

Recent progress on deep reinforcement learning (RL) methods has led to
significant breakthroughs such as training autonomous agents to play Atari
[33], Go [50], StarCraft [58], and Dota [3], or to perform complex robotic
tasks [24, 44, 26, 35, 16, 41]. In many of these cases, success relied on
having access to large-scale datasets of human demonstrations [58, 3, 41, 26].
Without access to such demonstrations, training RL agents to operate
effectively in these environments remains challenging due to the hard
exploration problem posed by their vast state and action spaces. In addition,
having access to a simulator is key for training agents that can discover new
strategies not exhibited by human demonstrations. Therefore, training agents
using a combination of offline data and online interaction has proven to be a
highly successful approach for solving a variety of challenging sequential
decision making tasks. However, this requires access to complex simulators and
large-scale offline datasets, which tend to be computationally expensive.

The NetHack Learning Environment (NLE) was recently introduced as a testbed
for RL, providing an environment which is both extremely challenging [15] and
exceptionally fast to simulate [23]. NLE is a stochastic, partially observed,
and procedurally generated RL environment based on the popular game of
NetHack. Due to its long episodes (i.e. tens or hundreds of thousands of
steps) and large state and action spaces, NLE poses a uniquely hard
exploration challenge for current RL methods. Thus, one of the most promising
research avenues towards progress on NetHack is leveraging human or symbolic-
bot demonstrations to bootstrap performance, which also proved successful for
StarCraft [58] and Dota [3].

In this paper, we introduce the NetHack Learning Dataset (NLD), an open and
accessible dataset for large-scale offline RL and learning from
demonstrations. NLD consists of three parts: first, NLD-NAO: a collection of
state-only trajectories from 1.5 million human games of NetHack played on
nethack.alt.org (NAO) servers between 2009 and 2020; second, NLD-AA: a
collection of state-action-score trajectories from 100,000 NLE games played by
the symbolic-bot winner of the 2021 NetHack Challenge [15]; third,
TtyrecDataset: a highly-scalable tool for efficient training on any NetHack
and NLE-generated trajectories and metadata. NLD, in combination with NLE,
enables computationally-accessible research in multiple areas including
imitation learning, offline RL, learning from sequences of only observations,
as well as combining learning from offline data with learning from online
interactions. In contrast with other large-scale datasets of demonstrations,
NLD is highly efficient in both memory and compute. NLD-NAO can fit on a $30
hard drive, after being compressed (by a factor of 160) from 38TB to 229GB. In
addition, NLD-NAO can be processed in under 15 hours, achieving a throughput
of 288,000 frames per second with only 10 CPUs. NLD’s low memory and
computational requirements makes large-scale learning from demonstrations more
accessible for academic and independent researchers.

To summarize, the key characteristics of NLD are that: it is a scalable
dataset of demonstrations (i.e. large and cheap) for a highly-complex
sequential decision making challenge; it enables research in multiple areas
such as imitation learning, offline RL, learning from observations of
demonstrations, learning from both static data and environment interaction;
and it has many properties of real-world domains such as partial
observability, stochastic dynamics, sparse reward, long trajectories, rich
environment, diverse behaviors, and a procedurally generated environment.

In this paper, we make the following core contributions:  (i) we introduce
NLD-NAO, a large-scale dataset of almost 10 billion state transitions, from
1.5 million NetHack games played by humans;  (ii) we also introduce NLD-AA, a
large-scale dataset of over 3 billion state-action-score transitions, from
100,000 games played by the symbolic-bot winner of the NeurIPS 2022 NetHack
Competition;  (iii) we open-source code for users to record, load, and stream
any collection of NetHack trajectories in a highly compressed form; and  (iv)
we show that, while current state-of-the-art methods in offline RL and
learning from demonstrations can effectively make use of the dataset, playing
NetHack at human-level performance remains an open research challenge.

##  2 Related Work

Offline RL Benchmarks. Recently, there has been a growing interest in
developing better offline RL methods [25, 39, 11, 60, 61, 21, 20, 6, 2, 49]
which aim to learn from datasets of trajectories. With it, a number of offline
RL benchmarks have been released [1, 62, 10, 42, 22, 43]. While these
benchmarks focus specifically on offline RL, our datasets enable research on
multiple areas including imitation learning, learning from observations only
(i.e. without access to actions or rewards), as well as learning from both
offline and online interactions. In order to make progress on difficult RL
tasks such as NetHack, we will likely need to learn from both human data and
environment interaction, as was the case with other challenging games like
StarCraft [58] or Dota [3]. In contrast, the tasks proposed in current offline
RL benchmarks are much easier and can be solved by training either only online
or only offline [10, 62]. In addition, current offline RL benchmarks test
agents on the exact same environment where the data was collected. As
emphasized by [57], imitation learning algorithms drastically overfit to their
environments, so it is essential to evaluate them on new scenarios in order to
develop robust methods. In contrast, NLE has long procedurally generated
episodes which require both long-term planning and systematic generalization
in order to succeed. This is shown in [30], which investigates transfer
learning between policies trained on different NLE-based environments.

For many real-world applications such as autonomous driving or robotic
manipulation, learning from human data is essential due to safety concerns and
time constraints [40, 10, 8, 5, 27, 52, 55, 17, 38, 18]. However, most offline
RL benchmarks contain synthetic trajectories generated by either random
exploration, pretrained RL agents, or simple hard-coded behaviors [10]. In
contrast, one of our datasets consists entirely of human replays, while the
other one is generated by the winner of the NetHack Competition at NeurIPS
2022 which is a complex symbolic bot with built-in knowledge of the game.
Human data (like the set contained in NLD-NAO) is significantly more diverse
and messy than synthetic data, as humans can vary widely in their expertise,
optimize for different objectives (such as fun or discovery), have access to
external information (such as the NetHack Wiki [34]), and even have different
observation or action spaces than RL agents. Hence, learning directly from
human data is essential for making progress on real-world problems in RL.

Large-Scale Human Datasets. A number of large-scale datasets of human replays
have been released for StarCraft [59], Dota [3], and MineRL [14]. However,
training models on these datasets requires massive computational resources,
which makes it unfeasible for academic or independent researchers. In
contrast, NLD strikes a better balance between scale (i.e. a large number of
diverse human demonstrations on a complex task) and efficiency (i.e. cheap to
use and fast to run).

For many real-world applications such as robotic manipulation, we only have
access to the demonstrator’s observations and not their actions [29, 48, 40,
8, 51]. Research on this setting has been slower [9, 56, 7], in part due to
the lack of efficient large-scale datasets. While there are some datasets
containing only observations, they are either much smaller than NLD [32, 57,
48], too computationally expensive [59, 3], or lack a simulator which prevents
learning from online interactions [12, 28, 8].

##  3 Background: The NetHack Learning Environment

The NetHack Learning Environment (NLE) is a gym environment [4] based on the
popular “dungeon-crawler” game, NetHack [19]. Despite the visual simplicity,
NetHack is widely considered one of the hardest video games in history since
it can take years for humans to win the game [54]. Players need to explore the
dungeon, manage their resources, as well as learn about the many entities and
their dynamics (often by relying on external knowledge sources like the
NetHack Wiki [34]). NetHack has a clearly defined goal, namely descend the
dungeon, retrieve an amulet, and ascend back to win the game. At the beginning
of each game, players are randomly assigned a given multidimensional character
defined by role, race, alignment, and gender (which have varying properties
and challenges), so they need to master all characters in order to win
consistently. Thus, NLE offers a unique set of properties which make it well-
positioned to advance research on RL and learning from demonstrations: it is a
highly complex environment, containing hundreds of entities with different
dynamics; it is procedurally generated, allowing researchers to test
generalization; it is partially observed, highly stochastic, and has very long
episodes (i.e. one or two orders of magnitude longer that Starcraft II [59]).

Following its release, several works have built on NLE to leverage its
complexity in different ways. MiniHack [45] allows researchers to design their
own environments to test particular capabilities of RL agents, by leveraging
the full set of entities and dynamics from NetHack. The NetHack Challenge [15]
was a competition at NeurIPS 2021, which sought to incentivise a showdown
between symbolic and deep RL methods on NLE. Symbolic bots decisively
outperformed deep RL methods, with the best performing symbolic bots
surpassing state-of-the-art deep RL methods by a factor of 5.

##  4 The NetHack Learning Dataset

The NetHack Learning Dataset (NLD) contains three components:

  1. 1.

NLD-NAO — a directory of ttyrec.bz2 files containing almost 10 billion state
trajectories and metadata from 1,500,000 human games of NetHack played on
nethack.alt.org.

  2. 2.

NLD-AA — a directory of ttyrec3.bz2 files containing over 3 billion state-
action-score trajectories and metadata from 100,000 games collected from the
winning bot of the NetHack Challenge [15].

  3. 3.

TtyrecDataset — a Python class that can scalably load directories of
ttyrec.bz2 / ttyrec3.bz2 files and their metadata into numpy arrays.

We are also releasing a new version of the NetHack environment, NLE v0.9.0,
which contains new features and ensures compatibility with NLD (see Appendix
C).

File Format. The ttyrec111https://nethackwiki.com/wiki/Ttyrec file format
stores sequences of terminal instructions (equivalent to observations in RL),
along with the times at which to play them. In NLE v0.9.0, we adapt this
format to also store keypress inputs to the terminal (equivalent to actions in
RL), and in-game scores over time (equivalent to rewards in RL), allowing a
reconstruction of state-action-score trajectories. This adapted format is
known as ttyrec3. The ttyrec.bz2 and ttyrec3.bz2 formats, compressed versions
of ttyrec and ttyrec3, are the primary data formats used in NLD. Using
TtyrecDataset, these compressed files can be written and read on-the-fly,
resulting in data compression ratios of more than 138. The files can be
decompressed into the state trajectory on a terminal, by using a terminal
emulator and querying its screen. For more details see Appendix D.

Throughout the paper, we refer to a player’s input at a given time as either
state or observation. However, note that NetHack is partially observed, so the
player doesn’t have access to the full state of the game. We also sometimes
use the terms score and reward interchangeably, since the increment in in-game
score is a natural choice for the reward used to train RL agents on NetHack.
Similarly, a human’s keypress corresponds to an agent’s action in the game.

Metadata. NetHack has an optional built-in feature for the logging of game
metadata, used for the maintenance of all-time high-score lists. At the end of
a game, 26 fields of data are logged to a common
xlogfile222https://nethackwiki.com/wiki/Xlogfile for posterity. These fields
include the character’s name, race, role, score, cause of death, maximum
dungeon depth, and more. See Appendix E for more details on these fields. With
NLE v0.9.0, an xlogfile is generated for all NLE recordings. These files are
used to populate all metadata contained in NLD.

State-Action-Score Transitions. As mentioned, NLD-AA contains sequences of
state-action-score transitions from symbolic-bot plays, while NLD-NAO contains
sequences of state transitions from human plays. These transitions are
efficiently processed using the TtyrecDataset. The states consist of:
tty_chars (the characters at each point on the screen), tty_colors (the colors
at each point on the screen), tty_cursor (the location of the cursor on the
screen), timestamps (when the state was recorded), and gameids (an identifier
for the game in question). Additionally, keypresses and score observations are
available for ttyrec3 files, as in the NLD-AA dataset. The states, keypresses,
and scores from NLD map directly to an agent’s observations, actions, and
rewards in NLE. More information about these transitions can be found in
Appendix F.

API. The TtyrecDataset follows the API of an IterableDataset defined in
PyTorch. This allows for the batched streaming of ttyrec.bz2 / ttyrec3.bz2
files directly into fixed NumPy arrays of a chosen shape. Episodes are read
sequentially in chunks defined by the unroll length, batched with a different
game per batch index. The order of these games can be shuffled, predetermined
or even looped to provide a never-ending dataset. This class allows users to
load any state-action-score trajectory recorded from NLE v0.9.0 onwards.

The TtyrecDataset wraps a small sqlite3 database where it stores metadata and
the paths to files. This design allows for the simple querying of metadata for
any game, along with the dynamic subselection of games streamed from the
TtyrecDataset itself. For example, in Figure 1, we generate sub-datasets from
NLD-NAO and NLD-AA, selecting trajectories where the player has completed the
game (‘Ascended’) or played a ‘Human Monk’ character, respectively. Appendix G
shows how to load only a subset of trajectories, for example where the player
has ascended, or a certain role has been used.

Scalability. The TtyrecDataset is designed to make our large-scale NLD
datasets accessible even when the computational resources are limited. To that
end, several optimizations are made to improve the memory efficiency and
throughput of the data. Most notably, TtyrecDataset streams recordings
directly from their compressed ttyrec.bz2 files. This format compresses the
30TB of frame data included in NLD-NAO down to 229GB, which can fit on a $30
SSD333https://www.amazon.com/HP-240GB-Internal-Solid-State/dp/B09KFHTYWH. This
decompression requires on-the-fly unzipping and terminal emulation. The
TtyrecDataset performs these in GIL-released C/C++. This process is fast and
trivially parallelizable with Python Threadpool, resulting in throughputs of
over 1.7 GB/s on 80 CPUs. This performance allows the processing of almost 10
billion frames of NLD-NAO in 15 hours, with 10 CPUs. See Table 1 for a
quantitative description of our two datasets.

Table 1: NLD-AA and NLD-NAO in numbers. | NLD-AA | NLD-NAO
---|---|---
Episodes | 109,545 | 1,511,228
Transitions | 3,481,605,009 | 9,858,127,896
Policies (Players) | 1 | 48,454
Policies Type | symbolic bot | human
Transition | (state, action, score) | state
Disk Size (Compressed) | 96.7 GB | 229 GB
Data Size (Uncompressed) | 13.4 TB | 38.0 TB
Compression Ratio | 138 | 166
Mean Epsiode Score | 10,105 | 127,218
Median Episode Score | 5,422 | 836
Median Episode Transitions | 28,181 | 1,724.0
Median Episode Game Turns | 20,414 | 3,766
Epoch Time (10 CPUs) | 4h 49m | 14h 37m

##  5 Dataset Analysis

In this section we perform an in-depth analysis of the characteristics of NLD-
AA and NLD-NAO.

###  5.1 NLD-AA

To our knowledge, AutoAscend444https://github.com/maciej-
sypetkowski/autoascend is currently the best open-sourced artificial agent for
playing NetHack 3.6.6, having achieved first place in the 2021 NeurIPS NetHack
Challenge by a considerable margin [15]. AutoAscend is a symbolic bot,
forgoing any neural network and instead relying on a priority list of hard-
coded subroutines. These subroutines are complex, context dependant, and make
significant use of NetHack domain knowledge and all NetHack actions. For
instances, the bot keeps track of multiple properties for encountered entities
and can even solve challenging puzzles such as Sokoban. A full description of
AutoAscend’s algorithm and behavior can be found in the NetHack Challenge
Report [15].

NLD-AA was generated by running AutoAscend on the NetHackChallenge-v0 [15]
task in NLE v0.9.0, utilising its built-in recording feature to generate
ttyrec3.bz2 files. It consists of over 3 billion state-action-score
transitions, drawn from 100,000 episodes generated by AutoAscend. Of the NLE
tasks, NetHackChallenge-v0 most closely reproduces the full game of NetHack
3.6.6, and was introduced in NLE v0.7.0 to grant NetHack Challenge competitors
access to the widest possible action space, and force an automated
randomisation of the starting character (by race, role, alignment, and
gender). By virtue of using ttyrec3.bz2 files, in-game scores and actions (in
the form of keypresses) are stored along with the states, and metadata about
the episodes.

Dataset Skill. The AutoAscend trajectories demonstrate a strong and reliable
NetHack player, far exceeding all the deep learning based approaches, but
still falling short of an expert human player. NetHack broadly defines a
character with less than 2000 score as a
‘Beginner’555https://nethackwiki.com/wiki/Beginner. AutoAscend comfortably
surpasses the ‘Beginner’ classification in more than 75% of games for all
roles, and in 95% of games for easier roles like Monk (see Figure 1). Given
the high variance nature of NetHack games, and the challenge of playing with
the more difficult roles, this is an impressive feat.

Compared to the human players in NLD-NAO, AutoAscend’s policy finds itself
just within the top 15% of all players when ranked by mean score. When ranked
by median score666Median score was the primary metric in the NetHack Challenge
it comes within the top 7%. However, these metrics are somewhat distorted by
the long tail of dilettante players who played only a few games. If instead we
define a ‘competent’ human player as one to have ever advanced beyond the
Beginner classification, then AutoAscend ranks in the top 33% of players by
mean score, and top 15% by median.

The competence of AutoAscend contrasts both with the poor performance of deep
RL bots, and with the exceptional performance required to beat the game. As
the winning symbolic bot of the NetHack Challenge, AutoAscend beat the winning
deep learning bot by almost a factor of 3 in median score, and close to a
factor of 5 in mean score. This performance is far outside what deep RL agents
can currently achieve, and in some domains NLD-AA may be considered an
“expert" dataset. However, when compared to the distribution of winning games
(i.e. those that have ‘ascended’), both mean and median score are orders of
magnitude short.

Dataset Coverage. Thanks to its scale, NLD-AA provides solid coverage of the
early-to-mid game of NetHack. NLD-AA also provides good coverage of all
possible actions and roles, with episodes that last for tens of thousands of
transitions. Nevertheless, the dataset contains many complex and diverse
behaviors. For the full breakdown of the symbolic bot’s behaviors found in
NLD-AA see Appendix E.

Datasest Noise. Since all trajectories were generated using NLE, these
trajectories have no terminal rendering noise. NLE uses a VT100 emulator to
record ttyrec3.bz2 files, and NLD uses the same emulator to load them.

Figure 1: Box plots of episode scores (left) and episode turns (right) for
NLD-NAO

and NLD-AA as well as for their corresponding subsets that contain only
winning games (‘Ascended’) or games played with a ‘Human Monk’ character,
respectively. These plots show the distribution in ability in NLD-NAO is
heavy-tailed, along with the typical scores and game length needed to win,
well above the median. By comparison, NLD-AA games are competent but not
expert, even for easy characters like ‘Human Monk’.

###  5.2 NLD-NAO

The NetHack community has a long history of using public servers, wherein
players can play together, stream and rewatch games of
NetHack.777https://nethackwiki.com/wiki/Public_server. The nethack.alt.org
server, known as NAO888https://nethackwiki.com/wiki/Nethack.alt.org, is one of
the longest running of such servers, having started in 2001, and hosts
recordings dating back to 2009. The server attempts to record all games in
ttyrec.bz2 format, and hosts them publicly on an S3
bucket999s3://altorg/ttyrec/. Unlike NLE, NAO offers players the option of
saving a game and resuming play later, allowing for games that span multiple
sessions, or last potentially days, weeks, months or years.

NLD-NAO was collected by downloading and filtering a dump of ttyrec.bz2
recordings and metadata from NAO. It consists of almost 10 billion state
transitions, drawn from approximately 1.5 million games played by just under
50,000 humans of mixed ability. These games were played between 2009 and 2020,
and span NetHack versions v3.4.0 to 3.6.6. Since NAO provides no link between
episode metadata and available ttyrec recordings, we developed an algorithm to
assign and correctly order ttyrec recordings to their corresponding episode
metadata, and to filter out trivial or empty games. The full procedure is
outlined in Appendix H.

Dataset Skill. The NAO trajectories were generated from humans of very
different skill levels, ranging from novice to expert, and cover a vast
distribution of outcomes. The distributions of episode scores, episode turns,
and average player scores are highly Zipfian, with long correlated tails
spanning several orders of magnitude. The correlation of these distributions
greatly improves the quality of the dataset. While the median episode score
does not progress beyond Beginner status, such short episodes contribute
relatively few transitions to the overall dataset. The bulk of the transitions
comes from medium-to-high scoring games that last a long time. The NLD-NAO
dataset therefore combines two desirable properties: it contains a large
number of diverse behaviors with varying levels of performance, as well as
many good trajectories from expert demonstrators.

Dataset Coverage. Thanks to its size and expert policies, NLD-NAO provides an
unparalleled coverage of the game of NetHack. The dataset contains many
examples of advanced game sections not yet achievable by symbolic or RL bots,
such as descending the dungeon and retrieving the amulet. Most importantly, it
includes 22,000 examples of winning the game (“Ascension”), and includes
several examples of doing so with special conducts such as “Nudist” (not
wearing any armour). The full distribution of such achievements and conducts
can be found in Appendix E.

Dataset Noise. Unlike NLE, NAO does not enforce any fixed terminal type on the
player, nor any fixed screen dimensions or configuration. Variations in all
these can lead to rendering noise that may include odd symbols or cropped
layouts. These types of artefacts are common in real-world applications of
learning from demonstrations, which makes NLD a suitable dataset for advancing
research in this area. More details about these artefacts, as well as other
types of noise (e.g. matching of recordings to players), see Appendix H.

##  6 Experiments

###  6.1 Methods

We demonstrate the utility and challenge of NLD by evaluating algorithms from
a range of fields including online RL, offline RL, imitation learning, and
learning from demonstrations.

All methods are implemented using the open-source asynchronous RL library
moolib [31], allowing for continual evaluation of agents in order to generate
training curves (even in the offline RL setting). In addition, all our methods
use the same core model architecture, which is based on a popular open-source
Sample Factory baseline101010https://github.com/Miffyli/nle-sample-factory-
baseline, adapted to only use tty_* observations but still remain fast to run.
Unless specified otherwise, all experiments were evaluated with 1024 episodes
from NLE’s NetHackChallenge-v0 task after training for 1.5B frames. The mean
and standard deviation of the episode return was computed over 10 training
seeds. Full details about the algorithms and their hyperparameters can be
found in Appendix I. All our implementations will be open-sourced to enable
fast research progress.

Online RL. For online RL baselines that do not make use of the NLD datasets,
we use Asynchronous Proximal Policy Optimization (APPO) [47, 37] and Deep
Q-Networks (DQN-Online). The former is a strong on-policy policy-gradient-
based method for NLE, and the latter is a popular off-policy value-based
method in the wider literature.

Offline RL. Since NLD-AA contains state-action-reward transitions, we can
train a number of offline RL baselines on it. More specifically, we use a Deep
Q-Network baseline trained on the dataset (DQN-Offline), as well as two state-
of-the-art offline RL methods, Conservative Q-Learning (CQL) [21] and Implicit
Q-Learning (IQL) [20]. Note that these methods cannot be trained on NLD-NAO
since it doesn’t contain actions or rewards.

Learning from Demonstrations. We also evaluate a number of baselines that do
not make use of rewards, but instead use state-action or state-only
transitions. For NLD-AA, we use Behavioural Cloning (BC), a popular imitation
learning method, which trains a policy to match the corresponding actions in
the dataset using a supervised learning objective. For NLD-NAO, we use
Behavioral Cloning from Observations (BCO) [56] which is a popular baseline
for learning from state-only demonstrations. This method performs BC on a
learned inverse dynamics model which was trained to predict the action taken
given two consecutive states. We then investigate the use of BC and BCO in an
online setting by augmenting our APPO baseline in two ways. First, we add the
BC or BCO loss between the agent’s policy and the expert’s action for
observations in the dataset resulting in APPO + BC and APPO + BCO
respectively. Second, we add a KL-divergence loss between the agent’s policy
and a pretrained BC(O) model resulting in Kickstarting BC(O), similar to [46].
These methods make use of state and action data but no rewards.

For NLD-AA, we investigate the performance of the above algorithms across two
partitions: one with a single character (‘Human Monk’), and one with all the
characters (i.e. the full dataset). Each of these is evaluated on their
corresponding environments. Full details about our experimental setup are
available in Appendix I.

Table 2: Average episode return across 10 seeds for a number of popular baselines using NLD and / or NLE. First, we evaluate a range of offline RL methods such as DQN-Offline, CQL, IQL and BC trained on NLD-AA and NLD-AA-Monk, as well as BCO trained on NLD-NAO. Second, we evaluate methods that combine offline and online learning such as Kickstarting + BC and APPO + BC trained on NLD-AA and NLD-AA-Monk, as well as Kickstarting + BCO and APPO + BCO trained on NLD-NAO. Finally, we evaluate a few methods that only have access to online interactions from NLE (no external datasets) such as a Random Policy, DQN-Online, and APPO. All evaluations are conducted with NLE’s NetHackChallenge-v0 task, using Random (@) characters for NLD-AA and NLD-NAO, and a Human Monk character for NLD-AA-Monk. We compare all these to the average episode return in the corresponding datasets. Methods that use both offline and online data outperform all others by a wide margin, but are still significantly worse than humans (by almost two orders of magnitude) and even the symbolic bot (by more than a factor of three). The top online algorithm, APPO, outperforms the top offline approach, BC. These results indicate that NLD poses a substantial challenge to state-of-the-art methods. |  |  | NLD-AA | NLD-AA-Monk | NLD-NAO
---|---|---|---
Character | Random | Human Monk | Random
offline only | $\left\\{\begin{tabular}[]{<EMAIL_ADDRESS>± 0.1&0.0 ± 0.1&-\\\ CQL&704 ± 112&732 ± 219&-\\\ IQL&341 ± 40&533 ± 178&-\\\ {BC(O)}&{1220 ± 261}&{1728 ± 615}&{12.5 ± 2.0}\\\ \phantom{Kickstarting + BC(O))}&\phantom{ PADDIN WORD}&\phantom{ PADDIN WORD}&\phantom{(see column 1}{}\\\\[-12.0pt] \end{tabular}\right.\kern-1.2pt$
|
offline + online | $\left\\{\begin{tabular}[]{@{}lccc@{}}Kickstarting + BC(O)&2250 ± 81&3907 ± 330&502 ± 62\\\ {APPO + BC(O)}&{2767 ± 284}&{4779 ± 1116}&{1972 ± 180}\\\ \phantom{Kickstarting + BC(O))}&\phantom{ PADDIN WORD}&\phantom{ PADDIN WORD}&\phantom{(see column 1}{}\\\\[-12.0pt] \end{tabular}\right.\kern-1.2pt$
|
online only | $\left\\{\begin{tabular}[]{@{}lccc@{}}Random Policy&0.2 ± 0.2&0.4 ± 0.4&(see col 1)\\\ DQN-Online&159 ± 68&247 ± 152&(see col 1)\\\ {APPO}&{1799 ± 156}&{4006 ± 323}&{(see col 1)}\\\ \phantom{Kickstarting + BC(O))}&\phantom{ PADDIN WORD}&\phantom{ PADDIN WORD}&\phantom{(see column 1}{}\\\\[-12.0pt] \end{tabular}\right.\kern-1.2pt$
|  | Dataset Average | 10105 | 17274 | 127356
---|---|---|---

###  6.2 Results and Discussion

As Table 2 and Figure 2 show, NLD-AA poses a difficult challenge to modern
offline RL algorithms. While CQL is able to use the offline trajectories to
outperform both a random policy, IQL, as well as online and offline variants
of DQN, it falls short of capturing the average performance of trajectories in
the dataset by over an order of magnitude. BC performs better due to the
higher quality of the supervised learning signal, which is typically denser
and less noisy than the RL one, but it too falls well short of the data
generating policy. Prior work has shown that offline RL methods struggle to
perform well in stochastic environments [36], and to generalize outside the
distribution of state-action pairs in the dataset [13]. These challenges are
particularly intensified with NLD, which is highly stochastic, partially
observed, procedurally generated, and has a very large state and action space.

Methods that combine offline and online training such as Kickstarting + BC(O)
or APPO + BC(O) outperform offline-only methods and outperform or match
online-only methods, while also speeding up training. In fact, APPO + BC sets
a new state-of-the-art for purely Deep RL on NLE, and would have won first
place on the Neural Track at the NetHack Challenge 2021, scoring a median of
2110 and beating the hybrid symbolic-deep RL winner by almost 400 points [15].
Despite being our best method, APPO + BC is still significantly worse than the
average performance in the dataset, further emphasizing the difficulty of NLD.

Finally, our results demonstrate the utility of NLD-NAO while highlighting the
challenge of learning from sequences of only observations. Our inverse model
is trained with trajectories from a random policy, and therefore experiences
an extremely small fraction of the entire state-action space. Even so, BCO on
NLD-NAO labeled with these actions is able to yield performance that
comfortably exceeds that of a random policy. While APPO + BCO performs
similarly to APPO, training appears to be much more stable. Thus, using more
sophisticated policies for training the inverse model (e.g. a mix between a
pretrained RL policy and a random policy or an exploration method that
maximizes the diversity of visited states and actions) could further improve
performance.

Figure 2: Mean episode return (with samples plotted) over 10 seeds during
training for APPO, APPO + BC(O), and Kickstarting + BC(O) using NLD-AA (left),
NLD-AA-Monk (center), and NLD-NAO (right). NLD-AA and NLD-NAO both are
evaluated with Random character (@), while NLD-AA-Monk is evaluated with only
the Human Monk character. Some sample curves associated with the pure online
APPO methods exhibit a behaviour wherein performance appears to plateau,
before bursting rapidly to a new performance level. This is a common pattern
in NLE and corresponds to an agent learning a new behaviour that greatly
unlocks yet more of the game, such as praying when starving or kicking down
locked doors. Since NLE is highly procedurally generated with a vast action
and state space, it is hard to predict when an agent will encounter sufficient
opportunities to “take off” in this way. Such “taking off” phenomena are less
visible when learning from NLD, as useful behaviours are present in the data
throughout.

##  7 Conclusion

In this paper, we introduce NLD, a large-scale dataset of demonstrations from
NetHack, a complex, stochastic, partially observed, procedurally generated
video game. NLD enables research in multiple areas such as imitation learning,
offline RL, learning from sequences of observations, and learning from both
offline data and online interactions. NLD allows researchers to learn from
very large datasets of demonstrations without requiring extensive memory or
compute resources. We train a number of state-of-the-art methods in online RL
(both on-policy and off-policy), offline RL, imitation learning, as well as
learning from observation-only sequences. Our results indicate that
significant research advances are needed to fully leverage large-scale
datasets in order to solve challenging sequential decision making problems.

In many real-world applications such as robotics, it is common to have access
to datasets of observations, without corresponding actions or rewards. Thus,
one important direction for future research is to learn from trajectories
containing only observations. NLD enables this type of research in a safe,
cheap, fast, and reproducible environment, while still exhibiting many
properties of real-world domains such as partial observability, high
stochasticity, and long episodes. Due to NetHack being procedurally generated,
in order to consistently win the game, an agent needs to learn a robust policy
that generalizes to new scenarios. Our strongest baseline learns from both
static data and environment interaction, so a promising avenue for future work
is to develop more effective methods that combine offline and online learning.
Existing algorithms for offline RL and learning from demonstrations struggle
to match the average performance of trajectories in our datasets, suggesting
that NLD is a good benchmark to probe the limits of these methods and inspire
the development of approaches that are effective in more challenging and
realistic domains.

##  8 Acknowledgements

We would like to thank Drew Streib and the NAO maintainers for their work in
the NetHack community and their help throughout this project. We would also
like to thank Michał Sypetkowski and Maciej Sypetkowski for their work on the
AutoAscend bot.

## References

  * [1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  Striving for simplicity in off-policy deep reinforcement learning.  ArXiv, abs/1907.04543, 2019.
  * [2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In ICML, 2020.
  * [3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang.  Dota 2 with large scale deep reinforcement learning.  ArXiv, abs/1912.06680, 2019.
  * [4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  arXiv preprint arXiv:1606.01540, 2016.
  * [5] Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerík, Oleg O. Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyun Wang.  A framework for data-driven robotics.  ArXiv, abs/1909.12200, 2019.
  * [6] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal.  Adversarially trained actor critic for offline reinforcement learning.  ArXiv, abs/2202.02446, 2022.
  * [7] Samuel Cohen, Brandon Amos, Marc Peter Deisenroth, Mikael Henaff, Eugene Vinitsky, and Denis Yarats.  Imitation learning from pixel observations for continuous control.  In Deep RL Workshop NeurIPS 2021, 2021.
  * [8] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn.  Robonet: Large-scale multi-robot learning.  ArXiv, abs/1910.11215, 2019.
  * [9] Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Lee Isbell.  Imitating latent policies from observation.  ArXiv, abs/1805.07914, 2019.
  * [10] Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  ArXiv, abs/2004.07219, 2020.
  * [11] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In ICML, 2019.
  * [12] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32:1231 – 1237, 2013.
  * [13] Caglar Gulcehre, Sergio Gómez Colmenarejo, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, Nando de Freitas, et al.  Addressing extrapolation error in deep offline reinforcement learning.  2020\.
  * [14] William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela M. Veloso, and Ruslan Salakhutdinov.  Minerl: A large-scale dataset of minecraft demonstrations.  ArXiv, abs/1907.13440, 2019.
  * [15] Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Minqi Jiang, Daejin Jo, Anssi Kanervisto, Jongmin Kim, et al.  Insights from the neurips 2021 nethack challenge.  arXiv preprint arXiv:2203.11889, 2022.
  * [16] Nicolas Manfred Otto Heess, TB Dhruva, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyun Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver.  Emergence of locomotion behaviours in rich environments.  ArXiv, abs/1707.02286, 2017.
  * [17] James Henderson, Oliver Lemon, and Kallirroi Georgila.  Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets.  Computational Linguistics, 34:487–511, 2008.
  * [18] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah J. Jones, Shixiang Shane Gu, and Rosalind W. Picard.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  ArXiv, abs/1907.00456, 2019.
  * [19] Kenneth Lorber.  NetHack Home Page.  https://nethack.org, 2020.  Accessed: 2020-05-30.
  * [20] Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  ArXiv, abs/2110.06169, 2021.
  * [21] Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  ArXiv, abs/2006.04779, 2020.
  * [22] Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and B. Leibe.  The atari grand challenge dataset.  ArXiv, abs/1705.10998, 2017.
  * [23] Heinrich Kuttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel.  The nethack learning environment.  ArXiv, abs/2006.13760, 2020.
  * [24] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter.  Learning quadrupedal locomotion over challenging terrain.  Science Robotics, 5, 2020.
  * [25] Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  ArXiv, abs/2005.01643, 2020.
  * [26] Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.  The International Journal of Robotics Research, 37:421 – 436, 2018\.
  * [27] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire.  A contextual-bandit approach to personalized news article recommendation.  ArXiv, abs/1003.0146, 2010.
  * [28] Jeffrey Mahler, Matthew Matl, Vishal Satish, Michael Danielczuk, Bill DeRose, Stephen McKinley, and Ken Goldberg.  Learning ambidextrous robot grasping policies.  Science Robotics, 4, 2019.
  * [29] Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart’in-Mart’in.  What matters in learning from offline human demonstrations for robot manipulation.  ArXiv, abs/2108.03298, 2021.
  * [30] Michael Matthews, Mikayel Samvelyan, Jack Parker-Holder, Edward Grefenstette, and Tim Rocktäschel.  Hierarchical kickstarting for skill transfer in reinforcement learning, 2022.
  * [31] Vegard Mella, Eric Hambro, Danielle Rothermel, and Heinrich Küttler.  moolib: A Platform for Distributed RL.  2022\.
  * [32] Raphael Memmesheimer, Ivanna Mykhalchyshyna, Viktor Seib, and Dietrich Paulus.  Simitate: A hybrid imitation learning benchmark.  2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5243–5249, 2019.
  * [33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller.  Playing atari with deep reinforcement learning.  ArXiv, abs/1312.5602, 2013.
  * [34] NetHack Wiki.  NetHackWiki.  https://nethackwiki.com/, 2020.  Accessed: 2020-02-01.
  * [35] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas A. Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei M. Zhang.  Solving rubik’s cube with a robot hand.  ArXiv, abs/1910.07113, 2019.
  * [36] Keiran Paster, Sheila McIlraith, and Jimmy Ba.  You can’t count on luck: Why decision transformers fail in stochastic environments.  arXiv preprint arXiv:2205.15967, 2022.
  * [37] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun.  Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning.  In International Conference on Machine Learning, pages 7652–7662. PMLR, 2020.
  * [38] Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet.  Sample-efficient batch reinforcement learning for dialogue management optimization.  ACM Trans. Speech Lang. Process., 7:7:1–7:21, 2011.
  * [39] Rafael Figueiredo Prudencio, Marcos R.O.A. Maximo, and Esther Luna Colombini.  A survey on offline reinforcement learning: Taxonomy, review, and open problems.  ArXiv, abs/2203.01387, 2022.
  * [40] Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu.  Neorl: A near real-world benchmark for offline reinforcement learning.  ArXiv, abs/2102.00714, 2021.
  * [41] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  ArXiv, abs/1709.10087, 2018.
  * [42] Sabela Ramos, Sertan Girgin, L’eonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphaël Marinier, Jeremiah Harmsen, Olivier Pietquin, and Nikola Momchev.  Rlds: an ecosystem to generate, share and use datasets in reinforcement learning.  ArXiv, abs/2111.02767, 2021.
  * [43] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In AISTATS, 2011.
  * [44] Andrei A. Rusu, Matej Vecerík, Thomas Rothörl, Nicolas Manfred Otto Heess, Razvan Pascanu, and Raia Hadsell.  Sim-to-real robot learning from pixels with progressive nets.  ArXiv, abs/1610.04286, 2017.
  * [45] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel.  Minihack the planet: A sandbox for open-ended reinforcement learning research.  arXiv preprint arXiv:2109.13202, 2021.
  * [46] Simon Schmitt, Jonathan J. Hudson, Augustin Zídek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami.  Kickstarting deep reinforcement learning.  ArXiv, abs/1803.03835, 2018.
  * [47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [48] Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Kumar Gupta.  Multiple interactions made easy (mime): Large scale demonstrations data for imitation.  In CoRL, 2018.
  * [49] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin A. Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  ArXiv, abs/2002.08396, 2020.
  * [50] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [51] Bradly C. Stadie, P. Abbeel, and Ilya Sutskever.  Third-person imitation learning.  ArXiv, abs/1703.01703, 2017.
  * [52] Alexander L. Strehl, John Langford, Lihong Li, and Sham M. Kakade.  Learning from logged implicit exploration data.  In NIPS, 2010.
  * [53] Michita Imai Takuma Seno.  d3rlpy: An offline deep reinforcement library.  In NeurIPS 2021 Offline Reinforcement Learning Workshop, December 2021.
  * [54] The Telegraph.  The 25 hardest video games ever.  2021\.  Accessed: 2021-05-05.
  * [55] Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill.  Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing.  In AAAI, 2017.
  * [56] Faraz Torabi, Garrett Warnell, and Peter Stone.  Behavioral cloning from observation.  ArXiv, abs/1805.01954, 2018.
  * [57] Sam Toyer, Rohin Shah, Andrew Critch, and Stuart J. Russell.  The magical benchmark for robust imitation.  ArXiv, abs/2011.00401, 2020.
  * [58] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  Nature, pages 1–5, 2019.
  * [59] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John P. Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, H. V. Hasselt, David Silver, Timothy P. Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing.  Starcraft ii: A new challenge for reinforcement learning.  ArXiv, abs/1708.04782, 2017.
  * [60] Yifan Wu, G. Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning.  ArXiv, abs/1911.11361, 2019.
  * [61] Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor K. Prasanna.  Brac+: Improved behavior regularized actor critic for offline reinforcement learning.  In ACML, 2021.
  * [62] Çaglar Gülçehre, Ziyun Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Jaymin Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Zheng Li, Mohammad Norouzi, Matthew D. Hoffman, Nicolas Manfred Otto Heess, and Nando de Freitas.  Rl unplugged: A collection of benchmarks for offline reinforcement learning.  In NeurIPS, 2020.

##  Appendix A Accessing the Dataset

For the duration of the review the information on how to download, install and
use NLD will be available at
https://github.com/dungeonsdatasubmission/dungeonsdata-neurips2022. This link
contains a README.md file with instructions on how to download zip files for
NLD-AA and NLD-NAO, as well as instructions on how to install and use the
TtyrecDataset, and includes code used in the running of experiment.

After review, this information will migrate to the README.md for the NLE repo,
hosted at https://github.com/facebookresearch/nle.

#### Hosting & Maintenance Plan

The dataset will eventually be hosted on a separate, dedicated site, with
infrastructure akin to that provided on https://ai.facebook.com/datasets/.
This site will be permanently available for the foreseeable future, with links
to the paper and NLE code. The site is currently planned for launch in advance
of the NeurIPS 2022 main conference, links will be available on the NLE repo’s
README.md.

##  Appendix B License

Data is provided under the NetHack General Public License - A GPL style
license that is used to covered the NetHack Game since 1989. The license can
be found at https://github.com/facebookresearch/nle/blob/main/LICENSE, and is
used by both the official NetHack
repository111111https://github.com/NetHack/NetHack and NLE.

##  Appendix C NLE v0.9.0

NLE v0.9.0 will be the latest version of NLE released, and will be released
with NLD. The release branch is will be cut from main branch on the repo,
hosted at (https://github.com/facebookresearch/nle/), and deployed to PyPI.
The branch currently holds all the changes needed for compatibility with NLD,
including the TtyrecDataset. A full changelog will be made public after
release but the changes include:

  * •

New ability to record ttyrec3.bz2 files directly from NLE. Previous versions
recorded a modified ttyrec.bz2.

  * •

The additional logging to an xlogfile for each episodes that finish naturally
(i.e. when done is True).

  * •

A C++ Converter class to load ttyrec.bz2/ttyrec3.bz2 files directly in to
NumPy arrays, based on a V100 Terminal emulator (from
libtmt121212https://github.com/deadpixi/libtmt).

  * •

The TtyrecDataset class, to marshall the Converter objects into an
Torch.IterableDataset interface, and handle metadata.

##  Appendix D The ttyrec format

The ttyrec is a file format that has historically been used to store
recordings of terminal-based NetHack
games.131313https://nethackwiki.com/wiki/Ttyrec The format consists of a stack
of frames where each frame consists of a 12-byte header immediately followed
by a variable-size buffer of terminal instructions. The header contains 8
bytes of time information (storing when these updates were recorded) and 4
bytes indicating the size of the subsequent buffer (storing what these updates
were). The buffer’s contents are then the instructions to be fed to a terminal
that will be rendering ttyrec. As such, this buffer will then generally
consist of terminal-specific Escape
Sequences141414https://en.wikipedia.org/wiki/Escape_sequence and text to be
printed.

NLE’s ttyrec3 has a 13-byte header, adding 1-byte channel to indicate what
kind of content is in the buffer: 0 for terminal output; 1 for terminal input
(i.e the keypresses corresponding to actions in the game); and, 2 for in-game
score. These latter two channels are written just after and just before an
action is taken (respectively).

It is worth noting that frames with channel 0 are written whenever NetHack
intends to write to the screen, and therefore several may written before an
action is required. An example of such an event occurs when zapping wand which
animates a beam from the player. Such animations may result in a stack of
frames with channels that resemble something like [0 2 1 0 0 0 2 1 ...], where
several different states might correspond to one action. In this case
ttyrec3.bz2 rendering loads the final state of the of the terminal NumPy
arrays, alongside the score and action, while ttyrec.bz2 rendering loads every
state/frame, having no clear delineation of where actions take place.

To further inspect ttyrec.bz2 and ttyrec3.bz2, NLE v0.9.0 comes with an
adapted read_tty.py script that displays the raw, decompressed ttyrec/ttyrec3
contents, shown in Figure 3.

Figure 3: A screenshot of a tool to read raw ttyrec.bz2/ttyrec3.bz2 files,
included in NLE v0.9.0. The numbers in green represent the frame count for
each channel [0, 1, 2], followed by the timestamp, and the buffer contents in
braces: channel 0 is displayed in yellow braces, 1 in blue braces, and 2 in
pink braces. Terminal Escape Sequences are printed in dark grey.

##  Appendix E NLD Gameplay Metadata Reference

The gameplay metadata in NLD is identical to that stored NetHacks’s xlogfile
at the end of a game, with the addition of a gameid which is used by NLD to
identify games. The metadata consists of:

  1. 1.

gameid (int) \- A unique id for the game, created by the local database.

  2. 2.

version (str) \- The version of NetHack played.

  3. 3.

points (int) \- The final in-game score of the episode.

  4. 4.

deathdnum (int) \- The dungeon number where you died. These correspond to the
dungeons found in dungeon.def file:

     * •

0 \- The Dungeons of Doom

     * •

1 \- Gehennom

     * •

2 \- The Gnomish Mines

     * •

3 \- The Quest

     * •

4 \- Sokoban

     * •

5 \- Fort Ludios

     * •

6 \- Elemental Planes

  5. 5.

deathlev (int) \- The dungeon level where you died.151515For more, see
https://nethackwiki.com/wiki/Mazes_of_Menace

  6. 6.

maxlvl (int) \- The deepest dungeon level you reached.

  7. 7.

hp (int) \- The number of hit points you ended the game with.

  8. 8.

maxhp (int) \- The number of hit points you’d have had at “full health” at the
game end.

  9. 9.

deaths (int) \- The number of times you died.161616In rare cases, you can die
more than once. See https://nethackwiki.com/wiki/Amulet_of_life_saving

  10. 10.

deathdate (int) \- The day the game ended (format YYYYMMDD).

  11. 11.

birthdate (int) \- The day the game started (format YYYYMMDD).

  12. 12.

uid (int) \- An id used (in conjunction with player name) to identify players
(for save games etc).

  13. 13.

role (str) \- The Role of the player. The 13 roles are:

     * •

Arc \- Archaelogist

     * •

Bar \- Barbarian

     * •

Cav \- Cave(wo)man

     * •

Hea \- Healer

     * •

Kni \- Knight

     * •

Mon \- Monk

     * •

Pri \- Priest(ess)

     * •

Ran \- Ranger

     * •

Rog \- Rogue

     * •

Sam \- Samurai

     * •

Tou \- Tourist

     * •

Val \- Valkyrie

     * •

Wiz \- Wizard

  14. 14.

race (str) \- The Race of the player. The 5 races are:

     * •

Dwa \- Dwarf

     * •

Elf \- Elf

     * •

Gno \- Gnome

     * •

Hum \- Human

     * •

Orc \- Orc

  15. 15.

gender (str) \- The Gender of the player. The 2 genders are:

     * •

Fem \- Female

     * •

Mal \- Male

  16. 16.

align (str) \- The Alignment of the player. The 3 alignments are:

     * •

Cha \- Chaotic

     * •

Law \- Lawful

     * •

Neu \- Neutral

  17. 17.

name (str) \- The name of the player. These are pseudonymised in the database
for NLD-NAO.

  18. 18.

death (str) \- A description of the manner in which the player died or ended
the game.

  19. 19.

conduct (str) \- a bitfield encoding the
conducts171717https://nethackwiki.com/wiki/Conduct completed in the game. The
bitfield encodes:

     * •

0x001: Foodless

     * •

0x002: Vegan

     * •

0x004: Vegetarian

     * •

0x008: Atheist

     * •

0x010: Weaponless

     * •

0x020: Pacifist

     * •

0x040: Illiterate

     * •

0x080: Polypileless

     * •

0x100: Polyselfless

     * •

0x200: Wishless

     * •

0x400: Artifact wishless

     * •

0x800: Genocideless

  20. 20.

turns (int) \- The number of in-game turns played by the player. This may not
correspond to transitions, as several moves do not advance the in-game clock
(eg checking the inventory or moving into a wall).

  21. 21.

achieve (str) \- a bitfield encoding the
achievements181818https://nethackwiki.com/wiki/Xlogfile attained in the game.
The bitfield encodes:

     * •

0x0001: Got the Bell of Opening

     * •

0x0002: Entered Gehennom

     * •

0x0004: Got the Candelabrum of Invocation

     * •

0x0008: Got the Book of the Dead

     * •

0x0010: Performed the Invocation

     * •

0x0020: Got the Amulet of Yendor

     * •

0x0040: Was in the End Game

     * •

0x0080: Was on the Astral Plane

     * •

0x0100: Ascended

     * •

0x0200: Got the Luckstone at Mines’ End

     * •

0x0400: Finished Sokoban

     * •

0x0800: Killed Medusa

     * •

0x1000: Zen conduct intact

     * •

0x2000: Nudist conduct intact

  22. 22.

realtime (int) \- the duration of the game in seconds

  23. 23.

starttime (int) \- the start time of the game as an (epoch) unix timestamp.

  24. 24.

endtime (int) \- the end time of the game as an (epoch) unix timestamp.

  25. 25.

gender0 (str) \- the starting gender of the player. Same options as gender.

  26. 26.

align0 (str) \- the starting alignment of the player. Same options as align.

  27. 27.

flags (str) \- a bitfield encoding some additional state of the game. The
bitfield encodes:

     * •

0x1: Wizard mode

     * •

0x2: Discover mode

     * •

0x4: Never loaded a Bones file

##  Appendix F NLD Observations Reference

The TtyrecDataset loads minibatches of observations, with batch size B, and
sequence/unroll length T. The following observations are present in each
minibatch dictionary:

Name | Type | Shape |  Description
---|---|---|---
tty_chars | np.uint8 | [B, T, H, W] |  The on-screen characters (default screen size 80 x 24).
tty_colors | np.int8 | [B, T, H, W] |  The on-screen colors for each character.
tty_cursor | np.int16 | [B, T, 2] |  The coordinates of the on-screen cursor.
timestamps | np.int64 | [B, T] |  The time each frame was recorded.
gameids | np.int32 | [B, T] |  The gameid for the episode being rendered.
done | np.uint8 | [B, T] |  An indicator that this frame is from a new gameid.
scores* | np.int32 | [B, T] |  The in-game score at this time (i.e. before keypress).
keypresses* | np.uint8 | [B, T] |  The keypress the player made in response to this state (i.e. after seeing tty_*, scores, etc).

The observations marked with an asterisk (*) are only available in ttyrec3.bz2
datasets.

##  Appendix G TtyrecDataset Reference

###  G.1 Design Overview

To load minibatches, the TtyrecDataset performs two main actions: first, at
construction time, it obtains the list of gameids and ttyrec.bz2/ttyrec3.bz2
filepaths that constitute dataset, and second, at usage time, it constructs an
Python generator191919https://peps.python.org/pep-0255/ that will sequentially
load these files into fixed NumPy arrays.

The first of these actions is performed by querying a local sqlite3 database
file, which acts as the metadata information hub for all datasets. This
database file need only be populated once, and from thenceforth can be copied,
shared or swapped out, without issue. For more on how to add datasets see the
Tutorial Notebook202020https://github.com/dungeonsdatasubmission/dungeonsdata-
neurips2022/blob/main/usage_tutorial_notebook.ipynb, or Appendix 4. The
metadata in this database includes the location of files for each game, the
association of which games are in which dataset, and the gameplay metadata for
each game (see Appendix E). For more on the database schema see Appendix G.2.

The second of these actions is performed by the TtyrecDataset itself, by
creating a lightweight C++ Converter object for each batch index in a
generator, and loading from these objects when next is called. This process
can be modified in a variety of different ways, including looping forever,
shuffling, or using a threadpool. For more on the way these can be modified
see Appendix G.3.

###  G.2 Database Layout

Table 3 shows the layout of the database, where fields of the same name map
directly to each other. Each dataset is associated with a string
(dataset_name), and once populated the dataset can be retrieved with this
string.

Table 3: The layout of the sqlite3 database used by the TtyrecDataset. The
roots table stores information about the root directory of the files in a
dataset, as well as ttyrec version, while ttyrecs table stores the data about
the files and what games they belong to. The datasets table then associates
the gameids to their dataset, and games contains the gameplay metadata for
each gameid.

roots |
---|---
dataset_name | TEXT
root | TEXT
ttyrec_version | INTEGER
datasets |
gameids | INTEGER
dataset_name | TEXT
ttyrecs |
path | TEXT
part | INTEGER
size | INTEGER
mtime | REAL
gameid | INTEGER
meta |
ctime | REAL
mtime | REAL

games |
---|---
gameid | INTEGER
version | TEXT
points | INTEGER
deathdnum | INTEGER
deathlev | INTEGER
maxlvl | INTEGER
hp | INTEGER
maxhp | INTEGER
deaths | INTEGER
deathdate | INTEGER
birthdate | INTEGER
uid | INTEGER
role | TEXT
race | TEXT
gender | TEXT
align | TEXT
name | TEXT
death | TEXT
conduct | TEXT
turns | INTEGER
achieve | TEXT
realtime | INTEGER
starttime | INTEGER
endtime | INTEGER
gender0 | TEXT
align0 | TEXT
flags | TEXT

###  G.3 TtyrecDataset API

The TtyrecDataset is documented primarily through Python docstrings, which
provide a helpful way automatically documenting code objects, that can be
queried by using Python’s help function. An example usage can be clearly seen
in the tutorial
notebook212121https://github.com/dungeonsdatasubmission/dungeonsdata-
neurips2022/blob/main/usage_tutorial_notebook.ipynb supplied with the
submission, that will also migrate to NLE.

The TtyrecDataset accepts the following arguments, with all but first taking
reasonable default values:

  1. 1.

dataset_name \- (str) \- The name of the dataset to load. Each datasets is
associated with a unique name.

  2. 2.

batch_size \- (int) \- The number of simulataneous episodes to load (i.e. size
B).

  3. 3.

seq_length \- (int) \- The number of steps to unroll for each episode in a
minibatch (i.e. size T).

  4. 4.

rows \- (int) \- The row size of the terminal to emulate (i.e. size H).

  5. 5.

cols \- (int) \- The column size of the terminal to emulate (i.e. size W).

  6. 6.

dbfilename \- (str) \- The path to the database file used.

  7. 7.

threadpool \- (Object) \- An executor class that has a map function (i.e.
concurrent.futures.ThreadPoolExecutor.

  8. 8.

gameids \- (List[str]) \- A predetermined list of gameids to iterate through.

  9. 9.

shuffle \- (bool) \- Whether to shuffle the order of gameids before iterating.

  10. 10.

loop_forever \- (bool) \- Whether to start again at the beginning of gameids
when the iterator runs out. If False, empty batch indexes are padded with 0.

  11. 11.

subselect_sql \- (str) \- A SQL string to dynamically generate subdataset, by
select gameids.

  12. 12.

subselect_args \- (Objects) \- Arguments to any SQL query provided in
subselect_sql.

###  G.4 Adding Datasets

Adding datasets to a the database is a very easy “one-off” operation. Examples
are shown in the Tutorial Notebook and also in Figure 4.

Since the process for adding ttyrec3.bz2 and ttyrec.bz2 files are a little
different from each other, two different methods should be used:
nle.dataset.add_nledata_directory for ttyrec3.bz2, and
nle.dataset.add_altorg_drectory for ttyrec.bz2 data from NAO. This is
documented in the code and in the Tutorial Notebook.

Figure 4: Example of how to simply create a database file and populate it with
datasets. Note that each directory must be associated with a dataset name
chosen by the user (i.e. ‘‘nld-aa’’). This is then used to access the data (as
in Figure 5).

###  G.5 ThreadPool Usage

All files are read independently, loaded into their own C++ Converter object
that provides lightweight terminal emulation. This independence provides the
potential for very effective parallelization along the batch index, B. In the
Converter object, we release the GIL, allowing for very true parallelization
of file processing with Python’s own threads. This can provide a many-fold
boost to performance and throughput of frames per second, as can be seen in
Table 4. Note, however, that the benefits of parallelization are mainly found
with long sequence lengths. Example usage is shown in Figure 5.

Figure 5: Example of how to use a ThreadPoolExecutor using Python 3’s built-in concurrent.futures module. This parallelism allows control of how many CPU workers are use to load data, without having to resort to Python’s multiprocessing. Table 4: The frames per second, mean (standard deviation), across 5 runs of 500k frames when served by the NLD dataset with different parameters for batch size, sequence length, and number of CPUs used (which corresponds to the number of threads in the threadpool). |  |  | Sequence Length
---|---|---|---
|  |  | 32 | 128 | 512
| Batch Size | 32 | 20.6k (325) | 19.4k (250) | 20.7k (499)
1 CPU | 128 | 16.5k (159) | 18.9k (235) | 21.2k (254)
| 512 | 15.1k (325) | 17.1k (218) | 17.9k (94)
| Batch Size | 32 | 73.8k (3.2k) | 110.8k (3.0k) | 142.7k (6.7k)
10 CPUs | 128 | 113.1k (1.5k) | 144.2k (1.9k) | 167.0k (3.3k)
| 512 | 114.2k (2.2k) | 129.8k (2.7k) | 139.1k (4.1k)
| Batch Size | 32 | 86.2k (2.3k) | 134.6k (3.9k) | 231.2k (7.2k)
80 CPUs | 128 | 177.8k (2.9k) | 372.9k (16.5k) | 492.4k (24.4k)
| 512 | 222.4k (7.9k) | 446.7k (22.5k) | 482.5k (10.4k)

###  G.6 SQL Usage

With such a large and diverse dataset, it is highly likely that many will wish
to train on subselections of the data according to certain criteria, contained
in the metadata. The TtyrecDataset enables this very easily, by exposing the
ability to subselect gameids on the basis of a SQL command. An example of this
is shown in Figure 6, and was used to generate the plots in Figure 6.

Figure 6: Example of how to use a SQL queries to subselect a part of a dataset
dynamically. This method was used to generate the NLD-AA-Monk runs only runs
in Table 2 and Figure 2

###  G.7 Creating Custom Datasets

It is easy to create and add your own custom dataset using NLE and NLD. NLE
includes the ability to record files to a given save directory. The directory
can then be directly added to NLD using the add_nledata_directory methods
referred to in Appendix G.4. An example is shown in Figure 7.

Figure 7: Example of how to create a custom dataset using NLE and NLD. A
variant of this method was used to create NLD-AA.

##  Appendix H NAO Details

Table 5: A table of the conducts and achievements accomplished as a percentage
of total episodes for each dataset. Most notably we can see only 1.46% of
episodes ascend in NLD-NAO, and 0.04% do this without any armour (Nudist). By
contrast NLD-AA achieves only the occasional solving of the Sokoban,
indicating NLD-NAO has a much better coverage of the late game.

Conducts |  NLD-AA (%) |  NLD-NAO (%)
---|---|---
Foodless | 0.89 | 21.16
Vegan | 1.35 | 29.50
Vegetarian | 1.41 | 32.18
Atheist | 2.36 | 52.13
Weaponless | 7.71 | 17.77
Pacifist | 0.26 | 8.53
Illiterate | 33.36 | 34.61
Polypileless | 100.00 | 96.88
Polyselfless | 87.65 | 93.59
Wishless | 98.91 | 95.88
Artifact wishless | 100.00 | 98.74
Genocideless | 100.00 | 96.90

Achievements |  NLD-AA (%) |  NLD-NAO (%)
---|---|---
Got the Bell of Opening | 0.00 | 2.15
Entered Gehennom | 0.00 | 2.06
Got the Candelabrum of Invocation | 0.00 | 1.73
Got the Book of the Dead | 0.00 | 1.65
Performed the Invocation | 0.00 | 1.62
Got the Amulet of Yendor | 0.00 | 1.60
Was in the End Game | 0.00 | 1.56
Was on the Astral Plane | 0.00 | 1.53
Ascended | 0.00 | 1.46
Got the Luckstone at Mines’ End | 0.00 | 3.95
Finished Sokoban | 0.17 | 5.93
Killed Medusa | 0.00 | 2.31
Zen conduct intact | 0.00 | 0.05
Nudist conduct intact | 0.00 | 0.04

###  H.1 Conducts and Achievements

By analysing the Conducts and Achievement flags in the conduct and achieve
metadata fields, one can see the variety of behaviours in the NLD-NAO dataset.
The aggregated % of episodes with each conduct/achievement is shown for each
dataset in Table 5

###  H.2 Metadata Matching Algorithm

On NAO, a game of NetHack can be saved and resumed, resulting in games split
across several ttyrec.bz2 files and time periods. In these situations, only
one entry is made to the communal xlogfile, and this is made at the very end
of the game with no link to which files are part of the game.

However, it is possible to assign files to games making use of three pieces of
information: first, that alt.org enforces that a user complete one game before
starting another; second, that the xlogfile records the starttime and endtime
of the game; and, third, that each file is saved with the same filename
template indicating the user and file creation time (
<username>/<file_creation_timestamp>.ttyrec.bz2).

Making use of these three pieces of information, we can assign the files to
the games, by lining up all episodes per user in an xlogfile by start and
endtime, and then assigning files created between that start and endtime to
that game. The full logic for this is carefully documented in the file
nle/datasets/populate_db.py.

In this way it is possible to remove games that have started but not finished
from the alt.org S3 bucket, whilst also filter out games that may have started
well before the first starttime in an xlogfile (indicating for whatever reason
the game was not logged). All this logic is vital in ensuring that NLD
episodes can be seamlessly stitched together correctly from many file parts.

###  H.3 Filtering ‘Bad’ Episodes for NLD-NAO

We filtered ‘bad’ episodes for NLD-NAO in three ways. First, we removed
episodes from the dataset where the player seems to have participated in
‘start-scumming’. This practice involves starting a game of NetHack and ending
it early if the randomly-generated starting attributes/inventory is not
favourable enough. We filtered these games by filtering episodes from the
dataset with fewer than 10 turns, where the death was ‘quit’ or ‘escaped’.

We also filtered an episode with negative in-game turns \- something which
should be impossible! This single player had managed to survive many orders of
magnitude longer than any other game, eventually successfully overflowing the
turns counter (a 32-bit integer). While surviving this long is impressive, it
is unlikely this episode demonstrate a drive to ‘ascend’ and may pose a large
bias to our dataset, and so is filtered out.

Finally we also decided to remove the files of players with highly offensive
names from our dataset. These accounted for very few episodes and frames (less
than 0.1%) and largely from poor players. To add an extra layer of privacy, we
pseudonymise all player names in the database, even though player names are
generally already pseudonyms.

###  H.4 Noise Investigation

The NLD-NAO dataset is a real-world dataset collected over a decade, and as
such contains some real world noise that can pose research problems.

The first problem is terminal rendering noise. Different terminals have
different escape sequences and different characters sets, and some of these
may not be interpretable by NLD’s lightweight VT100 emulator. While NLD has
made adjustments to support DEC
graphics222222https://en.wikipedia.org/wiki/DEC_Special_Graphics and IBM
graphics232323https://nethackwiki.com/wiki/IBMgraphics, it is still possible
for a user to define their own exotic graphics which could render noisily in
NLD, perhaps with extra, unusual symbols. This problem is also faced by
alt.org’s own webplayer, but thankfully, this issues is only encountered very
rarely.

Another source of rendering noise, aside from graphics, is terminal size.
Players can play NetHack with different terminal sizes, and NetHack’s clever
paging system will adapt the screen to the right size. In our case we chose by
default to render on the screen size used by NLE, which is also the smallest
"full size" you can play NetHack on (80 x 24). For episodes where the screen
size was larger than this, we make the decision to crop the screen rather than
wrap around the lines (as one typically would when emulating large commands on
a small screen). It is possible to control the emulator size scren by using
the rows and cols arguments to the constructor, if the user so wishes.

The second source of noise comes from errors introduced into the metadata
assignment process, outlined in Appendix H.2. In some cases, such as under
heavy traffic, NetHack can fail to write a line to an xlogfile, and drop the
line entirely. This introduces noise into the our assignment algorithm and
could result in a user’s games potentially including some ttyrec files from a
different one of their episodes. We check for discrepancies between metadata
and in-game play, by rendering the last 10 frames of all episodes in NLD-NAO
and checking whether the point and number of turns parsed from the screen
match the gameplay metadata in NLD-NAO. Since these two values are themselves
not always displayed or might change after game end, this itself is a noisy
process, yet we managed to obtain a upper bound on the errors. In the episodes
we found at most ~5% occurence of a discrepancy between the final frames of an
episode and the metadata for score and turns, and were able to verify correct
metadata for ~90% of episodes (~5% we were unable to classify).

##  Appendix I Experiment Hyperparameters and Details

###  I.1 Methods

All methods are implemented using the open-source RL library moolib [31]. The
online and online + offline RL methods are built on top of an APPO
implementation based on [37]. The offline RL methods are built on top of the
DQN algorithm. For CQL, we followed the implementation from [53], which is an
established library for offline RL algorithms. For IQL, we followed the
original implementation open-sourced by the authors [20]. We use the same base
architecture and processing of the observations and actions for all
algorithms. Please check out our open-sourced code for full details of these
implementations. All experiments were run with 20 CPUs in approximately 1 day
on single Volta 32GB GPUs.

###  I.2 Preprocessing

As a preprocessing step, a wrapper is used to render the observations
tty_chars and tty_colors as pixels, which are then downsampled and cropped
centered on tty_cursor. The resulting image is added as an observation known
as screen_image. This preprocessing step is standard for the Chaotic Dwarf
baseline model that we use. [15]

###  I.3 Model

We adapt the model from the standard Chaotic Dwarf Sample Factory baseline,
that was open sourced242424https://github.com/Miffyli/nle-sample-factory-
baseline during the NetHack Challenge. [15] This model relies on three small
encoders for observations, that combine their representations before passing
through an LSTM and taking policy and baseline heads from the output.

The original model relied on NLE’s blstats, message, and the preprocessed
screen_image (outlined in Appendix I.2) observations. Since the former two do
not exist in the raw terminal output, instead they are replaced with their
closest proxies - the top and bottom two lines of tty_chars respectively. This
involves slight changing to the normalization of the encoders. The full model
can be found in our open sourced code.

###  I.4 Hyperparameter Sweeps

For DQN, we performed a hyperparameter search over
$\epsilon_{start}\in[0.1,0.2,0.25,0.5,0.75,0.8,0.9]$,
$\epsilon_{decay}\in[2500,5000,10000,20000,25000,50000,75000,250000]$,
$target$ $update$ $\in[40,200,400,800,2000,4000]$ and tried both a MSE and a
Huber loss. We found the following to work best: $\epsilon_{start}=0.25$,
$\epsilon_{decay}=25000$, $target$ $update$ $=400$, and MSE loss. For the
common hyperparameters between CQL and IQL, we use the same values as the best
ones found for DQN without additional tuning. For CQL, we also performed a
hyperparameter search over $\tau\in[0.005,0.05,0.0005,0.1,0.01,0.001,0.0001]$
and for $cql$ $loss$ $coef$ $\in[1.0,0.5,0.1,2.0,10.0]$. We found $\tau=0.005$
and $cql$ $loss$ $coef$ $=2.0$ to work best. For IQL, we performed a
hyperparameter search over $expectile\in[0.8,0.7,0.9]$ and
$temperature\in[1.0,0.1,10.0,0.5,2.0]$. We found $expectile=0.8$ and
$temperature=1.0$ to work best. Table 6 contains a list with all relevant
hyperparameters used in our experiments (i.e. the best ones found following
our HP search).

Table 6: List of hyperparameters used to obtain the results in this paper. Hyperparameter | Value
---|---
activation function | relu
actor batch size | 512
adam betea1 | 0.9
adam beta2 | 0.999
adam eps | 0.0000001
adam learning rate | 0.0001
appo clip policy | 0.1
appo clip baseline | 1.0
baseline cost | 1
batch size | 256
crop dim | 18
grad norm clipping | 4
normalize advantages | True
normalize reward | False
num actor batches | 2
num actor cpus | 20
pixel size | 6
rms alpha | 0.99
rms epsilon | 0.000001
rms momentum | 0
reward clip | 10
reward scale | 1
unroll length | 32
use batchnorm | false
use lstm | true
virtual batch size | 128
rms reward norm | true
initialisation | orthogonal
use global advantage norm | norm
msg embedding dim | 32
msg hidden dim | 512
kickstarting loss | 1.0
ttyrec envpool size | 4
ttyrec batch size | 256
ttyrec unroll length | 32
ttyrec envpool size | 4
$\epsilon_{start}$ | 0.25
$\epsilon_{end}$ | 0.05
$\epsilon_{decay}$ | 25000
target update | 400
dqn loss | mse
$\tau$ | 0.005
cql loss coef | 2.0
expectile | 0.8
temperature | 1.0

###  I.5 Evaluation

We evaluate performance using the eval.py script in the experiment code. This
script creates a moolib.EnvPool and assigns each batch index a number of
episodes to run, to collect data. We collect 1024 episodes in this way, per
run.

It is important to note that this method of evaluating performance is slightly
different to the one generating curves in Figure 2. For training curves, we
add any episodes to a running mean as soon as they finish rolling out (i.e.
just before training). The end consequence of this is that short episodes with
low scores are sampled disproportionately more often that very long episodes
with high scores - in other words these curves are biased down. Accurately
sampling these very high scoring and very long rollouts account for the
discrepancy between the curves in Figure 2 and the values in Table 2.

##  Appendix J On the Scale and Performance of NLD

We compare the scale and performance of NLE and NLD to two other large scale
datasets of human demonstrations: MineRL [14] and StarCraft II [59]. In
particular, we present the size of the datasets, in terms of trajectories, and
benchmark the performance of the datasets and environments, measured in frames
per seconds (FPS). The overview can be found in Table 7.

Table 7: The mean frames per second (± standard deviation) across 5 runs for each environment or dataset. Although batch sizes and sequence length can affect performance, it is clear that NLD is considerably faster than rival datasets. Dataset | NLD | StarCraft II | MineRL
---|---|---|---
Trajectories | 100,000 - 1,500,000 | 971,000 | <5000
Environment FPS |  26400 ± 1200 | 200-700 | 68 ± 6.2
Dataset FPS |  113100 ± 1500 | 1800 ± 2.9 | 5000 ± 89

###  J.1 Overview

These results support the claim of NLD to be a large-scale dataset. NLD-NAO
has more trajectories than dataset used in AlphaStar[58], and possibly also
more transitions, since the average episode of NLE is longer than that of
StarCraft II. Even NLD-AA has an order of magnitude more trajectories than
MineRL.

These results also support the claims of NLD and NLE to be very performant in
terms of frames per second. The NLE environment is at least an order of
magnitude faster than either alternative environment, and NLD is potentially
several orders of magnitude faster at loading data. Further details in how the
batch size and unroll length affect the performance of NLD can be found in
Table 4.

###  J.2 FPS Benchmarking Method

To compute the throughput of NLD we used the threadpool with 80 threads on a
machine with 80 CPUs, a batch size of 128 and a sequence length of 32, taking
the mean and standard deviation from 5 runs to 500k frames. All assessments
are done with NLD-AA, the slower of the two datasets. For NLE we ran the
environment for 500k frames, with an agent sampling random actions and a max
episode length of 1000 frames.

For StarCraft II we include the throughput of the environment as reported in
the original paper, and measure the throughput of the dataset by running the
pysc2.bin.replay_actions function, provided in the github repo for batch
processing replays, for 5 minutes with 80 threads on 80 cpus. This too was
repeated 5 times.

For MineRL we used the MineRLTreechop-v0 environment and dataset. We measured
the environment speed by running headless on a machine with 80 cpus for 10k
steps, sampling random actions and assuming a max episode length of 30k steps
(note, the MineRL documentation suggests that running headless slows down the
environment by a factor of 2-3x because the rendering is performed on the CPU
instead of the GPU). We measured the MineRL dataset throughput using the
provided BufferedBatchIter with a batch size of 4096 for a full epoch. Once
again, we repeated this 5 times.

Table 7 presents the mean and standard deviation of the frames per second.

##  Appendix K Limitations of our Work

One potential limitation of our work is our human dataset NLD-NAO doesn’t
contain any actions or rewards, so it cannot directly be used for imitation
learning of offline RL. However, this is the case for many other domains of
interests where you may have access to sequences of observations (e.g. when
learning skills from human or agent videos in domains like robotics or
autonomous driving). Thus, we believe our human dataset can enable research on
this more realistic and important setting in a safe environment.

While NLD-AA contains both actions and rewards thus allowing research on
imitation learning and offline RL, it is less diverse than NLD-NAO as it
contains demonstrations from only one symbolic bot, the winner of the NetHack
Challenge at NeurIPS 2022 [15]. In the future, we hope the community will
contribute many more datasets to NLD collected using a variety of bots
(including learning-based ones). We believe that the current dataset is a good
starting point to develop even better agents at playing NetHack. Then, those
agents could be used to collect additional trajectories which can be added to
NLD, resulting in a feedback loop that constantly grows the size and diversity
of the dataset and develops better and better agents.

Another potential limitation of our work is that we use the in-game score to
train our RL agents, which is the standard practice from [23] and [15]. While
the in-game score roughly correlates with good performance on NetHack, it is
not exactly the same as ascending i.e. winning the game, which is ultimately
how we want to evaluate agents. For example, humans sometimes attempt to
ascend with as low a score as possible, as an additional challenge. This
misalignment is common in other real-world domains where it can be difficult
to design a good reward function that illicits the desired behavior (e.g.
autonomous driving). Thus, this is an open research challenge which could be
enabled by our dataset, but lies outside the scope of the current work.

##  Appendix L Broader Impact Statement

Our work introduces a new large-scale dataset based on a computer game to
enable research in multiple domains including learning from demonstrations,
imitation learning, and reinforcement learning. Our dataset is cheap to run in
order to democratize research on large-scale datasets of demonstrations, which
have historically been restricted to well-resourced (industry) labs. This
dataset contains human demonstrations, but these are already available online
and the demonstrations are de-identified such that only a user’s chosen
usernames can be retrieved. Thus, we don’t envision any direct negative social
impact of this work. Of course, people could use our dataset to develop new
methods for learning from demonstrations. Since these techniques can be quite
general, they could also be used for other real-world applications such as
autonomous driving, robotics, healthcare, financial services, or
recommendation systems (where large-scale datasets may be available). Of
course, these domains are more sensitive than computer games, so much more
care is needed to mitigate the risks and develop safe, robust systems. While
we believe this lies outside the scope of current work, researchers and
practitioners using our dataset should always keep an eye on potential
negative societal impacts.