# Flatland-RL : Multi-Agent Reinforcement Learning on Trains

Sharada Mohanty*
<EMAIL_ADDRESS>
AIcrowd Erik Nygren
<EMAIL_ADDRESS>
SBB CFF FFS These authors have equal contribution Florian Laurent
<EMAIL_ADDRESS>
AIcrowd Manuel Schneider
<EMAIL_ADDRESS>
ETH ZUrich Christian Scheller
<EMAIL_ADDRESS>
AIcrowd, FHNW Nilabha Bhattacharya
<EMAIL_ADDRESS>
AIcrowd Jeremy Watson
<EMAIL_ADDRESS>
AIcrowd Adrian Egli
<EMAIL_ADDRESS>
SBB CFF FFS Christian Eichenberger
<EMAIL_ADDRESS>
SBB CFF FFS Christian Baumberger
<EMAIL_ADDRESS>
SBB CFF FFS Gereon Vienken
<EMAIL_ADDRESS>
Deutsche Bahn Irene Sturm
<EMAIL_ADDRESS>
Deutsche Bahn Guillaume Sartoretti
<EMAIL_ADDRESS>
Nanyang Technological University, Singapore Giacomo Spigler
<EMAIL_ADDRESS>
Tilburg University

###### Abstract

Efficient automated scheduling of trains remains a major challenge for modern
railway systems. The underlying vehicle rescheduling problem (VRSP) has been a
major focus of Operations Research (OR) since decades. Traditional approaches
use complex simulators to study VRSP, where experimenting with a broad range
of novel ideas is time consuming and has a huge computational overhead. In
this paper, we introduce a two-dimensional simplified grid environment called
“Flatland” that allows for faster experimentation. Flatland does not only
reduce the complexity of the full physical simulation, but also provides an
easy-to-use interface to test novel approaches for the VRSP, such as
Reinforcement Learning (RL) and Imitation Learning (IL). In order to probe the
potential of Machine Learning (ML) research on Flatland, we (1) ran a first
series of RL and IL experiments and (2) design and executed a public Benchmark
at NeurIPS 2020 to engage a large community of researchers to work on this
problem. Our own experimental results, on the one hand, demonstrate that ML
has potential in solving the VRSP on Flatland. On the other hand, we identify
key topics that need further research. Overall, the Flatland environment has
proven to be a robust and valuable framework to investigate the VRSP for
railway networks. Our experiments provide a good starting point for further
research and for the participants of the NeurIPS 2020 Flatland Benchmark. All
of these efforts together have the potential to have a substantial impact on
shaping the mobility of the future.

### Keywords

multi-agent reinforcement learning, operations research, vehicle re-scheduling
problem, automated traffic management system, deep reinforcement learning

##  1 Introduction

The Swiss Federal Railway Company (SBB) operates the densest mixed railway
traffic network in the world. On a typical day of operations, more than 10,000
train runs are executed on a network of more than 13,000 switches and 32,000
signals. Almost 1.2 million passengers and 50% of all goods within Switzerland
are transported on the railway network each day. The demand for transportation
is forecasted to grow further in the next years, requiring SBB to increase the
current transportation capacity of the network by approximately 30%.

This increase in transport capacity can be achieved through different measures
such as denser train schedules, large infrastructure investments, and/or
investments in new rolling stock [1]. However, SBB currently lacks suitable
technologies and tools to quantitatively assess these different measures.

A promising solution to this dilemma is a complete railway simulation that
efficiently evaluates the consequences of infrastructure changes or schedule
adaptations for network stability and traffic flow. A complete railway
simulation consists of a full dynamical physics simulation as well as an
automated traffic management system (TMS).

The research group at SBB has developed a high performance simulator which
represents and simulates the railway infrastructure and the dynamics of train
traffic. The simulation is the basis for investigating the possibility of an
automated traffic management system (TMS) [2] that manages all traffic on the
network by selecting train routes and deciding on the train orders at switches
in order to optimize the flow of traffic through the network.

In essence, the TMS’s needs to (computationally) solve the so-called vehicle
scheduling problem (VSP). The VSP has been a main research topic for
operations research (OR) for many decades [3, 4] and was already described in
detail in the early 1980s [5]. In 2007, Li, Mirchandani and Borenstein
proposed the more general vehicle re-scheduling problem (VRSP) [6]:

The vehicle rescheduling problem (VRSP) arises when a previously assigned trip
is disrupted. A traffic accident, a medical emergency, or a breakdown of a
vehicle are examples of possible disruptions that demand the rescheduling of
vehicle trips. The VRSP can be approached as a dynamic version of the
classical vehicle scheduling problem (VSP) where assignments are generated
dynamically.

However, solving the VRSP while taking into account all aspects of the real
world that are represented in SBB’s high performance physical simulation,
means dealing with a NP-complete problem in an extremely complex setting: fast
iteration cycles for experimenting with new ideas are impossible. In order to
enable a faster pace of innovation, we propose a simpler 2D grid environment
called “Flatland” which allows for fast experimentation with ideas and,
eventually, applying the results back to the high performance physical
simulation. Flatland aims to address the VRSP by providing a simplistic grid
world environment. Flatland’s architecture is designed to facilitate exploring
novel approaches from Machine Learning Researchers, in particular from
Reinforcement Learning (RL) communities where the VRSP has become of interest
in recent years [7, 8, 9].

Flatland represent railway networks as 2D grid environments with restricted
transitions between neighbouring cells. On the 2D grid multiple train runs
have to be performed while minimizing the global delay on the network. In
Reinforcement Learning terms the problem can be described as multiple agents
(trains) with different objectives (schedules) who need to collaborate in
order to maximize a global long-term reward.

In 2019, we ran the first version of the Flatland Benchmark
111https://www.aicrowd.com/challenges/flatland-challenge/ a collaboration
between SBB and AIcrowd, in order to test the Flatland environment and invite
the AIcrowd community to tackle VRSP for the specific case of a railway
system. The second Flatland Benchmark was organized as a part of NeurIPS 2020.
The different tasks that were expected to be solved are explained in Section
3.2.

The key goal of Flatland is to foster innovation with respect to classical
methods of Operations Research (OR), Reinforcement Learning (RL) and,
importantly, combinations of OR and RL that leverage the strength of both
fields, in order to provide novel solutions for the VRSP on railway networks.

In section 2, we summarize the concept of the Flatland environment and the
implementation details. In section 3 we describe design of the Flatland
Benchmark and our own exploratory experiments are described in 4. We then
report the results of our experiments in section 5 and discuss the results and
the potential of the Flatland environment in the final section.

##  2 The Flatland Grid World

Figure 1:  Visualization of a simple Flatland environment with one train
station and one train.

The Flatland library is a comprehensive framework that allows to easily run
VRSP experiments for railway networks. Figure 1 shows the visualisation of a
simple Flatland environment. In the following, we describe the concepts and
implementation of Flatland in detail.

###  2.1 Environment

Flatland is a 2D grid environment of arbitrary size, where the most primitive
unit is a cell. A cell is a location in the grid environment represented by
two integer coordinates $x$ and $y$, where $x\in[0,w-1]$ and $y\in[0,h-1]$,
with $w$ beeing the width and $h$ the height of the grid world. Each cell has
the capacity to hold a single agent.222In principle, the implementation of the
environment allows cells to hold more than one agent, but for the sake of
simplicity/realism, we assume the capacity of a single cell to be 1 in the
context of the benchmark and the application to a railway system.

An agent is an entity which is located at a particular cell, and has a
direction value $d\in[0,3]$ representing its current orientation. The
direction values represent the 4 cardinal directions (North, East, South,
West). An agent can move to a subset of adjacent cells. The subset of adjacent
cells that an agent is allowed to transition to is defined by a transition
map.

Flatland is a discrete time simulation, i.e. it performs all actions with
constant time step. A single simulation step synchronously moves the time
forward by a constant increment, thus enacting exactly one action per agent.

###  2.2 Transition Maps and Emergent Patterns

Each cell of the simulation grid has a transition map which defines the
movement possibilities of the agent out of the cell.

A cell’s transition map is stored as a 4-bit bitmask that represents possible
movements to the four adjacent cells given the agent’s current direction.
Since the movement of agents in Flatland is directional (eg. a train cannot
move backwards on the track), each cell has four such 4-bit transition maps
(e.g., the transition map of an east-facing agent is different than that of a
south-facing agent). This setup allows to store the information about the
constrained transitions using 4x4 bits of information per cell. At the same
time it allows to efficiently access this data in $O(1)$ time at a minimal
memory expense ($O(w\cdot h)$). For example, to store the transition map for a
grid size of $1000x1000$ takes only 1.9 MB of space for the whole grid.
Section 2.2 describes in detail which movement patterns emerge from the
transition maps currently used in Flatland.

Figure 2: Each cell can have transitions to its $4$ neighbouring cells located
North, East, South or West.

For this railway specific problem, $8$ basic types of transition maps (as
outlined in Figure 3) are defined to describe a real world railway network.
All possible transitions maps are given by rotating the basic transitions maps
by 0, 1, 2 or 3 directions. As in real railway networks there are never more
than two transition options present in a given situation. This means that in
Flatland a train needs to take at most a binary decision. Changes of agent
orientation can happen for particular transitions and only if these are
allowed.

Figure 3: The railway configuration of Flatland consists of eight basic
transition maps. Each basic transition can occur in all four rotations. A
transition from one cell to another is allowed if there is a line connecting
one cell to the other. Curves are a special type of case 1 transition maps
where the transition includes a 90 degrees rotation.

Case 0:

represents an empty cell, thus no agent can occupy the tile at any time.

Case 1:

represent a passage through the cell. While on the cell the agent can make no
navigation decision. The agent can only decide to either continue, i.e.,
passing on to the next connected tile or wait.

Case 2:

represents a simple switch. An train incoming from South in this example has
to take a navigation decision (either West or North). In general, the straight
transition (S $\,\to\,$ N in the example) is less costly than the bent
transition. Therefore, these decisions could be rewarded differently in future
implementations. Case 6 is identical to case 2 from a topological point of
view but has symmetrical transition costs (both choices involve a turn).

Case 3:

can be seen as a superposition of two times Case 1. No turn can be taken when
coming from any direction.

Case 4:

represents a single-slit switch. In the example, a navigation choice is
possible when coming from West or South.

Case 5:

a navigation choice must be taken when coming from any direction.

Case 6:

represents a symmetrical switch with the same transition costs for both
choices. At this switch, the agent must either turn left or right if coming
from the South in this example. Coming from East or West, this cell acts like
a simple turn.

Case 7:

represents a dead-end. Therefore, only stopping or changing direction
(implemented by “continuing forward”, see Section 2.3) is possible when an
agent enters this cell from the South.

For consistency, all outgoing connections of a cell must be joined to incoming
connections of a neighboring cell. We use this consistency rule to generate
random valid railway network configurations (Figure 4) which are used for
training. We use a hold-out set of network configurations for validation and
evaluation for our experiments and for the benchmark.

Figure 4: a) Cells can only be connected if their outgoing transitions match
the incoming transitions of their neighbours. Following this rule any real
world railway network can be implemented within the Flatland environment. The
transition maps are direction dependent and thus the possible paths for an
agent on Flatland depend not only on its position but also its current
direction as shown in sub figures b) and c).

Agents can only travel in the direction they are currently facing. Thus, the
permitted transitions for any given agent depend both on its position and on
its direction (see Figure 4).

###  2.3 Action Space

The action space is $discrete(5)$ for this railway specific implementation of
Flatland, and consists of:

  * •

Go forward (or turn to opposite direction and continue forward if the agent is
at a dead end; see Case 7 before),

  * •

Select a left turn,

  * •

Select a right turn,

  * •

Halt on current cell, always valid, and

  * •

No-op, always valid.

This definition of the action space differs from typical environments used in
the RL community where agents move unconstrained and are able to change
direction by rotation of the agent.

The ”Go forward” action can only be executed if the transition map of the
current cell allows the agent to move to the adjacent cell in its current
direction. Similarly, selecting a left/right turn can only be executed when a
switch is present (in the transition map of the current cell) with an allowed
transition either to the left or to the right (see Figure 2; Cases 2,4,5 and
6). In cases where the turn is chosen, the direction of the agent is changed
simultaneously with the move to the next cell. Finally, the ”no-op action”
lets the agent continue what it was doing previously: if the agent was already
moving, then the no-op is equivalent to the “go forward” action, while a
halted agent will remain stopped.

In a multi-agent setting, any action is executed if and only if it is allowed
by the transition map of the current cell and if the subsequent cell will not
be occupied by another agent at the next time step. If an action cannot be
executed, the environment executes instead a no-op action.

###  2.4 Observations

This benchmark differs from many other reinforcement learning benchmarks
because the objective for participants is two fold. On one hand, participants
need to produce an efficient, intelligent agent which solves the VRSP tasks
(Section 3.2). On the other hand, we encourage participants to design their
own observation spaces in order to improve the performance of their agent.
Creating this secondary focus on observation design and presenting an
environment that provides maximum information for this task sets the Flatland
benchmark apart from other similar competitions.

As an inspiration and baseline, we provide two different implementations of
the observation space as well as access to the full information of the
internal state of the environment in order to allow participants to propose
and test their own representations. Furthermore, we provide wrapper functions
that can be used to generate novel observation space representations by the
participants.

####  2.4.1 Global Observation

The global observation is conceptually simple: every agent is provided with a
global view of the full Flatland environment. This can be compared to the
full, raw-pixel data used in many Atari games. The size of the observation
space is $h\times w\times c$, where $h$ is the height of the environment, $w$
is the width of the environment and $c$ s the number of channels of the
environment. These channels can be modified by the participants. In the
initial configuration we include the following $c=5$ channels:

  1. 1.

Channel 0: one-hot representation of the self agent position and direction.

  2. 2.

Channel 1: other agents’ positions and directions.

  3. 3.

Channel 2: self and other agents’ malfunctions.

  4. 4.

Channel 3: self and other agents’ fractional speeds.

  5. 5.

Channel 4: number of other agents ready to depart from that position.

####  2.4.2 Tree Observation

The tree observation is built by exploiting the graph structure of the railway
network. The observation is generated by spanning a four-branched tree from
the current position of the agent. Each branch follows the allowed transitions
(backward branch only allowed at dead-ends) until a cell with multiple
transitions is reached, e.g. a switch. The information gathered along the
branch until this decision point is stored as a node in the tree. In more
detail, the tree observation is constructed as follows (see also Figure 5):

From the agent’s location,

  1. 1.

probe all 4 directions starting with left and start a branch for every allowed
transition.

  2. 2.

Walk along the track of the branch until a dead-end, a switch or the target
destination is reached,

  3. 3.

Create a node and fill in the node information as stated below.

  4. 4.

If max depth of tree is not reached and there are possible transitions, start
new branches and repeat the steps 1 to 4.

  5. 5.

Fill up all non existing branches with -infinity such that tree size is
invariant to the number of possible transitions at branching points.

Note that we always start with the left branch according to the agent
orientation. Thus, the tree observation is independent of the orientation of
cells, and only considers the transitions relative to the agent’s orientation.

Each node is filled with information gathered along the path to the node.
Currently each node contains 9 features:

  1. 1.

if own target lies on the explored branch, the current distance from the agent
in number of cells is stored.

  2. 2.

if another agent’s target is detected, the distance in number of cells from
the current agent position is stored.

  3. 3.

if another agent is detected, the distance in number of cells from the current
agent position is stored.

  4. 4.

conflict detected on this branch

  5. 5.

if an unusable switch (for the agent) is detected we store the distance. An
unusable switch is a switch where the agent does not have any choice of path,
but other agents coming from different directions might.

  6. 6.

This feature stores the distance (in number of cells) to the next node (e.g.
switch or target or dead-end)

  7. 7.

minimum remaining travel distance from this node to the agent’s target given
the direction of the agent if this path is chosen

  8. 8.

agent in the same direction found on path to node

     * •

n = number of agents present in the same direction (possible future use:
number of other agents in the same direction in this branch)

     * •

0 = no agent present in the same direction

  9. 9.

agent in the opposite direction on path to node

     * •

n = number of agents present in the opposite direction to the observing agent

     * •

0 = no agent present in other direction to the observing agent

Figure 5:  Building a tree observation of depth 3. The branches are built in
the order left, forward, right, and backward of the agent’s current
orientation. Only valid transitions result in a branch whereas backward is
only allowed in a dead end. The right side of the figure shows the tree
corresponding to the railway network on the left, where the triangle depicts
the agent and its orientation. In the tree, a circle indicates a node
populated with the corresponding information.

Figure 5 illustrates the tree observation on a simple network where the
switches are highlighted and numbered in order to identify them with their
corresponding nodes in the tree.

###  2.5 Reward

Each agent receives a combined reward consisting of a local and a global
reward signal. For every time step $t$, an agent $i$ receives a local reward
$r_{l}^{i}=-1$ if it is moving or stopped along the way and $r_{l}^{i}=0$ if
it has reached its target location. In addition, a configurable penalty
$r_{p}^{i}$ is received if the move is illegal, i.e. the action cannot be
executed. This penalty is set to $0$ in the NeurIPS 2020 benchmark. If all
agents have reached their targets at time step $t$, a global reward $r_{g}=1$
is awarded to every agent. If not all the agents is not at its destination,
all agents get a global reward $r_{g}^{t}=0$. Overall, every agent $i$
receives the reward

$r_{i}(t)=\alpha\,r_{l}^{i}(t)+\beta\,r_{g}(t)+r_{p}^{i}(t)\hskip
28.45274pt\in{[-\alpha-2,\beta]},$ (1)

where $\alpha$ and $\beta$ are parameters for tuning collaborative behavior.
Therefore, the reward creates an objective of finishing the episode as quickly
as possible in a collaborative way. This structure is similar to that of
existing multi-agent path finding RL casts [10].

At the end of each episode, the return of each agent $i$ is given by the sum
over the time step rewards:

$g_{i}=\sum_{t=1}^{T}r_{i}(t),$ (2)

where $T$ is the time step when the episode terminated.

##  3 The Flatland Benchmark

The Flatland library is only a means to solve the VRSP at hand. A public
benchmark offers a great opportunity to collectively tackle this difficult
problem. Some specifics of NeurIPS 2020 Flatland Benchmark are described
below, please visit the dedicated
website333https://www.aicrowd.com/challenges/neurips-2020-flatland-challenge/
for more information.

###  3.1 Related Benchmarks

There have been other multi-agent reinforcement learning benchmarks where the
objective was to learn either collaboration or competition (or both) among
agents, e.g., Pommerman [11], Marlö [12], SC2LE [13]. The key aspects in which
the Flatland benchmark differs from the previous benchmarks are:

  * •

In Flatland, the tasks have to be solved by a large group of agents ($n>100$),
which means that collaboration becomes a much more complex problem. These
coordination tasks cannot be solved by relying just on local information (e.g.
limited view of agents). At the same time, scalability constraints limit the
use of centralized approaches, thus introducing the need for subtle new
observation spaces and training methods.

  * •

The allowed transitions in Flatland are much more constrained than free
movement on a grid. Participants are therefore forced to explore novel
approaches for representing the observation spaces in contrast to pixel or
grid view observations used in other environments. Furthermore, the
distribution and importance of actions in Flatland’s action-space is highly
imbalanced. In many situations only a single action is possible (e.g., go-
forward). Much less frequently, there is more than one option and the
decisions are highly critical to the task’s success (e.g., decisions to turn
at junctions). This imbalance results in more difficult exploration at
training time, and will likely require the need for more complex training
methods to allow agents to efficiently explore the state-action space, as well
as to identify and reason about key decisions along their way.

  * •

Flatland directly addresses a real world problem. Solutions submitted to the
benchmark have a practical impact on issues such as real-time railway re-
scheduling. The VRSP belongs to a wider class of collaborative, scalable
multi-agent cooperation tasks, which includes the job shop scheduling problem
[14], the aircraft landing problem [15], and many more problems connected to
logistics and transportation. Therefore, any promising solution found in
context of this benchmark, will be applicable in a broad range of industries,
where improved autonomous collaboration could have a large impact on society.

  * •

Along the same lines, the existing Flatland (open-source) code base provides
an extensible framework to define novel tasks beyond railway scheduling
problems. An adaptation of the Flatland could lead to easy and accessible
tools for studying other classes of related optimization problems, e.g.
optimization of ground traffic at airports in challenges and applications.

Typically, (Multi Agent) RL benchmarks are tailored to match specific research
questions and therefore present a fixed design of observation space and reward
functions. In contrast, Flatland exposes the full internal state of the
environment, presenting participants with larger freedom. This enables them to
apply their creativity also to the design of observations, an aspect that has
been found to be essential for the success of RL solutions and that is worth
to be incorporated into the core library.

Flatland also builds upon features of many successful Multi-Agent
Reinforcement Learning Environments. Participants can interact with the
environment using a familiar gym-flavoured API inspired by numerous other
Multi Agent RL environments.

###  3.2 Task and rounds

The NeurIPS 2020 Flatland benchmark introduces the VRSP to the participants in
two main rounds. In both rounds, the agents have to navigate from a starting
cell to a target cell at a constant speed. In addition, agents can be
inflicted by malfunction on their run. The occurrence of malfunctions is
defined in the environment by a malfunction probability and a malfunction
duration interval.

Each round introduces a separate set of test environments and a different set
of constraints.

The goal of the first round was to maximise the accumulated reward for solving
a set of 14 tests, each including several evaluation environments.
Prerequisite for a valid submission was that all evaluation environments had
to be solved within 8 hours, otherwise the submission failed and no score was
awarded.

In the second round, the goal was not only to maximize the score for the test
environments, but also to minimize computation time. This goal directly
relates to the business requirements of real world railway operations that
require short response times in case of malfunction/disruptions. Submissions
to round two had to solve a open-ended sequence of test environments, starting
with small grid sizes and small number of trains with progressively increasing
grid sizes and number of trains. The goal was to solve as many of these
environments as possible in 8 hours. The evaluation was stopped pre-maturely
if less than 25% of the agents reached their target, averaged over each test
of 10 episodes.

###  3.3 Custom Observations

Our own experiments showed that the design of the observation space is a
crucial element for all optimal solutions. Therefore, participants were given
the flexibility to design their own observations, either by modifying and
combining Flatland’s “stock” observations or using the full Flatland state
information and predefined wrapper functions.

###  3.4 Metrics

For each task, the agents submitted by the participants were evaluated on a
fixed set of environment configurations ordered from low to high complexity.
The solutions were evaluated using the normalized reward per agent across all
environments, which is calculated as follows :

$S=\sum_{j=0}^{N}\sum_{i=0}^{M_{j}}g_{i}^{j}.$ (3)

where the score ($S$) is the accumulated total reward (Section 2.5) for $N$
completed environment configurations with $M_{i}$ agents for the respective
configuration. The participants’ solutions were ranked by the score $S$ where
a higher score is better.

###  3.5 Tutorial and documentation

The Flatland project website444http://flatland.aicrowd.com features a
documentation for the Flatland environment with a step by step introduction to
the core components and API references, information on the ongoing and past
benchmarks, information on research approaches, and Frequently Asked Questions
(FAQ). The code for the environment, provided under an open-source license, is
available for the participants in the public Flatland
repository.555https://gitlab.aicrowd.com/flatland/flatland

Participants can make their first submission by following the instructions in
the benchmark starter
kit.666https://gitlab.aicrowd.com/flatland/neurips2020-flatland-starter-kit.
Beyond the starter-kit, we also provide baseline implementations which can act
as a good starting point for participants.

##  4 Experiments on Flatland

Flatland provides the basis to use RL to tackle VRSP problems. As a first step
towards demonstrating the potential of RL for this problem, we conducted a
series of experiments on Flatland. We evaluated a number of well-known RL
algorithms in standard and customized versions and applied additional
alterations to problem formulation and observation. Our results provide first
insights into the performance and the key challenges of RL for VRSP.

The experiments are built upon the RLlib framework [16], which provides a
diverse set of state-of-the-art reinforcement learning algorithms such as PPO
[17], IMPALA [18] and Ape-X [19]. We include results from:

  * •

Random agents, shortest-path agents and solutions based on OR alternatives as
baselines

  * •

Standard and customized RL algorithms:Standard PPO and Ape-X, PPO with
centralized critic (CCPPO) with custom observations and/or variants of action
handling

  * •

Imitation learning approaches

With Ape-X using a DQfD loss, we evaluate how expert demonstrations from OR
solutions can be used for imitation learning. Compared to other imitation
learning algorithms such as DAGGER [20] and Deeply AggreVaTe [21], a key
advantage of DQfD [22] is the ability to improve beyond the demonstrations
which is essential to master large Flatland instances in which OR approaches
do not scale.

###  4.1 Training, Evaluation and Test Protocols

For all our experiments, we use the sparse rail
generator777https://flatland.aicrowd.com/getting-
started/env/level_generation.html#sparse-rail-generator from the Flatland
library to generate the environments. All environments had a size of $25\times
25$ cells and 5 agents. The detailed configurations for the generator and the
experiments can all be found in the baseline
repository.888https://gitlab.aicrowd.com/flatland/neurips2020-flatland-
baselines/blob/flatland-paper-
baselines/envs/flatland/generator_configs/small_v0.yaml999
https://gitlab.aicrowd.com/flatland/neurips2020-flatland-
baselines/tree/flatland-paper-baselines/baselines

Training for pure RL algorithms like Ape-X and PPO were done for 15 million
steps. For Imitation Learning (IL) algorithms, which were run with saved
expert experiences, more training steps were run. MARWIL was run for 5 billion
steps whereas Ape-X FIXED IL(25%) was run for 1 billion steps. Ape-X FIXED
IL(100%) was run for 100 million steps. The number of steps for these IL
experiments are different as we saw significant improvement after 15 million
steps. Moreover, the expert experiences were saved separately for each agent
and each agent’s experience is considered as a step. Hence, for an environment
with 5 agents, one episode step in a typical MARL setting would be equivalent
to 5 steps in the MARWIL setting. For pure IL, we only report evaluation
results. Expert samples for IL were taken from the OR solution of 2019
Flatland Benchmark, by Roman
Chernenko101010https://bitbucket.org/roman_chernenko/flatland-final which
showed a 100% completion rate on the tested environments.

Training was done with 3 different environment seeds. For evaluation, we ran
50 evaluation episodes using held out random seeds for every 50 training
iterations. We took the checkpoint corresponding to the highest normalised
reward on the evaluation set starting at 1 million steps to avoid good results
due to lucky initialisation and easier environments in the beginning.

We then ran the model with this checkpoint on a separate set of 50 episodes
and report this as the test set. The test set is run against 3 different
training checkpoints for each algorithm. For Ape-X based runs, we sampled a
random action from the action space with probability 0.02 to allow for some
exploration in the experiment. The test result is shown as the mean and
standard deviation of these 3 runs corresponding to the 3 different training
seeds in section 5.

###  4.2 Baselines

To establish some conventional baselines for our experiments, we used:

  * •

Stochastic random agents (Random)

  * •

Agents that always choose the “go forward” action (Constant Go Forward Action)

  * •

Agents that use the shortest Path (Shortest Path)

###  4.3 Standard and customized RL algorithms

Our RL experiments were performed using

  * •

Standard Ape-X and PPO with the local tree search observation.

  * •

PPO with centralized critic (CCPPO) in a base variant and using a transformer
(details see below)

  * •

PPO and/or Ape-X with frame skipping, action masking or a global density
observation (details see below)

####  4.3.1 Centralized Critic

The Flatland environment reflects one challenge of real-world railway planning
very strongly: trains need to cooperate in sharing sections of railway
infrastructure. Therefore Flatland presents a challenge to learning algorithms
with respect to the agent’s behavior when faced with the presence of other
agents. Centralized critic methods address the problem of cooperation in
multi-agent training situations with partial observability. They can be
regarded as a middle ground between individual train agents that regard other
trains as part of the (thus non-stationary) environment, and a single agent
for all trains whose action space is the joint action space of all train
agents, a solution which has serious drawbacks when it comes to scalability.

This centralized critic approach was first implemented and evaluated in a
project of Deutsche Bahn in cooperation with InstaDeep. In the context of that
project, CCPPO was validated and directly compared to standard PPO on a
previous version of Flatland. That original implementation that used a custom
version of a sparse reward and a custom observation similar to Flatland’s tree
observation outperformed standard PPO which motivated the current experiments
with CCPPO.

The version implemented here follows the actor-attention critic approach [23]
and uses a centrally computed critic that learns based on the observation of
all agents, but that trains decentralized policies. In addition to a basic
version where the input of the centralized critic is just a combination of all
agents’ observations, we also provide a Transformer as proposed by Parisotto
et al. [24] that serves as an attention mechanism that learns to reduce the
concatenated observation to the relevant information.

The base architecture implemented here is a a fully connected network trained
using PPO. At execution time the agents will step through the environment in
the usual way. During training, however, a different network is used that
provides the evaluation of the current state on which to train the agent’s
network. This “central critic” is provided with a more holistic view of the
simulation state in order to better approximate an adequate reward for the
agent’s actions in the current situation. Both network consist of three hidden
layers of size 512.

In the basic version, the input to the central critic is a concatenation of
all trains’ observations in a consistent order. This leads to an observation
that contains no information about the position of the trains in the network,
nor about their relative positions. However, it provides a good starting point
for experimenting with the input for the critic.

In the version using the transformer, an additional processing layer is added
between the concatenated observations of the agents and the centralized
critic, in order to provide the critic with an embedding of the observations.
The transformer is built with self-attention layers that calculate the
embedding with trained linear projections. The dimensions of these outputs can
be decided: their number is the product of the number of parallel heads and of
the output dimension. The transformer in the current implementation represents
an attention mechanism that learns the best representation of the concatenated
observations.

For the experiments described in this paper, we the used the local tree search
observation and standard reward.

####  4.3.2 Frame Skipping and Action Masking

These modifications were introduced in order to ease the learning for RL
agents:

##### Skipping “no-choice” cells:

Between intersections, agents can only move forward or stop. Since trains stop
automatically when blocked by another train until the path is free, manual
stopping only makes sense right before intersection, where agents might need
to let other trains pass first in order to prevent deadlocks. To capitalize on
this, we skip all cells, where the agent is not on or next to an intersection
cell. This should help agents in the sense that they no longer need to learn
to keep going forward between intersections. Also skipping steps shortens the
perceived episode length.

##### Action masking

The Flatland action space consists of 5 discrete actions (no-op, forward,
stop, left, right). However, at any given time step, only a subset of all
actions is available. For example, at a switch with a left turn and a straight
continuation, the agent cannot turn right. Furthermore, the noop-action is
never required and can be removed completely. We test masking out all invalid
and noop actions so that the agents can focus on relevant actions only.

We evaluated skipping “no-choice” cells for both DQN Ape-X and PPO agents.
Action masking was only tested with PPO agents. All experiments were performed
with the stock tree observations. For policy and value functions, we employed
a small two layer neural network (each with 256 hidden units and ReLU
activation).

####  4.3.3 Global Density Map

Since in preliminary experiments the standard global observation didn’t
perform well, we experimented with alternative global observations.

The global density map observation is based on the idea that every cell of the
environment has a value representing an “occupation density”. For simplicity,
we assume that an agent follows the shortest path to its target and doesn’t
consider alternative paths. For every cell of an agent’s path the distance in
steps of the current position of the agent to this cell is calculated. These
values are aggregated for all agents into a estimated occupation density
measure. This measure encodes how many agents will likely occupy that cell in
the future. The contribution of an agent to the density value is weighted by
the agent’s distance to the cell.

For example, if all $M$ agents’ paths pass the same cell at the same time step
$t_{i}$, the density is high. If the agents pass the same cell at different
time steps $\\{t_{j}^{m}|0<m\leq M,j>i\\}$, the density is lower.

The occupation density value for a cell and agent is defined as:

$d(t)=\exp(-\frac{t}{\sqrt{t_{\textrm{max}}}})$

where $t$ is the number of time steps for the agent to reach the cell and
$t_{\textrm{max}}$ the maximum number of time steps included in the density
map. The density map present the agents the opportunity to learn from the
(projected) cell occupancy distribution.

The observation for each agent consists of two arrays representing the cells
of the environment. The first array contains the density values for the agent
itself including the agent’s own position, and the second array contains the
mean of the other agents’ values for each cell.

###  4.4 Imitation Learning

The Flatland environment presents two major challenges to learning algorithms.
On the one hand, agents not only have to learn to efficiently navigate to
their target, but also to deal with path conflicts with other agents. On the
other hand, rewards are sparse and delayed which renders their assignment to
the relevant actions complex. OR solutions which follow rule-based and
planning approaches have been shown to perform well in this environment. We
used imitation learning to take advantage of the OR capabilities while
potentially generalizing to larger instances than those considered in the
demonstrations.

We considered two different, broad Imitation Learning approaches:

####  4.4.1 Pure Imitation Learning

This approach involves training in a purely offline process from stored expert
experiences or training online by learning to mimic actions generated by an
expert.

For the offline process using stored experiences this is implemented using the
RLLib supported MARWIL [25] algorithm which is an on-policy algorithm as well
as the off-policy Ape-X algorithm.

We also developed a simple online training process to train a neural network
model that mimics expert actions that are generated as the actual environment
is run to overcome limited samples stored offline.

####  4.4.2 Mixed Learning

This step involves training in both an offline process via stored experiences
and actual environment simulation experiences. This is implemented using the
off-policy Ape-X algorithm with the stored experiences sampled from the replay
buffer based on a user configurable ratio of simulation to expert data. To
combine the losses from both the expert and the RL algorithm, we used a loss
function similar to DQfD [22].

We also used the simple online training process in the pure imitation learning
step along with the on-policy PPO algorithm [26]. This was done by running
entire episodes with either a Pure IL approach or a PPO Approach. The
algorithm to be run was selected based on the probability (this is user
configurable though for our experiments we used 50%) of a binomial
distribution. This has been adapted from the approach described in the Multi-
agent path finding (MAPF) framework PRIMAL [10].

The aim of the 2-step approach was to see the limit of what can be learned
using only expert data and then improve on that using a hybrid approach to
bootstrap the Reinforcement Learning process. For each of the two approaches,
both, on- and off-policy RL algorithms were run. The results can be directly
compared to the Ape-X and PPO runs that were also part of our experiments.

##  5 Results

Figure 6 and Figure 7 show the training progress (per approach) averaged over
three runs with different seeds.111111These figures and more are available on
Weights & Biases: https://wandb.ai/aicrowd/flatland-paper/reports/Flatland-
Paper-Results--VmlldzoxOTc5Mjk The training and test results are listed in
Table 1. Refer to Section 4.1 for details of the training, evaluation and test
protocols).

In summary, all RL and IL experiments outperform the “Random” agents and
“Always Forward” agents. Few experiments (Ape-X FIXED IL (25%), PPO SKIP) show
small improvements over the Standard Ape-X and PPO models. The Shortest-Path-
Agents perform better than the low-performing RL approaches quite well (local
grid observation and global density map observation) demonstrating the
strength of even a basic planning approach. In the following section, more
details for the individual experiments are provided.

###  5.1 Reinforcement Learning

Figure 6: Mean normalized score (left) and completion rate (right) of the RL
experiments. The figures show the mean across the respective training runs for
each experiment, individual runs are displayed as fine lines. We applied a
Gaussian filter for smoothing.

For the centralized critic, the results show that the base network version
outperformed the transformer variation in completion rate with 76.27% compared
to 71.87%. The base network also shows a higher standard deviation. However,
in our experiments CCPPO BASE was outperformed by both the Ape-X
implementation and several PPO versions. Even though, one of the best
submission to the NeurIPS 2020 Flatland benchmark is built on the CCPPO
implementation of our experiments.

The experiments which skip “no-choice” cells and mask unavailable actions show
comparable performance and sample-efficiency to their standard Ape-X and PPO
counterparts. Notably, The skipping of “no-choice” cells with Ape-X was
successfully used in a submission to the NeurIPS 2020 Flatland benchmark using
different hyperparameters.

The global density observation performs slightly better than the stock global
observation, illustrating that there is potential in carefully designing
global observations. However, the test performance is significantly lower than
the training performance which warrants further investigation.

###  5.2 Imitation learning

Figure 7: Mean normalized score (Left) and completion rate (Right) of the IL
experiments. The figures show the mean across the respective training runs for
each experiment, individual runs are displayed as fine lines. We applied a
Gaussian filter for smoothing.

The results show that a pure Imitation Learning can help push the mean
completion to 80% on the test results 1. This is achieved via a simple neural
network that mimics the expert actions. The MARWIL algorithm performs
similarly with a completion of 72.4%. The Ape-X based pure off-policy based IL
algorithm performs poorly with only 10% completion. However supplementing this
approach with 75% samples from the actual environment and using a loss
function similar to DQfD, leads to 86% completion which is a significant
improvement with respect to the best performing pure IL algorithm.

When the simple neural network is mixed with a PPO, we see a drop in
performance to a completion rate of 71.47%. Since this mixed approach was
trained only for 15 million steps and with a constant probability of IL/RL
ratio, this performance could be improved by using a decayed IL/RL ratio and
training for higher number of steps.

Combining both, the expert training and environment training with the fast
Ape-X or the PPO based mixed IL/RL algorithm, leads to a mean completion rate
comparable and higher than corresponding pure RL runs. A notable observation
is that Imitation Learning algorithms have a higher minimum completion rate.

METHOD | Train | Test
---|---|---
| % Complete | Reward | % Complete | Reward
Ape-X FIXED IL(25%) | 90.45±0.4 | -0.18±0 | 86±1.44 | -0.22±0.01
Ape-X FIXED IL(100%) |  |  | 22.93±11.83 | -0.84±0.08
Ape-X | 90.38±1.64 | -0.2±0.02 | 80.93±5.45 | -0.32±0.04
Ape-X SKIP | 89.51±1.09 | -0.21±0.01 | 79.73±0.92 | -0.33±0.01
CCPPO | 87.72±2.37 | -0.2±0.02 | 71.87±3.7 | -0.35±0.03
CCPPO BASE | 83.21±1.47 | -0.25±0.01 | 76.27±6.96 | -0.31±0.06
MARWIL FIXED IL(100%) |  |  | 72.4±3.27 | -0.35±0.02
PPO + Online IL(50%) | 83.46±1.09 | -0.23±0.01 | 71.47±4.01 | -0.35±0.04
PPO | 94.78±0.29 | -0.13±0.01 | 81.33±5.86 | -0.26±0.05
PPO MASKING | 93.4±0.27 | -0.15±0 | 80.53±9.59 | -0.28±0.09
PPO SKIP | 93.48±0.66 | -0.15±0.01 | 82.67±5.79 | -0.26±0.05
Ape-X Global density | 57.87±1.85 | -0.51±0.01 | 34.4±9.23 | -0.71±0.07
Online IL(100%) |  |  | 80±3.27 | -0.27±0.03
Random |  |  | 20.4 | -0.85
Constant Forward Action |  |  | 22.4 | -0.8
Shortest Path |  |  | 67.2 | -0.38
Table 1: Evaluation and test results of the experiments.

##  6 Discussion

Our experiments demonstrate that RL agents are able to learn strategies for
solving most of the test environments. Although there is a gap between the
train and test mean completion rate, most completion rates range between 71
and 86% for the test environments, demonstrating that RL models are able to
generalize from the training samples to unseen problem instances. Even though
the RL results presented here demonstrate the feasibility of the approach, RL
solutions are clearly outperformed by OR methods that dominate the public
benchmark’s overall leader board.

We note that standard Ape-X and PPO with tree observations perform as well as,
or better, than other models using the same observation. There is no visible
difference in sample efficiency between these experiments. Therefore, the
chosen RL approaches are capable to identify the relevant information from the
observation alone. Unexpectedly, PPO with the centralized critic (CCPPO)
performed slightly worse than the standard PPO counterpart. This means that
the assumed benefit of the centralized critic with it’s global view in the
training could not be proven, but on the contrary seems to affect
generalization. This is supported by the finding that the basic CCPPO has a
smaller drop between training and testing performances than the transformer
CCPPO. This could mean that the attention mechanism makes the CCPPO’s policy
more specific to the states it encountered. In summary, the principle of the
centralized critic that has been introduced in a basic version, needs further
experimentation in combination with other models beyond PPO.

Similarly, providing the agents with the global view of the global density map
leads to significantly lower completion rates. This approach outperformed the
baseline agents with random and constant forward action, but performed worse
than a simple planning approach given by the shortest path baseline. It is
also noteworthy that the training performance oscillated much more than the
approaches using the tree observation. A tentative explanation of the lower
performance and the lack of stability during training could relate to the fact
that the global density experiments contain a representation of the track
layout of the respective environment. Therefore, the agents are likely to
”memorize” the layout of the environments they were trained on rather than
being able to generalize to new environment. A possible improvement could be
to build the global density observation in a way that it abstracts from the
track layout.

The approach using a combination of IL and RL (fixed 25% IL) shows the highest
mean completion rate in the test set with only a minor drop from the training
performance. This could be due to the fact that it could learn from many
successful demonstrations (the IL part) and that all actions in a
demonstration are optimal with regard to the original OR strategy used to
generate the demonstrations.

An obvious shortcoming of the RL approach seems to be the lack of coordination
between trains. Although the tree observation provides some information about
possible collisions with other agents, the agents do not seem to be able to
coordinate a response to that information, resulting in frequent dead-lock
situations. A way to improve the coordination between agents could be to add
communication, for instance enabling a mechanism that allows agents to
communicate their intended next action.

###  6.1 Limitations

A common question concerns the transferability of solutions from Flatland’s
simplified grid environment to real railway networks and real world railway
planning tasks.

In Flatland, the movement physics of trains is simplified as much as possible:
Trains either travel at full speed or do not move. Even though this may seem
highly unrealistic, such simplifications are common in today’s scheduling and
re-scheduling methods. Today, all planning is done using simplified
calculations of minimal travel-time with additional buffer times. Even though
these models are still more complex than the one used in Flatland, the general
idea of planning with simplified models and additional buffers is the same.

Flatland’s core principle of mutual exclusive occupancy of grid cells directly
reflects the structure of signalling and safety systems: in today’s railway
systems the railway network is divided into discrete blocks. Each block can
only be occupied by a single train. First and foremost this is a safety
principle that is ensured by axle counters and signals. However, this
structure also simplifies planning. In the process of digitization and
automation the block signaling techniques will be replaced by the “continuous”
moving block techniques at some point. However, the discrete problem reflects
the vast majority of today’s railway networks and also is relevant for
discrete modelling of the “continuous” moving block scenario.

The patterns of possible transitions between grid cells were modelled directly
after the real railway infrastructure. All transition maps of the Flatland
environment reflect real infrastructure elements of the Swiss Railway Network,
such as single slip switches, double slip switches, and crossings.

Exploiting the capacity of the railway infrastructure is one of the most
challenging aspects of railway planning in all railway networks due to the
ever increasing demand for service and to the high effort of building new
infrastructure. Flatland directly reflects these aspects by presenting
problems with a large ratio of single-track sections that are one source of
complexity for the re-scheduling in daily railway operations. Furthermore,
Flatland’s problems with high numbers of trains that fill the railway network
directly address the problem of handling traffic loads close to a network’s
maximum capacity.

###  6.2 Conclusions

Despite the simplifications in the Flatland environment, the essence of the
real-world challenges from railway networks are still well represented in the
problem formulation. Algorithmic solutions found for the Flatland benchmark
can be integrated into current prototypes for traffic management systems
without major modifications. For instance, some of the top methods of the 2019
Flatland benchmark were successfully adapted to real-world internal scheduling
and data models by the Swiss Railway (SBB). Flatland instances can be
transformed into the SBB-internal data format for further investigation. At
Deutsche Bahn (DB) first prototypes pf a planning system were developed on a
previous version of Flatland and are now extended to work in a more realistic
railway simulation that models the key characteristics of a digitalized
railway system of the future.

With Flatland, we introduced a framework that allows to experiment complex
VRSP scenarios through OR and RL approaches on a grid world. Our results
demonstrate that RL has the potential to effectively and efficiently solve
VRSP problems. This is further underlined by the preliminary results from the
NeurIPS 2020 Flatland Benchmark that successfully used the approaches from our
experiments as well as custom built solutions with even better performance.
However, more research is needed to make reinforcement learning a solid
alternative to operations research and eventually address problems for which
the latter is ill-equipped.

##  7 Acknowledgments

The CCPPO baseline was developed in a joint project of Deutsche Bahn and
InstaDeep with a substantial contributions from Alain-Sam Cohen
<EMAIL_ADDRESS>from InstaDeep. We would like to thank the efforts
from Shivam Khandelwal (AIcrowd) and Jyotish Poonganam (AIcrowd) in helping
ensure a stable execution of the competition. We would like to thank Mattias
Ljungström for contributions to the many discussions in the initial formative
phases of this project. We would like to thank Francois Ramond and the team at
SNCF for their continued support for executing this project. We would like to
thank our collaborators at NVIDIA for offering prizes for the NeurIPS 2020
Flatland Benchmark. We would like to thank our collaborators at SBB, DB and
SNCF for supporting this project with resources and foundational ideals. We
would like to thank the AIcrowd community for their continued engagement with
this problem.

## References

  * [1] “smartrail 4.0: An innovation programme from the Swiss railway industry”, https://smartrail40.ch/index.asp?inc=&lang=en
  * [2] Adrian Egli and Erik Nygren  “Real World Application of Multi-Agent Deep Reinforcement Learning: Autonomous traffic flow management”  In _GTC Europe 2018_ , 2018  DOI: 10.13140/RG.2.2.13164.62085
  * [3] Jean-Yves Potvin and Jean-Marc Rousseau  “A parallel route building algorithm for the vehicle routing and scheduling problem with time windows”  In _European Journal of Operational Research_ 66.3 Elsevier, 1993, pp. 331–340
  * [4] Brian A Foster and David M Ryan  “An integer programming approach to the vehicle scheduling problem”  In _Journal of the Operational Research Society_ 27.2 Springer, 1976, pp. 367–384
  * [5] Lawrence Bodin and Bruce Golden  “Classification in vehicle routing and scheduling”  In _Networks_ 11.2, 1981, pp. 97–108  DOI: 10.1002/net.3230110204
  * [6] Jing-Quan Li, Pitu B Mirchandani and Denis Borenstein  “The vehicle rescheduling problem: Model and algorithms”  In _Networks: An International Journal_ 50.3 Wiley Online Library, 2007, pp. 211–229
  * [7] Eric Larsen et al.  “Predicting Tactical Solutions to Operational Planning Problems under Imperfect Information”  In _arXiv preprint arXiv:1901.07935_ , 2019
  * [8] Darja Šemrov et al.  “Reinforcement learning approach for train rescheduling on a single-track railway”  In _Transportation Research Part B: Methodological_ 86 Elsevier, 2016, pp. 250–267
  * [9] Jamal Shahrabi, Mohammad Amin Adibi and Masoud Mahootchi  “A reinforcement learning approach to parameter estimation in dynamic job shop scheduling”  In _Computers & Industrial Engineering_ 110 Elsevier, 2017, pp. 75–82
  * [10] Guillaume Sartoretti et al.  “PRIMAL: Pathfinding via reinforcement and imitation multi-Agent learning”  In _IEEE Robotics and Automation Letters_ 4.3 IEEE, 2019, pp. 2378–2385
  * [11] Cinjon Resnick et al.  “Pommerman: A Multi-Agent Playground”, 2018  eprint: arXiv:1809.07124
  * [12] Diego Perez-Liebana et al.  “The Multi-Agent Reinforcement Learning in MalmÖ (MARLÖ) Competition”, Challenges in Machine Learning (NIPS Workshop), 2018, 2019  eprint: arXiv:1901.08129
  * [13] Oriol Vinyals et al.  “StarCraft II: A New Challenge for Reinforcement Learning”, 2017  eprint: arXiv:1708.04782
  * [14] Albert Jones, Luis Rabelo and Abeer T.  “Survey of Job Shop Scheduling Techniques”, 1999  DOI: 10.1002/047134608X.W3352
  * [15] Li Zipeng and Wang Yanyang  “A Review for Aircraft Landing Problem”  In _MATEC Web of Conferences_ 179, 2018, pp. 03016  DOI: 10.1051/matecconf/201817903016
  * [16] Eric Liang et al.  “RLlib: Abstractions for Distributed Reinforcement Learning”  In _International Conference on Machine Learning (ICML)_ , 2018
  * [17] Barret Zoph, Vijay Vasudevan, Jonathon Shlens and Quoc V Le  “Learning transferable architectures for scalable image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8697–8710
  * [18] Lasse Espeholt et al.  “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures”  In _Proceedings of the 35th International Conference on Machine Learning_ 80, 2018, pp. 1407–1416
  * [19] Dan Horgan et al.  “Distributed Prioritized Experience Replay”  In _International Conference on Learning Representations_ , 2018  URL: https://openreview.net/forum?id=H1Dy---0Z
  * [20] Stephane Ross, Geoffrey J. Gordon and J. Bagnell  “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, 2010  arXiv:1011.0686 [cs.LG]
  * [21] Wen Sun et al.  “Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction”, 2017  arXiv:1703.01030 [cs.LG]
  * [22] Todd Hester et al.  “Deep q-learning from demonstrations”  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018
  * [23] Shariq Iqbal and Fei Sha  “Actor-attention-critic for multi-agent reinforcement learning”  In _International Conference on Machine Learning_ , 2019, pp. 2961–2970
  * [24] Emilio Parisotto et al.  “Stabilizing transformers for reinforcement learning”  In _arXiv preprint arXiv:1910.06764_ , 2019
  * [25] Qing Wang et al.  “Exponentially Weighted Imitation Learning for Batched Historical Data”  In _Advances in Neural Information Processing Systems 31_ Curran Associates, Inc., 2018, pp. 6288–6297  URL: http://papers.nips.cc/paper/7866-exponentially-weighted-imitation-learning-for-batched-historical-data.pdf
  * [26] John Schulman et al.  “Proximal Policy Optimization Algorithms”, 2017  arXiv:1707.06347 [cs.LG]