fn1fn1footnotetext: The authors are with the Department of Electrical
Engineering & Computer Science, University of Cincinnati, Cincinnati, OH,
45221 USA

# Rapid Learning of Spatial Representations for Goal-Directed Navigation Based
on a Novel Model of Hippocampal Place Fields

Adedapo Alabi<EMAIL_ADDRESS>Dieter Vanderelst<EMAIL_ADDRESS>Ali Minai<EMAIL_ADDRESS>
###### Abstract

The discovery of place cells and other spatially modulated neurons in the
hippocampal complex of rodents has been crucial to elucidating the neural
basis of spatial cognition. More recently, the replay of neural sequences
encoding previously experienced trajectories has been observed during
consummatory behaviour – potentially with implications for rapid learning,
quick memory consolidation, and behavioral planning. Several promising models
for robotic navigation and reinforcement learning have been proposed based on
these and previous findings. Most of these models, however, use carefully
ingineered neural networks, and sometimes require long learning periods. In
this paper, we present a self-organizing model incorporating place cells and
replay, and demonstrate its utility for rapid one-shot learning in non-trivial
environments with obstacles.

##  1 Introduction

Animals have over time evolved to possess, among other things, advanced
spatial cognition. In a bid to understand the computational processes and
neural mechanisms underpinning this ability, the rodent in particular has been
studied extensively. These studies have lead to the current consensus that
rodent spatial cognition is supported by a context dependent topological map
encoded in the hippocampus with mechanisms for quick one-shot memory
consolidation and recall [50, 57, 78, 61, 59]. Computational models of this
system have the potential to improve robot spatial cognition.

Bio-inspired approaches to navigation are particularly interesting because
they naturally lend themselves to neuromorphic implementation with its
inherent advantages of energy efficiency and parallelizability, which is
better suited to maintaining multiple pose estimates under uncertainty than
the traditional von Neumann architecture. More ambitiously, it has been
hypothesized that the same basic circuitry that is used for spatial cognition
in the hippocampal complex evolved into the cortical columns of the cerebral
cortex that enable more general cognition or navigation in other abstract (non
physical) spaces [32]. Thus, understanding how this system works and building
computational models of it can serve as a precursor to realizing the elusive
dream of artificial general intelligence. The use of biologically-inspired
features in our model is also driven by the strong conviction that the
solutions to many of the deep, unsolved problems in learning systems, e.g.,
rapid reinforcement learning, can be found by moving closer to the biological
systems that already solve these problems.

Over the years, many research groups have developed mapping and navigation
methods inspired by the hippocampal system [9, 67, 63, 20]. The model
presented in this paper builds on some of the ideas in these models, and adds
others from recent discoveries in neuroscience such as replay and preplay. The
model shows how a self-organized dynamical system can learn to encode
actionable place representations using a biologically plausible reinforcement
learning mechanism, and exploit it to learn and generalize goal-oriented
spatial representations from very limited experience.

##  2 Motivation

The work in this paper had two primary motivations:

  1. 1.

Animals are able to learn quickly and generalize from limited experience. This
is in sharp contrast to most of today’s artificial intelligence systems which
require extensive training. Rodents appear to augment their real-time
exploration with offline replay – a discovery that has inspired a number of
successful reinforcement learning algorithms such as AlphaZero and Deep
Q-learning [66, 48, 47]. This is putatively coupled with the use of mental
preplay to ”imagine” the outcome of possible actions at decision points [61,
59, 65]. The current model integrates both mechanisms in a model that can
learn to demonstrate efficient goal-directed navigation from limited
experience. The model shows that replay can be used to assign reward values to
previously explored states rapidly and preplay can be used to exploit these
reward values during subsequent navigation.

  2. 2.

Place cell models typically include carefully tuned connections between the
place cells and/or with their afferent input to generate convex place fields
[63, 26]. We show how a competitive self-organized place cell network
requiring minimal pre-configurations can learn convex place fields with
implicit directional information, resulting in a spatial representation that
supports efficient goal-directed navigation in complex environments with
multiple obstacles.

##  3 Background

It is now generally accepted that the hippocampal complex plays a central role
in spatial cognition in mammals [50, 57, 78]. Place cells in the hippocampus
[56, 57, 52, 51], grid cells in the entorhinal cortex [27, 25, 50, 41, 11],
and head direction cells in the postsubiculum [76] and entorhinal cortex are
some of the spatially modulated cells found in this region. In a new
environment, place cells exhibit spatially localized activity called place
fields [57, 52]. Grid cells effectively code for displacement, firing in
multiple locations across an environment in a regular hexagonal lattice [27,
25] and are believed to perform path integration to support localization in
place cells [41, 68, 10]. Both systems work together to allow animals to form
spatial cognitive maps [55, 62] supporting localization and navigation by
integrating sensory and ideothetic information. As the name suggests, the head
direction cells of the entorhinal cortex [64] and postsubiculum [77] encode
the allocentric heading of the animal, firing maximally when the heading
matches their preferred directions.

Along with mapping, intelligent agents must also learn important locations
within the map and how to navigate efficiently to them. Animals are known to
exhibit quick one-shot learning of key locations in new environments [49, 79].
In both biological and artificial agents, this problem is solved through
reinforcement learning (RL) [72], where the agent learns through rewards and
penalties elicited by exploratory behavior. RL is particularly useful when
complete knowledge of the environment is unavailable, rendering supervised
training impossible

A major difference between RL in animals and computational systems is that the
former can learn very rapidly from just a few trials while the latter requires
extensive training. The reasons for this difference are not well-understood.
The availability of good pre-configured priors, transfer learning, and
accurate extrapolation may play a role, but an important contributor may be
the ability to learn from mental rehearsal. The discovery that previously
experienced neural sequences are replayed in the hippocampus during sleep [40,
37, 16] and consummatory behaviour [12, 13, 19] suggests the possibility that
animals augment real-time learning with off-line replay. The replay that
occurs upon reward receipt is particularly interesting as it theoretically
provides a mechanism for the quick assignment of expected reward values to
previously traversed locations if accompanied by spike time-dependent
potentiation (STDP) between place cells and neurons that encode rewards [1,
42, 58].

Beyond replay, rodents navigating towards known goals have also been observed
to pause and look down the available paths at decision points in real-time
navigation. This is accompanied by a mental preplay of the place cell
sequences encoding the paths in a process termed vicarious trial-and-error
(VTE) [61, 59, 65]. Assuming that expected reward values have previously been
assigned to the place cells - either by replay or a different means; this
could hypothetically allow the rodent evaluate the outcomes of its available
actions so it can decide which one to take. This approach has also been used
in some hippocampally-inspired navigation models [20, 21]. However, the
success of both replay-based RL and preplay-based decision-making depends on
the agent’s ability to generalize by extrapolation as well as interpolation.
The model presented in this paper shows such generalization.

A number of research groups have developed robotic navigation algorithms
incorporating findings in neuroscience. RatSLAM [46, 82, 45] is one such
example and has been shown to navigate office spaces [44] and map large
outdoor environments [45]. Extensions of RatSLAM have also been shown to be
sensor-agnostic and to support sensor fusion [5, 33, 71]. Some algorithms also
incorporate the experimentally observed replay [34, 23] for memory
consolidation. We have recently proposed and developed a model [1] to enable
quick one-shot learning and generalization in simple open mazes using replay.
In this work, we refine that model by defining it as a self-organizing
dynamical system and demonstrate it’s ability to learn quickly in more complex
mazes with obstacles.

##  4 Model Description

Figure 1: An overview of our model architecture

The model consists of four types of cells [1]: the head-direction cells;
boundary vector cells; place cells, and reward cells. These four types of
cells are denoted by the symbols $h$ (head-direction cells), $b$ (boundary
vector cells), $p$ (place cells), and $r$ (reward cells). The time-varying
firing rate of the $i^{th}$ cell from cell type $k$ is denoted as
${v_{i}^{k}}(t)$ or ${v_{i}^{k}}$ for brevity. The time-varying synapse from
the $j^{th}$ cell of type $l$ to the $i^{th}$ cell of type $k$ as
${W_{ij}^{kl}}(t)$ (or ${W_{ij}^{kl}}$). The current model does not include
grid cells, which will be added in the future.

The networks operate in three states: regular, replay and preplay. In the
regular state, the networks directly encode the sensory input and the
environment is learnt. In preplay, the agent uses the learnt map to “imagine”
the expected network states if it were to move a step from its location in a
specific direction. Replay is a transient spreading activation state that
occurs upon reward receipt and serves to learn the value map of the
environment.

###  4.1 Head Direction Network

The head-direction cells comprise a single layer of neurons, each of which has
a preferred direction in allocentric coordinates. It fires maximally when the
animal’s heading is in its preferred direction, with a symmetric dropoff on
both sides. The model by Erdem & Hasselmo [22] is used to represent the head-
direction cells. In particular, the firing rate of head-direction cell $i$ is
given by:

${v^{h}_{i}}=\sum_{j=0}^{2}{x^{\prime}_{j}}\left[\begin{array}[]{c}\cos({\theta_{i}^{h}}+\theta_{0})\\\\[4.30554pt]
\sin({\theta_{i}^{h}}+\theta_{0})\\\ \end{array}\right]$ (1)

where $\theta_{0}$ is the heading angle of the anchor cue, ${\theta_{i}^{h}}$
is the preferred direction of head direction cell, and ${x^{\prime}}$ is the
instantaneous velocity of the agent in Cartesian coordinates.

###  4.2 Place Cell Network

Figure 2: Examples of place fields developed in the mazes. These plots were
generated using a color coded hexagonal bin plot of the firing rate of the
probed place cell during a random run after exploration. It should be noted
that here and in other plots, the environment is continuous and the binning is
only done for plotting. Unvisited bins have no values and are left unfilled,
with the grey arena underneath visible. The red circles indicate the goal
locations in the mazes.

Place cells have localized, convex activity fields (_place fields_) in
specific locations in a given environment. While their activity is known to be
based on several input sources, two of the primary sources are boundary vector
cells [54, 38] and other place cells via recurrent connections [43, 69, 3],
which are the sources considered in the current model.

The regular state activity of place cells in the model is based on input from
boundary vector cells (BVCs). BVCs are neurons that fire when the animal is at
specific distance and allocentric directions from obstacles or boundaries.
These cells are found in the subiculum [38]. Following the model by Barry et
al [4], for BVC $i$ tuned to obstacles at displacement $(d_{i},\;\phi_{i})$ in
polar coordinates with tuning widths $(\sigma_{r},\;\sigma_{\theta})$, the
firing rate in response to an obstacle at displacement $(r,\;\theta)$,
subtending an angle $\delta\theta$ is given by:

$\delta{v_{i}^{b}}=\frac{\exp\left[-\left(r-d_{i}\right)^{2}/2\sigma^{2}_{r}\right]}{\sqrt{2\pi\sigma^{2}_{r}}}\times\frac{\exp\left[-\left(\theta-\phi_{i}\right)^{2}/2\sigma^{2}_{\theta}\right]}{\sqrt{2\pi\sigma^{2}_{\theta}}}\delta\theta$
(2)

The equations for the firing rate of the $i^{th}$ place cell are then:

$\tau_{p}\frac{d{s_{i}^{p}}}{dt}=-{s_{i}^{p}}+\sum_{j=0}^{n_{b}}{W_{ij}^{pb}v_{j}^{b}}-\Gamma^{pb}\sum_{j=0}^{n_{b}}{v_{j}^{b}}-\Gamma^{pp}\sum_{j=0}^{n_{p}}{v_{j}^{p}}$
(3a)

${v_{i}^{p}}=\tanh{\left(\left[\psi{s_{i}^{p}}\right]_{+}\right)}$ (3b)

Equation (3a) models the dynamics of ${s_{i}^{p}}$, the membrane potential of
the place cell as a function of its BVC input. This response is subject to
inhibition from both feedforward and feedback sources to ensure spatial
specificity. The feedforward inhibition term $\sum_{j=0}^{n_{b}}{v_{j}^{b}}$
scaled by gain $\Gamma^{pb}$ sets a threshold on the amount of feedforward
input to a place cell required for depolarization while the recurrent
inhibition term $\sum_{j=0}^{n_{p}}{v_{j}^{p}}$ scaled by gain $\Gamma^{pp}$
serves to keep the total network activity from diverging. Equation (3b)
determines the firing rate ${v_{i}^{p}}$ as the hyperbolic tangent of its
rectified membrane potential scaled by a parameter $\psi$.

It is important to ensure that place cells cover the entire environment. To
guarantee uniform coverage over environments, competitive learning is used in
the place cell network. Place cells can develop localized place fields by
competing for boundary vector cell inputs. The following rule proposed by Oja
[53] captures competitive learning. Specifically, the synaptic strength from
BVC cell $j$ to place cell $i$ evolves as:

$\tau_{w^{pb}}\frac{d{W^{pb}_{ij}}}{dt}={v^{p}_{i}}\left({v^{b}_{j}}-\frac{1}{\alpha_{pb}}{v^{p}_{i}W^{pb}_{ij}}\right)$
(4)

where $\tau_{w^{pb}}$ parameterizes the speed of learning,
${W^{pb}_{ij}}(0)=1$ with a probability $p_{pb}$ and $\alpha_{pb}$ is a
normalizing factor.

We propose that place cells switch to being driven by recurrent input during
preplay state. This distinction between the contributions of the recurrent and
boundary vector cell input is inspired by both the morphology and physiology
of the hippocampus. Afferent input to place cells such as that from boundary
vector cells targets distal apical dendritic tufts while excitatory recurrent
synapses are typically at basal dendrites [35, 73, 43]. Negative recurrent
feedback – mediated by oriens lacunosum-moleculare (OLM) cells – also target
the apical dendritic tufts [6], justifying the negative feedback term in
equation 3a.

Beyond the physical separation of the input sources, it has been proposed
that, similar to what has been observed in the piriform cortex [28, 29, 75],
selective suppression of recurrent but not feedforward synaptic transmission
by acetylcholine and other neurotransmitters in the hippocampus can allow for
the learning of new information over recall[31, 30, 80, 7]. We extend this
hypothesis in Section 4.4.

During preplay, the place cells are driven by recurrent input from other place
cells along the direction $a\in{\theta^{h}}$.

${\bar{v}_{i}^{p}}\,|\,a=\tanh{\left(\left[\sum_{j=0}^{n_{p}}{W^{pp}_{aij}}{v^{p}_{j}}-{v^{p}_{i}}\right]_{+}\right)}$
(5)

${\bar{v}_{i}^{p}}$ is the resultant value of ${v_{i}^{p}}$ when the agent
imagines taking one step in direction $a$. The synapse ${W^{pp}_{aij}}$
between place cell $i$ and place cell $j$ encodes the proximity of their place
fields along direction $a$.

While synapses have conventionally been represented as dyadic objects, newer
evidence is beginning to make clear that their effect can depend on signals
from sources other than just the activity of pre-synaptic and post-synaptic
neurons. For example, external modulation and local computations occurring
within dendritic branches creates complex dependencies between synapses and
with modulating inputs [83, 14, 73, 60, 39]. We extend a previous proposal
that co-active pre-synaptic neurons that respond to related sensory input
preferentially innervate the same dendritic branches [74, 36, 17, 2] to
suggest that the dendritic branches of place cell recurrent collaterals can be
modulated in direction-specific ways, thus resulting in triadic synapses. We
represent the direction encoding branches as the first dimension of a three-
dimensional weight tensor, $[{W^{pp}_{aij}}]$. Potentation occurs on
individual branches in a direction-specific manner so that ${W^{pp}_{aij}}$ –
the synapse from place cell $j$ to place cell $i$ along the branch responsive
to direction $a$ encodes the proximity of their place fields going in the
direction $a$ from the place field of $j$ to the place field of $i$.

Along with the directionality in the spatial relationship between two place
fields, the temporal order is also crucial, i.e., which of the place fields
leads to the other along the specified direction. To capture this causality or
temporal order, a static rule focusing only on the instantaneous firing rate
would be insufficient. We formulated a learning rule that integrates the pre-
synaptic firing rate of ${v^{p}_{j}}$ as well as the head direction cell
activity ${v^{h}_{k}}$ over the preceding time period $\tau_{pre}$ to
determine temporal order.

To update the synaptic strengths in the weight tensor, $W^{pp}_{kij}$, in a
directionality-specific way, we compute variables $\Upsilon^{p}_{j}$,
$\Upsilon^{p}_{i}$, and $\Upsilon^{p}_{k}$, each representing the integrated
recent activation of pre-synaptic place cell $j$, post-synaptic place cell
$i$, and head direction cell $k$, respectively, along direction
$\theta_{k}^{h}$:

$\tau_{h}\frac{d{\Upsilon^{p}_{j}}}{dt}=-{\Upsilon^{p}_{j}}+{v^{p}_{j}}$ (6a)
$\tau_{h}\frac{d{\Upsilon^{p}_{i}}}{dt}=-{\Upsilon^{p}_{i}}+{v^{p}_{i}}$ (6b)
$\tau_{h}\frac{d{\Upsilon^{h}_{k}}}{dt}=-{\Upsilon^{h}_{k}}+{v^{h}_{k}}$ (6c)

where $\tau_{h}$ is a time constant parameterizing the length of the history
taken into account.

The weights in the tensor then change as:

$\tau_{w^{pp}}\frac{d{W^{pp}_{kij}}}{dt}={\Upsilon^{h}_{k}}\left({v^{p}_{i}}{\Upsilon^{p}_{j}}-{v^{p}_{j}}{\Upsilon^{p}_{i}}\right)$
(7)

where $\tau_{w^{pp}}$ parameterizes the speed of learning, and ,
${W^{pp}_{kij}}(0)=0$. The product ${\Upsilon^{p}_{i}}{v^{p}_{j}}$ has a high
value if post-synaptic place cell $i$ has strong activation following strong
recent activation of pre-synaptic place cell $j$, indicating movement
_towards_ the place field of $i$ from the place field of $j$. Similarly,
${\Upsilon^{p}_{j}}{v^{p}_{i}}$ has a high value if pre-synaptic place cell
$i$ has strong activation following strong recent activation of post-synaptic
place cell $i$, indicating movement _away from_ the place field of $i$ towards
the place field of $j$. The sign of the difference term in equation (6) thus
indicates the direction of movement relative to the place fields of $i$ and
$j$, and the ${\Upsilon^{h}_{k}}$ term gates the difference term by the
directionality. As a result, weight ${W^{pp}_{kij}}$ increases when the path
in direction ${\theta_{k}^{h}}$ activates the place field of $j$ before the
place field of $i$, and decreases if the order is reversed.

###  4.3 Reward Cell Network

This network comprises reward cells that exhibit a binary response to the
receipt of their preferred rewards. Reward cells receive synapses from place
cells, which are modified to learn the proximity of each place cell’s place
field to the preferred reward locations for reward cells. These weights are
learnt during replay states as described further in Section 4.4.1. Once
learnt, the weights enable the evaluation of actions as explained in Section
4.4.2.

###  4.4 Modes of Operation

The simulated scenario is that of an environment with visible boundaries, a
variety of possible obstacles, and one or more goal locations that elicit
rewards when the agent reaches them. The environment is initially unfamiliar
to the agent, and the goal locations are not known. The agent operates in two
modes:

Figure 3: Implied values maps developed in maze 1 after the first runs
depicted in Fig. 4. The value map was derived by allowing the robot to roam
aimlessly around the maze after the reward had been found on the first run
without any synaptic modulation. At each location the firing rate of the
reward cell as a result of place cell input was recorded. This was then plot
as a color coded hexagonal bin plot.

(a) Trial 1

(b) Trial 2

(c) Trial 3

Figure 4: Three trials in the open maze showing the ability of the model to
quickly learn and exploit a useful value map of the environment. In each
trial, the robot is tasked with navigating to the goal shown in green (which
is invisible to the robot) four times from the same starting point. On the
first runs, it has no prior knowledge of the environment, however, subsequent
runs exploit and build on the previously learnt maps.

####  4.4.1 Exploration

In this mode, the agent explores randomly to map the environment. Place cells
develop place fields using equation (4) and the environment’s topology is
learnt using equation (7). On encountering a goal, the agent receives a
reward, and initiates the backward replay state. Such time-compressed replay
of experienced trajectories by rodents during consummatory behaviour has been
observed experimentally [12, 13]. The result of this process is to associate
expected reward values with the recently experienced states by modifying the
place-to-reward cell synapses as described next.

Upon receiving a reward, a specific reward cell is activated. On an initial
encounter with a goal, this cell is chosen randomly, but is thereafter
associated with that goal. The synapses from the currently active place cells
to this reward cell are then potentiated. The currently active place cells
then activate the place cells with adjacent place fields via the recurrent
synapses that have been potentiated previously according to equation 7, and
the place cell to reward cell synapses for these cells are also potentiated.
Activity then spreads out through the recurrent place cell synapses generating
a backward replay of place activity and associating it with the active reward
cell. This continues for a certain time, causing the reward to become
associated with the path taken to reach the goal. At each iteration, the total
amount of potentiation in the place-to-reward cell synapses is subject to
normalization by the total place cell network activity to enforce competition
as well as a temporal decay from the time when the replay started. This
process is summarized in Algorithm 1.

$\Delta W^{rp}=0$;

for  _$t_{r}\leftarrow 0$ to $n_{rs}$_ do

$\Delta
W_{ij}^{rp}\mathrel{{+}{=}}\frac{\bar{v}_{j}^{p}}{||\bar{v}^{p}||_{\infty}}e^{\frac{-t_{r}}{\tau_{r}}}$;

$\bar{v}_{j}^{p}=\tanh{\left(\left[\bar{v}_{j}^{p}+\sum_{l=0}^{n_{p}}(\max_{k}{W_{kjl}^{pp}})\bar{v}_{l}^{p}\right]_{+}\right)}$;

end for

$W_{ij}^{rp}\mathrel{{+}{=}}\frac{\Delta W_{ij}^{rp}}{||\Delta
W^{rp}||_{\infty}}$;

Algorithm 1 Replay Generation Method

In Algorithm 1, the synaptic modulation term $\Delta{W^{rp}}$ is initialized
to $0$ for all synapses. For a preset number of time steps $n_{rs}$;

  1. 1.

Each synapse’s modulation term is increased by its pre-synaptic place cell’s
firing rate max normalized and scaled by a temporal decay. This captures how
active the place cell is at this point in the replay and the temporal decay
serves to discount activity further from the reward.

  2. 2.

Place cells activate connected place cells along their most strongly connected
direction - indicating the most efficient path between them.

After this two-step process is completed $n_{rs}$ times, the resulting
synaptic modulation term $\Delta{W^{rp}}$ is normalized by the maximum and
added to ${W^{rp}}$. Here (and in the preplay described below), the activation
of place cell $j$ is denoted by ${\bar{v}_{j}^{p}}$ to indicate that this
activation is because of the agent’s imagination and not because it is
actually located in the place field of neuron $j$.

As the agent encounters a goal along more paths, more locations in the
environment become associated with the reward at that goal, creating a _reward
map_ of the environment with respect to that goal. Three aspects of this
process are worth noting: 1) The normalization serves to ensure that place
cells that are activated later in the replay and are thus presumably further
away have their connections to the reward cell potentiated less, thus causing
the reward map to encode the distance to the reward; 2) Given that activity
spreads out along experienced paths, the learnt reward map is a topological
rather than a metric map enabling it to take obstacles into account to the
extent they are experienced; and 3) Because place fields have significant
width, the map covers not only the exact paths traversed but also a swath of
locations around them, resulting in significant generalization to unvisited
locations. As a result of this field effect, a reasonably accurate and
actionable reward map is built for a given goal even with a small amount of
exploration though, of course, the reward map only covers the general region
that has been explored.

####  4.4.2 Exploitation

In previously explored environments, the agent exploits the learnt place field
connectivity and reward map values to navigate to the reward locations. It
achieves this by using a 1-step preplay to recall the outcome of taking the
actions available in ${\theta^{h}}$ and evaluating the expected reward value
of the resulting states using the reward map.

The agent “imagines” the outcomes of taking actions by letting activity spread
in the place cell network via the recurrent synapses that have been
potentiated through exploration. The place cells encoding its current location
activate the place cells encoding locations that have been learnt to be
adjacent along the direction of the action being evaluated. This imagined
place cell activity then drives the reward cell network giving a measure of
the value of the imagined states, allowing the robot to pick the action that
maximizes the expected value state.

The imagined activation of place cell $i$ denoted by ${\bar{v}_{i}^{p}}\,|\,a$
is defined in (5). The expected reward value, ${\bar{v}_{g}^{r}}\,|\,a$, of
the imagined state is computed by allowing the imagined place cell network
activity, ${\bar{v}}^{p}$ to activate the reward cell for the current goal $g$
via the place to reward cell synapses and normalizing this by the sum of
${\bar{v}_{i}^{p}}$. The normalization is done so that the potential reward
evaluation can be made relative to other options without any bias from the
total place cell network activity. We represent the effective weight matrix as
${\bar{W}_{gi}^{rp}}$ for the duration of the exploitation episode. It is
initialized as ${{W}_{gi}^{rp}}$. Thus for all $a\in{\theta^{h}}$:

${\bar{v}_{g}^{r}}\,|\,a=\frac{\sum_{i=0}^{n_{p}}{\bar{W}_{gi}^{rp}}{\bar{v}_{i}^{p}}}{\sum_{i=0}^{n_{p}}{\bar{v}_{i}^{p}}}$
(8)

With the expected reward values of each action estimated, the optimal action
$a^{*}$ is then computed as the circular mean direction of the expected reward
values as shown in (9) below. The optimal action is taken when
$\max_{a}\left({\bar{v}_{g}^{r}}|a\in{\theta^{h}}\right)\geq\xi$ otherwise the
robot explores. Here, $\xi$ is a (possibly stochastic) threshold that
determines the explore/exploit trade-off.

$a^{*}=\arctan(\sum_{j=1}^{n}{\bar{v}_{g}^{r}}|{a_{j}}\sin{{a_{j}}},\sum_{j=1}^{n}{\bar{v}_{g}^{r}}|{a_{j}}\cos{{a_{j}}})$
(9)

where $n$ is the number of possible step directions, and
$a_{j}\in{\theta^{h}}$. Synapses from active place cells are temporarily
depressed for the rest of the episode by a constant factor $\lambda$ to
disincentivize the agent from revisiting locations.

${\bar{W}_{gi}^{rp}}={\bar{W}_{gi}^{rp}}-\lambda{\bar{W}_{gi}^{rp}}{v_{i}^{p}}$
(10)

After each action ${v}_{i}^{p}$ is reset from the imagined state to actual
state according to (3).

###  4.5 Parallels with Traditional Reinforcement Learning

The components of the model for neural reinforcement learning described above
have many similarities with the traditional Markov Decision Process (MDP)
formalism of RL, where an MDP is a 4-tuple
$\left(\mathcal{S},\mathcal{A},\mathcal{T}_{a},\mathcal{V}\right)$. The model
in this paper is specialized to a place cell-based spatial representation and
navigation task, using specific features such as network connectivity and
replay of neural activity, and is, thus, not intended to be a general-purpose
state-of-the-art RL model. Nevertheless, it might be useful for the reader to
view the model through the lens of the RL framework. The MDP variables and
their parallels in the model are described below:

####  4.5.1 $\mathcal{S}$ \- state space

This is the set of agent-environment states. In the model, the place cells
encode the possible locations or states of the agent. Thus $\mathcal{S}$ can
be taken as ${v^{p}}$ the vector of place cell firing rates..

####  4.5.2 $\mathcal{A}$ \- action space

This is the set of actions that are available to the agent. While we left this
unconstrained during exploration, we restrict this to ${\theta^{h}}$ \- the
preferred head directions of the head direction cells i.e the eight
allocentric cardinal directions during exploitation. This gives a discrete
action space as is the case in most RL models.

####  4.5.3 $\mathcal{T}_{a}$ \- state transition function

This gives the expected state if an action is taken from the current state.
During exploration, the place cell recurrent synapses are potentiated in an
action specific manner to learn the expected network activity if an action is
taken from an initial network state. As we’ve already stated that the place
cell activity can be seen as the state, the place cell recurrent synapses
${W^{pp}}$ can be taken as the state transition function.

####  4.5.4 $\mathcal{V}$ \- value function

This gives a measure of the expected reward starting from the current state.
This is typically a value that is discounted by the number of steps required
to reach a rewarded state from the current state. Through the replay process,
the place to reward cell synapses encode a proxy of this value. Consequently,
the firing rate of the reward cell associated with the currently sought reward
due to place cell input is a direct analog of the value function. It is
assumed that a higher level brain structure that is not explicitly modeled
here assigns reward cells to the rewards discovered, and selects those reward
cells for evaluation when seeking known goals.

Values | MDP | Proposed Model
---|---|---
State Space | $\mathcal{S}$ | ${v^{p}}$
Action Space | $\mathcal{A}$ | ${\theta^{h}}$
Transition Probability Function | $\mathcal{T}_{a}$ | ${W^{pp}}$
Reward Function | $\mathcal{V}$ | ${v^{r}}$
Table 1: Interpretation of the proposed model as an MDP

##  5 Experimental Results

###  5.1 Experimental Setup

The model was deployed on a simulated iRobot Create 2 robot, equipped with an
omnidirectional rangefinder in Webots [81] – an open-source robot simulation
platform. The standard delayed-match-to-place (DMTP) procedure of the Morris
water maze task [79, 18, 8] was followed, with the agent seeking to find a
disk-shaped goal that becomes visible only when reached. Unlike the original
water maze experiment, which used a cylindrical environment, the simulations
here is a square one. In addition to simulating the classic water maze test
with an obstacle-free open environment, simulations were also carried out with
various obstacle configurations that required the agent to learn topological
rather than metric maps.

Ideally, the model agent would develop place cells in is initial exploration
of the environment. However, the small size of the simulated model meant that
the number of place cells competing for activity at any one time was small,
resulting in place fields developing “tails” along the receptive fields of
their BVC [4]. To overcome this, an initial “dry run exploration” without the
goal present was done where only the BVC-to-place cell synapses were plastic
to “clean” the place fields. This step would not be needed with enough place
cells enforcing competition for place fields. The robot was then reintroduced
into the same environment and allowed to explore without any restrictions on
the network processes previously described. Upon finding the goal, the robot
was reinitialized in the environment to exploit its learnt (partial) value
map. This was done three times, making a total of four runs – the first being
naive.

###  5.2 Open Maze Performance

Figure 5: The path indirectness over the four runs is depicted here with a box
plot. It shows a negative trend - indicating the ability of the robot to learn
a reward map from experience and exploit it to compute more direct paths.
Figure 6: The path indirectness on the second run shows an inverse correlation
to the distance covered on the first run. This indicates that covering more
ground on the initial run makes it more likely that the starting location of
the second run has previously been encountered and thus incorporated into the
reward map.

The model was first deployed and evaluated in an open maze environment with no
obstacles. Starting from the same location, the robot was able to improve its
path over the four runs. Having no prior experience of the maze, the first run
showed random undirected searching. Once the goal had been found the first
time, the replay process enabled the creation of an implicit reward value map
which is visualized in Fig. 3. This value map was then exploited on subsequent
runs so that the robot follows the previously traversed locations that connect
the starting location to the goal. The efficiency of the computed path is
constrained by the previously learnt place field connectivity - in turn
constrained by the size of the place fields themselves.

Fig. 4 illustrates three sample runs. On the first two runs, the robot starts
from the same location towards the same goal location in each trial. After
finding the goal by random exploration on the first run, the robot takes
progressively more direct paths towards the goal by exploiting its implicit
reward map shown in 3. In trial 3, the robot begins each run from a different
location with the same goal location. Since the first run involved wide
sampling of the maze before finding the goal, an extensive reward map is built
and the subsequent runs from random spots were directly towards the goal.

To further quantify the performance of the model, 20 trials were run where the
robot started from a different random location on each of the four runs. On
each run, a path indirectness metric was computed to measure the efficiency of
the path taken. This is calculated as the _relative excess length_ of the path
taken compared to the length straight line path from the starting point to the
closest point on the edge of the goal disk, which is always the shortest path
in an open environment:

$d=\sum_{t=0}^{T-1}||x(t+1)-x(t)||_{2}\,dt$ (11a)
$d^{*}=||{x}(T)-{g^{x}}||_{2}-\rho$ (11b) $indirectness=\frac{d-d^{*}}{d^{*}}$
(11c)

where $d$ is the length of the path taken, $d^{*}$ is the length of the
shortest path to the edge of the goal, $T$ is the duration of the run, ${x}$
is the robot’s trajectory in Cartesian coordinates, ${g^{x}_{i}}$ is the
center of the goal location, and $\rho$ is the goal radius.

The results obtained are shown as box plots in Fig. 5. As expected, the path
indirectness is seen to decrease over the four runs, given that more of the
environment has been learnt on previous runs and a more direct path can be
computed. The greatest improvement. however. occurs from the first run to the
second as most information is typically learnt in the first run. First runs
that result in the robot finding the goal quickly with little exploration are
more likely to cause subsequent runs to have higher indirectness values
because the initial run did not enable the robot to build a full map. Fig. 6
shows the indirectness on the second run plotted against the total length of
the path taken on Run 1. A negative Pearson correlation of $-0.53$ was
observed, in line with expectations.

(a) Maze 1

(b) Maze 2

(c) Maze 3

Figure 7: The path taken to the goal from the same starting point in the
complex mazes is shown over four runs in Maze 1 (left column), Maze 2 (middle
column), and Maze 3 (right column). The paths in the top panels for each
column are the initial exploratory paths without prior experience of the goal
location. Each subsequent trial includes the learning from previous trials.
The robot is able to quickly learn a useful value map in Mazes 1 and 2, but
struggles with perceptual ambiguity in Maze 3.

Figure 8: The implied reward maps learnt from the first runs depicted in Fig.
7 are shown here for Maze 1 (top panel), Maze 2 (middle panel), and Maze 3
(bottom panel). The reward locations are indicated by the red circles. The
plots were generated using the same method as Fig. 3.

Figure 9: Place cells start to lose spatial specificity as the maze complexity
grows so that locations begin to appear perceptually similar to the robot.
This is evident in maze 4 where the obstacle divides the environment into four
similar sub-compartments. The place cells probed here developed place fields
at perceptually similar locations of three of the four sub-compartments.

###  5.3 Performance in Complex Mazes

To test whether the model would work in more complex environments, it was
evaluated in three mazes with various obstacle configurations. In each maze,
the robot started from the same location on each of four trial runs. The paths
taken in these runs are shown in Fig. 7. The paths in the top row of panels
are the purely exploratory ones that the robot took without prior experience
of the reward location. As can be seen, the exploratory path taken in Maze 2
(middle column) was much more extensive than in the others. In Maze 1 (left
column), the path samples most parts of the environment sparsely, whereas in
Maze 3 (right column), exploration is dense in the upper right quadrant while
other regions are unexplored. The reward maps generated after the first run in
each case are shown in Fig.8. AS expected, the inferred reward maps are highly
dependent on the exploratory runs. In the subsequent exploitatory runs (roows
2 through 4 in Fig. 7), the effects of the exploration are apparent. In Maze 1
(left column, the robots goes directly to the goal along a path that stays
within the previous explored regions. It is worth noting, however, that the
path represents a strong generalization, and is not just a subset of the
exploratory path. In Maze 2 (middle column), the robout again finds a strongly
generalized efficient path, and the quality of the path improves further on
subsequent trials (since learning is still on in each trial). For Maze 3
(right column), the situation is less clear. While the robot does show some
amount of generalization, it still tends to meander around – especially if it
enters a previously unexplored region. There are two reasons for this. First,
the initial exploration was confined to a small part of the environment, so
the robot does not have a reward map that ranges across the environment. The
second problem is that the obstacle configuration divides the environment into
four compartments that appear perceptually similar from the perspective of a
robot whose only sensory apparatus is a rangefinder. Thus, the robot tends to
suffer from _perceptual aliasing_ , mistaking its location for one of the
other equivalent locations in some other quadrant, and heads off in the wrong
direction – sometimes entering perceptually similar regions to the goal
location where it exhibits scanning behaviour.

These limitations in the face of similar boundary conditions (or more broadly
sensory input) are expected in the model. In fact, rodent place cells are also
known to present multiple place fields in geometrically similar regions of
environments, suggesting the same perceptual aliasing [67, 70, 24, 15]. Fig. 9
shows two instances of this effect in the present model, where the sampled
place cells each developed place fields in perceptually similar regions of two
quadrants. In rodents however, the dorsal place cells that are putatively
driven by BVC input coexist with ventral place cells with larger place fields
which are believed to encode context. Such a multiscale configuration would
allow the place cell network to encode the knowledge of what quadrant the
robot is in and is a natural extension to this work. Moreover, it would allow
for the value map to spread out farther during replay and for the computation
of more efficient paths as the preplay can then be done at a coarser
resolution further away from the goal – smoothing out the finer details of the
exact trajectory previously experienced. In addition to a multiscale
configuration of place fields, rodents are believed to combine sensory
information via place cells with ideothetic information from grid cells [41,
68, 10]. This path integration input can potentially be used to disambiguate
position as well. Adding these biologically-motivated features to the current
model is the subject of ongoing work and will be reported in the future.

(a)

(b)

(c)

Figure 10: The reward maps from a robot placed at random reward locations
without any further exploration after initially exploring the open maze
extensively in (a), partially in (b), and Maze 2 extensively in (c). The top
row shows the exploratory path followed in each case. Rows 2 through 4 show
the inferred reward maps obtained by placing the reward at various locations
(depicted with the green circle) and sampling the inferred reward map for that
location over the whole environment without further learning. The reward maps
spread over the known environment when the reward location had been previously
encountered. This spread is more limited when the reward location was
previously unknown as seen in the last map in scenario (b).

###  5.4 Generalization Across Different Reward Locations

An important issue for the model is its ability to generalize after learning,
i.e., once it has explored an environment (and thus established place fields),
it should be able to infer a reward map for a reward at any location in the
environment without further exploration.

To test for this spatial generalization, we allowed the robot to first explore
the maze in the absence of any reward location. The robot was then placed at
arbitrary locations that were designated as the reward locations, and allowed
to infer a reward map relative to that location purely by using the backward
replay mechanism described earlier, i.e., without any further exploration.
This map was then sampled by letting the robot run extensively through the
maze without any further learning, generating a sampled view of the inferred
reward map for visualization.

Since the inference of the reward map would clearly depend on how extensive
the initial exploration was, we simulated two cases in the open maze: a)
Extensive initial exploration of the maze; and b) A very limited initial
exploration along a specific path. Additionally, we simulated a scenario with
extensive exploration of Maze 2 (c), which is a more complex maze with
obstacles. The results from this experiment are shown in Fig. 10. The first
experiment with extensive exploration in the open maze is shown in the left
column, that with limited exploration in the open maze is shown in the center
column, while the experiment with extensive exploration of complex Maze 2 is
shown in the right column. The paths followed during the exploratory runs are
shown in the first row. The remaining rows show observed reward maps with the
reward placed at various arbitrary locations. It can be seen that, in each
case, the reward map generalizes over the known environment without further
exploration or any prior knowledge of the reward location. However, as
expected, the inferred reward map works only for the regions included in the
initial exploration, though not confined only to the exploratory paths because
of the field generalization effect discussed earlier. This also means that
when the new goal location is placed in a region not visited during prior
exploration, the robot is unable to compute a useful reward map. This can be
seen in the last reward map of Fig. 10(c) where the goal was placed in an area
of the map it had not previously visited at all.

##  6 Conclusions

In this work, the goal was to show how replay and preplay, which have been
proposed to support the efficiency of biological reinforcement learning, might
be used in artificial agents with a neural place representation and
reinforcement learning system inspired by the hippocampus and related regions
in the rodent brain. Another goal was to develop a place cell network model
that generates convex place fields from minimal preconfiguration. Including
this component in the model has three benefits: 1) The place field model is
not assumed ad-hoc but is built using an explicit, well-defined, and
biologically plausible process; 2) The specific pattern of neural connectivity
required by the navigation process is ensured by the model; and 3) The model
represents an interesting hypothesis for the rapid formation of convex place
fields in animals.

The proposed self-organized model is able to develop convex localized place
fields from randomly initialized BVC-to-place cell connections, learn the
topology of non-trivial environments, and then compute and exploit value maps
of these environments for efficient goal-directed navigation. The model also
displays very rapid learning with continued improvements from further
experience, and can thus be considered a one-shot or few-shot reinforcement
learning system. The model is, however, limited by the fidelity of its
perception. Performance suffers in the presence of perceptual aliasing, though
this is expected of any spatial cognition system: Even humans get lost in
mazes.

The model can potentially be extended in several ways. First, as previously
stated, rodents are known to combine the BVC-driven place cells which have
been modelled here with context-driven place cells that have larger place
fields in a hierarchical manner. Such a multiscale configuration can allow for
the disambiguation of similar areas at the BVC place cell representation level
based on the representation at the lower resolution contextual level.

Second, moving beyond a multiscale configuration, the idea of place cells
tuned to boundary vectors can be abstracted to place cells tuned to specific
views – be it boundary vectors, visual cues, cues of any other sensory
modality, or a combination of these. Using richer sensory modalities and/or
fusing complementary modalities reduces the incidence of scene ambiguity and
can increase the robustness of the model.

Further work is being done to incorporate these into the model, as well as
extensions that allow for the learning of multiple maps by the same place cell
network without interference.

## References

  * Alabi et al. [2020] Alabi, A., Minai, A.A., Vanderelst, D., 2020.  One shot spatial learning through replay in a hippocampus-inspired reinforcement learning model, in: 2020 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8.
  * Alabi et al. [2022] Alabi, A., Minai, A.A., Vanderelst, D., 2022.  Context-dependent spatial representations in the hippocampus using place cell dendritic computation, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8.
  * Bains et al. [1999] Bains, J.S., Longacher, J.M., Staley, K.J., 1999.  Reciprocal interactions between ca3 network activity and strength of recurrent collateral synapses.  Nature neuroscience 2, 720–726.
  * Barry et al. [2006] Barry, C., Lever, C., Hayman, R., Hartley, T., Burton, S., O’Keefe, J., Jeffery, K., Burgess, N., 2006\.  The boundary vector cell model of place cell firing and spatial memory.  Reviews in the Neurosciences 17, 71\.
  * Berkvens et al. [2015] Berkvens, R., Weyn, M., Peremans, H., 2015.  Asynchronous, electromagnetic sensor fusion in RatSLAM.  2015 IEEE SENSORS - Proceedings doi:10.1109/ICSENS.2015.7370552.
  * Bloss et al. [2016] Bloss, E.B., Cembrowski, M.S., Karsh, B., Colonell, J., Fetter, R.D., Spruston, N., 2016\.  Structured dendritic inhibition supports branch-selective integration in ca1 pyramidal cells.  Neuron 89, 1016–1030.
  * Booker et al. [2020] Booker, S.A., Harada, H., Elgueta, C., Bank, J., Bartos, M., Kulik, A., Vida, I., 2020.  Presynaptic gabab receptors functionally uncouple somatostatin interneurons from the active hippocampal network.  Elife 9, e51156.
  * Brandeis et al. [1989] Brandeis, R., Brandys, Y., Yehuda, S., 1989.  The use of the morris water maze in the study of memory and learning.  International Journal of Neuroscience 48, 29–69.
  * Burgess and O’Keefe [1996] Burgess, N., O’Keefe, J., 1996\.  Neuronal computations underlying the firing of place cells and their role in navigation.  Hippocampus 6, 749–762.
  * Bush et al. [2014] Bush, D., Barry, C., Burgess, N., 2014.  What do grid cells contribute to place cell firing?  Trends in neurosciences 37, 136–145.
  * Bush et al. [2015] Bush, D., Barry, C., Manson, D., Burgess, N., 2015\.  Using grid cells for navigation.  Neuron 87, 507–520.
  * Buzsáki [1989] Buzsáki, G., 1989.  Two-stage model of memory trace formation: a role for “noisy” brain states.  Neuroscience 31, 551–570.
  * Carr et al. [2011] Carr, M.F., Jadhav, S.P., Frank, L.M., 2011.  Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval.  Nature neuroscience 14, 147\.
  * Chiu et al. [2013] Chiu, C.Q., Lur, G., Morse, T.M., Carnevale, N.T., Ellis-Davies, G.C., Higley, M.J., 2013\.  Compartmentalization of gabaergic inhibition by dendritic spines.  Science 340, 759–762.
  * Derdikman et al. [2009] Derdikman, D., Whitlock, J.R., Tsao, A., Fyhn, M., Hafting, T., Moser, M.B., Moser, E.I., 2009.  Fragmentation of grid cell maps in a multicompartment environment.  Nature neuroscience 12, 1325–1332.
  * Diekelmann and Born [2010] Diekelmann, S., Born, J., 2010\.  The memory function of sleep.  Nature Reviews Neuroscience 11, 114–126.
  * Druckmann et al. [2014] Druckmann, S., Feng, L., Lee, B., Yook, C., Zhao, T., Magee, J.C., Kim, J., 2014\.  Structured synaptic connectivity between hippocampal regions.  Neuron 81, 629–640.
  * D’Hooge and De Deyn [2001] D’Hooge, R., De Deyn, P.P., 2001\.  Applications of the morris water maze in the study of learning and memory.  Brain research reviews 36, 60–90.
  * Epsztein [2022] Epsztein, J., 2022.  Mental replays enable flexible navigation.
  * Erdem and Hasselmo [2012] Erdem, U.M., Hasselmo, M., 2012\.  A goal-directed spatial navigation model using forward trajectory planning based on grid cells.  European Journal of Neuroscience 35, 916–931.
  * Erdem and Hasselmo [2014a] Erdem, U.M., Hasselmo, M.E., 2014a.  A biologically inspired hierarchical goal directed navigation model.  Journal of Physiology - Paris 108, 28–37.
  * Erdem and Hasselmo [2014b] Erdem, U.M., Hasselmo, M.E., 2014b.  A biologically inspired hierarchical goal directed navigation model.  Journal of Physiology-Paris 108, 28–37.
  * Foster and Knierim [2012] Foster, D.J., Knierim, J.J., 2012\.  Sequence learning and the role of the hippocampus in rodent navigation.  Current opinion in neurobiology 22, 294–300.
  * Fuhs et al. [2005] Fuhs, M.C., VanRhoads, S.R., Casale, A.E., McNaughton, B., Touretzky, D.S., 2005.  Influence of path integration versus environmental orientation on place cell remapping between visually identical environments.  Journal of neurophysiology 94, 2603–2616.
  * Fyhn et al. [2008] Fyhn, M., Hafting, T., Witter, M.P., Moser, E.I., Moser, M.B., 2008.  Grid cells in mice.  Hippocampus 18, 1230–1238.
  * Grieves et al. [2018] Grieves, R.M., Duvelle, É., Dudchenko, P.A., 2018.  A boundary vector cell model of place field repetition.  Spatial Cognition & Computation 18, 217–256.
  * Hafting et al. [2005] Hafting, T., Fyhn, M., Molden, S., Moser, M.B., Moser, E.I., 2005.  Microstructure of a spatial map in the entorhinal cortex.  Nature 436, 801–806.
  * Hasselmo and Bower [1991] Hasselmo, M.E., Bower, J.M., 1991\.  Selective suppression of afferent but not intrinsic fiber synaptic transmission by 2-amino-4-phosphonobutyric acid (ap4) in piriform cortex.  Brain research 548, 248–255.
  * Hasselmo and Bower [1992] Hasselmo, M.E., Bower, J.M., 1992\.  Cholinergic suppression specific to intrinsic not afferent fiber synapses in rat piriform (olfactory) cortex.  Journal of neurophysiology 67, 1222–1229.
  * Hasselmo and Bower [1993] Hasselmo, M.E., Bower, J.M., 1993\.  Acetylcholine and memory.  Trends in neurosciences 16, 218–222.
  * Hasselmo and Schnell [1994] Hasselmo, M.E., Schnell, E., 1994\.  Laminar selectivity of the cholinergic suppression of synaptic transmission in rat hippocampal region ca1: computational modeling and brain slice physiology.  Journal of Neuroscience 14, 3898–3914.
  * Hawkins et al. [2019] Hawkins, J., Lewis, M., Klukas, M., Purdy, S., Ahmad, S., 2019.  A framework for intelligence and cortical function based on grid cells in the neocortex.  Frontiers in neural circuits 12, 121\.
  * Jacobson et al. [2015] Jacobson, A., Chen, Z., Milford, M., 2015.  SLAM, Autonomous Multisensor Calibration and Closed-loop Fusion for SlM.  Journal of Field Robotics 32, 85–122.  doi:10.1002/rob, arXiv:10.1.1.91.5767.
  * Johnson and Redish [2005] Johnson, A., Redish, A.D., 2005\.  Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model.  Neural Networks 18, 1163 – 1171.  URL: http://www.sciencedirect.com/science/article/pii/S0893608005001991, doi:https://doi.org/10.1016/j.neunet.2005.08.009. computational Theories of the Functions of the Hippocampus.
  * Klausberger and Somogyi [2008] Klausberger, T., Somogyi, P., 2008\.  Neuronal diversity and temporal dynamics: the unity of hippocampal circuit operations.  Science 321, 53–57.
  * Kleindienst et al. [2011] Kleindienst, T., Winnubst, J., Roth-Alpermann, C., Bonhoeffer, T., Lohmann, C., 2011.  Activity-dependent clustering of functional synaptic inputs on developing hippocampal dendrites.  Neuron 72, 1012–1024.
  * Klinzing et al. [2019] Klinzing, J.G., Niethard, N., Born, J., 2019.  Mechanisms of systems memory consolidation during sleep.  Nature neuroscience 22, 1598–1610.
  * Lever et al. [2009] Lever, C., Burton, S., Jeewajee, A., O’Keefe, J., Burgess, N., 2009.  Boundary vector cells in the subiculum of the hippocampal formation.  Journal of Neuroscience 29, 9771–9777.
  * Losonczy and Magee [2006] Losonczy, A., Magee, J.C., 2006\.  Integrative properties of radial oblique dendrites in hippocampal ca1 pyramidal neurons.  Neuron 50, 291–307.
  * Maingret et al. [2016] Maingret, N., Girardeau, G., Todorova, R., Goutierre, M., Zugaro, M., 2016.  Hippocampo-cortical coupling mediates memory consolidation during sleep.  Nature neuroscience 19, 959–964.
  * McNaughton et al. [2006] McNaughton, B.L., Battaglia, F.P., Jensen, O., Moser, E.I., Moser, M.B., 2006.  Path integration and the neural basis of the’cognitive map’.  Nature Reviews Neuroscience 7, 663–678.
  * Michon et al. [2019] Michon, F., Sun, J.J., Kim, C.Y., Ciliberti, D., Kloosterman, F., 2019.  Post-learning hippocampal replay selectively reinforces spatial memory for highly rewarded locations.  Current Biology 29, 1436–1444.
  * Miles et al. [2014] Miles, R.M., Le Duigou, C., Simonnet, J., Telenczuk, M., Fricker, D., 2014.  Recurrent synapses and circuits in the ca3 region of the hippocampus: an associative network.  Frontiers in cellular neuroscience 7, 262.
  * Milford and Wyeth [2010] Milford, M., Wyeth, G., 2010\.  Persistent navigation and mapping using a biologically inspired slam system.  The International Journal of Robotics Research 29, 1131–1153.
  * Milford and Wyeth [2008] Milford, M.J., Wyeth, G.F., 2008\.  Mapping a suburb with a single camera using a biologically inspired slam system.  IEEE Transactions on Robotics 24, 1038–1053.
  * Milford et al. [2004] Milford, M.J., Wyeth, G.F., Prasser, D., 2004.  Ratslam: a hippocampal model for simultaneous localization and mapping, in: IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04. 2004, IEEE. pp. 403–408.
  * Mnih et al. [2013] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M., 2013.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602 .
  * Mnih et al. [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al., 2015\.  Human-level control through deep reinforcement learning.  nature 518, 529–533.
  * Morris [1984] Morris, R., 1984.  Developments of a water-maze procedure for studying spatial learning in the rat.  Journal of neuroscience methods 11, 47–60.
  * Moser et al. [2008] Moser, E.I., Kropff, E., Moser, M.B., 2008.  Place cells, grid cells, and the brain’s spatial representation system.  Annu. Rev. Neurosci. 31, 69–89.
  * Muller and Kubie [1989] Muller, R.U., Kubie, J.L., 1989\.  The firing of hippocampal place cells predicts the future position of freely moving rats.  Journal of Neuroscience 9, 4101–4110.
  * Muller et al. [1987] Muller, R.U., Kubie, J.L., Ranck, J.B., 1987.  Spatial firing patterns of hippocampal complex-spike cells in a fixed environment.  Journal of Neuroscience 7, 1935–1950.
  * Oja [1982] Oja, E., 1982.  Simplified neuron model as a principal component analyzer.  Journal of mathematical biology 15, 267–273.
  * O’Keefe and Burgess [1996] O’Keefe, J., Burgess, N., 1996\.  Geometric determinants of the place fields of hippocampal neurons.  Nature 381, 425–428.
  * O’Keefe and Conway [1978] O’Keefe, J., Conway, D., 1978\.  Hippocampal place units in the freely moving rat: why they fire where they fire.  Experimental brain research 31, 573–590.
  * O’Keefe and Dostrovsky [1971] O’Keefe, J., Dostrovsky, J., 1971\.  The hippocampus as a spatial map: Preliminary evidence from unit activity in the freely-moving rat.  Brain research .
  * O’keefe and Nadel [1978] O’keefe, J., Nadel, L., 1978\.  The hippocampus as a cognitive map.  Oxford: Clarendon Press.
  * Ólafsdóttir et al. [2018] Ólafsdóttir, H.F., Bush, D., Barry, C., 2018.  The role of hippocampal replay in memory and planning.  Current Biology 28, R37–R50.
  * Papale et al. [2016] Papale, A.E., Zielinski, M.C., Frank, L.M., Jadhav, S.P., Redish, A.D., 2016.  Interplay between hippocampal sharp-wave-ripple events and vicarious trial and error behaviors in decision making.  Neuron 92, 975–982.
  * Poirazi et al. [2003] Poirazi, P., Brannon, T., Mel, B.W., 2003.  Pyramidal neuron as two-layer neural network.  Neuron 37, 989–999.
  * Redish [2016] Redish, A.D., 2016.  Vicarious trial and error.  Nature Reviews Neuroscience 17, 147–159.
  * Redish and Touretzky [1997] Redish, A.D., Touretzky, D.S., 1997\.  Cognitive maps beyond the hippocampus.  Hippocampus 7, 15–35.
  * Samsonovich and McNaughton [1997] Samsonovich, A., McNaughton, B.L., 1997\.  Path integration and cognitive mapping in a continuous attractor neural network model.  Journal of Neuroscience 17, 5900–5920.
  * Sargolini et al. [2006] Sargolini, F., Fyhn, M., Hafting, T., McNaughton, B.L., Witter, M.P., Moser, M.B., Moser, E.I., 2006.  Conjunctive representation of position, direction, and velocity in entorhinal cortex.  Science 312, 758–762.
  * Schmidt et al. [2013] Schmidt, B., Papale, A., Redish, A.D., Markus, E.J., 2013\.  Conflict between place and response navigation strategies: effects on vicarious trial and error (vte) behaviors.  Learning & Memory 20, 130–138.
  * Silver et al. [2017] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al., 2017\.  Mastering chess and shogi by self-play with a general reinforcement learning algorithm.  arXiv preprint arXiv:1712.01815 .
  * Skaggs and McNaughton [1998] Skaggs, W.E., McNaughton, B.L., 1998\.  Spatial firing properties of hippocampal ca1 populations in an environment containing two visually identical regions.  Journal of Neuroscience 18, 8455–8466.
  * Solstad et al. [2006] Solstad, T., Moser, E.I., Einevoll, G.T., 2006.  From grid cells to place cells: a mathematical model.  Hippocampus 16, 1026–1031.
  * Solstad et al. [2014] Solstad, T., Yousif, H.N., Sejnowski, T.J., 2014.  Place cell rate remapping by ca3 recurrent collaterals.  PLoS computational biology 10, e1003648.
  * Spiers et al. [2015] Spiers, H.J., Hayman, R.M., Jovalekic, A., Marozzi, E., Jeffery, K.J., 2015.  Place field repetition and purely local remapping in a multicompartment environment.  Cerebral Cortex 25, 10–25.
  * Steckel et al. [2013] Steckel, J., Boen, A., Peremans, H., 2013.  Broadband 3-D sonar system using a sparse array for indoor navigation.  IEEE Transactions on Robotics 29, 161–171.
  * Sutton and Barto [2018] Sutton, R.S., Barto, A.G., 2018\.  Reinforcement learning: An introduction.  MIT press.
  * Takahashi and Magee [2009] Takahashi, H., Magee, J.C., 2009\.  Pathway interactions and synaptic plasticity in the dendritic tuft regions of ca1 pyramidal neurons.  Neuron 62, 102–111.
  * Takahashi et al. [2012] Takahashi, N., Kitamura, K., Matsuo, N., Mayford, M., Kano, M., Matsuki, N., Ikegaya, Y., 2012.  Locally synchronized synaptic inputs.  Science 335, 353–356.
  * Tang and Hasselmo [1994] Tang, A.C., Hasselmo, M.E., 1994\.  Selective suppression of intrinsic but not afferent fiber synaptic transmission by baclofen in the piriform (olfactory) cortex.  Brain research 659, 75–81.
  * Taube [2007] Taube, J.S., 2007.  The head direction signal: origins and sensory-motor integration.  Annu. Rev. Neurosci. 30, 181–207.
  * Taube et al. [1990] Taube, J.S., Muller, R.U., Ranck, J.B., 1990.  Head-direction cells recorded from the postsubiculum in freely moving rats. i. description and quantitative analysis.  Journal of Neuroscience 10, 420–435.
  * Ulanovsky and Moss [2007] Ulanovsky, N., Moss, C.F., 2007\.  Hippocampal cellular and network activity in freely moving echolocating bats.  Nature neuroscience 10, 224–233.
  * Vorhees and Williams [2006] Vorhees, C.V., Williams, M.T., 2006\.  Morris water maze: procedures for assessing spatial and related forms of learning and memory.  Nature protocols 1, 848\.
  * Wallenstein and Hasselmo [1997] Wallenstein, G.V., Hasselmo, M.E., 1997\.  Gabaergic modulation of hippocampal population activity: sequence learning, place field development, and the phase precession effect.  Journal of neurophysiology 78, 393–408.
  * [81] Webots, .  http://www.cyberbotics.com.  URL: http://www.cyberbotics.com. commercial Mobile Robot Simulation Software.
  * Wyeth and Milford [2009] Wyeth, G., Milford, M., 2009\.  Spatial cognition for robots.  IEEE Robotics & Automation Magazine 16, 24–32.
  * Yang et al. [2016] Yang, G.R., Murray, J.D., Wang, X.J., 2016.  A dendritic disinhibitory circuit mechanism for pathway-specific gating.  Nature communications 7, 1–14.