# Neural Network based Successor Representations of Space and Language

Paul Stoewer Cognitive Computational Neuroscience Group, University Erlangen-
Nuremberg, Germany Pattern Recognition Lab, University Erlangen-Nuremberg,
Germany  Christian Schlieker Cognitive Computational Neuroscience Group,
University Erlangen-Nuremberg, Germany Pattern Recognition Lab, University
Erlangen-Nuremberg, Germany  Achim Schilling Cognitive Computational
Neuroscience Group, University Erlangen-Nuremberg, Germany Neuroscience Lab,
University Hospital Erlangen, Germany  Claus Metzner Neuroscience Lab,
University Hospital Erlangen, Germany  Biophysics Lab, University Erlangen-
Nuremberg, Germany  Andreas Maier Pattern Recognition Lab, University
Erlangen-Nuremberg, Germany  Patrick Krauss Cognitive Computational
Neuroscience Group, University Erlangen-Nuremberg, Germany Pattern Recognition
Lab, University Erlangen-Nuremberg, Germany  Neuroscience Lab, University
Hospital Erlangen, Germany  Linguistics Lab, University Erlangen-Nuremberg,
Germany

###### Abstract

How does the mind organize thoughts? The hippocampal-entorhinal complex is
thought to support domain-general representation and processing of structural
knowledge of arbitrary state, feature and concept spaces. In particular, it
enables the formation of cognitive maps, and navigation on these maps, thereby
broadly contributing to cognition. It has been proposed that the concept of
multi-scale successor representations provides an explanation of the
underlying computations performed by place and grid cells. Here, we present a
neural network based approach to learn such representations, and its
application to different scenarios: a spatial exploration task based on
supervised learning, a spatial navigation task based on reinforcement
learning, and a non-spatial task where linguistic constructions have to be
inferred by observing sample sentences. In all scenarios, the neural network
correctly learns and approximates the underlying structure by building
successor representations. Furthermore, the resulting neural firing patterns
are strikingly similar to experimentally observed place and grid cell firing
patterns. We conclude that cognitive maps and neural network-based successor
representations of structured knowledge provide a promising way to overcome
some of the short comings of deep learning towards artificial general
intelligence.

Keywords:
cognitive maps, successor representations, hippocampus, entorhinal cortex,
spatial navigation, linguistic constructions, mental spaces, neural networks,
artificial intelligence

## Introduction

Cognitive maps are mental representations that serve an organism to acquire,
code, store, recall, and decode information about the relative locations and
features of objects [1]. Electrophysiological research in rodents suggests
that the hippocampus [2] and the entorhinal cortex [3] are the neurological
basis of cognitive maps. There, highly specialised neurons including place [4]
and grid cells [5] support map-like spatial codes, and thus enable spatial
representation and navigation [6], and furthermore the construction of multi-
scale maps [7, 8]. Also human fMRI studies during virtual navigation tasks
have shown that the hippocampal and entorhinal spatial codes, together with
areas in the frontal lobe, enable route planning during navigation [9], e.g.
detours [10], shortcuts or efficient novel routes [11], and in particular
hierarchical spatial planning [12] based on distance preserving
representations [13].

Recent human fMRI studies even suggest that these map-like representations
might not be restricted to physical space, i.e. places and spatial relations,
but also extend to more abstract relations like in social and conceptual
spaces [14, 15, 16], thereby contributing broadly to other cognitive domains
[17], and thus enabling navigation and route planning in arbitrary abstract
cognitive spaces [18].

The hippocampus also plays a crucial role in episodic and declarative memory
[19, 20]. Furthermore, the hippocampal formation, as a hub in brain
connectivity [21], receives highly processed information via direct and
indirect pathways from a large number of multi-modal areas of the cerebral
cortex including language related areas in the frontal, temporal, and parietal
lobe [22]. Finally, some findings indicate that the hippocampus even
contributes to the coding of narrative context [23, 24], and that memory
representations, similar to the internal representation of space,
systematically vary in scale along the hippocampal long axis [25]. This scale
might be used for goal directed navigation with different horizons [26] or
even encode information from smaller episodes to more complex concepts [27].
This geometry can also be modeled in artificial neural networks when
performing an abstraction task [28]. Cognitive maps therefore enable flexible
planning through re-mapping of place cells and through the continuous
(re-)scaling, generalization or detailed representation of information [29].

A number of computational models try to describe the hippocampal-entorhinal
complex. For instance, the Tolman-Eichenbaum Machine describes hippocampal and
entorhinal cell types and allows flexible transfer of structural knowledge
[30]. Another framework that aims to describe the firing patterns of place
cells in the hippocampus uses the successor representation (SR) as a building
block for the construction of cognitive or predictive maps [31, 32]. The
hierarchical structure in the entorhinal cortex can also be modeled by means
of multi-scale successor representations [33]. Here, SR can be for example
learned with a feature set of boundary vector cells [34] or with a sequence
generation model inspired by the entorhinal-hippocampal circuit [35].

To further investigate both the biological plausibility and potential machine
learning applications of multi-scale SR and cognitive maps, we developed a
neural network based simulation of place cell behavior under different
circumstances. In particular, we trained a neural network to learn the SR for
a simulated spatial environment and a navigation task in a virtual maze as
proposed by Alvernhe et al. [36]. In addition, we investigated if the
applicability of our model extends from space to language as the hippocampus
is known to also contribute to language processing [37][38]. Therefore, we
created a model to learn a simplified artificial language. In particular, the
model’s task was to learn the underlying grammatical structure in terms of SR
of words by observing exemplary input sentences only.

## Methods

### Successor Representation

The developed model is based on the principle of the successor representation
(SR). As proposed by Stachenfeld et al. the SR can model the firing patterns
of the place cells in the hippocampus [32]. The SR was originally designed to
build a representation of all possible future rewards $V(s)$ that may be
achieved from each state $s$ within the state space over time [39]. The future
reward matrix $V(s)$ can be calculated for every state in the environment
whereas the parameter $t$ indicates the number of time steps in the future
that are taken into account, and $R(s_{t})$ is the reward for state $s$ at
time $t$. The discount factor $\gamma[0,1]$ reduces the relevance of states
$s_{t}$ that are further in the future relative to the respective initial
state $s_{0}$ (cf. Equation 1).

$\centering
V(s)=E[\sum^{\infty}_{t=0}\gamma^{t}R(s_{t})|s_{0}=s]\@add@centering$ (1)

Here, $E[]$ denotes the expectation value.

The future reward matrix $V(s)$ can be re-factorized using the SR matrix $M$,
which can be computed from the state transition probability matrix $T$ of
successive states (cf. 2). In case of supervised learning, the environments
used for our model operate without specific rewards for each state. For the
calculation of these SR we choose $R(s_{t})=1$ for every state.

$\displaystyle V(s)=\sum_{s^{\prime}}M(s,s^{\prime})R(s^{\prime})$
$\displaystyle M=\sum^{\infty}_{t=0}\gamma^{t}T^{t}$ (2)

### Spatial Environment

The spatial environments created in our framework are designed as discrete
grid-like spatial rooms which can be freely explored by the agent. The
neighboring states of a particular state are defined as direct successor
states. Walls and barriers are not counted as possible successor state for a
neighboring initial state. The squared room from the spatial exploration task
consisted of 100 states arranged as a $10\times 10$ rectangular grid (cf.
Figure 1). The maze from the spatial navigation task was represented as a
$15\times 15$ rectangular grid, whereas only 94 states from all 225 states
were ”allowed” states that could be observed by the agent (cf. Figure 3).

### Language Environment

Additionally, we set up a state space with a non-spatial structure, i.e. a
linguistic environment. The environment consists of 40 discrete states
representing the vocabulary. Each state corresponds to a particular word, and
each word belongs to one of the five different word classes: adjectives,
verbs, nouns, pronouns and question words. The transition probabilities
between subsequent words are defined according to a simplified syntax which
consists of three types of linguistic constructions: an adjective-noun
construction (cf. rule 3a), a descriptive construction (cf. rule 3b) and an
interrogative construction (cf. rule 3c).

$adjective\rightarrow noun$ (3a) $pronoun\rightarrow verb\rightarrow
adjective$ (3b) $question\rightarrow pronoun\rightarrow verb$ (3c)

The syntax rules, i.e. constructions, determine the transition probabilities
for randomly chosen starting states and a word from the picked word group is
set as label for the training data. The individual words from the successor
word class are chosen with equal probability. The constructed sentences have
therefore no particular meaning. For the language data set, 5,000 training
samples were generated and the network was trained for 50 epochs.

### Neural network architectures

To be able to learn the SR by just observing the environment, we set up three-
layered neural networks that learn the transition probabilities of the
different environment. The input to the network is the momentary state encoded
as one-hot vectors. Thus the number of neurons in the input layer is 100 in
the exploration task, 225 in the navigation task and 45 in the linguistic
task.

The hidden layer neurons have a ReLU activation function, whereas the number
of neurons is equal to the corresponding number of input neurons in all
networks. We also tested network architectures with a smaller number of hidden
layer neurons (bottleneck). In all cases, the results where very similar.

The softmax output layer gives a probability distribution for all successor
states. Thus, the number of neurons in the output layer corresponds to the
number of possible states and thus to the number of input neurons in all
networks.

The training data set is created by sampling trajectories through the spatial
structure of the simulated environment. First, a random starting state is
chosen as input and subsequently another random possible successor state is
chosen from its neighbors as desired output. Walls are excluded as input
states. For the first experiment we sampled 50,000 states and successor
states, and trained for 10,000 epochs.

#### Reinforcement learning

In an attampt to reproduce experimental data, we simulated a maze as proposed
by Alvernhe et al. [36]. Furthermore, a reward system is required for
reinforcement learning (RL). Our RL approach enables us to define rewards in
the spatial environments, which we use to simulate the food trays of the
original experiment. The network structure is again a three-layered network,
with a ReLU activation function for the hidden layer neurons and a softmax
output layer, which yields the probabilities for the next actions. A DQN agent
can choose from several actions depending on the number of the neighboring
states belonging to the current state. If the agent chooses a wall state
during training, the momentary training run is terminated and a new random
starting state is chosen randomly. Training was performed for 10,000 epochs.

#### Transition probability and successor representation matrix

After the training process, the network can predict all probabilities of
successor states for any given initial state. Concatenating the predictions of
all states leads to the transition probability (TP) matrix of our
environments, which we use to calculate the SR matrix (cf. Equation 2). In
case of the supervised learning approach (spatial exploration task and
language task), the output of the network is a vector shaped like a row of the
respective environment’s TP or SR matrix and can therefore directly be used to
fill the TP or SR matrix, respectively. The reinforcement learning network
however only yields the probabilities for the direct successors of a given
state, which therefore need to be further extended to a vector containing all
possible states of the environment.

#### Reproducing experimental grid cell firing patterns

After training the network, the resulting SR matrices are evaluated.
Therefore, each state encoded as one-hot vector is fed in as input to the
network, and the resulting softmax output vectors are concatenated to built
the SR matrix. The resulting SR matrix can be used to calculate their
Eigendecompostion. The different Eigenvectors can be ordered according to
their size, and are subsequently reshaped to fit the shape of the
corresponding state space, i.e. simulated environment. The reshaped
Eigenvectors are supposed to form grid like patterns, and to be a
representation of the grid cells’ receptive fields [32].

### Multi-dimensional scaling

A frequently used method to generate low-dimensional embeddings of high-
dimensional data is t-distributed stochastic neighbor embedding (t-SNE) [40].
However, in t-SNE the resulting low-dimensional projections can be highly
dependent on the detailed parameter settings [41], sensitive to noise, and may
not preserve, but rather often scramble the global structure in data [42, 43].
In contrats, multi-Dimensional-Scaling (MDS) [44, 45, 46, 47] is an efficient
embedding technique to visualize high-dimensional point clouds by projecting
them onto a 2-dimensional plane. Furthermore, MDS has the decisive advantage
that it is parameter-free and all mutual distances of the points are
preserved, thereby conserving both the global and local structure of the
underlying data.

When interpreting patterns as points in high-dimensional space and
dissimilarities between patterns as distances between corresponding points,
MDS is an elegant method to visualize high-dimensional data. By color-coding
each projected data point of a data set according to its label, the
representation of the data can be visualized as a set of point clusters. For
instance, MDS has already been applied to visualize for instance word class
distributions of different linguistic corpora [48], hidden layer
representations (embeddings) of artificial neural networks [49, 50], structure
and dynamics of recurrent neural networks [51, 52, 53], or brain activity
patterns assessed during e.g. pure tone or speech perception [54, 48], or even
during sleep [55, 56]. In all these cases the apparent compactness and mutual
overlap of the point clusters permits a qualitative assessment of how well the
different classes separate.

### Code Implementation

The models were coded in Python. The neural networks were design using the
Keras [57] and Keras-RL [58] libraries. Mathematical operations were performed
with numpy [59] and scikit-learn [60] libraries. Visualizations were realised
with matplotlib [61] and networkX [62].

## Results

### Spatial environment

#### Supervised learning reproduces basic firing patterns of place cells in
rodents

In the supervised learning approach, the transition probabilities between
neighboring places and hence the SR is learned by randomly observing places
(states) and exploring their potential successors. In the simplest case of a
2D square environment without any obstacles, the transition probabilities from
any starting place to all its eight neighbors are identical, i,e. uniformly
distributed. Places at the walls (corners) of the room however only have five
(three) neighboring states, so the transition probabilities corresponding to
the remaining neighboring states representing those walls are zero,
respectively. The resulting successor representations learned by the neural
network are almost identical to the the ground truth (cf. Figure 1).
Furthermore, these firing patterns are strikingly similar to those of place
cells in rodents. They reflect the environment’s spatial structure depending
on obstacles and room shape [63], i.e. the intensity of the firing patterns is
centered around the starting position (as this is also the most probable next
state) and directed away from any walls into the open space (cf. Figure 1 1),
as described e.g. in [64], and found experimentally [65].

Figure 1: Supervised learning to explore a spatial environment: The SR for a
2D squared environment is learned with a supervised neural network. The small
green squares indicate two sample starting positions (a, d). The corresponding
SR are calculated and serve as ground truth (b, e). The neural network learns
the transition probabilities for its direct neighbors and estimates the SR for
a sequence length of t=10 (c, f).

#### Eigenvectors of learned SR resemble firing patterns of grid cells in
rodents

Stachenfeld et al. [64] propose that the grid-like firing patterns of grid
cells in the entorhinal cortex of rodents [66] may be explained by an
Eigendecomposition of the SR matrix, whereas each individual grid cell would
correspond to one Eigenvector. To test this assumption in the context of our
framework, we calculated the Eigenvectors of the learned SR matrix as shown in
Figure 1, and reshaped them to the shape of the environment. We find that this
procedure actually leads to grid cell-like firing patterns (cf. Figure 2). The
first 30 Eigenvectors ordered by increasing value of the corresponding
Eigenvalues are shown in Figure 2. As known from neurobiology [66], the grid-
like patterns vary in orientation and mesh size (i.e. frequency). In
particular, the smaller the corresponding Eigenvalue of the Eigenvector, the
smaller the mesh size, i.e. more fine-grained the resulting grid, becomes (cf.
Figure 2). Furthermore it is known that, the individual orientation of the
grid cells’ firing patterns follows no particular order, whereas the mesh size
of the grids varies systematically along the long axis of the entorhinal
cortex [67]. This feature, especially, is thought to enable multi-scale
mapping, route planing and navigation [33].

Figure 2: Grid cell-like Eigenvectors of the SR matrix: The firing patterns of
grid cells in the entorhinal cortex are proposed to represent the Eigenvectors
of the SR matrix [32]. For the squared room depicted in Figure 1, the
Eigenvectors for the first 30 Eigenvalues of the SR matrix are shown (re-
shaped to the shape of the squared environment). Indeed, they resemble grid
cell-like firing patterns. Furthermore, the grids vary in orientation and
scaling, as observed in electrophysiological experiments in rodents.

#### Reinforcement learning reproduces basic firing patterns of place cells in
rodents

In contrast to a goal-free random walk in order to explore a novel environment
(as in the previous setting), navigation is usually driven by a specific goal
or reward [68] like, e.g. food. The task is therefore ideally suited for
reinforcement learning (RL) [69]. In our simulation, we reproduced a classical
rodent maze experiment presented by Alvernhe et al. [36]. As in the supervised
learning setting, the successor representations learned in the RL setting are
very similar to the ground truth (cf. Figure 3). Also, the resulting place
cell firing patterns closely resemble those of place cells in rodents during
maze navigation tasks. The SR place fields are clearly different from those
obtained without any reward in the goal-free exploration task (cf. Figure 1).
A position close to a reward state is associated with highly localized firing
patterns, whereas the highest successor probabilities are directed towards the
reward states. Furthermore, places in the middle of the maze are associated
with firing patterns that are stretched parallel to the orientation of the
maze’s main corridor. Highest successor probabilities are localized around the
starting position, but still also in reach of the reward states. The side arms
of the maze are a detour to the goal (reward position), and are therefore
mainly ignored by the network, i.e. associated with the lowest successor state
probabilities.

Figure 3: Reinforcement learning to navigate a spatial environment: We
reproduced the rat maze experiment published by Alvernhe et al. [36] (left
column). Therefore, we simulated the corresponding maze environment, small
green squares indicate three different sample starting positions (second
column). Based on the transition probabilities to neighboring states we
calculated the successor representation of the maze as ground truth (third
column). The predicted SR of the trained network, i.e. the firing patterns of
the artificial place cells are very similar to the underlying ground truth
(right column).

### Linguistic structures

#### Linguistic constructions define a network-like linguistic map

Cognitive maps are however not restricted to physical space. On the contrary,
cognitive maps may also be applied to arbitrary abstract and complex state
spaces. In general, any state space can be represented as a graph. In this
case, nodes correspond to states, and edges to state transitions. A prime
example of such graph-like (or network-like) state space representations is
language. In cognitive linguistics, there is an overall agreement on the fact,
that language is represented as a network in the human mind [70, 71, 72, 73,
74, 75, 76], whereas the nodes correspond to linguistic units at different
hierarchical levels from phonemes, through words, to idioms and abstract
argument structure constructions [70]. In particular, ”the nodes at one level
of analysis are networks at another level of analysis” [77]. Hence, multi-
scale SR [33] appears to be an ideal theoretical framework to explain language
representation and processing in the human mind, whereas the systematically
varying grid-scale along the long axis of the entorhinal cortex [67] might
explain its implementation in the human brain. To investigate this hypothesis,
we constructed as a first step a simplified language as described in detail in
the Methods section. The lexicon together with the three linguistic
constructions result in a network-like linguistic map (cf. Figure 4), that has
to be learned by the neural network.

Figure 4: Network-like map of linguistic constructions: The simplified
language model consists of five different word classes and three linguistic
constructions defining allowed word class transitions. The word transition
matrix can be visualized as a graph or network-like map, whereas each word
corresponds to a node, and edges represent possible word transitions.
Different node colors indicate different word classes. Note that, edges for
transition probabilities smaller than $10^{-4}$ are not shown for better
readability.

#### The neural network learns state TP and SR matrices

The learned behaviour of the network in the state space can be displayed as a
state TP or SR matrix. After training, the TP matrix which is predicted by the
network is very similar to the ground truth (cf. Figure 5a,c). However the
network also predicts adjectives (states 0-10) as successors of nouns (states
20-30), even though this transition does not explicitly exist in the three
pre-defined constructions. Consequently, the network’s SR matrix is also
slightly different from the ground truth (cf. Figure 5b,d).

Figure 5: Word transition probability and word successor representation
matrices: After training on the linguistic data, the neural network predicts
the word transition probabilities (TP) and the word successor representations
(SR). As ground truth, the calculated TP matrix (a) and the SR matrix (b) for
$t=2$ and $\gamma=1$ are shown. The corresponding predictions learned by the
network are very similar to the ground truth in both cases (c, d). States
(0-39) correspond to words. 0-9: adjectives, 10-19: verbs, 20-29: nouns,
30-34: pronouns, 35-40: question words.

#### Word classes spontaneously emerge as clusters in the TP and SR vector
space

The transition probabilities from a given word to all other 40 words (rows in
the TP matrix), as well as the corresponding successor probabilities (rows in
the SR matrix) can be represented as vectors, and hence may be interpreted as
points in a 40-dimensional TP or SR space respectively, whereas each word
corresponds to a particular point. In order to further investigate the
properties of these high-dimensional representations, we visualize both the TP
and the SR space using multi-dimensional scaling. In particular, the
40-dimensional TP and SR vector representations of each word are projected
onto a two-dimensional plane as described in detail in the Methods section. By
color-coding each word according to its word class, we observed putative
clustering of the vocabulary. Remarkably, the words actually cluster according
to their word classes (cf. Figures 6 and 7), even though, this information was
not provided (e.g. as an additional label for each word) to the neural network
at any time during training.

Figure 6: MDS of the word transition probability vectors: A two-dimensional
projection of the 40-dimensional word TP vectors (rows in TP matrix) for the
calculated ground truth (a) and the learned TP matrix (b). In both cases, the
words build clearly separated, dense clusters according to the respective word
class. Different colors correspond to word classes. Note that, scaling of the
axes is in arbitrary units since coordinates have no particular meaning other
than indicating the relative positions of the projected vectors. Figure 7: MDS
of the word successor representation vectors: A two-dimensional projection of
the 40-dimensional word SR vectors (rows in SR matrix) for the calculated
ground truth (a) and the learned SR matrix (b). In both cases, the words
cluster according to the respective word class. Different colors correspond to
word classes again. However, the resulting clusters are less dense and located
closer to each other than for the TP vectors. Since SR vectors cover several
time steps, whereas TP vectors only cover a single time step in the future,
this result is intuitive. Note that, scaling of the axes is in arbitrary units
since coordinates have no particular meaning other than indicating the
relative positions of the projected vectors.

## Discussion

In this study, we demonstrated that efficient successor representations can be
learned by artificial neural networks in different scenarios. The emerging
representations share important properties with network-like cognitive maps,
enabling e.g. navigation in arbitrary abstract and conceptual spaces, and
thereby broadly supporting domain-general cognition, as proposed by Bellmund
et al. [18].

In particular, we created a model, which can learn the SR for spatial and non-
spatial environments. The model successfully reproduced experimentally
observed firing patterns of place and grid cells in simulated spatial
environments in two different scenarios. First, an exploration task based on
supervised learning in a squared room without any obstacles, and second, a
navigation task based on reinforcement learning in a simulated maze.
Furthermore our neural network model learned the underlying word classes of a
simplified artificial language framework just by observing sequences of words.

The involvement of the entorhinal-hippocampal complex – as being the most
probable candidate structure underlying network-like cognitive maps and multi-
scale navigation [64, 16, 15, 29, 33] – in language processing has already
been experimentally demonstrated [37, 38]. Our study further supports, in
particular, the involvement of place cells, as being the nodes of the
”language network” as suggested in cognitive linguistics [70, 71, 72, 73, 74,
75, 76]. Early language acquisition, especially, is driven by passive
listening [78] and implicit learning [79]. Our model replicates learning by
listening and therefore resembles a realistic scenario.

The varying grid cell scaling along the long axis of the entorhinal cortex is
known to be associated with hierarchical memory content [25]. The Eigenvectors
of the SR matrix are strikingly similar to the firing patterns of grid cells,
and therefore provide a putative explanation of the computational mechanisms
underlying grid cell coding. These multi-scale representations are perfectly
suited to map hierarchical linguistic structures from phonemes through words
to sentences, and even beyond, like e.g. events or entire narratives. Indeed,
recent neuroimaging studies provide evidence for the existence of ”event
nodes” in the human hippocampus [23].

Since our neural network model is able learn the underlying structure of a
simplified language, we speculate that also the human hippocampal-entorhinal
complex similarly encodes the complex linguistic structures of the languages
learned by a given individual. Therefore learning further languages of similar
structure as previously learned languages might be easier due to the fact that
the multi-scale representations and cognitive maps in the hippocampus can be
more easily transferred and re-mapped [16, 15], i.e. re-used, in major parts
to the new language.

Whether the hippocampus is actually involved in multi-scale representation and
processing of linguistic structures across several hierarchies needs to be
verified experimentally and theoretically. Neuroimaging studies during natural
language perception and production, like for instance listening to audiobooks
[48], need to be performed. Only continuous, connected speech and language
provides such corpus-like rich linguistic structures, being crucial to assess
putative multi-scale processing. Additionally, further theoretical studies are
needed to extend the presented model, and to apply it to more complex and
naturalistic linguistic tasks, like e.g. word prediction in a natural language
scenario.

As recently suggested, the neuroscience of spatial navigation might be of
particular importance for artificial intelligence research [80]. A neural
network implementation of hippocampal successor representations, especially,
promises advances in both fields. Following the research agenda of Cognitive
Computational Neuroscience proposed by Kriegeskorte et al. [81], neuroscience
and cognitive science benefit from such models by gaining deeper understanding
of brain computations [82, 83, 50]. Conversely, for artificial intelligence
and machine learning, neural network-based multi-scale successor
representations to learn and process structural knowledge (as an example of
neuroscience-inspired artificial intelligence [84]), might be a further step
to overcome the limitations of contemporary deep learning [85, 86, 87, 88] and
towards human-level artificial general intelligence.

## Acknowledgments

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation): grant KR5148/2-1 to PK – project number 436456810.

## Author contributions

PS and CS performed computer simulations. PK and AM designed and supervised
the study. All authors discussed the results and wrote the manuscript.

## Competing interests

The authors declare no competing financial interests.

## References

  * [1] Edward C Tolman.  Cognitive maps in rats and men.  Psychological review, 55(4):189, 1948.
  * [2] John O’keefe and Lynn Nadel.  The hippocampus as a cognitive map.  Oxford university press, 1978.
  * [3] Edvard I Moser, May-Britt Moser, and Bruce L McNaughton.  Spatial representation in the hippocampal formation: a history.  Nature neuroscience, 20(11):1448–1464, 2017.
  * [4] John O’Keefe and Jonathan Dostrovsky.  The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat.  Brain research, 1971.
  * [5] Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I Moser.  Microstructure of a spatial map in the entorhinal cortex.  Nature, 436(7052):801–806, 2005.
  * [6] Edvard I Moser, Emilio Kropff, and May-Britt Moser.  Place cells, grid cells, and the brain’s spatial representation system.  Annu. Rev. Neurosci., 31:69–89, 2008.
  * [7] Maya Geva-Sagiv, Liora Las, Yossi Yovel, and Nachum Ulanovsky.  Spatial cognition in bats and rats: from sensory acquisition to multiscale maps and navigation.  Nature Reviews Neuroscience, 16(2):94–108, 2015.
  * [8] Lukas Kunz, Shachar Maidenbaum, Dong Chen, Liang Wang, Joshua Jacobs, and Nikolai Axmacher.  Mesoscopic neural representations in spatial navigation.  Trends in cognitive sciences, 23(7):615–630, 2019.
  * [9] Hugo J Spiers and Eleanor A Maguire.  Thoughts, behaviour, and brain dynamics during navigation in the real world.  Neuroimage, 31(4):1826–1840, 2006.
  * [10] Hugo J Spiers and Sam J Gilbert.  Solving the detour problem in navigation: a model of prefrontal and hippocampal interactions.  Frontiers in human neuroscience, 9:125, 2015.
  * [11] Tom Hartley, Eleanor A Maguire, Hugo J Spiers, and Neil Burgess.  The well-worn route and the path less traveled: distinct neural bases of route following and wayfinding in humans.  Neuron, 37(5):877–888, 2003.
  * [12] Jan Balaguer, Hugo Spiers, Demis Hassabis, and Christopher Summerfield.  Neural mechanisms of hierarchical planning in a virtual subway network.  Neuron, 90(4):893–903, 2016.
  * [13] Lindsay K Morgan, Sean P MacEvoy, Geoffrey K Aguirre, and Russell A Epstein.  Distances between real-world locations are represented in the human hippocampus.  Journal of Neuroscience, 31(4):1238–1245, 2011.
  * [14] Russell A Epstein, Eva Zita Patai, Joshua B Julian, and Hugo J Spiers.  The cognitive map in humans: spatial navigation and beyond.  Nature neuroscience, 20(11):1504–1513, 2017.
  * [15] Seongmin A Park, Douglas S Miller, and Erie D Boorman.  Inferences on a multidimensional social hierarchy use a grid-like code.  bioRxiv, pages 2020–05, 2021.
  * [16] Seongmin A Park, Douglas S Miller, Hamed Nili, Charan Ranganath, and Erie D Boorman.  Map making: Constructing, combining, and inferring on abstract cognitive maps.  BioRxiv, page 810051, 2020.
  * [17] Daniela Schiller, Howard Eichenbaum, Elizabeth A Buffalo, Lila Davachi, David J Foster, Stefan Leutgeb, and Charan Ranganath.  Memory and space: towards an understanding of the cognitive map.  Journal of Neuroscience, 35(41):13904–13911, 2015.
  * [18] Jacob LS Bellmund, Peter Gärdenfors, Edvard I Moser, and Christian F Doeller.  Navigating cognition: Spatial codes for human thinking.  Science, 362(6415), 2018.
  * [19] Endel Tulving and Hans J Markowitsch.  Episodic and declarative memory: role of the hippocampus.  Hippocampus, 8(3):198–204, 1998.
  * [20] Leila Reddy, Benedikt Zoefel, Jessy K Possel, Judith Peters, Doris E Dijksterhuis, Marlene Poncet, Elisabeth CW van Straaten, Johannes C Baayen, Sander Idema, and Matthew W Self.  Human hippocampal neurons track moments in a sequence of events.  Journal of Neuroscience, 41(31):6714–6725, 2021.
  * [21] Francesco P Battaglia, Karim Benchenane, Anton Sirota, Cyriel MA Pennartz, and Sidney I Wiener.  The hippocampus: hub of brain network communication for memory.  Trends in cognitive sciences, 15(7):310–318, 2011.
  * [22] Gregory Hickok and David Poeppel.  Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language.  Cognition, 92(1-2):67–99, 2004.
  * [23] Branka Milivojevic, Meryl Varadinov, Alejandro Vicente Grabovetsky, Silvy HP Collin, and Christian F Doeller.  Coding of event nodes and narrative context in the hippocampus.  Journal of Neuroscience, 36(49):12412–12424, 2016.
  * [24] Neal W Morton and Alison R Preston.  Concept formation as a computational cognitive process.  Current Opinion in Behavioral Sciences, 38:83–89, 2021.
  * [25] Silvy HP Collin, Branka Milivojevic, and Christian F Doeller.  Memory hierarchies map onto the hippocampal long axis in humans.  Nature neuroscience, 18(11):1562–1564, 2015.
  * [26] Iva K Brunec and Ida Momennejad.  Predictive representations in hippocampal and prefrontal hierarchies.  bioRxiv, page 786434, 2019.
  * [27] Branka Milivojevic and Christian F Doeller.  Mnemonic networks in the hippocampal formation: From spatial maps to temporal and conceptual codes.  Journal of Experimental Psychology: General, 142(4):1231, 2013.
  * [28] Silvia Bernardi, Marcus K Benna, Mattia Rigotti, Jérôme Munuera, Stefano Fusi, and C Daniel Salzman.  The geometry of abstraction in the hippocampus and prefrontal cortex.  Cell, 183(4):954–967, 2020.
  * [29] Ida Momennejad.  Learning structures: Predictive representations, replay, and generalization.  Current Opinion in Behavioral Sciences, 32:155–166, 2020.
  * [30] James CR Whittington, Timothy H Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens.  The tolman-eichenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation.  Cell, 183(5):1249–1263, 2020.
  * [31] Kimberly L Stachenfeld, Matthew Botvinick, and Samuel J Gershman.  Design principles of the hippocampal cognitive map.  Advances in neural information processing systems, 27:2528–2536, 2014.
  * [32] Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman.  The hippocampus as a predictive map.  Nature neuroscience, 20(11):1643, 2017.
  * [33] Ida Momennejad and Marc W. Howard.  Predicting the future with multi-scale successor representations.  bioRxiv, 2018.
  * [34] William De Cothi and Caswell Barry.  Neurobiological successor features for spatial navigation.  BioRxiv, page 789412, 2019.
  * [35] Daniel C McNamee, Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman.  Flexible modulation of sequence generation in the entorhinal–hippocampal system.  Nature Neuroscience, 24(6):851–862, 2021.
  * [36] Alice Alvernhe, Etienne Save, and Bruno Poucet.  Local remapping of place cell firing in the tolman detour task.  The European journal of neuroscience, 33:1696–705, 03 2011.
  * [37] Vitória Piai, Kristopher L. Anderson, Jack J. Lin, Callum Dewar, Josef Parvizi, Nina F. Dronkers, and Robert T. Knight.  Direct brain recordings reveal hippocampal rhythm underpinnings of language processing.  Proceedings of the National Academy of Sciences of the United States of America, 113:11366–11371, 2016.
  * [38] Natalie V. Covington and Melissa C. Duff.  Expanding the language network: Direct contributions from the hippocampus.  Trends in cognitive sciences, 20:869–870, December 2016.
  * [39] Peter Dayan.  Improving Generalization for Temporal Difference Learning: The Successor Representation.  Neural Computation, 5(4):613–624, 07 1993.
  * [40] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [41] Martin Wattenberg, Fernanda Viégas, and Ian Johnson.  How to use t-sne effectively.  Distill, 1(10):e2, 2016.
  * [42] Catalina A Vallejos.  Exploring a world of a thousand dimensions.  Nature biotechnology, 37(12):1423–1424, 2019.
  * [43] Kevin R Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B Burkhardt, William S Chen, Kristina Yim, Antonia van den Elzen, Matthew J Hirn, Ronald R Coifman, et al.  Visualizing structure and transitions in high-dimensional biological data.  Nature biotechnology, 37(12):1482–1492, 2019.
  * [44] Warren S Torgerson.  Multidimensional scaling: I. theory and method.  Psychometrika, 17(4):401–419, 1952.
  * [45] Joseph B Kruskal.  Nonmetric multidimensional scaling: a numerical method.  Psychometrika, 29(2):115–129, 1964.
  * [46] Joseph B Kruskal.  Multidimensional scaling.  Number 11. Sage, 1978.
  * [47] Michael AA Cox and Trevor F Cox.  Multidimensional scaling.  In Handbook of data visualization, pages 315–347. Springer, 2008\.
  * [48] Achim Schilling, Rosario Tomasello, Malte R Henningsen-Schomers, Alexandra Zankl, Kishore Surendra, Martin Haller, Valerie Karl, Peter Uhrig, Andreas Maier, and Patrick Krauss.  Analysis of continuous neuronal activity evoked by natural speech with computational corpus linguistics methods.  Language, Cognition and Neuroscience, 36(2):167–186, 2021.
  * [49] Achim Schilling, Andreas Maier, Richard Gerum, Claus Metzner, and Patrick Krauss.  Quantifying the separability of data classes in neural networks.  Neural Networks, 139:278–293, 2021.
  * [50] Patrick Krauss, Claus Metzner, Nidhi Joshi, Holger Schulze, Maximilian Traxdorf, Andreas Maier, and Achim Schilling.  Analysis and visualization of sleep stages based on deep neural networks.  Neurobiology of sleep and circadian rhythms, 10:100064, 2021.
  * [51] Patrick Krauss, Alexandra Zankl, Achim Schilling, Holger Schulze, and Claus Metzner.  Analysis of structure and dynamics in three-neuron motifs.  Frontiers in Computational Neuroscience, 13:5, 2019.
  * [52] Patrick Krauss, Karin Prebeck, Achim Schilling, and Claus Metzner.  Recurrence resonance” in three-neuron motifs.  Frontiers in computational neuroscience, page 64, 2019.
  * [53] Patrick Krauss, Marc Schuster, Verena Dietrich, Achim Schilling, Holger Schulze, and Claus Metzner.  Weight statistics controls dynamics in recurrent neural networks.  PloS one, 14(4):e0214541, 2019.
  * [54] Patrick Krauss, Claus Metzner, Achim Schilling, Konstantin Tziridis, Maximilian Traxdorf, Andreas Wollbrink, Stefan Rampp, Christo Pantev, and Holger Schulze.  A statistical method for analyzing and comparing spatiotemporal cortical activation patterns.  Scientific reports, 8(1):1–9, 2018.
  * [55] Patrick Krauss, Achim Schilling, Judith Bauer, Konstantin Tziridis, Claus Metzner, Holger Schulze, and Maximilian Traxdorf.  Analysis of multichannel eeg patterns during human sleep: a novel approach.  Frontiers in human neuroscience, 12:121, 2018.
  * [56] Maximilian Traxdorf, Patrick Krauss, Achim Schilling, Holger Schulze, and Konstantin Tziridis.  Microstructure of cortical activity during sleep reflects respiratory events and state of daytime vigilance.  Somnologie, 23(2):72–79, 2019.
  * [57] François Chollet et al.  Keras, 2015.
  * [58] Matthias Plappert.  keras-rl, 2016.
  * [59] Charles R. Harris, K. Jarrod Millman, St’efan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern’andez del R’ıo, Mark Wiebe, Pearu Peterson, Pierre G’erard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  Nature, 585(7825):357–362, September 2020.
  * [60] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [61] J. D. Hunter.  Matplotlib: A 2d graphics environment.  Computing in Science & Engineering, 9(3):90–95, 2007.
  * [62] Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart.  Exploring network structure, dynamics, and function using networkx.  In Gaël Varoquaux, Travis Vaught, and Jarrod Millman, editors, Proceedings of the 7th Python in Science Conference, pages 11 – 15, Pasadena, CA USA, 2008.
  * [63] Julija Krupic, Marius Bauza, Stephen Burton, Caswell Barry, and John O’Keefe.  Grid cell symmetry is shaped by environmental geometry.  Nature, 518, 2015.
  * [64] Kimberly L. Stachenfeld, Matthew M. Botvinick, and Samuel J. Gershman.  The hippocampus as a predictive map.  Nature Neuroscience, 21(6):895–895, 2017.
  * [65] John O’Keefe and Neil Burgess.  Geometric determinants of the place fields hippocampal neurons.  Nature, 381:425–8, 06 1996.
  * [66] Marianne Fyhn, Torkel Hafting, Menno P. Witter, Edvard I. Moser, and May-Britt Moser.  Grid cells in mice.  Hippocampus, 18(12):1230–1238, 2008.
  * [67] Vegard Heimly Brun, Trygve Solstad, Kirsten Brun Kjelstrup, Marianne Fyhn, Menno P Witter, Edvard I Moser, and May-Britt Moser.  Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex.  Hippocampus, 18(12):1200–1212, 2008.
  * [68] Richard C Gerum, André Erpenbeck, Patrick Krauss, and Achim Schilling.  Sparsity through evolutionary pruning prevents neuronal networks from overfitting.  Neural Networks, 128:305–312, 2020.
  * [69] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [70] Adele E Goldberg.  Constructions: A construction grammar approach to argument structure.  University of Chicago Press, 1995.
  * [71] Adele E Goldberg.  Constructions: A new theoretical approach to language.  Trends in cognitive sciences, 7(5):219–224, 2003.
  * [72] Adele E Goldberg.  Constructions at work: The nature of generalization in language.  Oxford University Press on Demand, 2006.
  * [73] Adele Goldberg and Adele E Goldberg.  Explain me this.  Princeton University Press, 2019.
  * [74] Holger Diessel, Ewa Dabrowska, and Dagmar Divjak.  Usage-based construction grammar.  Cognitive linguistics, 2:50–80, 2019.
  * [75] Holger Diessel.  A dynamic network approach to the study of syntax.  Frontiers in Psychology, page 3196, 2020.
  * [76] Natalia Levshina.  The grammar network: How linguistic structure is shaped by language use by holger diessel.  Language, 97(4):825–830, 2021.
  * [77] Holger Diessel.  The grammar network.  Cambridge University Press, 2019.
  * [78] Patricia K. Kuhl.  Brain Mechanisms in Early Language Acquisition.  Neuron, 67:713–727, 2010.
  * [79] Ewa Dabrowska.  Implicit lexical knowledge.  Linguistics, 52(1):205–223, 2014.
  * [80] Edgar Bermudez-Contreras, Benjamin J Clark, and Aaron Wilber.  The neuroscience of spatial navigation and the relationship to artificial intelligence.  Frontiers in Computational Neuroscience, 14:63, 2020.
  * [81] Nikolaus Kriegeskorte and Pamela K Douglas.  Cognitive computational neuroscience.  Nature neuroscience, 21(9):1148–1160, 2018.
  * [82] Achim Schilling, Richard Gerum, Alexandra Zankl, Holger Schulze, Claus Metzner, and Patrick Krauss.  Intrinsic noise improves speech recognition in a computational model of the auditory pathway.  bioRxiv, 2020.
  * [83] Patrick Krauss, Konstantin Tziridis, Achim Schilling, and Holger Schulze.  Cross-modal stochastic resonance as a universal principle to enhance sensory processing.  Frontiers in neuroscience, 12:578, 2018.
  * [84] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick.  Neuroscience-inspired artificial intelligence.  Neuron, 95(2):245–258, 2017.
  * [85] Gary Marcus.  Deep learning: A critical appraisal.  arXiv preprint arXiv:1801.00631, 2018.
  * [86] Zijin Yang, Achim Schilling, Andreas Maier, and Patrick Krauss.  Neural networks with fixed binary random projections improve accuracy in classifying noisy data.  In Bildverarbeitung für die Medizin 2021, pages 211–216. Springer, 2021.
  * [87] Richard C Gerum and Achim Schilling.  Integration of leaky-integrate-and-fire neurons in standard machine learning architectures to generate hybrid networks: A surrogate gradient approach.  Neural Computation, 33(10):2827–2852, 2021.
  * [88] Andreas K Maier, Christopher Syben, Bernhard Stimpel, Tobias Würfl, Mathis Hoffmann, Frank Schebesch, Weilin Fu, Leonid Mill, Lasse Kling, and Silke Christiansen.  Learning with known operators reduces maximum error bounds.  Nature machine intelligence, 1(8):373–380, 2019.