# aypoint Exploration using a Single Demonstration

Mara Levy, Nirat Saini, and Abhinav Shrivastava
University of Maryland, College Park

###### Abstract

We propose WayEx, a new method for learning complex goal-conditioned robotics
tasks from a single demonstration. Our approach distinguishes itself from
existing imitation learning methods by demanding fewer expert examples and
eliminating the need for information about the actions taken during the
demonstration. This is accomplished by introducing a new reward function and
employing a knowledge expansion technique. We demonstrate the effectiveness of
ur waypoint exploration strategy, across six diverse tasks, showcasing its
applicability in various environments. Notably, our method significantly
reduces training time by $\sim$50% as compared to traditional reinforcement
learning methods. WayEx obtains a higher reward than existing imitation
learning methods given only a single demonstration. Furthermore, we
demonstrate its success in tackling complex environments where standard
approaches fall short. Appendix is available at: https://waypoint-
ex.github.io.

##  I INTRODUCTION

Humans have a natural ability to learn tasks by observing a single
demonstration which they can follow step-by-step. For instance, we can watch a
video and grasp how to open a vault, then practice until we succeed without
requiring further instructions. Drawing inspiration from this ability, the
combination of learning from demonstrations and reinforcement learning
techniques has become a popular and potent approach for training robots [1, 2,
3, 4]. However, compared to humans who can learn simple tasks from a single
demonstration, robots require a multitude of diverse expert instances. For
example, to learn how to open a vault, the robot must observe successful
demonstrations for different views and locations of the vault handle with
respect to the robot’s location. Moreover, each demonstration must contain
information about the location of the vault (state), the precise joint
rotations (action) to reach the vault, and knowledge about how close it is to
completing the task (reward). Hence, most common methods in learning from
demonstrations, such as Imitation learning [5, 6, 4] and Inverse reinforcement
learning [7, 8, 9] require a set of expert demonstrations, with a defined
state, action and reward space. Collecting all the data in real time and
computing the definitive action and reward space is impractical and
inefficient. Therefore, in this work, we strive to reduce these inefficiencies
by using only a single demonstration for training. Additionally, our setup
does not require knowledge about the action space, relying on just the state
space and a corresponding reward function.

Figure 1: A comparison of our approach to general imitation learning
techniques. (a) Traditional Imitation learning approaches require multiple
expert trajectories with a known action space for training (4 shown here). (b)
For our proposed method e use only one expert trajectory, and expand knowledge
from this one trajectory to learn how to solve the task. During training with
a single initial state ($s_{0}$) and a single goal state ($g_{0}$), our model
learns to navigate back to the expert trajectory from points that are not part
of the trajectory (all dotted states, which can be a combination of 4 expert
trajectories shown on the top). This enables the model to successfully reach
the goal state. We further introduce additional start and goal states
([$s_{1},g_{1}$],[ $s_{2},g_{2}$], [$s_{3},g_{3}$]).

Prior works identify key states (waypoints [6, 5]) along the robot’s
trajectory, to help it navigate towards the goal. We also employ waypoints to
solve the task, without having access to the action space. Each observation
within the demonstration is defined as an individual waypoint. While other
approaches [6] need access to waypoints during the testing phase, nly requires
waypoints during training. This prevents the need for training a model to
predict the waypoints during inference. We leverage the waypoints during
training via an augmented reward structure based on the known Q-Values [10]
associated with each waypoint.

A common approach to solving a task without a dataset of expert trajectories
is to use dense rewards, based on the key steps of a task. Existing studies in
the field of reinforcement learning acknowledge that employing dense rewards
is difficult since it requires practitioners to engineer specific reward
functions for each task. Additionally these rewards, if ill-designed, can lead
to unforeseen behavior. To circumvent these challenges, we assume a sparse
reward structure when determining the new reward. With sparse rewards, the
robot receives a reward of 0 if the actions lead to a goal state; otherwise,
the reward is -1. Acquiring this reward solely through the process of
exploration can prove to be highly challenging, making certain tasks
unachievable with sparse rewards alone [11]. Our approach strikes a balance,
allowing the model to receive frequent rewards without exposing it to the
typical risks associated with dense rewards.

By utilizing this new reward structure, our model can solve tasks that closely
resemble the demonstration setup. However, despite its ability to learn from a
single example, our approach still encounters a common limitation of learning
from demonstrations. If the robot encounters a state beyond the scope of what
it has previously encountered it will not know how to proceed. To overcome
this challenge, the robot needs to acquire experience beyond the confines of
the provided demonstrated space. A commonly employed method to achieve this is
to integrate learning from demonstrations with conventional model-free
reinforcement learning algorithms [12, 13]. Strict model-free reinforcement
learning involves learning optimal state-action pairs through trial and error
rather than relying on expert trajectories. We combine model-free
reinforcement learning with our waypoint reward and introduce an additional
expansion method that further enhances the model’s knowledge.

In this work, we propose erived from Waypoint Exploration, a novel approach
that enables the training of a reinforcement learning model using a single
expert demonstration and without any prior knowledge of the action space. It
can serve as a wrapper around any reinforcement learning algorithm,
facilitating its applicability as the field advances. Our primary technical
contributions are: (1) the introduction of a new reward function based on
sparse rewards, which provides additional rewards without introducing
unforeseen consequences, and (2) a method for expanding knowledge beyond a
single demonstration to encompass the entire spectrum of both the state and
goal spaces. We demonstrate that our approach enables faster learning of tasks
compared to previous reinforcement learning methods while requiring minimal
additional information.

##  II RELATED WORK

Goal-Conditioned Reinforcement Learning. We are interested in investigating
tasks that involve a robot reaching a specific end state specified by an
initial “goal.” In the reinforcement learning (RL) community, these problems
are known as Goal-Conditioned Tasks. Prior works have studied how to use RL in
many different ways in order to solve these tasks [14, 15, 16, 17, 18, 19].
Early works such as [15, 18] show that it is possible to use standard RL
methods such as [12, 13, 20], but it can be time-consuming, and there are some
types of tasks that these methods alone cannot solve. To combat this, other
works have suggested the use of hindsight re-labeling [14, 21], which speeds
up the process, but still requires a large amount of data to reach a
successful trajectory [15].

Figure 2: Visualization of Grid World Toy Example. (a) shows the environment
setup with a sparse reward. (b) shows the reward for each state once the
entire environment has been solved using the bellman equation [22]. (c)
represents here with a single demonstration, we compute a close approximation
of the reward for each state along the path to the goal.

Imitation Learning. Learning from demonstrations, also known as imitation
learning, is a common approach for solving goal conditioned tasks. Our
approach uses ideas similar to prior works such as [5, 4, 2, 3, 1, 23]. These
works take recorded successful episodes of a task and use them to aid in
training the RL model. Several of these papers operate by adding successful
demonstrations to the replay buffer [1, 23]; however, these papers require
accurate knowledge of the action space. This type of information is only
accessible in datasets specifically designed for robot training. In order to
learn from other sources, such as videos, the field must evolve past this
method. Additionally, aside from [5, 2], these works all require a large
number of demonstrations. [5] can use a smaller number, but needs a task-
specific reward function. [2] only uses one example, but requires different
start states. [4] proposes pre-training the model and then fine tuning for
each individual task, but their pre-training is data intensive and not always
generalizable.

Inverse Reinforcement Learning. Another closely related area, Inverse
Reinforcement Learning (IRL), uses a model to predict the reward values based
on the state and action space. Methods such as [7, 9, 8], employ a hybrid
approach involving both IRL and adversarial learning to solve a task with
limited demonstrations. Similar to our method, IRL methods are useful because
they do not require a defined reward function. Nonetheless, despite their
claim of using a small number of demonstrations, these methods still require
at least 50 demonstrations. Additionally, these methods struggle to generalize
beyond the initial demonstrations. Our approach differs from IRL approaches
because we do not require the training of a model to define our generalized
reward, which leaves less room for error and requires less data.

Modified Sparse Reward. Several methods [24, 25, 26, 27] attempt to modify the
sparse reward function in different ways to make learning more efficient.
Despite employing a similar reward structure to ours and looking at the
maximum of two different functions, [24] still requires a large number of
demonstrations as well as access to the action space. Another way to learn
with sparse rewards is to split multi-step tasks into several different tasks
[25]. This allows the sparse rewards to be more frequent, but requires a
precise definition of each auxiliary task which makes creating a generalized
model more difficult.

##  III Method

###  III-A Preliminaries

We formulate our problem as a Markov Decision Process (MDP) consisting of an
[n]-tuple $(S,A,R,\tau_{D},\gamma)$. The elements of this tuple are the state
space $S$, the action space $A$, the reward function $R\colon S\times
A\rightarrow\mathbb{R}$, the demonstration trajectory
$\tau_{D}\colon(s^{*}_{0},\dots,s^{*}_{N})$ and the discount factor $\gamma$.
We will refer to a random episode trajectory as $\tau$, where
$\tau=(s_{0},a_{0},\dots,s_{N},a_{N})$. $N$ is the total number of states and
actions to reach every state. A policy is represented as $\pi_{\theta}\colon
S\rightarrow A$, with parameters $\theta$. We use a sparse reward paradigm for
our method, where the action space is unknown. A sparse reward is defined as a
reward function $R$ that receives a reward of $0$ when in the goal state, $g$.
At all other times the reward is $-1$. If $g$ represents the goal state, and
$s_{n}$ and $a_{n}$ are the $n^{\text{th}}$ state and action respectively,
then the sparse reward function $R$ can be represented as

$R(s_{n},a_{n})=\begin{cases}0&\text{if}\;s_{n}=g,\\\ -1&\text{if}\;s_{n}\neq
g.\end{cases}$ (1)

###  III-B Overview

In this section, we describe our method or learning goal-conditioned skills
from a single demonstration with no information about the demonstration’s
action space. An illustrative overview of our method is provided in Figure 2,
which shows a much simpler grid world demonstration. The leftmost grid in
Figure 2 represents the environment, where an agent must traverse the boxes to
find the goal, which gives a sparse reward of 0. The middle grid shows the
reward for each box using a bellman equation [22], which requires access to
rewards for all states. Finally, the right grid shows our approach which
traverses a single path and then recursively determines the reward of each
state along the path. Access to one successful demonstration allows o
determine the pseudo ground truth rewards for each box along the path.

ses a sparse reward, along with bellman’s equation to compute the value for
each waypoint along the demonstration path. During exploration, a new state
obtains a reward if its distance from a known waypoint is less than a
threshold $d_{\text{thresh}}$ (captured by is$\\_$prox$\\_$wp$()$). We use
Nearest Neighbors to determine the waypoint the new state is compared to.
After training and achieving some success, we improve our method’s ability to
generalize to unseen start and goal states, by expanding on possible start and
goal states. We present an algorithmic overview of n Algorithm 1, followed by
a comprehensive breakdown of each step for a clearer understanding.

###  III-C Proximal Waypoint

Each state of the environment can be represented as
$s_{i}=\\{p_{1},p_{2},\dots,p_{K}\\}$, $\forall$ i $\in(1,\dots,N)$, where
$p_{k}$ represents an environmental parameter such as object pose and gripper
velocity and $K$ represents the number of environmental parameters. Note that
each parameter, $p_{k}$, is a relative value, with respect to the object’s
location, instead of an absolute value with respect to the world coordinate
system. This ensures better generalizability of our method. We use our policy
$\pi_{\theta}$ to determine an action that allows us to reach an unseen state
$s_{e}$. Following other reinforcement learning algorithms [13, 12], we add
random noise to the action space, in order to increase the amount of
exploration done during training. Once we have reached $s_{e}$ we will
determine if it is within close proximity of a waypoint along our
demonstration trajectory $\tau_{D}$. To do this we first use the Nearest
Neighbor function to find the waypoint with the smallest total L2 distance
between the parameters in the waypoint and the parameters in $s_{e}$ as

$\displaystyle s^{*}_{t}={\mathbf{NN}}(s_{e},\tau_{D})$
$\displaystyle=\operatorname*{arg\,min}_{\forall
s^{*}_{i}\in\tau_{D}}\left\lVert s_{e}-s^{*}_{i}\right\rVert_{2},$ (2)

where $s^{*}_{t}\in\tau_{D}$ is the closest state to (or the proximal waypoint
for) $s_{e}$ from the states within the expert trajectory $\tau_{D}$. For an
agent to receive a reward at state $s_{e}$, the distance between the
parameters of $s_{e}$ and $s^{*}_{t}$ should be less than a threshold. For
instance, if $d_{\text{thresh}}=\\{d_{1},d_{2},\dots,d_{K}\\}$ is the
corresponding threshold for parameters between
$s_{e}=\\{p_{1},p_{2},\dots,p_{K}\\}$ and
$s^{*}_{t}=\\{p^{*}_{1},p^{*}_{2},\dots,p^{*}_{K}\\}$, then we compute the
Boolean hasProxWP as

$\displaystyle\text{{is$\\_$prox$\\_$wp}}(s_{e},s^{*}_{t})=\begin{cases}\text{True},&\text{if}\;\left\lVert
p^{*}_{i}-p_{i}\right\rVert\leq d_{i},\forall i\in K,\\\
\text{False},&\text{otherwise.}\end{cases}$

We define one $d_{\text{thresh}}$ for each of the waypoints in $\tau_{D}$. For
instance $s^{*}_{t}$ has its own threshold, $d^{*}_{t}$, and when $s^{*}_{t}$
is the nearest neighbor to $s_{e}$, we use $d^{*}_{t}$ as the threshold. In
order to encourage progression, if hasProxWP is False 10 consecutive times for
a waypoint $s^{*}_{t}$, then we increase $d^{*}_{t}$ by $\epsilon\;(=0.001)$.
We repeat this until we explore a point that falls within the threshold of the
waypoint $s^{*}_{t}$.

Algorithm 1 rior to expanding knowledge

1:$\tau_{D}=(s^{*}_{0},\dots,s^{*}_{T}):$ A successful demonstration

2:$\pi_{\theta}\colon$ The policy that we will follow and update

3:while true do

4: $\tau\leftarrow(s_{0},a_{0},\dots,s_{N},a_{N})$ : an episode roll out

5: for all $s_{n},a_{n}\in\tau$ do

6: $s^{*}_{t}\leftarrow{\mathbf{NN}}(s_{n},\tau_{D})$, where
$s^{*}_{t}\in\tau_{D}$

7: hasProxWP $\leftarrow$ is$\\_$prox$\\_$wp$(s_{e},s^{*}_{t})$

8:
$r\leftarrow\text{reward}(\text{{hasProxWP}},\;t,\;l_{\text{D}},l_{\text{max}})$

9: $R\leftarrow\max(r,\;\gamma*\text{critic}(s_{n+1}))$

10: end for

11:end while

Figure 3: The environments that we experimented on with e show results on 4
different tasks: (a) pick and place, (b) peg assembly, (c) open door and (d)
peg insertion. These tasks are ideal because they have a clear definition of
success and therefore a clear sparse reward. However, most of these tasks
cannot be solved with sparse rewards alone. Figure 4: (a,b,c) shows the
results of the Meta World [28] environments when trained using SAC [13] and a
batch size of 2048. (a) Open Door Task, (b) Peg Insertion Task, (c) Peg
Assembly Task. (d) Pick and Place task is from OpenAI [15]. TABLE I:
Requirements for different baselines and n terms of state, action and number
of expert demonstrations.

Pre-requisites | SAC (S) | SAC (D) | HER | SAC + RB | SAC + MCAC | AWAC | AWAC + MCAC |
---|---|---|---|---|---|---|---|---
Requires Action Space | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗
Requires Pre-training | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗
# Expert Demonstrations | 0 | 0 | 0 | 1 or 100 | 1 or 100 | 1 or 100 | 1 or 100 | 1

###  III-D New Reward Function

Given the nearest neighbor waypoint, $s^{*}_{t}$, and the boolean result,
hasProxWP, we can solve for the reward function, $r$, for our state action
pair $(s_{e},a_{e})$. $l_{\text{max}}$ represents the maximum length of an
episode, $l_{\text{D}}$ represents the length of the demonstration $\tau_{D}$
and $t$ represents the time at which $s^{*}_{t}$ occurs. We compute the
proposed reward, $r$ as

$r=\begin{cases}\displaystyle\sum_{i=0}^{l_{\text{D}}-t}-\gamma^{i}&\text{{hasProxWP},}\\\
\displaystyle\sum_{i=1}^{l_{\text{max}}}-\gamma^{i}&\text{not
{hasProxWP}.}\end{cases}$ (3)

If hasProxWP is false, we still want to account for the possibility that our
state, $s_{e}$, is on a successful trajectory. To do this, we say that the
final reward $R=\text{max}(r,\gamma*\text{critic}(s_{e+1}))$. In order for
this to work we need to warm up the critic. We do this by training it for
$1000$ time steps on just $r$ before we include the critic reward. For more
information on actor critic methods please refer to [13, 12].

###  III-E Expanding Knowledge

Following the demonstration, eaches the policy how to solve the task from a
fixed start and goal state. We now need to expand to every possible start and
goal location. To achieve this, we slowly increase the number of possible
start locations and goal locations, by adding random noise to the initial
state space.

Given our current demonstration trajectory $\tau_{D}$ with states $s^{*}_{0}$
and goal $g^{*}$, let $\mathcal{N}^{*}(\mu^{*},\,\sigma^{*})$ be the
distribution representing the amount of noise we will add to the start state
and goal state. We will set the mean, $\mu^{*}$, to $0$ and only increase the
standard deviation $\sigma^{*}$.

At the start of training $\sigma^{*}$ is set to 0. $\sigma^{*}$ uses a
modification strategy applied every 25 episodes. The updated value of
$\sigma^{*}$ is described as $\sigma^{\prime}$, where:

$\sigma^{\prime}=\begin{cases}\sigma^{*}+0.001,&\text{if success rate}\geq
0.05,\\\ \sigma^{*},&\text{otherwise.}\\\ \end{cases}$ (4)

Figure 5: This figure shows the results of several baselines when the are
given one expert demonstration (a,b,c) shows the results of the Meta World
[28] environments when trained using AWAC [29], MCAC [24], SAC + RB and AWAC +
MCAC. (d,e) shows the results of these same baselines on the robosuite tasks
[30]

##  IV Experiments and Results

###  IV-A Environment Setup

We use MuJoCo [31] to simulate our tasks. The implementation of our
reinforcement learning algorithms was based on modified versions of the open-
source code provided by stable-baselines3 [32] and our baselines were based on
[24]. Although we used only one demonstration for each task, we varied the
starting demonstration across different seeds to demonstrate the adaptability
of our method to different initial demonstrations. To ensure robustness, we
conducted each experiment four times with different seeds and present the mean
and standard deviation of these seeds in our graphs. Each epoch consists of
40,000 simulated timesteps in MuJoCo. The pick and place environment, the
robosuite door environment and the robosuite lift environment have 50 time
steps per episode, and the remaining environments have 150 time steps per
episode. We pre-train AWAC for 25000 timesteps.

###  IV-B Tasks

s trained on six distinct tasks, each designed to showcase the robot’s
capability to accomplish simple goal-conditioned objectives. The pick and
place environment is from the OpenAI Fetch tasks [15], the next three (open
door, peg insertion, and peg assembly) tasks are from Meta World [28]. The
final two tasks, Robosuite (RS)-Open Door and RS-Lift come from [30]. Images
of these tasks can be seen in Figure 3. Our tasks are: (a) Pick and Place: The
task is to grasp a box and move it towards a goal in the air. (b) Peg
Assembly: The task is to pick up a round nut and then place the round part
over a peg. (c) Open Door: The task is to grasp a door handle and then open
the door until it reaches a goal location. (d) Peg Insertion: The task is to
pick up a peg and insert it into a hole. (e) RS-Open Door: The task is to
grasp a door handle and then open the door very slightly. (f) RS-Lift: The
task is to pick up a block and lift it into the air.

###  IV-C Baslines

We test our method (Table III-C) against following baselines:

  * •

SAC. This baseline uses the Soft Actor Critic (SAC) algorithm [13] as the
reinforcement learning algorithm. It can be initialized in three ways: (1)
Sparse Reward (SAC (S)), (2) Dense Reward (SAC (D)), (3) SAC + Replay Buffer
(SAC + RB): we initialize the replay buffer with expert demonstrations.

  * •

Hindsight Experience Replay (HER) is a well known technique [14], which uses
the previous experiences to learn overtime.

  * •

SAC+MCAC. This baseline uses the soft actor critic (SAC) algorithm [13] as
well as Monte Carlo augmented Actor-Critic [24].

  * •

Advantage Weighted Actor Critic (AWAC) follows the methods described in [29].

  * •

AWAC+MCAC uses [29] along with a modified reward as describe in [24].

###  IV-D Results

Figure 6: This figure shows the results of several baselines when they are
given one hundred expert demonstrations (a,b,c) shows the results of the Meta
World [28] environments when trained using AWAC [29], MCAC [24], SAC + RB and
AWAC + MCAC. (d,e) shows the results of these same baselines on the robosuite
tasks [30]. Our method continues to use only one demonstration.

We evaluate several different combinations of baselines each of which have
different requirements in comparison to our method.

####  IV-D1 No Action Space

First we look at methods that do not require access to the action space of an
expert demonstration in order to learn. In Figure 4(d), we analyze the
performance of our method compared to other conventional reinforcement
learning algorithms for the pick and place task. These graphs reveal two
significant observations: (1) xhibits a remarkable acceleration in the
learning process with just a single demonstration, over SAC. (2) emonstrates a
considerably lower standard deviation compared to the other algorithms. We
hypothesize that in general, methods relying on sparse rewards requires a
degree of luck. For that, the environment must be explored extensively until a
reward is obtained, allowing the method to learn the task. However, ircumvents
this by guiding the agent towards the goal, irrespective of the initial start
point.

In Figure 4(a,b,c), we examine the outcomes of training three Meta World
environments [28] using the SAC algorithm. These results are compared against
a sparse reward and an episode-specific dense reward. Notably, we encountered
difficulties in implementing hindsight experience replay with these
environments. The Figure 4(a) represents the results of the open door task,
(b) displays the results of the peg insertion task, and (c) showcases the peg
assembly task. Across all these environments, we observed that SAC was unable
to solve the tasks effectively when utilizing sparse rewards. For the open
door and peg insertion task erforms similar or better than the dense reward.
The dense rewards have been finely tuned to each task and can require a lot of
trial and error to finalize, while s a general reward that can be applied to
any method. Regarding the peg assembly task, we discovered that SAC was unable
to solve the task even with the dense reward during the training period. This
is due to increased complexity of the task, which results in a noisier reward
signal. This task is more difficult because there are a greater number of
objectives to be accomplished. However, our method proved capable of swiftly
solving the task despite these challenges.

####  IV-D2 One Expert Demonstration

Next, we look at our method compared to baselines that require just 1 example.
We compare against AWAC, AWAC + MCAC, SAC + RB and SAC + MCAC when just one
expert demonstration is used. The results of this are shown in Figure 5. We
find that for the three meta world tasks our approach significantly
outperforms the other approaches. AWAC and AWAC + MCAC work in the Peg
Assembly and Door Open tasks, but they are not able to solve the problem as
well as our approach is. For the RS-Door Open task, we find that our approach
performs very similar to all other baselines. This is likely because the task
is very easy and requires only a small amount of data. The RS-Lift task
results look similar to the MetaWorld results where our method significantly
outperforms the others. This task was more difficult than others due to the
rotation of the block being different in each episode, but our method is able
to handle it nonetheless.

####  IV-D3 100 Expert Demonstrations

Finally, we look at our method compared to the same baselines when the
baselines use 100 expert demonstrations. The results are shown in Figure 6.
Note that AWAC, which is the method that performs the best in these scenarios
has to be pre-trained in addition to the online training. We find that
although our method takes more online training time with just one
demonstration, versus 100 demonstrations, our method performs equal to or
better than the baselines in all of the tasks.

##  V Conclusion

We present new approach that enables training reinforcement learning models
using a single demonstration. Unlike other imitation learning methods, which
typically rely on multiple demonstrations and access to detailed action
information, an operate with limited information and single demonstration. In
order to achieve this, we introduce a novel universal reward function and
leverage a knowledge expansion technique that extends beyond initial start and
goal states. This makes it highly suitable for learning tasks with minimal
information across different environments. We show that s faster than standard
reinforcement learning models, in cases where the rewards are sparse or dense,
and showcase its ability to succeed where other approaches fall short.
Additionally, we show that erforms similar to or better than a variety of
imitation learning methods when these methods use either one or one hundred
expert demonstrations. In future research, we aim to explore the use of
expansion for non-linear states and investigate the utilization of image-based
state spaces rather than joint locations.

Acknowledgements: This work was partially supported by DARPA SAIL-ON
(W911NF2020009) program and NSF CAREER Award (#2238769) to AS.

## References

  * [1] Ashvin Nair et al. “Overcoming Exploration in Reinforcement Learning with Demonstrations” In _CoRR_ abs/1709.10089, 2017 arXiv: http://arxiv.org/abs/1709.10089
  * [2] Tim Salimans and Richard Chen “Learning Montezuma’s Revenge from a Single Demonstration” In _CoRR_ abs/1812.03381, 2018 arXiv: http://arxiv.org/abs/1812.03381
  * [3] Yuke Zhu et al. “Reinforcement and Imitation Learning for Diverse Visuomotor Skills” In _CoRR_ abs/1802.09564, 2018 arXiv: http://arxiv.org/abs/1802.09564
  * [4] Karl Pertsch, Youngwoon Lee, Yue Wu and Joseph J. Lim “Demonstration-Guided Reinforcement Learning with Learned Skills” In _CoRR_ abs/2107.10253, 2021 arXiv: https://arxiv.org/abs/2107.10253
  * [5] Xue Bin Peng, P. Abbeel, Sergey Levine and Michiel Panne “DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills” In _ACM Trans. Graph._ 37, 2018, pp. 143
  * [6] Kei Ota et al. “Deep Reactive Planning in Dynamic Environments” In _CoRR_ abs/2011.00155, 2020 arXiv: https://arxiv.org/abs/2011.00155
  * [7] Jonathan Ho and Stefano Ermon “Generative Adversarial Imitation Learning” In _CoRR_ abs/1606.03476, 2016 arXiv: http://arxiv.org/abs/1606.03476
  * [8] Siddharth Reddy, Anca D. Dragan and Sergey Levine “SQIL: Imitation Learning via Regularized Behavioral Cloning” In _CoRR_ abs/1905.11108, 2019 arXiv: http://arxiv.org/abs/1905.11108
  * [9] Justin Fu, Katie Luo and Sergey Levine “Learning Robust Rewards with Adversarial Inverse Reinforcement Learning”, 2018 arXiv:1710.11248 [cs.LG]
  * [10] Christopher Watkins and Peter Dayan “Technical Note: Q-Learning” In _Machine Learning_ 8, 1992, pp. 279–292 DOI: 10.1007/BF00992698
  * [11] Volodymyr Mnih et al. “Playing Atari with Deep Reinforcement Learning” In _CoRR_ abs/1312.5602, 2013 arXiv: http://arxiv.org/abs/1312.5602
  * [12] Timothy P. Lillicrap et al. “Continuous control with deep reinforcement learning.” In _ICLR_ , 2016 URL: http://dblp.uni-trier.de/db/conf/iclr/iclr2016.html#LillicrapHPHETS15
  * [13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor” In _CoRR_ abs/1801.01290, 2018 arXiv: http://arxiv.org/abs/1801.01290
  * [14] Marcin Andrychowicz et al. “Hindsight Experience Replay” In _CoRR_ abs/1707.01495, 2017 arXiv: http://arxiv.org/abs/1707.01495
  * [15] Matthias Plappert et al. “Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research” In _CoRR_ abs/1802.09464, 2018 arXiv: http://arxiv.org/abs/1802.09464
  * [16] Soroush Nasiriany, Vitchyr H. Pong, Steven Lin and Sergey Levine “Planning with Goal-Conditioned Policies” In _CoRR_ abs/1911.08453, 2019 arXiv: http://arxiv.org/abs/1911.08453
  * [17] Minghuan Liu, Menghui Zhu and Weinan Zhang “Goal-Conditioned Reinforcement Learning: Problems and Solutions”, 2022 arXiv:2201.08299 [cs.AI]
  * [18] Richard Sutton et al. “Horde : A Scalable Real-time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction Categories and Subject Descriptors” In _Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems_ 2, 2011
  * [19] Meng Fang et al. “Curriculum-guided Hindsight Experience Replay” In _Neural Information Processing Systems_ , 2019
  * [20] Volodymyr Mnih et al. “Playing Atari with Deep Reinforcement Learning” cite arxiv:1312.5602Comment: NIPS Deep Learning Workshop 2013, 2013 URL: http://arxiv.org/abs/1312.5602
  * [21] Yevgen Chebotar et al. “Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills” In _CoRR_ abs/2104.07749, 2021 arXiv: https://arxiv.org/abs/2104.07749
  * [22] Richard Bellman “Dynamic programming” In _Science_ 153.3731 American Association for the Advancement of Science, 1966, pp. 34–37
  * [23] Tom Le Paine et al. “Making Efficient Use of Demonstrations to Solve Hard Exploration Problems” In _CoRR_ abs/1909.01387, 2019 arXiv: http://arxiv.org/abs/1909.01387
  * [24] Albert Wilcox et al. “Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations”, 2022 arXiv:2210.07432 [cs.LG]
  * [25] Martin A. Riedmiller et al. “Learning by Playing - Solving Sparse Reward Tasks from Scratch” In _CoRR_ abs/1802.10567, 2018 arXiv: http://arxiv.org/abs/1802.10567
  * [26] Alexander Trott, Stephan Zheng, Caiming Xiong and Richard Socher “Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards” In _CoRR_ abs/1911.01417, 2019 arXiv: http://arxiv.org/abs/1911.01417
  * [27] Xingyu Lu, Stas Tiomkin and P. Abbeel “Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards” In _ArXiv_ abs/1912.13414, 2019
  * [28] Tianhe Yu et al. “Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning” In _Conference on Robot Learning (CoRL)_ , 2019 arXiv:1910.10897 [cs.LG]
  * [29] Ashvin Nair, Abhishek Gupta, Murtaza Dalal and Sergey Levine “AWAC: Accelerating Online Reinforcement Learning with Offline Datasets”, 2021 arXiv:2006.09359 [cs.LG]
  * [30] Yuke Zhu et al. “robosuite: A Modular Simulation Framework and Benchmark for Robot Learning” In _arXiv preprint arXiv:2009.12293_ , 2020
  * [31] Emanuel Todorov, Tom Erez and Yuval Tassa “MuJoCo: A physics engine for model-based control.” In _IROS_ IEEE, 2012, pp. 5026–5033 URL: http://dblp.uni-trier.de/db/conf/iros/iros2012.html#TodorovET12
  * [32] Antonin Raffin et al. “Stable-Baselines3: Reliable Reinforcement Learning Implementations” In _Journal of Machine Learning Research_ 22.268, 2021, pp. 1–8 URL: http://jmlr.org/papers/v22/20-1364.html


# Symphony of high-dimensional brain.
Reply to comments on “The unreasonable effectiveness of small neural ensembles
in high-dimensional brain”

Alexander N. Gorban<EMAIL_ADDRESS>Valeri A. Makarov<EMAIL_ADDRESS>Ivan Y. Tyukin<EMAIL_ADDRESS>Department of Mathematics, University of
Leicester, Leicester, LE1 7RH, UK  Lobachevsky University, Nizhni Novgorod,
Russia  Instituto de Matemática Interdisciplinar, Faculty of Mathematics,
Universidad Complutense de Madrid, 28040 Madrid, Spain  Saint-Petersburg State
Electrotechnical University, Saint-Petersburg, Russia

###### keywords:

Complexity; Big data; Non-iterative learning; Error correction; Measure
concentration; Blessing of dimensionality; Linear discriminant; Memory

This reply is not a ‘comment on comments’ but rather a ‘selective echo’ that
resonates with some ideas and questions raised by the commentators [1, 2, 3,
4, 5, 6, 7, 8, 9]. We would like to thank all of them. All the individual
comments and ideas come together as a symphony of thought, and each individual
part changes the way the multidimensional brain is understood. The comments
include many deep thoughts and inspiring questions and deserve a close
reading.

### The flight from independence and uniformity

Kůrková [1] has emphasized that many classical results about measure
concentration assume either uniform probability distribution (with respect to
the Lebesgue measure in space or the rotationally invariant measures on
spheres [10, 11]), or independence (‘product measures’ [12]), or both. In
classical machine learning theory, data are typically assumed to be i.i.d.
samples. On the contrary, in real life data samples are neither i.i.d. nor
uniformly distributed.

A similar comment about the difference between the theoretical random
distributions and “extremely non-random” real distributions has been received
from Geoffrey Hinton (personal communication to ANG). The obvious gap between
the current theory and practice is a problem, indeed, and the machine learning
theory should be revised to deal with non-independent and extremely non-
uniformly distributed data. Kůrková with Sanguineti proposed their approach to
this problem in a very recent publication [13]. Our first two stochastic
separation theorems used the classical assumptions about either uniformity of
distributions (uniform distribution in balls) or independence (product
distributions in cubes) [14]. After addressing these basic cases we started
our ‘flight from independence and uniformity’ in the paper “Augmented
Artificial Intelligence: a Conceptual Framework” [15] and developed this work
further in [16, 17].

The intermediate conclusion is as follows. Stochastic separation theorems do
not need hypotheses about independence and uniform distribution of data (as
well as any other hypothesis about special distributions like the Gaussian
one). The main condition we used instead of these simplifications is: sets of
small volume should not have a high probability (further specifications in
what ‘small’ and ‘large’ mean here can be found, for example in Theorem 1,
Definition 3, and Theorem 6 from [17], Theorem 7 and Sec. 6 “Quasiorthogonal
sets and Fisher separability of not i.i.d. data” in [16]). In particular,
instead of uniform or Gaussian distributions general log-concave distributions
can be used, and this is just an example. Fisher separability of random points
holds for more general distributions that satisfy the SmAC (SMeared Absolute
Continuity) condition (see [17], Definition 3 or [16], Definition 4). Donoho
and Tanner [18] observed ‘surprising’ universality of linear separation in
numerical experiments with high-dimensional random datasets. They proved this
property for Gaussian distributions and formulated the problem to determine a
general definition of distributions with such separability. We now know that
explicit and useful Fisher’s separability is also typical, and the general
definitions of distributions with such properties are understood much better
now.

The qualitative essence of these definitions is always the same: small volume
sets should not have a high probability. If such sets exist then an additional
challenge arises: extract the sets of small volume and relatively large
probability from the data space. The remaining part will satisfy the
stochastic separation theorems with Fisher’s separability, while the extracted
set will have effectively lower dimension and can be approximated by the low-
dimensional continua [19, 20, 21]. The extracted low-dimensional subsets can
have complex topology and analysis of this topology is important for many
modern applications like single-cell omics in bioinformatics [23]. Here we
reveal a special complementarity principle [22]: the data space is split into
a low volume (low dimensional) subset, which requires nonlinear methods for
data approximation and analysis, and a high dimensional subset, where the
linear methods (Fisher discriminants) work well.

Following the Kůrková comment [1] and papers [13, 15, 16, 17], we invite all
readers to join the flight from independence and uniformity. This is a call
for revisiting the existing general machine learning theory to bring it closer
to real life problems.

### Linear algebra vs topology

Tozzi and Peters [2] used the Borsuk–Ulam Theorem (BUT) to explain the
blessing of dimensionality. In the topological form (proven by Fet) [24] BUT
is: For every continuous involution $a\mapsto a^{*}$ ($a^{**}=a$) of a sphere
$\mathbb{S}^{n}$ and every continuous map $f:\mathbb{S}^{n}\to\mathbb{R}^{n}$
there exists a point $x\in\mathbb{S}^{n}$ such that $f(x)=f(x^{*})$. The most
well-known form of BUT uses the central symmetry $a\mapsto-a$ instead of a
general continuous involution. The gluing of opposite points is interpreted by
Tozzi as a decrease in information in topological sense. Reversely, the
dimensionality increase ‘unglues’ points, and information increases. In more
detail, this point of view is presented and illustrated with applications in
the review paper [25]. It is necessary to add that for a continuous map
$f:\mathbb{S}^{n}\to\mathbb{R}^{n}$ ‘almost every’ point is glued with another
point (just imagine a projection of a 2D sphere on a plane). We should stress,
however, that the volume/probability concentration arguments with
concentration of the volume near equator are invariant with respect to any
orthogonal transformation that preserves the whitened distribution with unit
covariance matrix. This property will be destroyed by a typical homeomorphism.
On the contrary, the BUT in the topological form and other topological
statements are invariant with respect to any homeomorphism. Both arguments,
topological and linear, are important, but further work is needed to clarify
the relations between them (at least, for us). Another intriguing problem with
interplay of linear algebra and topology is the definition of ‘intrinsic
dimension’ of data. Real data never are i.i.d. sample from a regular and fixed
distribution. There are correlations between data points, the distribution
(assuming that it exists) can change over time, there may exist a hidden
“concept drift” and many other complications. We proposed to use the Fisher
separability property for definition of intrinsic dimension of data: it is the
dimension of a sphere having the same separability properties for i.i.d.
samples from a uniform distribution (the ‘equivalent sphere’) [16]. This
concept was developed further and tested on biological data [26]. Zinovyev et
al [26] showed how the suggested approach can be used to explore the structure
of the data types that are generally considered hard to analyze (mutation and
single cell RNA-Seq data). We expect further interesting and useful
discoveries at the boundary between topology and geometry of data.

### Communication between neuroscience and Artificial Intelligence (AI)

Varona [3] discussed relations between machine learning and brain studies,
from their common roots to a long period of divergence followed by the modern
trend towards convergence. The “bidirectional communication between machine
learning and neuroscience” is an important way to enrich both disciplines.
Problems arising in the ontogeny of the brain and in machine learning can be
similar. Both in the brain and in machine learning, re-training large
ensembles of neurons is in fact impossible to realize in many real-life
situations/applications. The robust low-dimensional neural dynamics in high
dimensional brains is important for understanding implementation of cognitive
functions (see, for example, [28]). However, there is a difference in this
similarity. We agree with Varona that AI is far from incorporating all highly
heterogeneous, dynamically robust, and error-handling elements of real brain.
Further bi-directional exchange of ideas and methods between AI and
neuroscience is needed.

### Moving from “brainland” to “flatland” and backwards

Barrio [4] has directed our attention to the intriguing interplay between the
low and high-dimensional worlds, between the high-dimensional brainland (the
whole world of brain) and low-dimensional flatland of small neural ensembles.
A flexible walk along the stairs of models of different dimensions is
necessary to understand the brain dynamics. In data analysis, we can find a
partial realization of this idea in our complementarity principle [22]:
extraction of a lower dimensional subset for non-linear analysis [19] and
application of simple linear methods to essentially high-dimensional part of
data [16]. Some open access software for this problems is available on github
[27].

### Jewels of low-dimensional representation in complexity of high dimensional
systems

Kreiman [5] analyzed three forms of the curse of dimensionality and high-
dimensional pitfalls in neuroscience: (1) Huge diversity of possible stimuli
and tasks; (2) Huge number of neurons; (3) No adequate tools – most modeling
and analysis tools are developed for small dimensional worlds. Nevertheless,
there are many examples in neuroscience, where small dimensional models
explain a large fraction of a cognitive behavior. Most of them are
manifestations of the concentration of measure phenomena. Kreiman attracted
attention to the mechanism of removing the curse of dimensionality by
projection on random bases. This approach is based on the Johnson and
Lindenstrauss lemma and related topics [29, 31, 30]. According to this lemma
and its probabilistic proof, a set of points in a high-dimensional space can
be embedded into a space of much lower dimension in such a way that ratios of
distances between the points are nearly preserved. The map used for the
embedding can even be taken to be a random orthogonal projection. This method
of random projections was used in a recent publication [32] to explain a
striking simplicity underlying multi-neuronal data. It is worth to mention
that the Johnson and Lindenstrauss lemma and the method of random projections
are based on the measure concentration phenomena [30]. The ‘brainland’ can
turn out to be a small dimensional world after all.

### Coupling of stochastic separation theory with nonlinear dynamics
techniques

Fortuna [6] proposed to combine the data analysis methods based on statistical
physics of data and advanced experimental based nonlinear dynamics techniques.
He demonstrated the power of dynamical models in neuroscience by three
examples: (i) a multi jump resonance behavior of an ensemble of neurons that
explains selectivity, (ii) stochastic resonance in a group of neurons (when
the signal to noise ratio at the output of the neuron ensemble is greater than
the input signal to noise ratio), and (iii) synchronization effect in networks
in the brain. The challenge is in effective combination of the stochastic
separation technique and topological data approximation with low-dimensional
dynamical models and model reduction methods.

### Models differ from reality

van Leeuwen [7] presented an anti-cybernetic manifesto. He sought to show that
the classical cybernetic understanding of the brain had run its course.
According to his comment, cybernetic approach to brain study is a misleading
metaphor. For example, it does not take into account non-local field
interaction of neurons. The brain is actively generating predictions about
what will happen and engages in controlled hallucination. The brain, unlike
legacy software, is constantly updating itself. These and other important
properties of the brain may deplete the cybernetic approach and make it
useless. Such a serious gap between modern AI and real brain can ruin our hope
for the efficient communication between neuroscience and AI. According to van
Leeuwen, we have to modify our mathematical and computer science approaches so
that this communication becomes useful for both parts. What can we say here?
First of all, models always differ from reality. The question is not, however,
whether there exist important properties of the brain that are neglected by a
model. They always exist. The question is whether the model reflects any
important aspects of reality. A model is an intelligent device that should
perform useful work. Simple models, ranged from stochastic separation, concept
cells, and Hebb’s rule described in our review [17] to direct simulation of
large networks of relatively simple neurons [34] and rule based models of
development neuronal network generating behavior [35], have not yet exhausted
their capabilities. Nevertheless, we are ready to accept the expert opinion of
van Leeuwen and agree that a new conceptual basis of brain modelling is
needed. Let us indeed hope that a new theoretical framework will be eventually
developed in neuroscience that will appeal to mathematician and computer
scientists, and there will be a decent place for simple basic models.

### Akaki Akakievich and the modern point of view on Concept Cells

Quian Quiroga [8] clarified and commented on modern ideas about Concept Cells
and sparse coding in the form of a revised Lettvin story about grandmothers
cells. He stressed that Concept Cells are not involved in identifying a
particular stimulus or concept. They are rather involved in creating and
retrieving associations and can be seen as the “building blocks of episodic
memory”. We analyzed basic dynamical models of such associations [33] (see
also [17], Sec. 4. “Encoding and rapid learning of memories by single
neurons”) but this analysis should be extended and compared quantitatively to
experimental data. The professional and clearly presented explanation of the
basic concepts in Quian Quiroga’s commentary is a very useful addition to our
review, and we are happy to recommend this commentary to all readers.

### “Happiness is when you are understood”

The beautiful text of Kreinovich [9] does not need comments. It is our
pleasure just to mention that in our internal discussions of the manuscript
[17], we have many times quoted Pasternak’s poetry about the “heresy of
unheard-of simplicity”. Kreinovich found the same golden words of Pasternak to
describe the apparent simplicity in a multidimensional brain. This is an
example of an amazing intellectual resonance.

## Acknowledgments

The work was supported by the Ministry of Science and Higher Education of the
Russian Federation (Project No. 14.Y26.31.0022). Work of ANG and IVT was also
supported by Innovate UK (Knowledge Transfer Partnership grants KTP009890;
KTP010522) and University of Leicester. VAM acknowledges support from the
Spanish Ministry of Economy, Industry, and Competitiveness (grant
FIS2017-82900-P).

## References

## References

  * [1] Kůrková V. Some insights from high-dimensional spheres: Comment on “The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by Alexander N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.03.014
  * [2] Tozzi A, Peters JF. The Borsuk-Ulam theorem solves the curse of dimensionality: Comment on ”The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by Alexander N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.04.008
  * [3] Varona P. High and Low dimensionality in Neuroscience and Artificial Intelligence: Comment on “The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by A.N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.02.008
  * [4] Barrio R. “Brainland” vs. “flatland”: how many dimensions do we need in brain dynamics? Comment on the paper “The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by Alexander N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.02.010
  * [5] Kreiman G. It’s a small dimensional world after all: Comment on “The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by Alexander N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.03.015
  * [6] Fortuna L. Nonlinear effects for the reinforcement of small neural ensembles in high dimensional brain: Comment on “The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by A.N. Gorban, V.A. Makarov, I.Y. Tyukin. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.03.004
  * [7] Van Leeuwen C. The reasonable ineffectiveness of biological brains in applying the principles of high-dimensional cybernetics: Comment on “The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by Alexander N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.03.005
  * [8] Quian Quiroga R. Akakhievitch revisited: Comment on ”The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by Alexander N. Gorban et al. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.02
  * [9] Kreinovich V. The heresy of unheard-of simplicity: Comment on ”The unreasonable effectiveness of small neural ensembles in high-dimensional brain” by A.N. Gorban, V.A. Makarov, and I.Y. Tyukin. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2019.04.006
  * [10] Lévy P. Problèmes concrets d’analyse fonctionnelle. Paris: Gauthier-Villars, 1951.
  * [11] Ball K. An Elementary Introduction to Modern Convex Geometry. Flavors of Geometry, Vol. 31. Cambridge, UK: MSRI Publications; 1997. https://doi.org/10.1007/BF02699376
  * [12] Talagrand M. Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathematiques de l’IHES 1995;81:73–205.
  * [13] Kůrková V, Sanguineti M. Probabilistic Bounds for Binary Classification of Large Data Sets. In: Oneto L, Navarin N, Sperduti A, Anguita D. (eds) Recent Advances in Big Data and Deep Learning. INNSBDDL 2019. Proceedings of the International Neural Networks Society, vol 1. Springer, pp. 309–319. https://doi.org/10.1007/978-3-030-16841-4_32
  * [14] Gorban AN, Tyukin IY. Stochastic separation theorems. Neural Netw. 2017;94:255–259. https://doi.org/10.1016/j.neunet.2017.07.014
  * [15] Gorban AN, Grechuk B, Tyukin IY. Augmented Artificial Intelligence: a Conceptual Framework, arXiv preprint 2018, arXiv:1802.02172 [cs.AI]. https://arxiv.org/abs/1802.02172
  * [16] Gorban AN, Golubkov A, Grechuk B, Mirkes EM, Tyukin IY. Correction of AI systems by linear discriminants: Probabilistic foundations. Inf. Sci. 2018; 466:303-22. https://doi.org/10.1016/j.ins.2018.07.040.
  * [17] Gorban AN, Makarov VA, Tyukin IY. The unreasonable effectiveness of small neural ensembles in high-dimensional brain. Phys Life Rev 2019; this issue. https://doi.org/10.1016/j.plrev.2018.09.005
  * [18] Donoho D, Tanner J. Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Phil. Trans. R. Soc. A 2009;367:4273–4293. https://doi.org/10.1098/rsta.2009.0152
  * [19] Gorban AN, Zinovyev A. Principal manifolds and graphs in practice: from molecular biology to dynamical systems. International Journal of Neural Systems 2010;20(03):219–232. https://doi.org/10.1142/S0129065710002383
  * [20] Gorban AN, Kégl B, Wunsch D, Zinovyev A. (eds.) Principal Manifolds for Data Visualisation and Dimension Reduction. Lecture Notes in Computational Science and Engineering, Vol. 58. Berlin – Heidelberg – New York: Springer, 2008.
  * [21] Zinovyev A, Mirkes E. Data complexity measured by principal graphs. Comput. Mat. Appl. 2013;65(10):1471–1482. https://doi.org/10.1016/j.camwa.2012.12.009
  * [22] Gorban AN, Tyukin IY, Blessing of dimensionality: mathematical foundations of the statistical physics of data, Phil. Trans. R. Soc. A 2018;376:20170237. https://doi.org/10.1098/rsta.2017.0237
  * [23] Chen H, Albergante L, Hsu JY, Lareau CA, Bosco GL, Guan J, Zhou S, Gorban AN, Bauer DE, Aryee MJ, Langenau DM, Zinovyev A, Buenrostro JD, Yuan G-C, Pinello L. Single-cell trajectories reconstruction, exploration and mapping of omics data with STREAM. Nature Communications 2019;10:1903. https://doi.org/10.1038/s41467-019-09670-4
  * [24] Fet AI. Generalization of a theorem of Lusternik-Schnirelman on covering of spheres. Dokl. Akad. Nauk SSSR 1954;95:1149-1152.
  * [25] Tozzi A. The multidimensional brain. Phys Life Rev 2019. https://doi.org/10.1016/j.plrev.2018.12.004
  * [26] Albergante L, Bac J, Zinovyev A. Estimating the effective dimension of large biological datasets using Fisher separability analysis, 2019, arXiv:1901.06328 [cs.LG]. https://arxiv.org/pdf/1901.06328.pdf
  * [27] Laboratory of advanced methods for high-dimensional data analysis https://github.com/lamhda
  * [28] Rabinovich MI, Varona P. Discrete Sequential Information Coding: Heteroclinic Cognitive Dynamics. Front Comput Neurosci 2018;12:73. https://doi.org/10.3389/fncom.2018.00073
  * [29] Johnson W, Lindenstrauss J. Extensions of Lipschitz maps into a Hilbert space.Conference in modern analysis and probability (New Haven, Conn., 1982), Contemp. Math., 1984;26:189–206. https://doi.org/10.1090/conm/026/737400
  * [30] Gorban AN, Tyukin I, Prokhorov D, Sofeikov K. Approximation with random bases: Pro et contra. Inf. Sci. 2016:364–365;129–145. https://doi.org/10.1016/j.ins.2015.09.021
  * [31] Indyk P, Motwani R. Approximate nearest neighbors: towards removing the curse of dimensionality. In STOC’98 Proceedings of the thirtieth annual ACM symposium on Theory of computing, Dallas, Texas, USA – May 24-26, 1998, ACM New York, NY, 1998, pp. 604-613. https://doi.org/10.1145/276698.276876
  * [32] Gao P, Trautmann E, Yu B, Santhanam G, Ryu S, Shenoy K, Ganguli S. A theory of multineuronal dimensionality, dynamics and measurement, bioRxiv 2017;214262. https://doi.org/10.1101/214262
  * [33] Tyukin IY, Gorban AN, Calvo C, Makarova J, Makarov VA. High-dimensional brain. A tool for encoding and rapid learning of memories by single neurons, Bull. Math. Biol. 2018 https://doi.org/10.1007/s11538-018-0415-5.
  * [34] Izhikevich EM, Edelman GM. Large-scale model of mammalian thalamocortical systems. Proc. Natl. Acad. Sci. U.S.A. 2008 105(9):3593-8. https://doi.org/10.1073/pnas.0712231105
  * [35] Roberts A, Conte D, Hull M, Merrison-Hort R, al Azad AK, Buhl E, Borisyuk R, Soffe SR. J. Neurosci. 2014 34(2):608-21. https://doi.org/10.1523/JNEUROSCI.3248-13.2014