$\tilde{u}_{1}=u_{1\ }^{\ 1}+u_{1\ }^{\ 2}+u_{1\ }^{\ 21}+u_{1\ }^{\
22}\Rightarrow 1/4\tilde{u}_{1}=u_{1\ }^{\ 1}=u_{1\ }^{\ 2}=u_{1\ }^{\
21}=u_{1\ }^{\ 22}$

then simplifying the remaining equations of the two input units

$\displaystyle 0=w_{2\ }^{\ 11}u_{2\ }^{\ 11}=w_{2\ }^{\ 12}u_{2\ }^{\
12}\text{ and }0=u_{2\ }^{\ 11}=u_{2\ }^{\ 12}\Rightarrow\forall w_{2\ }^{\
11},\forall w_{2\ }^{\ 12}$ $\displaystyle\bar{u}_{2\ }^{\ i}=\frac{w_{1\ }^{\
2i}u_{1\ }^{\ 2i}(u_{2}+w_{1\ }^{\ 21}u_{1\ }^{\ 21}+w_{1\ }^{\ 22}u_{1\ }^{\
22})}{w_{1\ }^{\ 21}u_{1\ }^{\ 21}+w_{1\ }^{\ 22}u_{1\ }^{\ 22}}=\frac{w_{1\
}^{\ 2i}(u_{2}+1/4\tilde{u}_{1}(w_{1\ }^{\ 21}+w_{1\ }^{\ 22}))}{w_{1\ }^{\
21}+w_{1\ }^{\ 22}}$

and adding restrictions from the predetermined output units

$\displaystyle\tilde{u}^{1}=w_{1\ }^{\ 1}u_{1\ }^{\ 1}+w_{2\ }^{\ 1}u_{2\ }^{\
1}=w_{1\ }^{\ 1}\frac{\tilde{u_{1}}}{4}+w_{2\ }^{\ 1}u_{2\ }^{\ 1}$
$\displaystyle\tilde{u}^{2}=w_{1\ }^{\ 2}u_{1\ }^{\ 2}+w_{2\ }^{\ 2}u_{2\ }^{\
2}=w_{1\ }^{\ 2}\frac{\tilde{u_{1}}}{4}+w_{2\ }^{\ 2}u_{2\ }^{\ 2}$

would lead to the underdetermined weight system

$\displaystyle\begin{dcases}&0w_{2\ }^{\ 1i}=0\\\ &4(\tilde{u}^{i}-w_{1\ }^{\
i}\tilde{u}_{1})(w_{1\ }^{\ 21}+w_{1\ }^{\ 22})=w_{1\ }^{\ i}w_{1\ }^{\
2i}(4\tilde{u}_{2}+\tilde{u}_{1}(w_{1\ }^{\ 21}+w_{1\ }^{\ 22}))\\\
&4\tilde{u}^{i}-w_{1\ }^{\ i}\tilde{u}_{1}=4w_{2\ }^{\ i}u_{2\ }^{\
i}\end{dcases}$

Simulations with different choices of weights solving this system and initial
conditions (always satisfying $u_{1\ }^{\ ml}=u_{1\ }^{\
m^{\prime}l^{\prime}}$ since other behaviors of the system are unreachable
with the restriction (83)) were performed where a desired output vector
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}$ was repeatedly obtained from the constant
input vector $[\tilde{u}_{1},\tilde{u}_{2}]^{T}$. Analogously, we can consider
the original algebraic system as a polynomial system with both the potentials
and the weights as unknowns. Because of the unbiased features condition (83),
the attractor $(MN-1)$-plane is replaced by an equilibrium point and the
original system is fully determined in all potentials. We can can therefore
express these potentials entirely in terms of the fixed input and output
potentials, obtain equations only for the weights, and use these to place the
partial (output only) equilibrium in a desired location within the subspace
spanned by the output units. Because the full equilibrium points satisfying
the system are stable, the network will converge to the desired configuration.
Because the network is feedforward only and because of (83) there is just one
such equilibrium point and it is the $\omega$-limit set of all system
trajectories. In this way we can guarantee the reachability of the steady
state in addition to its existence.

The lack of uniqueness of the solutions of this system can be used in order to
match multiple pairs of inputs and equilibria. For instance, in order to
obtain the equilibrium $[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[2,2]^{T}$ from the
input $[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[1,1]^{T}$ and the equilibrium
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[1,2]^{T}$ from the input
$[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[1,2]^{T}$ , we have to form a new system
from the solutions of the two systems to obtain a more restricted solution
$w_{1\ }^{\ 1}=12+r$, $w_{1\ }^{\ 2}=8$, $w_{1\ }^{\ 22}=0$, $w_{2\ }^{\
1}=-1$, $w_{1\ }^{\ 21}=r$, $w_{2\ }^{\ 2}=s$, $w_{2\ }^{\ 11}=p$, , $w_{2\
}^{\ 12}=q$. This can still be done manually but for sufficiently many systems
we need to use at least a symbolic solver. It is important to note that the
extra degrees of freedom cannot be useful in all possible cases of input-
output associations and finding weights for multiple such might result in
trivial or no solution. The reason that it is possible to find weights for
some cases at all possibly has to do with the fact that the equations are
polynomials of degree larger than one. It can be seen in the stability
analysis of the motifs of (76) that their balancing equilibria are even higher
degree polynomials in the weights and look more promising for such endeavor.
At the same time, such endeavor is necessary as we will show next.

The original cognitive interpretation of the attractor networks usually
considers the initial conditions in the phase space as the actual clue that is
used for the recall. The memories are then synthesized as asymptotically
stable equilibria so that their basins of attraction represent their volumes
of robustness with respect to these initial conditions. Such robustness with
respect to previous states of the network makes a lot less sense than
robustness with respect to the external input to the system which is usually
kept constant or zero in attractor network studies. The external input which
is nonzero only for input units is a lot more natural way to represent
external information passed to the network than its initial condition. It also
makes more sense as a conceptualization of cognitive tasks like recognition
and prediction. Therefore, we will ultimately require that we have robustness
with respect to the external input rather than the initial conditions.

We started considering the robustness with respect to external input for the
2-unit feedforward network (80) by obtaining weights that match multiple
inputs with multiple desired output equilibria. The 2-output version of (76)
can be fully stated as

$\displaystyle\begin{dcases}&0=u_{1\ }^{\ 21}=u_{1\ }^{\ 22}=u_{2\ }^{\
11}=u_{2\ }^{\ 12}\\\ &\tilde{u}^{i}/6=u^{i\ }_{\ 1}=u^{i\ }_{\ 2}=u^{i\ }_{\
11}=u^{i\ }_{\ 12}=u^{i\ }_{\ 21}=u^{i\ }_{\ 22}\\\ &\tilde{u}_{i}=U_{i}+w^{1\
}_{\ i}u^{1\ }_{\ i}+w^{2\ }_{\ i}u^{2\ }_{\ i}=U_{i}+(w^{1\ }_{\
i}\tilde{u}^{1}+w^{2\ }_{\ i}\tilde{u}^{2})/6\\\ &\tilde{u}^{i}=w_{1\ }^{\
i}u_{1\ }^{\ i}+w_{2\ }^{\ i}u_{2\ }^{\ i}\\\ &(6u_{1\ }^{\ i}-w^{1\ }_{\
1i}\tilde{u}^{1}-w^{2\ }_{\ 1i}\tilde{u}^{2})(u_{1\ }^{\ 1}+u_{1\ }^{\
2})=6u_{1\ }^{\ i}\tilde{u}_{1}\\\ &(6u_{2\ }^{\ i}-w^{1\ }_{\
2i}\tilde{u}^{1}-w^{2\ }_{\ 2i}\tilde{u}^{2})(u_{2\ }^{\ 1}+u_{2\ }^{\
2})=6u_{2\ }^{\ i}\tilde{u}_{2}\end{dcases}$

While the feedforward case is easy enough to be done by a symbolic solver,
here this becomes computationally expensive and we have to rely on numeric
solvers only. Using iterating method, we had to specify initial guess for the
search for which we used unity vector and obtained a root for the system which
was then reproduced via simulation. For the simulation we use a specialized
algorithm which plays the role of network motif calculator, i.e. we can
quickly construct and simulate any network motif of sufficiently small size
(this includes all the above mentioned networks). Because the current network
involves cycles, approximation errors for the weights tend to accumulate and
we had to increase the precision of the system root to at least
$1\text{\times}{10}^{-16}$ to obtain good estimation for the equilibrium
state. Furthermore, we had to restrict the potentials to the nonnegative
region which was done by replacing the unknowns in the objective function by
their squares and then recalculating the answer as square of the found root.
Since the solution we can obtain can also be within the nonnegative region
without this restriction and yields more accurate approximation in some cases,
we took the liberty to use this as well. Conversely, we sometimes required
nonnegativeness for the weights of the connections to restrict our focus even
though there is no immediate reason for such requirement.

Some additional preliminary results include storing multiple input-output
pairs, multiple outputs with the same input, and multiple inputs with the same
output. This was done by generating additional equations and replacing the
input and output unknowns with the fixed values for each pair. Here are
conclusions we draw for each of the three cases respectively:

  * •

While numerical solutions were found for all attempted cases of input-output
pairs, including ones with overlapping, close or distant (in terms of simple
Euclidean norm) inputs, the accuracy of the recall deteriorated with the
number of stored pairs. This could be due to increased sensitivity to the
exact solution when it is a solution of many possibilities for equilibria.
There were also spurious intermediary equilibria if the network is provided
with noisy input outside of the learned pairs. This is usually expected in the
sense that we change the external input which changes the dynamics and
therefore the resulting equilibrium point. However, we are interested in
precisely mapping multiple inputs to the same equilibrium which is the third
case here.

  * •

The observations in the case of multiple equilibria from a single external
input match the theoretical predictions drawn from previous sections. In
particular, there can either be one isolated stable equilibrium point or an
attractor $(NM-1)$-plane and only the second option is possible if we require
multiple equilibria. We considered for instance a case of the equilibrium
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[1,1]^{T}$ from the input
$[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[1,1]^{T}$ and
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[2,2]^{T}$ from the input
$[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[1,1]^{T}$ and obtained forward connection
weights $w_{1\ }^{\ 1}=w_{2\ }^{\ 1}=$1.5297$$, $w_{1\ }^{\ 2}=w_{2\ }^{\
2}=$1.4714$$, near-zero backward connection weights $w^{1\ }_{\ 1}=w^{2\ }_{\
1}=$1.0336\text{\times}{10}^{-15}$$, $w^{1\ }_{\ 2}=w^{2\ }_{\
2}=$-1.2584\text{\times}{10}^{-15}$$, and near-zero backward metaconnection
weights $w^{1\ }_{\ 11}=w^{2\ }_{\ 11}=$-1.7451\text{\times}{10}^{-14}$$,
$w^{1\ }_{\ 12}=w^{2\ }_{\ 12}=$1.9641\text{\times}{10}^{-14}$$, $w^{1\ }_{\
21}=w^{2\ }_{\ 21}=$-1.3985\text{\times}{10}^{-14}$$, $w^{1\ }_{\
22}=$0.013\,908$$, and $w^{2\ }_{\ 22}=$-0.013\,908$$. This implies that the
numerical result has excluded any influence on the forward connections by
setting the metaconnection weights to zero. The result from excluding all
backward connections’ influence is that the network reduces to a feedforward
network where we already know that we should have at least one feature unit
and therefore attractor $(NM-1)$-plane. Simulations for these weights and
external input $[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[1,1]^{T}$ converge to an
intermediate value on the attractor
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[$1.5297$,$1.4714$]^{T}$ or nearby values
using different initial conditions.

  * •

The most important case is of course the one about robustness to external
input, i.e. obtaining the same equilibrium from multiple external inputs.
Numerical solutions for such weights like obtaining
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[1,1]^{T}$ from both
$[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[1,1]^{T}$ and
$[\tilde{u}_{1},\tilde{u}_{2}]^{T}=[2,2]^{T}$ resulted in either overshoot
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[1.33,1.33]^{T}$ or undershoot
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}=[0.66,0.66]^{T}$ of the desired equilibrium
depending on the choice of desired input. Single attractor point is therefore
only the answer for obtaining robustness with respect to initial conditions
and not with respect to external input. To obtain the same attractor for a set
of external inputs, we need at least a degenerate equilibrium or more
specifically - an attractor coinciding with the locus of
$[\tilde{u}^{1},\tilde{u}^{2}]^{T}$ for all external inputs in the set.
However, to do this we have to consider cases where such locus could be a line
and have to remove the simplifying assumption of unbiased features (83) which
goes beyond a simple review of possibilities for metanetwork synthesis.

Finally, to demonstrate a more usable (mostly larger in size) implementation
of pattern classification, we need networks of more than two input units.
Larger networks introduce a rapid increase in the number of free parameters
and need numerical solvers to find roots for high-dimensional systems. Such
roots represent states where the system is at equilibrium in both the
potential and weight spaces, i.e. for particular connection strengths and
potentials. The unit potentials are kept fixed to a desired value
corresponding to a preselected memory as usual: the input units to a training
data sample and the output units to a training label. Each training pair
generates a system for the weights and we can calculate a composite system or
average over all of the solutions of the simpler systems. Measuring the
accuracy of classification on the test set then happens as each test sample is
presented to the network until it converges and upon convergence the
equilibrium potentials of the output units are extracted and compared with the
right answer.

Preliminary results on the MNIST handwritten digits as one of the classical
machine learning datasets used for image classification required the solution
of systems with 9910 equations and 118810 unknowns, 28x28 of which input units
fixed to a particular image and 10 of which output units fixed to a binary
vector with 1 for the correct digit class and 0 for the rest. Since solving a
system for all cases of input-output pairs would require the same number of
equations for each of a total of 50000 training samples and it doesn’t
guarantee robust solution as observed from simpler experiments, a more
preferred way for training was to average over multiple cases. The numerical
observations coincided with the predictions from the 2-output version of (76)
for the case of multiple input-output pairs but from a larger implementation.

##  4 Discussion

###  4.1 Additional considerations

Here we will list some additional considerations that we couldn’t place
anywhere in the text but are still important for a complete understanding of
the various topics we had to cover. First we will counter some oversimplified
criticisms of the attractor networks as associative memories, then we will
briefly explore some possible extensions of these in terms of external inputs
and classical stability, and finally we will relate our choice of mathematical
theory to both the artificial and biological neural networks.

A major motivation for using attractors in applications of object recognition
is in the fact that the network could handle a continuous input stream (or
equivalently act as an online algorithm). While proponents of the attractor
recall idea could argue that this is more biologically plausible in general,
opponents could argue that brain activity would never become steady following
such mathematical description. However, processing a stream of information
would never lead to complete convergence since the input is practically never
steady and such convergence could be optimal at the very best. An objective in
learning would then be to minimize the processing time or maximize the
convergence rate so that the next time the same input is encountered it is
handled faster and contextualized better.

Another oversimplifying idea we have to disqualify is the need to reset the
initial conditions when changing input suggested by [37]. The main argument of
Hirsch is that we cannot define a map from the external input to the
equilibrium state $H(U)=u^{*}$. If we switch the input $U$ with another
$\bar{U}$ while converging to $u^{*}$, it will change the dynamics and the
convergence to a possibly different equilibrium $\bar{u}^{*}$. Switching back
to $U$ will no longer guarantee convergence to $u^{*}$ since the initial
condition at switching can now be within the basin of attraction of a
different point $\hat{u}^{*}\neq u^{*}$. Therefore, one cannot map an input to
an attractor and needs to reset the initial condition on change of the input.
While these conclusions are true, not being able to associate equilibria with
constant external inputs might actually be a desired effect. In particular, we
can still map a sequence of constant external inputs to a final equilibrium
state and can thus encode sequential information and have a network that
recalls different memories depending on the sequence of perceived inputs.

Our conclusions on robustness with respect to external input, i.e. recall of
the same attractor from multiple external inputs, can be extended to
sustaining the same attractor for a complete external input sequence. The
simplest possible sequence is of course a single constant input and is the
expected first step in the increase of complexity of any stored information. A
sequence of constant inputs is equivalent to a piecewise constant external
input and can be generalized to any time dependent external input. The
reversed form of sustaining the same attractor for a time dependent input then
is generating time dependent output of this attractor. A cognitive
interpretation for the first case was already given - recalling the same
concept from time evolving stimulus like recognizing a known melody or
predicting a known trajectory. A cognitive interpretation for the second case
could be stimulus generation like producing handwriting or telling a verbal
story. In all cases and exciting as it is, the mathematical and engineering
realization of these suggestions deviates away from our current scope.

Another extension in the way we interpret stability for neural networks that
we had to introduce here is the concept of partial stability. A recalled
pattern is generally expected to be stored in a part of the network rather
than the entire network. The general pattern sparsity is usually estimated to
be somewhere in between of all cells and a single ”grandmother” cell but
neither of the two extremes. Storing in a subregion in a network also reduces
the required processing and coupling needed for recall. Thus, we may be
interested in convergence on a particular manifold as opposed to the full
phase space which is also easier to handle mathematically. The behavior of the
system outside of the manifold could oscillate, be chaotic or even unstable as
long as such instability is unreachable in practice due to some preventive
conditions. We are satisfied with it as long as the set of possible
computational models we consider contains a set of models achieving the high
cognitive tasks we aim at.

This brings us to a final point about the recurring topic of biological
plausibility. We prefer performing classification with an attractor network,
investigating any principles of its memory consolidation or synthesis
shortcuts, instead of backpropagating error from a prescribed correct label.
Our preference for feature detection involves two types of units instead of
simpler lateral inhibition alternatives. One might argue that we need
validation from reality and therefore are striving for optimal biological
realism and successful numerical or empirical predictions of the model would
affirm the approaches it was build on and in turn reveal new information about
the biological reality itself. In the case of the feature detector, this would
be a prediction of the extra unit type for the perception of black which could
be a subtype of a known cell in the retina inhibited and inhibiting the first
type, a new type of interaction, or something else. However, our interest is
specifically in mathematical modelling in engineering, i.e. we are not
concerned with whether or not our approach matches the biological reality as
long as it achieves its purpose. The modelling performed here is merely a
provider for engineering directions that we need in order to search for a
common solution of a wide spectrum of problems in artificial intelligence.

None of our models are centered around engineering alone either. Our main
criticism of the computationally efficient and specifically accurate yet very
artificial neural networks is exactly their excessively large deviation from
simpler and more natural solutions. At the same time these solutions are not
available from the still difficult to generalize but truly biological neural
networks. Engineering freedom in solving a problem is unquestionable but how
far this freedom can go will remain a grey area. Nevertheless, the possibility
of existence of a universal solution and its large attractiveness is one of
the main inspirations for using neural networks for many computational tasks
instead of support vector machines or other powerful and simpler machine
learning methods. Even though such multifunctional solution could be a lot
harder to identify, it must be chased ardently and developments in instances
solving particular problems should try to be as close to it as possible.

###  4.2 Current limitations

As this is a newly introduced model, it has many limitations. Here, we will
briefly outline some of them:

  * •

Since the current metanetwork is a rate-based model, it suffers from all the
limitations that the firing rate approximation carries. Most important of
these is the fact that it cannot account for sequential encoding and time
related phenomena as do pulsed models which are also trying to be as minimal
as possible in terms of biological detail.

  * •

Existence of connections and usage of zero weights is not the same for the
current model. While to define the existence of a connection by a nontrivial
weight makes sense and ties very well mathematical generality with specific
implementation, the context term (61) in the equations is not compatible with
this concept. Adding weights to the term is also not possible since the
denominator could become zero if all weights in it are zero.

  * •

Many parts of the model can be modified later on and are currently
experimental. This is one of the main reasons it was introduced as a family of
models with greater modularity of the added dynamics.

  * •

The added structure increases significantly the computational requirements for
simulation because it adds equations for all connections which are all pairs
of units and equations for all metaconnections which are all pairs of units
and connections.

  * •

As mentioned before, robustness with respect to external input and input
sequences is a lot more intuitive for the concept of memory recall and is
currently lacking.

###  4.3 Future developments

Besides finding better ways to deal with the limitations the current model is
facing, two major extensions which are still not well-developed and well-
studied are

  * •

plasticity - specifically learning which can bring to adaptation and
generalization of input

  * •

abstraction - a multilayered network is capable of hierarchical
representations where lower-level features are combined in higher level ones

Regarding the first direction, finding a way to make the network rearrange
itself properly is the only way of storing the information it needs to know to
perform even the most basic cognitive tasks. This poses further challenges in
terms of supervision from how to obtain feedback about the quality or success
of storing to could we form known network structures like feedforward and
feedback distinction, connection cycles, cell specializations and specialized
cell arrangements.

Regarding the second direction, multiple layers can be helpful as multiple
levels of abstraction for tasks like object detection and classification
similarly to the case of DNNs. They could allow the network to perform
classification of images on more challenging datasets like CIFAR-10,
CIFAR-100, STL-10, SVHN, and ILSVRC2012 task 1. In the most realistic case a
multilayer learning capable version can be tested on video data sets which are
the closest to a real environment data allowing for cognitive development.

## Visuals

Figure 19: Visual experiments with the feature detector - each row is a
separate case with two stages for the first two boxes and results in the
second two boxes: a) white and grey uniform input - outputs for both unit
types for both stages; b) contour line of high and low contrast - output after
sufficiently long time (both unit types for the high contrast and the first
unit type for the low contrast) and short-term output of the low contrast (for
first unit type, remains long-term for second unit type); c) two simultaneous
contour lines of high and low contrast and a gradient - output of second unit
type throughout for the two lines and at a later stage for the gradient; d)
shapes introducing corners - output for both unit types for the triangle and
for the first unit type for the square (fixed stage); e) shapes hidden by
noise - output for the first unit type for both the star and the square (fixed
stage);

$\begin{array}[]{cc}\includegraphics[scale={0.4}]{point_motion}&\includegraphics[scale={0.4}]{triangle_motion}\\\
\includegraphics[scale={0.36}]{video_motion}&\includegraphics[scale={0.27}]{camera_motion}\end{array}$

Figure 20: Motion detection as temporal features: a) static, moving, and
blinking dots (top-left) with two snapshots in white and outputs of the first
and second unit type in black; b) rotating triangle (top-right) with two
snapshots in white and outputs of the first and second unit type in black; c)
video (bottom-left) with two snapshots to the left and the first unit type
response on the right; d) camera (bottom-right) with overadaptation due to
long exposure;

$\begin{array}[]{cc}\includegraphics[scale={0.8}]{tracker_periphery}&\includegraphics[scale={0.8}]{tracker_center}\end{array}$

Figure 21: Object tracker resolutions - peripheral vision (left) and central
vision (right) corresponding to the subregion of directed attention.

## Technologies used

Our main language of choice is python (2.7) together with numpy and OpenCV
packages for numerical and respectivelly computer vision processing. We also
made use of one of the main MPI bindings packages in the open source world
namely mpi4py [16]. The porting and development of any prototype code
specifically for the MPI (message passing interface) was necessary in order to
make the best use of the supercomputing resources provided by Caliban - the
high performance cluster in the University of L’Aquila - mainly for parameter
exploration regarding the feature detector and investigation of multiple types
of cellular neural network connectivity. Parallelizing the code was possible
because of its design which involves step synchronized virtual time for the
simulation, locally to loosely connected regions making use of a halo swapping
technique, and memory mapped and region restricted data file loading.

We also made additional use of software like octave and maxima for the network
synthesis part and wrote and tested some small code sections on theano, torch,
and tensorflow.

## References

  * [1] A Paul Alivisatos, Miyoung Chun, George M Church, Ralph J Greenspan, Michael L Roukes, and Rafael Yuste, _The brain activity map project and the challenge of functional connectomics_ , Neuron 74 (2012), no. 6, 970–974.
  * [2] B Ambrosio and MA Aziz-Alaoui, _Synchronization and control of coupled reaction–diffusion systems of the fitzhugh–nagumo type_ , Computers & Mathematics with Applications 64 (2012), no. 5, 934–943.
  * [3] David Anderson, _Your brain is more than a bag of chemicals_ , 2016.
  * [4] Francesco P Battaglia and Alessandro Treves, _Stable and rapid recurrent processing in realistic autoassociative memories_ , Neural Computation 10 (1998), no. 2, 431–450.
  * [5] C Buchel, _Different activation patterns in the visual cortex of late and congenitally blind subjects_ , Brain 121 (1998), no. 3, 409–419.
  * [6] John H. Byrne, _Introduction to neurons and neuronal networks_ , 2016.
  * [7] Jinde Cao and Qing Tao, _Estimation on domain of attraction and convergence rate of hopfield continuous feedback neural networks_ , Journal of Computer and System Sciences 62 (2001), no. 3, 528–534.
  * [8] S. Carpenter, G.A.Grossberg, _The art of adaptive pattern recognition by a self-organizing neural network_ , Computer 21 (1988), no. 3, 77–88.
  * [9] Tianping Chen and Shun Ichi Amari, _Stability of asymmetric hopfield networks_ , IEEE Transactions on Neural Networks 12 (2001), no. 1, 159–163.
  * [10] Xiaofeng Chen, Yafei Zhou, and Qiankun Song, _Multistability of delayed neural networks with discontinuous activations_ , International Symposium on Neural Networks, Springer, 2010, pp. 603–610.
  * [11] Chang-Yuan Cheng, Kuang-Hui Lin, and Chih-Wen Shih, _Multistability in recurrent neural networks_ , SIAM Journal on Applied Mathematics 66 (2006), no. 4, 1301–1320.
  * [12] , _Multistability and convergence in delayed neural networks_ , Physica D: Nonlinear Phenomena 225 (2007), no. 1, 61–74.
  * [13] Michael A Cohen and Stephen Grossberg, _Absolute stability of global pattern formation and parallel memory storage by competitive neural networks_ , IEEE transactions on systems, man, and cybernetics (1983), no. 5, 815–826.
  * [14] C. F. Connor, J. A.Stevens, _Prediction of repetitive firing behaviour from voltage clamp data on an isolated neurone soma_ , The Journal of Physiology 213 (1971), no. 1, 31–53.
  * [15] Francis Crick, _The recent excitement about neural networks_ , Nature 337 (1989), no. 6203, 129–132.
  * [16] Lisandro Dalcin, _MPI for Python_ , http://pythonhosted.org/mpi4py/, 2016\.
  * [17] Peter Dayan and L. F. Abbott, _Theoretical neuroscience: Computational and mathematical modeling of neural systems_ , The MIT Press, 2005.
  * [18] The Economist, _Brain work_ , 2011.
  * [19] Rubem Erichsen Junior and Leonardo Gregory Brunnet, _Multistability in networks of hindmarsh-rose neurons_ , Physical review. E, Statiscal, nonlinear, and soft matter physics 78 (2008), no. 6, 061917, 6 p.
  * [20] Yuguang Fang and Thomas G Kincaid, _Stability analysis of dynamical neural networks_ , IEEE Transactions on Neural Networks 7 (1996), no. 4, 996–1006.
  * [21] A. F. Filippov and F. M Arscott, _Differential equations with discontinuous righthand sides_ , Kluwer Academic Publishers, 1988.
  * [22] Richard FitzHugh, _Impulses and physiological states in theoretical models of nerve membrane_ , Biophysical Journal 1 (1961), no. 6, 445–466.
  * [23] M Forti and P Nistri, _Global convergence of neural networks with discontinuous neuron activations_ , IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 50 (2003), no. 11, 1421–1435.
  * [24] Kunihiko Fukushima, _Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position_ , Biol. Cybernetics 36 (1980), no. 4, 193–202.
  * [25] Wulfram Gerstner, _Time structure of the activity in neural network models_ , Physical review E 51 (1995), no. 1, 738.
  * [26] , _Population dynamics of spiking neurons: fast transients, asynchronous states, and locking_ , Neural computation 12 (2000), no. 1, 43–89.
  * [27] Wulfram Gerstner and Werner M Kistler, _Spiking neuron models_ , Cambridge University Press, 2002.
  * [28] Xavier Glorot and Yoshua Bengio, _Understanding the difficulty of training deep feedforward neural networks_ , In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’10). Society for Artificial Intelligence and Statistics, 2010.
  * [29] Charles G Gross, _Genealogy of the “grandmother cell”_ , The Neuroscientist 8 (2002), no. 5, 512–518.
  * [30] Stephen Grossberg, _Adaptive pattern classification and universal recoding: I. parallel development and coding of neural feature detectors_ , Biological cybernetics 23 (1976), no. 3, 121–134.
  * [31] Allon Guez, Vladimir Protopopsecu, and Jacob Barhen, _On the stability, storage capacity, and design of nonlinear continuous neural networks_ , IEEE Transactions on Systems, Man, and Cybernetics 18 (1988), no. 1, 80–87.
  * [32] Jack K Hale, _Ordinary differential equations_ , Wiley-Interscience, 1969.
  * [33] Himesh Handa and BB Sharma, _Synchronization of a set of coupled chaotic fitzhugh–nagumo and hindmarsh–rose neurons with external electrical stimulation_ , Nonlinear Dynamics (2016), 1–16.
  * [34] D. O Hebb, _The organization of behavior_ , Wiley, 1949.
  * [35] G. Hinton, P Dayan, B. Frey, and R. Neal, _The ”wake-sleep” algorithm for unsupervised neural networks_ , Science 268 (1995), no. 5214, 1158–1161.
  * [36] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh, _A fast learning algorithm for deep belief nets_ , Neural Computation 18 (2006), no. 7, 1527–1554.
  * [37] Morris W Hirsch, _Convergent activation dynamics in continuous time networks_ , Neural Networks 2 (1989), no. 5, 331–349.
  * [38] Sepp Hochreiter and Jürgen Schmidhuber, _Long short-term memory_ , Neural Computation 9 (1997), no. 8, 1735–1780.
  * [39] Alan Hodgkin and Andrew Huxley, _A quantitative description of membrane current and its application to conduction and excitation in nerve_ , Bulletin of Mathematical Biology 52 (1990), no. 1-2, 25–71.
  * [40] John J Hopfield, _Neural networks and physical systems with emergent collective computational abilities_ , Proceedings of the national academy of sciences 79 (1982), no. 8, 2554–2558.
  * [41] , _Neurons with graded response have collective computational properties like those of two-state neurons_ , Proceedings of the national academy of sciences 81 (1984), no. 10, 3088–3092.
  * [42] Kurt Hornik, Maxwell Stinchcombe, and Halbert White, _Multilayer feedforward networks are universal approximators_ , Neural Networks 2 (1989), no. 5, 359–366.
  * [43] Gan Huang and Jinde Cao, _Multistability of neural networks with discontinuous activation function_ , Communications in Nonlinear Science and Numerical Simulation 13 (2008), no. 10, 2279–2289.
  * [44] Yujiao Huang, Huaguang Zhang, and Zhanshan Wang, _Dynamical stability analysis of multiple equilibrium points in time-varying delayed recurrent neural networks with discontinuous activation functions_ , Neurocomputing 91 (2012), 21–28.
  * [45] D. H. Hubel and T. N. Wiesel, _Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex_ , The Journal of Physiology 160 (1962), no. 1, 106–154.
  * [46] Independent, _The human brain is the most complex structure in the universe. let’s do all we can to unravel its mysteries_ , 2014.
  * [47] E. M. Izhikevich, _Simple model of spiking neurons_ , IEEE Transactions on Neural Networks 14 (2003), no. 6, 1569–1572 (English).
  * [48] , _Which model to use for cortical spiking neurons?_ , IEEE Transactions on Neural Networks 15 (2004), no. 5, 1063–1070 (English).
  * [49] Teuvo Kohonen, _Self-organized formation of topologically correct feature maps_ , Biol. Cybern. 43 (1982), no. 1, 59–69.
  * [50] Andrey Kurenkov, _A ’brief’ history of neural nets and deep learning_ , 2015\.
  * [51] Louis Lapicque, _Recherches quantitatives sur l’excitation électrique des nerfs traitée comme une polarisation_ , J. Physiol. Pathol. Gen 9 (1907), no. 1, 620–635.
  * [52] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, _Backpropagation applied to handwritten zip code recognition_ , Neural Computation 1 (1989), no. 4, 541–551.
  * [53] Daniel S Levine and Stephen Grossberg, _Visual illusions in neural networks: Line neutralization, tilt after effect, and angle expansion_ , Journal of Theoretical Biology 61 (1976), no. 2, 477–504.
  * [54] J-H Li, AN Michel, and W Porod, _Analysis and synthesis of a class of neural networks: Variable structure systems with infinite grain_ , IEEE Transactions on circuits and systems 36 (1989), no. 5, 713–731.
  * [55] Jian-Hua Li, Anthony N Michel, and Wolfgang Porod, _Qualitative analysis and synthesis of a class of neural networks_ , IEEE Transactions on Circuits and Systems 35 (1988), no. 8, 976–986.
  * [56] Rodolfo Llinas, _Neuron_ , Scholarpedia 3 (2008), no. 8, 1490.
  * [57] Wenlian Lu, Lili Wang, and Tianping Chen, _On attracting basins of multiple equilibria of a class of cellular neural networks_ , IEEE Transactions on Neural Networks 22 (2011), no. 3, 381–394.
  * [58] Wolfgang Maass, _Networks of spiking neurons: The third generation of neural network models_ , Neural Networks 10 (1997), no. 9, 1659 – 1671\.
  * [59] Stephen L Macknik and Susana Martinez-Conde, _The spatial and temporal effects of lateral inhibitory networks and their relevance to the visibility of spatiotemporal edges_ , Neurocomputing 58 (2004), 775–782.
  * [60] Hanspeter A Mallot, _Computational neuroscience_ , Springer, 2013.
  * [61] Susana Martinez-Conde, Stephen L Macknik, and David H Hubel, _The role of fixational eye movements in visual perception_ , Nature Reviews Neuroscience 5 (2004), no. 3, 229–240.
  * [62] Warren S. McCulloch and Walter Pitts, _A logical calculus of the ideas immanent in nervous activity_ , Bulletin of Mathematical Biophysics 5 (1943), no. 4, 115–133.
  * [63] Blausen Medical, _Blausen gallery 2014_ , 2014.
  * [64] Anthony N Michel and Jay A Farrell, _Associative memories via artificial neural networks_ , IEEE Control Systems Magazine 10 (1990), no. 3, 6–17.
  * [65] Anthony N Michel, Jay A Farrell, and Wolfgang Porod, _Qualitative analysis of neural networks_ , IEEE Transactions on Circuits and Systems 36 (1989), no. 2, 229–243.
  * [66] Marvin Minsky and Seymour Papert, _Perceptrons_ , MIT Press, 1988.
  * [67] Catherine Morris and Harold Lecar, _Voltage oscillations in the barnacle giant muscle fiber._ , Biophysical journal 35 (1981), no. 1, 193.
  * [68] Michael Okun and Ilan Lampl, _Balance of excitation and inhibition_ , Scholarpedia 4 (2009), no. 8, 7467.
  * [69] D. Pinault, Y. Smith, and M. Deschenes, _Dendrodendritic and axoaxonic synapses in the thalamic reticular nucleus of the adult rat_ , J. Neurosci. 17 (1997), no. 9, 3215–3233.
  * [70] Wilfrid Rall, _Core conductor theory and cable properties of neurons_ , ch. 3, pp. 39–97, John Wiley and Sons, Inc., 2011.
  * [71] Wilfrid Rall, G.M. Shepherd, T.S. Reese, and M.W. Brightman, _Dendrodendritic synaptic pathway for inhibition in the olfactory bulb_ , Experimental Neurology 14 (1966), no. 1, 44–56.
  * [72] V. Ramachandran, _The perception of phantom limbs. the d. o. hebb lecture_ , Brain 121 (1998), no. 9, 1603–1630.
  * [73] Josef P. Rauschecker, _Compensatory plasticity and sensory substitution in the cerebral cortex_ , Trends in Neurosciences 18 (1995), no. 1, 36–43.
  * [74] R. M. Rose and J. L. Hindmarsh, _The assembly of ionic currents in a thalamic neuron i. the three-dimensional model_ , Proceedings of the Royal Society B: Biological Sciences 237 (1989), no. 1288, 267–288.
  * [75] F. Rosenblatt, _The perceptron: A probabilistic model for information storage and organization in the brain._ , Psychological Review 65 (1958), no. 6, 386–408.
  * [76] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, _Learning representations by back-propagating errors_ , Nature 323 (1986), no. 6088, 533–536.
  * [77] E. Salinas and L. F. Abbott, _Transfer of coded information from sensory to motor networks_ , J. Neurosci. 15 (1995), no. 10, 6461–6474.
  * [78] H. S. Seung, _How the brain keeps the eyes still_ , Proceedings of the National Academy of Sciences 93 (1996), no. 23, 13339–13344.
  * [79] H. Sompolinsky and R. Shapley, _New perspectives on the mechanisms for orientation selectivity_ , Curr. Opin. Neurobiol. 7 (1997), no. 4, 514–522.
  * [80] Subramania I Sudharsanan and Malur K Sundareshan, _Equilibrium characterization of dynamical neural networks and a systematic synthesis procedure for associative memories_ , IEEE transactions on neural networks 2 (1991), no. 5, 509–521.
  * [81] A. L. Taylor, J. M. Goaillard, and E. Marder, _How multiple conductances determine electrophysiological properties in a multicompartment model_ , J. Neurosci. 29 (2009), no. 17, 5573–5586.
  * [82] Roger B Tootell, Martin S Silverman, Eugene Switkes, and Russell L De Valois, _Deoxyglucose analysis of retinotopic organization in primate striate cortex_ , Science 218 (1982), no. 4575, 902–904.
  * [83] T. Trappenberg, _Fundamentals of computational neuroscience_ , Fundamentals of Computational Neuroscience, OUP Oxford, 2010.
  * [84] Lili Wang, Wenlian Lu, and Tianping Chen, _Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions_ , Neural Networks 23 (2010), no. 2, 189–200.
  * [85] Qing Wang, Feng Chen, Wenli Xu, and Ming-Hsuan Yang, _An experimental comparison of online object-tracking algorithms_ , SPIE Optical Engineering+ Applications, International Society for Optics and Photonics, 2011, pp. 81381A–81381A.
  * [86] Heiko Wersing, Wolf-Jürgen Beyn, and Helge Ritter, _Dynamical stability conditions for recurrent neural networks with unsaturating piecewise linear transfer functions_ , Neural Computation 13 (2001), no. 8, 1811–1825.
  * [87] B. Widrow, _Adaptive ”Adaline” neuron using chemical ”memistors”._ , Tech. Report 1553-2, Stanford Electron. Labs., Stanford, CA, October 1960.
  * [88] Hugh R. Wilson and Jack D. Cowan, _Excitatory and inhibitory interactions in localized populations of model neurons_ , Biophysical Journal 12 (1972), no. 1, 1–24.
  * [89] Zhang Yi, Kok Kiong Tan, and Tong Heng Lee, _Multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions_ , Neural computation 15 (2003), no. 3, 639–662.
  * [90] Zhigang Zeng and Jun Wang, _Multiperiodicity and exponential attractivity evoked by periodic external inputs in delayed cellular neural networks_ , Neural Computation 18 (2006), no. 4, 848–870.
  * [91] Huaguang Zhang, Zhanshan Wang, and Derong Liu, _A comprehensive review of stability analysis of continuous-time recurrent neural networks_ , IEEE Transactions on Neural Networks and Learning Systems 25 (2014), no. 7, 1229–1262.
  * [92] Lei Zhang, Zhang Yi, and Jiali Yu, _Multiperiodicity and attractivity of delayed recurrent neural networks with unsaturating piecewise linear transfer functions_ , IEEE Transactions on Neural Networks 19 (2008), no. 1, 158–167.