reduce the complexity of the overall system in order to study in depth and
validate the newly introduced structure and potential dynamics. For the sake
of completeness of this text, we will at least suggest a learning rule that
could be coupled with any of the forms above.

To add plasticity to any metanetwork, we simply have to add dynamics for the
weights of all connections and metaconnections. Despite the popularity of
Hebbian learning, we will suggest a different learning rule based not on
correlation of presynaptic and postsynaptic neurons but solely on the activity
of presynaptic neurons. The reasoning behind this choice is related with the
phenomenon of consolidating information just by recalling it which also causes
inherent difficulty in intentional forgetting. A simple such access-based
learning rule that satisfies many additional requirements is

$\tau_{w}\frac{dw}{dt}=-uw+u^{2}$ (63)

where $w$ carries the index of $u$, i.e. it is the weight of the connection
$u$. Of course the time scale for the weight dynamics $\tau_{w}$ must be
significantly larger than the one for the potential $\tau_{r}$. The specific
form of this learning rule avoids forgetting everything that is not recalled
within a fixed time window by simply turning off modifications of the weight
for $u=0$ when the connection is not used. Otherwise the weight slowly
converges to the current value of the potential but still with rate
proportional to the potential and the difference $u(u-w)$. The fact that
larger potential has more effect on changing the weight has to do with
avoiding simple linear learning, i.e. requiring that higher importance of
learned information needs fewer repetitions. The greater effect of larger
difference adds the importance of surprise and the difficulty of sustaining
large weight with small potential for long. A discrete version of (63) is the
update rule

$w\rightarrow w+\epsilon(-uw+u^{2})$ (64)

The learning rule is stable and the convergence of $w$ to $u$ makes more sense
if both of them are bounded from above by unity. Otherwise, a more complex
rule can be considered where $w$ saturates as $u\rightarrow\infty$. The
learning rule is also completely local - it couples only with the local
potential and uses no global information from the network. However, it poses
more challenges to obtain meaningful stored information with such a learning
rule rather than Hebbian rule. For instance, to provide any feedback about the
success or failure of an action potential further along the processing path,
it is necessary to provide feedback connections from the upper layers back to
the lower ones and a fully recurrent network is a crucial requirement. The
learning rule also cannot change the sign of the weight for any positive
potential and inhibitory connections must be predetermined. Both of these
computational drawbacks match actual anatomical observations.

####  3.1.2 Motivation for the family of models

We complete this part with motivation about the currently formulated family of
models. The simplest form (49) has approximating power similar to that of
regular multilayer networks (Section 3.2.1). The internal potential form (50)
can be interpreted as an expanded graph where each node can represent a unit
or a connection, all with the dynamics of an additive fully recurrent network
(Section 3.2.2). This interpretation helps us use existing theory for its
well-posedness and multistability and preserve all cognitive capabilities of
(28, 29, 30) which are only particular cases of an additive network. At the
same time, it reveals some of the limitations of the dynamics and the reasons
they are also extended in later models in addition to the more elaborate
network structure.

The full mathematical development of all later forms, i.e. models with
sufficient complexity to motivate our choice of dynamics, is separated from
its motivation which is the main topic of this section and starts with the
main sections of the next part. Phase space decomposition of (53, 54) provides
with sufficient conditions on manifold-restricted stability (Section 3.2.3).
Case study of (57, 58, 59, 60) network motifs extends this to attractors in
the full phase space and derives the network level dynamics necessary for the
desired extra capabilities we will show here (Section 3.2.4). Throughout the
entire analysis we use (62) reduced to the various forms above wherever we
don’t require more restricted network topology.

The main purpose for introducing this family of models is to be able to
describe cognitive level phenomena only through their network level dynamics
similarly to the selective amplification, input integration, etc. in (28),
(29), and (30) but for activities involving compression of information and
minimization of uncertainty. The following figures are some examples of such
activities and network circuits that realize them. One could argue that
organisms with a central nervous system that minimizes the uncertainty in the
environment have an evolutionary advantageous processing capabilities but this
is by no means the starting point for our definitions.

Figure 8: Network circuits realizing attention to input.

Figure 8 shows a circuit which although oversimplified is exemplary for many
aspects of network’s behavior. The input $u_{1}u_{2}u_{3}$ could be of
different modality: a 3-pixel/photoreceptor grayscale image where each input
contains a luminosity value, 3-letter word $ABC$ where each input is binary
and reflects whether the letter was parsed from a text stream, etc. The
outputs $u^{1}$ and $u^{2}$ are stored _concepts_ or rather abstractions from
the input. The two output connections of $u_{1}$ respectively to $u^{1}$ and
$u^{2}$ can be denoted as $u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ and are the
different _interpretations_ of the $u_{1}$ input, one for each concept.
Reversely, the input connections $u_{1\ }^{\ 1}$ and $u_{3\ }^{\ 1}$ are
_representations_ of the concept $u^{1}$.

The connectivity in the circuit 8 is preselected so that more local
definitions can be taken to the surface. If we consider an input like
$A\bar{B}\bar{C}$, i.e. $u_{1}=1$, $u_{2}=0$, and $u_{3}=0$, the only active
input $u_{1}$ will _broadcast_ its signal to both $u_{1\ }^{\ 1}$ and $u_{1\
}^{\ 2}$. _Uncertainty_ results from the input $A\bar{B}\bar{C}$ because no
single interpretation of $u_{1}$ is available. However, due to the
connectivity in the circuit, this uncertainty will eventually be _resolved_ by
activating the metaconnection $u_{2\ }^{\ 11}$ and therefore increasing the
potential in $u_{1\ }^{\ 1}$. In this way, $u_{2}$ or more precisely its
interpretation $u_{2\ }^{\ 11}$ is a critical _context_ for the right
interpretation $u_{1\ }^{\ 1}$ of the input $u_{1}$. The same would be true
for $u_{3}$ if it was active throughout and the input was rather $A\bar{B}C$
\- $u_{2}$ would be the context to _explain_ the additional input $u_{3}$ as
$u_{3\ }^{\ 1}$. Since $u_{2}$ directs both $u_{1}$ and $u_{3}$ to $u^{1}$, it
can be considered as a _feature_ of $u^{1}$ that has greater importance for
activating $u^{1}$ and helping it win against _alternative_ (same
representations) concepts like $u^{2}$. Reversely, $u_{2}$ together with
$u_{1}$ and $u_{3}$ are _components_ (same interpretations) with respect to
$u^{2}$.

Taking a deeper look at the motivation behind some of the dynamics, we have to
trace the processing of the input $A\bar{B}\bar{C}$ in more detail. Due to the
broadcasting of $u_{1}$ and no previous stored potential in the two output
connections, we have that both $u_{1\ }^{\ 1}=0.5$ and $u_{1\ }^{\ 2}=0.5$.
Treating the potential of $u_{1}$ as a _resource_ that is split among the
output connections helps in pathological cases where an input is learned so
well (many interpretations) that it overtakes or _cannibalizes_ the network in
terms of activation. In this case, the emitted signal will _diffuse_ among the
many output connections, so that each connection will have insignificant
effect unless it is _contextualized_ (learned with more metaconnections to it)
and _supported_ by the present context. Although $u_{1\ }^{\ 1}=0.5<\alpha$ if
the threshold $\alpha=1$, the potential storage as a property of the network
will eventually lead to the activation of $u^{1}$ if $u_{1}$ keeps firing with
the same rate for longer. At the same time, a threshold in general will help
avoiding reaction to arbitrarily small inputs (e.g. a diffused input) and
bring to sparse overall activity. Emitting through a feedback connection
$u^{1\ }_{\ 2}$ represents network’s _assumption_ and control over its own
input as well as the _imagination_ /reconstruction of a learned input
$AB\bar{C}$ achieved through the recall of the input $B$. The cycle
$u^{1}\rightarrow u^{1\ }_{\ 2}\rightarrow u_{2}\rightarrow u_{2\ }^{\
11}\rightarrow u_{1\ }^{\ 1}$ carries feedback for $u_{1\ }^{\ 1}$ from its
own activation but this and any later analysis is beyond the scope of the
current text. Increasing the emitting through $u_{1\ }^{\ 1}$ together with
the signal resource policy leads to reduction of the emitting through $u_{1\
}^{\ 2}$ and therefore a form of _weak inhibition_ which does not involve
specialized inhibitory connections. This is in accordance with empirical
observations which show a much smaller number of inhibitory connections (in
particular inhibitory interneurons or neurons withing the central nervous
system) instead of a ratio of 1 [68] which is often employed in regular
networks that need sufficient control of activation.

Taking another step towards credibility of such circuitry, we have to mention
some cognitive level properties like the high heterogeneity of representations
visualized by figure 9. A concept as an abstraction of the input, e.g. a
triangle, can be very insensitive (robust) with respect to a lot of noise and
corruption of the input as long as such variations are not features, and very
sensitive to differences as small as a single extra activated input if this
input is a feature of a concept. In the case of triangle, a fourth properly
placed point would direct all previous points towards a square concept.
Although features are most influential context for the rest of the overall
input, they are also contextually influenced (however still broadcasting in
comparison to fully resolved/explained inputs) and therefore are not
structurally fixed for a given concept.

Figure 9: Heterogeneity of representations of a triangle.

Another exemplary circuits could demonstrate cognitive level phenomena like
vertical attention (towards uncertainty higher in the abstraction hierarchy),
competition of concepts by spreading the influence of their features and
concentrating the influence of competitive ones, etc. Of course, the case
studied currently is an oversimplification of actual network circuits aimed at
demonstrating a good amount of such phenomena with unrealistically few
connections. Many definitions arising here should in fact be a lot more
general:

  * •

interpretations/representations of a concept could be combinations of
input/output connections also called _internal states_

  * •

a concept can be stored as an _active state_ (as a set of active units) or an
_active pattern_ (temporal sequence of states) rather than a single unit
(grandmother cell [29])

All demonstrated behaviors are not guaranteed to match any empirical reality
and all the models are not guaranteed to reproduce these behaviors. The first
is not necessary as long as the network is capable of performing goals of
sufficiently high generality, subject to evaluation through numerical
implementations. The second is one of the main objectives of our current
mathematical analysis together with providing guidance for new possible
behaviors and dynamics to produce them.

###  3.2 Mathematical analysis

The mathematical development of the proposed models includes analysis of their
approximating power (what kinds of input can be learned and represented),
well-posedness (existence and uniqueness of their solutions or restriction to
regions where these are provided), stability (multistability and
multiperiodicity of simpler forms, partial stability for more complex forms),
and overall dynamics (most complex forms, rather from simpler to more complex
network motifs).

####  3.2.1 Universal approximation

The model (49) is a universal approximator for any Borel measurable function
similarly to a multilayer network with an arbitrary number of hidden units. In
order to demonstrate this, we need some definitions and theorems from [42].

######  Definition 3.1.

For any $r\in\mathbb{N}$, $A^{r}$ is the set of all affine functions
$A:\mathbb{R}^{r}\rightarrow\mathbb{R}:A(\mathbf{x})=\mathbf{w}\cdot\mathbf{x}+b$
where $\mathbf{w},\mathbf{x}\in\mathbb{R}^{r}$ and $b\in\mathbb{R}$.

######  Definition 3.2.

For any (Borel) measurable function $G(\cdot):\mathbb{R}\rightarrow\mathbb{R}$
and $r\in\mathbb{N}$, $\Sigma^{r}(G)$ is a class of functions
$\\{f:\mathbb{R}^{r}\rightarrow\mathbb{R}:f(\mathbf{x})=\sum_{j=1}^{q}\beta_{j}G(A_{j}(\mathbf{x}))\\}$
where $\mathbf{x}\in\mathbb{R}^{r},\beta_{j}\in\mathbb{R},A_{j}\in
A^{r},q\in\mathbb{N}$.

######  Definition 3.3.

For any (Borel) measurable function $G(\cdot):\mathbb{R}\rightarrow\mathbb{R}$
and $r\in\mathbb{N}$, $\Sigma\Pi^{r}(G)$ is a class of functions
$\\{f:\mathbb{R}^{r}\rightarrow\mathbb{R}:f(\mathbf{x})=\sum_{j=1}^{q}\beta_{j}\prod_{k=1}^{l_{j}}G(A_{jk}(\mathbf{x}))\\}$
where $\mathbf{x}\in\mathbb{R}^{r},\beta_{j}\in\mathbb{R},A_{jk}\in
A^{r},l_{j}\in\mathbb{N},q\in\mathbb{N}$.

######  Definition 3.4.

A subset $S$ in a metric space $(C^{r},\rho_{k})$ where for $f,g\in C^{r}$ we
have the metric $\rho_{k}(f,g)=\sup_{x\in K}|f(x)-g(x)|$ is said to be
uniformly dense on compacta in $C^{r}$ if for every compact subset
$K\subset\mathbb{R}^{r}$ we have that $S$ is $\rho_{k}$-dense in $C^{r}$.

######  Definition 3.5.

Given a probability measure $\mu$ on $(\mathbb{R}^{r},B^{r})$ define the
metric $\rho_{\mu}:M^{r}\times M^{r}\rightarrow\mathbb{R}$ by
$\rho_{\mu}(f,g)=\inf\\{\epsilon>0:\mu\\{x:|f(x)-g(x)|>\epsilon\\}<\epsilon\\}$.

######  Theorem 3.1.

Let $C^{r}$ be the set of continuous functions from $\mathbb{R}^{r}$ to
$\mathbb{R}$. Let $G$ be any continuous nonconstant function from $\mathbb{R}$
to $\mathbb{R}$. Then $\Sigma\Pi^{r}(G)$ is uniformly dense on compacta in
$C^{r}$.

######  Theorem 3.2.

Let $M^{r}$ be the set of Borel measurable functions from $\mathbb{R}^{r}$ to
$\mathbb{R}$ and $B^{r}$ the Borel $\sigma$-algebra of $\mathbb{R}^{r}$. For
any continuous nonconstant function $G:\mathbb{R}\rightarrow\mathbb{R}$, every
$r\in\mathbb{N}$, and every probability measure $\mu$ on
$(\mathbb{R}^{r},B^{r})$ we have that $\Sigma\Pi^{r}(G)$ is $\rho_{\mu}$-dense
in $M^{r}$.

Thus a function representing the instant output of a neuron
$f(\mathbf{x})\in\Sigma^{r}(G)\subset\Sigma\Pi^{r}(G)$ can approximate any
continuous function arbitrary well when the input is restricted to a compact
interval $x_{i}\in[a,b]$ from the first theorem and any Borel measurable
function in a probability measure from the second theorem. A network with an
input layer size $r$, hidden layer size $q$, and output layer size $1$ has an
output unit of the form $f\in\Sigma\Pi^{r}(G)$, i.e.
$f=\sum_{j=1}^{q}\beta_{j}\prod_{k=1}^{1}G(\mathbf{w}_{jk}\cdot\mathbf{x}+b_{jk})=\sum_{j=1}^{q}\beta_{j}G(\mathbf{w}_{j}\cdot\mathbf{x}+b_{j})=\sum_{j=1}^{q}\beta_{j}G(\sum_{i=1}^{r}w_{ij}x_{i}+b_{j})$.
The input units are respectively $x_{i}$, the weights to a hidden unit are
represented by the vector $\mathbf{w}_{j}$ while its potential is
$A_{j}=\mathbf{w}_{j}\cdot\mathbf{x}+b_{j}$ with $b_{j}$ representing the bias
term (external input) for the $j$-th hidden unit. The activation function for
a hidden unit is $G(A_{j})$ and each weight to the output unit is represented
by $\beta_{j}$. For an output layer of size $s$ we simply take a vector valued
function so that $f_{i}\in\Sigma^{r}(G)$ for $i\in[0;s]$.

We will now show that (49) has the same approximating power by considering the
output of a neuron $v_{i}=g(\mathbf{x})$. In particular, consider a linear
variant of (49) with $\alpha=0$

$\displaystyle\begin{dcases}&\tau_{r}\frac{dv_{i}}{dt}=-v_{i}+\sum_{j=1}^{N}w_{j}^{i}c_{j}^{i}\\\
&\tau_{r}\frac{dc_{j}^{i}}{dt}=-c_{j}^{i}+\sum_{\begin{subarray}{c}k=1\\\
k\neq j\end{subarray}}^{N}w_{k}^{ji}c_{k}^{ji}+u_{j}\\\
&\tau_{r}\frac{dc_{k}^{ji}}{dt}=-c_{k}^{ji}+u_{k},\ k\neq j\end{dcases}$

Assuming instantaneous influence of the variables we get

$\displaystyle\begin{dcases}&v_{i}=\sum_{j=1}^{N}w_{j}^{i}c_{j}^{i}\\\
&c_{j}^{i}=\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k}^{ji}c_{k}^{ji}+u_{j}\\\ &c_{k}^{ji}=u_{k},\ k\neq
j\end{dcases}\Rightarrow\begin{dcases}&v_{i}=\sum_{j=1}^{N}w_{j}^{i}(\sum_{\begin{subarray}{c}k=1\\\
k\neq j\end{subarray}}^{N}w_{k}^{ji}u_{k}+u_{j})\\\
&c_{j}^{i}=\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k}^{ji}u_{k}+u_{j}\\\ &c_{k}^{ji}=u_{k},\ k\neq
j\end{dcases}$

Thus, dropping the index $i$ and considering a single output $v$ ($M=1$) we
obtain that

$v=g(\mathbf{u})=\sum_{j=1}^{N}w_{j}(\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k}^{j}u_{k}+u_{j})=\sum_{j=1}^{N}w_{j}(\sum_{\begin{subarray}{c}k=1\end{subarray}}^{N}w_{k}^{j}u_{k}),\quad
w_{j}^{j}=1$

Finally, changing notation to $u_{k}=x_{k}$, $w_{j}=\beta_{j}$, $k=i$,
$G(u)=u$, $q=r=N$, and using $b_{j}=0$ we can finally see that

$g(\mathbf{x})=\sum_{j=1}^{q}\beta_{j}G(\sum_{i=1}^{r}w_{i}^{j}x_{i})\Rightarrow
g(\mathbf{x})\in\Sigma^{r}(G)\subset\Sigma\Pi^{r}(G)$

and since $G(u)=u$ is continuous nonconstant function and $x_{i}\in[0,1]$ by
theorem 3.1 we can approximate any continuous function. Alternatively, using
theorem 3.2 for $G(u)=u$ continuous nonconstant function we can conclude that
the linear variant of (49) is a universal approximator to any Borel measurable
function on $r$ dimensions. We can consider the original (49) without the
dynamics and threshold on the output units in which case $G(u)=(u-\alpha)^{+}$
is still continuous and nonconstant. Therefore, we can still apply the
theorems and conclude that an input-output layer network of the form (49) is a
universal approximator. An important distinction here is that while the
original consideration of [42] was about any neural network with at least
three layers (Section 1.3), the network (49) has only two layers. This is
intuitive since (49) has more complex dynamics like dynamics on the
connections but also fits better the observation that the neocortex has only
six layers (Section 1.2) to perform all its complex operations.

####  3.2.2 Expanded graph interpretation

In the following we will establish some results available from the literature
regarding a simplified version of (62), namely the tensor form of (50). We
will show that (50) is a particular case of the additive model (34) which is
the most widely studied neural network with well established theory for its
well-posedness and stability. We can derive (34) both by reducing the dynamics
of (62) to the dynamics of (50) or by constructing a tensor form of (50)
similarly to the way it was done for (62) but using the simpler dynamics. Here
we will take the first approach and reduce (62) to a fully recurrent network
of the form

$\displaystyle\tau_{r}\frac{du_{kji}}{dt}=-u_{kji}+c_{kji}\sum_{l=1}^{N}w_{lki}(u_{lki}-\alpha)^{+}+b_{kji}(u_{00k}-\alpha)^{+}+U_{kji}=$
$\displaystyle=-d_{kji}u_{kji}+c_{kji}\sum_{l=1}^{N}w_{lki}a(u_{lki})+b_{kji}a(u_{00k})+U_{kji}$

where $d_{kji}=1$ and $a(u)=(u-\alpha)^{+}$ is the activation function.

Figure 10: Expanded graph version of a simple circuit.

If we interpret the network as an expanded graph where each unknown, be it a
unit, connection, or a metaconnection, is a node of the graph and the
dependence among two unknowns is an edge (figure 10), we can order the
unknowns in a vector and obtain a sparse weight matrix of $(N+1)^{3}$
rows/columns. In particular,

$\displaystyle\tau_{r}\frac{du_{kji}}{dt}=-d_{kji}u_{kji}+c_{kji}\sum_{l=1}^{N}w_{lki}a(u_{lki})+b_{kji}a(u_{00k})+U_{kji}=$
$\displaystyle=-d_{kji}u_{kji}+c_{kji}\sum_{l=0}^{N}\sum_{m=0}^{N}\sum_{n=0}^{N}w^{kji}_{lmn}a(u_{lmn})+U_{kji}=$
$\displaystyle=-d_{kji}u_{kji}+\sum_{l=0}^{N}\sum_{m=0}^{N}\sum_{n=0}^{N}\tilde{w}^{kji}_{lmn}a(u_{lmn})+U_{kji}$

with $w_{lki}=w^{kji}_{lki}$, $\tilde{w}^{kji}_{lmn}=0$ if $l=0$, $m\neq k$ or
$n\neq i$, and $\tilde{w}^{kji}_{00k}=b_{kji}$ (from
$w^{kji}_{00k}=b_{kji}/c_{kji}$) which is well-defined since $b_{kji}$ depends
only on $k$ and implies that $\tilde{w}^{kji}_{00k}=0\text{ if }k=0\text{ else
}1$. Absorbing $c_{kji}$ in addition implies that $\tilde{w}^{kji}_{lmn}=0$
also if $j\neq 0$ with the exception of the case $\tilde{w}^{kji}_{00l}=1$.

After reindexing of $kji$ into $i$ and $lmn$ into $j$ to arrange $u_{kji}$ in
a vector, the equation becomes

$\tau_{r}\frac{du_{i}}{dt}=-d_{i}u_{i}+\sum_{j=1}^{(N+1)^{3}}\tilde{w}_{ij}a(u_{i})+U_{i}\iff\tau_{r}\frac{d\mathbf{u}}{dt}=-diag(\mathbf{d})\mathbf{u}+\mathbf{\tilde{W}}\mathbf{a}(\mathbf{u})+\mathbf{U}$
(65)

which is the additive model (34). We can therefore apply all theory for (34)
to draw conclusions on the existence and uniqueness of its solutions, as well
as of its attractors. Most of the available results involve sufficient
conditions to guarantee the uniqueness of a stable equilibrium point
(monostability) due to the large applicability of such networks to
optimization [91] but we are more interested in the existence of multiple
stable equilibrium points (multistability) and multiple stable limit cycles
(multiperiodicity) of (65).

The existence and uniqueness of solutions clearly depends on the choice of
activation function $a(\cdot)$. They are guaranteed for a smooth as well as
Lipschitz continuous $a(\cdot)$ by the Picard-Lindelöf theorem from the
standard theory for ODEs. In the case of discontinuous activation functions we
need to make use of Filippov theory and define a solution (also if it is an
equilibrium point or a limit cycle) in the sense of differential inclusion
[21].

The existence (but not uniqueness) of attractors is studied predominantly for
bounded activation functions (saturating or also called squashing functions)
which can be interpreted as a maximal output firing rate. However, unbounded
activation functions are also of great relevance and included in the stability
results of many papers [91]. A final distinction in the type of activation
function is about monotonicity where the first assumptions are on strictly
monotonically increasing $a(\cdot)$ but later on extended to nondecreasing
cases.

As discussed in the study of (34), the activation function
$a(u)=(u-\alpha)^{+}$ is globally Lipschitz continuous so existence,
uniqueness and continuity with respect to initial data are all immediately
verifiable and hold for all times. If we take a step of extension towards
discontinuous activation functions, we need some more general definitions
similar to the ones found in the first investigations of the topic for
monostability [23] and multistability [54].

######  Definition 3.6 (Convex hull).

For a set $E\subset\mathbb{R}^{n}$, the smallest convex set containing $E$ is
called the convex hull of $E$. The convex hull always exists and is the
intersection of all convex sets containing $E$. The closure of the convex hull
of $E$ will be denoted as $K[E]$.

The convex hull is a linear operator, i.e. $K[\alpha E_{1}+\beta E_{2}]=\alpha
K[E_{1}]+\beta K[E_{2}]$. This property is used in the next definition.

######  Definition 3.7 (Solution).

Let us consider the set-valued map $\phi:\mathbb{R}^{n}\mapsto\mathbb{R}^{n}$
defined as

$\phi(\mathbf{u})=\bigcap_{\epsilon>0}\bigcap_{\mu(N)=0}K[\mathbf{f}(B(\mathbf{u},\epsilon)\setminus
N)]=-diag(\mathbf{d})\mathbf{u}+\mathbf{\tilde{W}}K[\mathbf{a}(\mathbf{u})]+\mathbf{U}$

where $N$ is an arbitrary set with measure zero and $B$ is a ball centered at
$\mathbf{u}$ with a radius $\epsilon$. A solution of (65) on an interval
$[t_{0},t_{1}]$, $t_{0}\leq t_{1}\leq+\infty$, with initial condition
$\mathbf{u}(t_{0})=\mathbf{u_{0}}$, is an absolutely continuous function
$\mathbf{u}(t)$ defined on $[t_{0},t_{1}]$, such that
$\mathbf{u}(t_{0})=\mathbf{u_{0}}$, and for almost all $t\in[t_{0},t_{1}]$,
$\mathbf{u}(t)$ satisfies the differential inclusion
$\dot{\mathbf{u}}(t)\in\phi(\mathbf{u}(t))$.

Here,
$K[\mathbf{a}(\mathbf{u})]=([a_{1}(u_{1}^{-}),a_{1}(u_{1}^{+})],\cdots[a_{N}(u_{N}^{-}),a_{N}(u_{N}^{+})])^{T}$
is a vector of intervals defined by each point of discontinuity of
$\mathbf{a}(\mathbf{u})$. Since equilibria are themselves solutions, we define
them here as well.

######  Definition 3.8 (Equilibrium point).

An equilibrium point $\mathbf{u^{*}}\in\mathbb{R}^{n}$ of (65) is a constant
solution of (65) $\mathbf{u}(t)=\mathbf{u^{*}},\forall t\in[0,+\infty)$. Thus
$\mathbf{u^{*}}$ satisfies

$\mathbf{0}\in\phi(\mathbf{u^{*}})=-diag(\mathbf{d})\mathbf{u^{*}}+\mathbf{\tilde{W}}K[\mathbf{a}(\mathbf{u^{*}})]+\mathbf{U}$

######  Definition 3.9 (Output equilibrium point).

If $\mathbf{u^{*}}$ is an equilibrium point of (65), then there exists a
vector $\mathbf{v^{*}}\in\mathbb{R}^{n}$ such that

$\mathbf{v^{*}}\in
K[a(\mathbf{u^{*}})]\qquad\mathbf{0}=-diag(\mathbf{d})\mathbf{u^{*}}+\mathbf{\tilde{W}}\mathbf{v^{*}}+\mathbf{U}$

which is called an output equilibrium point of (65) corresponding to
$\mathbf{u^{*}}$.

It is important to consider the output equilibria separately from the state
equilibria because the activation function is no longer assumed to be
continuous, i.e. convergence in the state space does not guarantee convergence
in the outputs as was the case before.

######  Theorem 3.3 (Local existence of solutions).

Suppose that $a:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ is such that each
component $a_{i}$ of $a$ is nondecreasing, bounded, and almost everywhere
continuous with finite left and right limits at each point of discontinuity.
Then $\forall\mathbf{u_{0}}\in\mathbb{R}^{n}$ there is at least a local
solution of $\mathbf{u}(t)$ of (65) with initial condition
$\mathbf{u}(0)=\mathbf{u_{0}}$. Furthermore, any solution is bounded and hence
defined on $[0,+\infty)$.

The local existence is a consequence of [21] and can also be derived in the
case where $a_{i}$ is not necessarily bounded. The global existence is proved
by [23] with the ultimate goal of providing sufficient conditions for
monostability. On the other hand, [54] used phase space decomposition to study
multistability of (65) with $a(u)=sgn(u)$ as an activation function.

There are numerous studies in the case of smooth activation [55, 80, 20, 65,
9, 31, 7] using techniques from custom energy functionals [7] to matrix
measures [20] and providing results on local and global exponential stability
[55, 80, 20, 65, 9, 31], convergence rate [20, 7], basins of attraction [65,
80, 7], and network synthesis/design [55, 80, 31]. More recent results using
continuous saturating (often piecewise linear) activation functions have
identified the possibility of $3^{N}$ equilibria from which $2^{N}$ are
asymptotically stable and the rest unstable. In particular, [90] analyzed
n-neuron network with time-varying delay and found conditions for the
existence of $2^{N}$ locally exponential attractive limit cycles (equilibria
being a special case and a corollary). In [11, 12], the number of possible
equilibria was extended to $3^{N}$ with the $3^{N}-2^{N}$ equilibria proven to
be unstable by [84, 57] who also showed that $3^{N}$ equilibria is a
particular case for an activation function with 2 corner points (assuming
piecewise linearity), the more general result for $2r$ corner points being
$(2r+1)^{N}$ equilibria, from which $(r+1)^{N}$ locally exponentially stable
and the rest unstable. Similar results on the stable equilibria is shown by
[43, 10, 44] also for discontinuous piecewise constant activation functions
with $s$ segments, namely $s^{N}$ locally exponentially stable equilibrium
points (or limit cycles if considering periodic external input) with
$a(u)=sgn(u)$ as a particular case with $s=2$.

Choosing a continuous (piecewise linear) activation function with arbitrary
number of corners or a discontinuous activation function with arbitrary number
of segments, one may argue that we can finally achieve arbitrary high capacity
for the number of the stored memories instead of the very limited first
estimation of $0.15N$ for (34). This is not true since one also needs
synthesis procedure in order to store preselected memories and Hopfield’s
early estimate of $0.15N$ is also due to his early synthesis procedure (37).
In particular, if we consider a large number of memories that are all
correlated in some units (vectors correlated in some components), we can
obtain large weights and violate the sufficient conditions for the $3^{n}$
equilibria by [90, 11, 84].

There are also some other drawbacks in the associative memory interpretation
of (34) and therefore (50) that are indicative of the excessive simplicity of
its dynamics:

  * •

The existence of unstable equilibria does not have proper interpretation in
terms of memories although the instability will leave the equilibrium
undetectable in practice due to unavoidable perturbations.

  * •

Under the sufficient conditions for existence of stable equilibria [90, 11],
each equilibrium would also have a counterpart which is symmetric with respect
to the origin, leaving only a total of $1^{n}$ equilibria in the fully
positive region which might sometimes be a meaningful restriction.

  * •

Results on convergence [90, 11, 84] might be too restrictive since the main
methods to obtain them involve decomposing the phase space and detecting
positive invariant regions where the equations remain fully decoupled. This
means that an equilibrium along a dimension will remain constant along any
other dimension and more interesting cases cannot be considered.

  * •

Results on the basins of attraction [11, 84] are usually descriptions of
convex positive invariant sets bounded by the stable manifolds of unstable
equilibria. In many cases this might be too simple definition of the kinds of
initial conditions that are pooled together for a robust response, possibly
involving partial convergence or a different notion of proximity of clues.

  * •

Intensive partitioning of the activation function would also lead to many
small attraction regions rendering the network useless for pattern completion,
classification, or any other purpose. It is also biologically less meaningful
compared to simpler activation functions like $a(u)=(u-\alpha)^{+}$ that
better reflect the firing rate approximation.

Because it is unbounded, the activation function $a(u)=(u-\alpha)^{+}$ has
more complex dynamics, fits better observations that neurons rarely operate in
upper saturating regime, and handles spurious equilibrium points in winner-
takes-all networks [86]. Results on this type of linear threshold function
involve sufficient conditions on boundedness of the trajectories (for
symmetric and nonsymmetric connectivity) [86, 89, 92], determination of global
attractive sets [89, 92], complete convergence [89], and existence of
exponentially stable limit cycles (and therefore equilibria) in invariant sets
[92].

####  3.2.3 Phase space decomposition

We can derive a tensor form of (53, 54) from (62) similarly to the way we did
it in the previous section or analogically to the way we obtained (62) from
(57) to get

$\displaystyle\tau_{r}\frac{du_{kji}}{dt}=-d_{kji}u_{kji}+c_{kji}\sum_{l=1}^{N}w_{lki}a(u_{lki})+b_{kji}\bar{e}(u_{kji},u_{00k},\dots)+U_{kji}$
$\displaystyle\tau_{r}\frac{du_{kji}}{dt}=-a(u_{kji})+c_{kji}\sum_{l=1}^{N}w_{lki}a(u_{lki})+b_{kji}\bar{e}(u_{kji},u_{00k},\dots)+U_{kji}$

We will now emphasize on the first equation (remembering that $w_{kki}=0$) and
write it explicitly with the context term expanded as the sum of indicator
functions for each condition

$\begin{split}&\tau_{r}\frac{du_{kji}}{dt}=-d_{kji}u_{kji}+c_{kji}\sum_{l=1}^{N}w_{lki}a(u_{lki})+U_{kji}+\\\
&b_{kji}[\chi(u_{k01},\dots,u_{kNN})\frac{u_{kji}}{\sum\limits_{l=0}^{N}\sum\limits_{m=1}^{N}u_{klm}}+(1-\chi(u_{k01},\dots,u_{kNN}))\frac{1}{N(N+1)}]a(u_{00k})\end{split}$
(66)

where we use the indicator function

$\chi(u_{k01},\dots,u_{kNN})=\left\\{\begin{array}[]{ll}1\text{ if
}\sum_{l=0}^{N}\sum_{m=1}^{N}u_{klm}\neq 0\\\ 0\text{
otherwise}\end{array}\right.$ (67)

Decomposing the phase space into regions according to the choice of activation
function helps us obtain simpler dynamics and conditions on local stability in
each region. This is one of the major approaches in the study of
multistability of neural networks in literature [91] and is done both for
continuous and discontinuous activation functions. In our case however, we
will also consider affine subspaces where the context term simplifies to
broadcasting dynamics and their positive side where the distributed emitting
is nonsingular. Without loss of generality, we will also assume that
$\alpha=0$, i.e. $a(u)=(u)^{+}=\max(0,u)$.

More specifically, if we fix $\bar{k}\in[1;N]$, then in the half-space
$\\{u_{00\bar{k}}\leq\alpha=0\\}$ we have that $\forall j\in[0;N],\forall
i\in[1;N]$ the equations simplify to

$\tau_{r}\frac{du_{\bar{k}ji}}{dt}=-d_{\bar{k}ji}u_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}+U_{\bar{k}ji}$

where $\mathcal{N}\subset[1;N]$ is such that $\bar{l}\in\mathcal{N}$ if
$u_{\bar{l}\bar{k}i}>\alpha=0$. The same reduction is true for the entire
phase space for the case $\bar{k}=0$ since $b_{0ji}=0$ and the following
conclusions extend to this case without effort. If we vary along a specific
dimension $u_{\bar{k}\bar{j}\bar{i}}$ and fix all the rest
($k\neq\bar{k},j\neq\bar{j},i\neq\bar{i}$), we can find a stable equilibrium
along the resulting 1-dimensional manifold as

$u_{\bar{k}\bar{j}\bar{i}}=\frac{c_{\bar{k}\bar{j}\bar{i}}}{d_{\bar{k}\bar{j}\bar{i}}}\sum_{l\in\mathcal{N}}w_{l\bar{k}\bar{i}}u_{l\bar{k}\bar{i}}+\frac{U_{\bar{k}\bar{j}\bar{i}}}{d_{\bar{k}\bar{j}\bar{i}}}$

This partial equilibrium is stable because the differential equation
simplifies to the somewhat canonical form $\dot{y}=-y+c$ with a stable
equilibrium at $c$. It varies linearly with $u_{l\bar{k}\bar{i}}$ and is
constant with respect to all other unknowns. In both cases the dynamical
behavior along the other dimensions remains qualitatively the same as long as
$u_{00\bar{k}}\leq 0$ but in the second case it also remains quantitatively
the same. If we vary $u_{\bar{k}ji}$ and fix all the rest ($k\neq\bar{k}$), we
can find a stable equilibrium on the resulting $(N+1)N$-dimensional manifold
as

$u_{\bar{k}ji}=\frac{c_{\bar{k}ji}}{d_{\bar{k}ji}}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}+\frac{U_{\bar{k}ji}}{d_{\bar{k}ji}}$

remembering again that $w_{\bar{k}\bar{k}i}=0$ and therefore that the above
$N(N+1)$ equations are mutually decoupled. Furthermore, if $c_{\bar{k}ji}=0$
or $u_{l\bar{k}i}\leq\alpha=0$ for $l,i\in[1;N]$ (and therefore
$\mathcal{N}=\emptyset$) holds then
$u_{\bar{k}ji}=\frac{U_{\bar{k}ji}}{d_{\bar{k}ji}}$ is an equilibrium in the
$N(N+1)$-dimensional manifold as before but constant with respect to all other
dimensions, i.e. with identical quantitative behavior along $u_{kji}$ for
$k\neq\bar{k}$. Finally, if we are in the subthreshold region
$u_{kji}\leq\alpha=0$ for all unknowns, we can find a stable equilibrium in
the phase space

$u_{kji}=\frac{U_{kji}}{d_{kji}}\quad\forall k,i\in[1;N],j\in[0;N]$

which would imply noninteracting units whose potentials balance at the ratio
of each external input and decay rate.

If we restrict the dynamics to the subspace
$\\{\sum_{l=0}^{N}\sum_{m=1}^{N}u_{\bar{k}lm}=0\\}$ intersected with the
remaining half-space $\\{u_{00\bar{k}}>\alpha=0\\}$, we get that
$\chi(u_{k01},\dots,u_{kNN})=0$ and $\forall j\in[0;N],\forall i\in[1;N]$ the
equations simplify to

$\tau_{r}\frac{du_{\bar{k}ji}}{dt}=-d_{\bar{k}ji}u_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}+b_{\bar{k}ji}\frac{u_{00\bar{k}}}{N(N+1)}+U_{\bar{k}ji}$

resulting in partial equilibrium on the $N(N+1)$-dimensional manifold (and its
lower dimensional counterparts) of the form

$u_{\bar{k}ji}=\frac{c_{\bar{k}ji}}{d_{\bar{k}ji}}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}+\frac{b_{\bar{k}ji}u_{00\bar{k}}}{d_{\bar{k}ji}N(N+1)}+\frac{U_{\bar{k}ji}}{d_{\bar{k}ji}}$

If $u_{l0\bar{k}}\leq 0$ and $u_{l\bar{k}i}\leq 0$ (or $c_{\bar{k}ji}=0$) for
$l\in[1;N],l\neq\bar{k}$, we have in addition that
$u_{00\bar{k}}=\frac{U_{00\bar{k}}}{d_{00\bar{k}}}$ and therefore

$u_{\bar{k}ji}=\frac{b_{\bar{k}ji}U_{00\bar{k}}}{d_{\bar{k}ji}d_{00\bar{k}}N(N+1)}+\frac{U_{\bar{k}ji}}{d_{\bar{k}ji}}$

which is fully decoupled and therefore constant with respect to all other
dimensions. Notice that all of this holds on a $(N-1)N(N+1)$-dimensional
manifold which is the subspace
$\\{\sum_{l=0}^{N}\sum_{m=1}^{N}u_{\bar{k}lm}=0\\}$.

Up until this point we haven’t added much to the dynamics but a scaled down
version of a particular interaction with a particular unknown $u_{00\bar{k}}$.
The remainder of the phase space, namely the region
$\\{\sum_{l=0}^{N}\sum_{m=1}^{N}u_{\bar{k}lm}\neq
0\\}\cap\\{u_{00\bar{k}}>\alpha=0\\}$ is the main regime of operation of the
network and leads to equations like

$\tau_{r}\frac{du_{\bar{k}ji}}{dt}=-d_{\bar{k}ji}u_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}+b_{\bar{k}ji}\frac{u_{\bar{k}ji}u_{00\bar{k}}}{\sum\limits_{l=0}^{N}\sum\limits_{m=1}^{N}u_{\bar{k}lm}}+U_{\bar{k}ji}$

where we have that $u_{\bar{k}\bar{k}m}=0$ from the definition of (53). To
obtain the partial equilibria now, we will define the remainder sum

$r_{kji}:=(\sum_{l=0}^{N}\sum_{m=1}^{N}u_{klm})-u_{kji}$ (68)

and use geometrical arguments. For the purpose of readability and since we
won’t modify anything about the indices until the end of the section, we will
use a simplified abusive notation that drops them altogether as follows

$\displaystyle a$ $\displaystyle:=r_{\bar{k}ji}$ $\displaystyle c$
$\displaystyle:=U_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}$
$\displaystyle b$ $\displaystyle:=b_{\bar{k}ji}u_{00\bar{k}}$ $\displaystyle
d$ $\displaystyle:=d_{\bar{k}ji}$

This implies that the equilibria satisfy

$0=-du_{\bar{k}ji}+\frac{bu_{\bar{k}ji}}{a+u_{\bar{k}ji}}+c\Rightarrow
f(u_{\bar{k}ji})=du_{\bar{k}ji}=\frac{bu_{\bar{k}ji}}{a+u_{\bar{k}ji}}+c=g(u_{\bar{k}ji})$

which has three different cases a), b), and the rest shown on figure 11.

Figure 11: The different qualitative cases of the stability of
$u_{\bar{k}ji}$: a) two locally stable equilibria; b) one globally stable
equilibrium; c) no stable equilibria; d,e) one stable and one unstable
equilibrium in reversing order; f) one semistable equilibrium (only one of two
cases shown);

When $-ba>0\Rightarrow b_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}<0$, we obtain
case a) where we have two stable equilibria $\bar{u}_{\bar{k}ji}^{1}$ and
$\bar{u}_{\bar{k}ji}^{2}$ on the two sides of a singularity which is located
at $u_{\bar{k}ji}=-a=-r_{\bar{k}ji}$. Clearly, the context term will dominate
the equation for $\dot{u}_{\bar{k}ji}$ for values $-a\pm\epsilon$ for an
arbitrary small $\epsilon>0$ where it will be positive for $-a+\epsilon$.
Since $f<g$ in
$(-\infty,\bar{u}_{\bar{k}ji}^{1})\cup(-b,\bar{u}_{\bar{k}ji}^{2})$ and $f>g$
in $(\bar{u}_{\bar{k}ji}^{1},-b)\cup(\bar{u}_{\bar{k}ji}^{2},+\infty)$, the
stability of each equilibrium can be verified by noticing that the derivative
$\dot{u}_{\bar{k}ji}$ increases on its left and decreases on its right side.
In particular, on the left of each equilibrium point
$u_{\bar{k}ji}=\bar{u}_{\bar{k}ji}^{1/2}-\epsilon$ we have that
$f(u_{\bar{k}ji})<g(u_{\bar{k}ji})\Rightarrow
du_{\bar{k}ji}<\frac{bu_{\bar{k}ji}}{a+u_{\bar{k}ji}}+c\Rightarrow
0<du_{\bar{k}ji}+\frac{bu_{\bar{k}ji}}{a+u_{\bar{k}ji}}+c=\dot{u}_{\bar{k}ji}$
and analogically on the right both implying positively invariant region
$(\bar{u}_{\bar{k}ji}^{1/2}-\epsilon,\bar{u}_{\bar{k}ji}^{1/2}+\epsilon)$ and
by the arbitrariness of $\epsilon>0$ the stability of
$\bar{u}_{\bar{k}ji}^{1/2}$. The stability is local since $f<g$ and $f>g$ each
hold on a sufficiently small interval and the basins of attraction are
respectively $(-\infty,-a)$ for $\bar{u}_{\bar{k}ji}^{1}$ and $(-a,\infty)$
for $\bar{u}_{\bar{k}ji}^{2}$. The asymptote for arbitrary large
$u_{\bar{k}ji}$ is $b+c$ which is related with the case b).

When $-ba=0\Rightarrow b_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}=0$, we obtain
case b) where we have one globally stable equilibrium $\bar{u}_{\bar{k}ji}$
and no singularity. Our $g(u_{\bar{k}ji})=b+c$ is constant function equal to
the asymptotic limit and since $f(u_{\bar{k}ji})=du_{\bar{k}ji}$ where $d$ is
always positive (as decay rate), $\bar{u}_{\bar{k}ji}$ will be in the positive
half-space of $u_{\bar{k}ji}$ if
$b+c=b_{\bar{k}ji}u_{00\bar{k}}+U_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}>0$.
The stability of $\bar{u}_{\bar{k}ji}$ can be verified similarly to the way it
was done in a) and the equilibrium can be found as

$\bar{u}_{\bar{k}ji}=\frac{c_{\bar{k}ji}}{d_{\bar{k}ji}}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}+\frac{b_{\bar{k}ji}}{d_{\bar{k}ji}}u_{00\bar{k}}+\frac{U_{\bar{k}ji}}{d_{\bar{k}ji}}$

When $-ba<0\Rightarrow b_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}>0$, we obtain
cases c), d), e), and f) according to the number of $f=g$ intersections (0, 1,
or 2) that we have. In order to derive conditions to distinguish among these
cases, we need to compare the position of the points of $g$ that have the same
slope as $f$. This means that we have to solve

$\displaystyle
d=g^{\prime}(u_{\bar{k}ji}^{1/2})=(\frac{bu_{\bar{k}ji}^{1/2}}{a+u_{\bar{k}ji}^{1/2}}+c)^{\prime}=\frac{ba}{(a+u_{\bar{k}ji}^{1/2})^{2}}$
$\displaystyle\Rightarrow
u_{\bar{k}ji}^{1/2}=-a\pm\sqrt{\frac{ba}{d}}=-r_{\bar{k}ji}\pm\sqrt{\frac{b_{\bar{k}ji}u_{00\bar{k}}}{d_{\bar{k}ji}}r_{\bar{k}ji}}$

where we always have $u_{\bar{k}ji}^{1/2}\in\mathbb{R}$ and
$u_{\bar{k}ji}^{2}<u_{\bar{k}ji}^{1}$ since we already assumed
$b_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}>0$ and $d_{\bar{k}ji}>0$. As a result
we calculate

$\displaystyle
f(u_{\bar{k}ji}^{1/2})=d_{\bar{k}ji}u_{\bar{k}ji}^{1/2}=-d_{\bar{k}ji}r_{\bar{k}ji}\pm\sqrt{b_{\bar{k}ji}d_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}}$
$\displaystyle
g(u_{\bar{k}ji}^{1/2})=\frac{b(-a\pm\sqrt{\frac{ba}{d}})}{a-a\pm\sqrt{\frac{ba}{d}}}+c=\frac{\mp
ab+b\sqrt{\frac{ba}{d}}}{\sqrt{\frac{ba}{d}}}+c=\mp\sqrt{abd}+b+c=$
$\displaystyle=\mp\sqrt{b_{\bar{k}ji}d_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}}+b_{\bar{k}ji}u_{00\bar{k}}+U_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l\in\mathcal{N}}w_{l\bar{k}i}u_{l\bar{k}i}$

Thus, if $f(u_{\bar{k}ji}^{2})<g(u_{\bar{k}ji}^{2})$ and
$f(u_{\bar{k}ji}^{1})>g(u_{\bar{k}ji}^{1})$ we have case c) with no
intersection and the trajectories converging towards the singularity at $-a$.
If $f(u_{\bar{k}ji}^{2})=g(u_{\bar{k}ji}^{2})$ or
$f(u_{\bar{k}ji}^{1})=g(u_{\bar{k}ji}^{1})$, we have case f) with one
semistable equilibrium point to the left or to the right of $-a$ with
trajectories slowing down and stopping at the point and then accelerating away
from it and towards $-a$. Finally, if
$f(u_{\bar{k}ji}^{1})>g(u_{\bar{k}ji}^{1})$ and
$f(u_{\bar{k}ji}^{2})>g(u_{\bar{k}ji}^{2})$ we have case d) and if
$f(u_{\bar{k}ji}^{1})<g(u_{\bar{k}ji}^{1})$ and
$f(u_{\bar{k}ji}^{2})<g(u_{\bar{k}ji}^{2})$ we have case e). In the case d) we
have one stable and one unstable equilibrium point $\hat{u}_{\bar{k}ji}$ and
$\check{u}_{\bar{k}ji}$ to the left of the singularity with basin of
attraction of the stable point being $(-\infty,\check{u}_{\bar{k}ji})$. In the
case e), $\hat{u}_{\bar{k}ji}$ is unstable and $\check{u}_{\bar{k}ji}$ stable,
both are to the right of the singularity, and the basin of attraction of the
stable point is $(\hat{u}_{\bar{k}ji},+\infty)$.

In our use of the model (66), we want to avoid the singularity surface
encountered in the above decomposition as well as the entire dynamics on the
negative side of $u_{\bar{k}ji}$ which doesn’t make much sense in the
phenomenological representation of the context term. Even though the output
firing rate must always be positive from its definition and the internal
potential is generally allowed to be negative from its derivation, we will
require nonnegative internal potential, considering inhibition to play the
role of depletion of any positive such. Requiring all $u_{kji}$ to be
nonnegative however means that some trajectories of the system are no longer
admissible since the autonomous dynamics of the network can drive some units
to negative potentials. This wasn’t the case for all output firing rate models
since $a(u)=(u-\alpha)^{+}\geq 0$ and each output firing rate would decay to a
positive value. Instead of modifying the dynamics to turn off decrease of
potential when approaching zero, we will take the simpler approach of using a
resetting condition like

$u\leftarrow 0\text{ if }u<0$

Assuming $u_{kji}\geq 0$ for $k,j,i\in[0;N]$ (with the exception of
$u_{000}=1$) implies that

$-a=-r_{\bar{k}ji}\leq 0\qquad-ba=-b_{\bar{k}ji}u_{00\bar{k}}r_{\bar{k}ji}\leq
0$

and therefore case a) is no longer possible. More importantly, the singularity
is locked on the negative side and not reachable by the constrained dynamics
of (66) since $-a\leq 0$ and if $-a=0\Rightarrow-ba=0$ and we are in case b)
which doesn’t have a singularity. Being some $\epsilon$ positive distance away
from the singularity also allows us to still guarantee the existence and
uniqueness of solutions in the positive region. In addition, we are interested
in obtaining nontrivial stability in the positive region. Since $-a\leq 0$
cases c) and d) reduce to trivial stability where $u_{\bar{k}ji}$ converge to
zero. The same behavior is true for case b) with $b+c\leq 0$, case e) with
nonpositive stable equilibrium, and case e) with
$f(u_{\bar{k}ji}^{2})=g(u_{\bar{k}ji}^{2})$. Somewhat more interesting
behavior appears in e) if $f(u_{\bar{k}ji}^{1})=g(u_{\bar{k}ji}^{1})$ and the
semistable equilibrium point is in the positive region ($u_{\bar{k}ji}^{1}>0$
since the equilibrium coincides with $u_{\bar{k}ji}^{1}$) where convergence to
zero will slow down to some positive value $u_{\bar{k}ji}^{1}$. In case b)
with $b+c>0$ we already concluded the existence of a strictly positive
equilibrium which we also specified above but this is also not too interesting
since the context term is reduced to just the input from $u_{00\bar{k}}$.
Finally, the most interesting case of e) includes nontrivial stable state in
the positive region with an unstable equilibrium that could be pushed back to
the negative region if one requires $u_{\bar{k}ji}^{1}=0$ leaving just one
stable state in the nonnegative region, fixed at zero and thus adding an
unstable state at zero, or even kept positive and thus adding a second stable
state at zero.

Notice however that a stable equilibrium on an $n$-dimensional manifold does
not say anything about the overall stability of the point or about the
existence of one, more or no equilibria in the entire phase space. In many of
the previously mentioned results on stability, the main approach is to find a
region or a bound with fully decoupled dynamics, thus producing an equilibrium
in one dimension that is constant with respect to the rest. Satisfying any
equilibrium existence conditions in all dimensions then would imply existence
of the equilibrium in the full phase space. We could obtain full instead of
partial equilibria in most of the cases above if we make some additional
simplifying assumptions like using piecewise constant activation function
(e.g. $a(u)=sgn(u)$) or saturating piecewise linear activation function (e.g.
$a(u)=(|u+1|-|u-1|)/2$) so that the reduced equations will contain constants
$\tilde{c}$ for many unknowns in certain saturation regions. The only more
complicated case is

$\displaystyle\tau_{r}\frac{du_{\bar{k}ji}}{dt}=-d_{\bar{k}ji}u_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l=0}^{N}w_{l\bar{k}i}\tilde{c}_{l\bar{k}i}+b_{\bar{k}ji}\frac{u_{\bar{k}ji}\tilde{c}_{00\bar{k}}}{\sum\limits_{l=0}^{N}\sum\limits_{m=1}^{N}u_{\bar{k}lm}}+U_{\bar{k}ji}\leq$
$\displaystyle\leq-
d_{\bar{k}ji}u_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l=0}^{N}w_{l\bar{k}i}\tilde{c}_{l\bar{k}i}+b_{\bar{k}ji}|\tilde{c}_{00\bar{k}}|+U_{\bar{k}ji}$
$\displaystyle\tau_{r}\frac{du_{\bar{k}ji}}{dt}\geq-
d_{\bar{k}ji}u_{\bar{k}ji}+c_{\bar{k}ji}\sum_{l=0}^{N}w_{l\bar{k}i}\tilde{c}_{l\bar{k}i}-b_{\bar{k}ji}|\tilde{c}_{00\bar{k}}|+U_{\bar{k}ji}$

which is possible assuming nonnegative potentials and implies conclusions
similar to the ones in [90, 11, 84]. However, we are not interested in full
equilibria. Partial equilibria represent convergence only in some subset of
the unknowns, which is reasonable when part of the network has to sustain
oscillations while another part stabilizes or when two relatively independent
parts of the input are recalled separately (e.g. object segmentation). We
might also add upper bound as a resetting condition although we already have
sufficient conditions on boundedness from [92] which could be satisfied on the
nonconverging part of the network.

We obtained some additional stability conditions for (66) from many
alternative methods like requirements on the trace and determinant signs of
the Jacobian, bounding the real parts of the eigenvalues in the negative half-
plane using Gershgorin disks, and trying some standard energy functionals. In
most of these cases the conditions weren’t easy to interpret or practical
enough. Consider for instance the energy functional (35) used for the additive
model (34). For the expanded graph interpretation of (66) with
$\chi(u_{k01},\dots,u_{kNN})=1$, it can be extended to

$E(\mathbf{u})=\sum_{i,j,k=0}^{N}[-\frac{1}{2}\sum_{l,m,n=0}^{N}a(u_{kji})w^{kji}_{lmn}a(u_{lmn})-U_{kji}a(u_{kji})-\hat{E}_{kji}+\tilde{E}_{kji}]$
(69)

where the term $\tilde{E}$ is one of the two options for (35), for instance

$\tilde{E}_{kji}=d_{kji}\int_{0}^{u_{kji}}sa^{\prime}(s)ds$

and the new term $\hat{E}$ is such that

$\hat{E}_{kji}=b_{kji}a(u_{00k})\int_{0}^{u_{kji}}\frac{sa^{\prime}(s)}{s+r_{kji}}ds$
(70)

Under the conditions of integrability of (70), special symmetry of the weight
tensor, and proper boundedness of the functional, we can see that

$\displaystyle\frac{dE(\mathbf{u})}{dt}=\sum_{i,j,k=0}^{N}\frac{dE}{du_{kji}}\frac{du_{kji}}{dt}=$
$\displaystyle=\sum_{i,j,k=0}^{N}[(-\sum_{l,m,n=0}^{N}w^{kji}_{lmn}a(u_{lmn})-U_{kji}-\dots+d_{kji}u_{kji})\frac{da(u_{kji})}{du_{kji}}]\frac{du_{kji}}{dt}=$
$\displaystyle=-\sum_{i,j,k=0}^{N}a^{\prime}(u_{kji})(\frac{du_{kji}}{dt})^{2}\leq
0\text{ if }a^{\prime}(u_{i})\geq 0$

where we can only perform the second passage if we assume that all extra terms
(in dots) resulting from the differentiation and the multiple additional
dependencies of the integral sum up to zero. Additional impractical
requirement from this is the symmetry of the 6-order tensor which doesn’t map
back to admissible metanetwork in the sense of expanded graph.

Using the more natural but unbounded activation like before on the decay term
would yield the continuous tensor form of (54). The differences in the
superthreshold region $\\{u_{kji}>\alpha\ =0\\}$ are only that we have assumed
a special case $d_{kji}=1$. In the subthreshold part $\\{u_{kji}\leq\alpha\
=0\\}$ we have instability where $u_{kji}$ acts like an integrator. However,
the accumulated potential does not take effect on output since we have to
substract it as $\alpha$. A discontinuous version of the previous activation
is therefore more reasonable and also differs in the superthreshold region but
requires the use of Filippov theory for discontinuous flows.

####  3.2.4 Network motif study

The feedforward, reduced lateral, reduced excitatory-inhibitory, and reduced
and fully recurrent networks (57, 58, 59, 60, 62) can be explicitly stated if
we use a set identifier function like

$\chi(c,u)=\theta(c)\theta(u)=\mathbf{1}_{\\{c>\alpha\\}}\mathbf{1}_{\\{u>\alpha\\}}$
(71)

If we exclude unit subthreshold cases where we have unstable but integrator
behavior with requirements like $\alpha=0$ and $u_{00k}>0,k\in[1;N]$, we will
obtain continuous activation functions in all equations. Furthermore, if we
confine the initial conditions and trajectories to the intersection of the
nonnegative region $u_{kji}\geq 0,k,j,i\in[0;N]$ and the nonbroadcasting
region $\bar{h}(C_{i})>0$ where $h$ is defined as (61) in accordance with the
choice of model, the context term will also simplify to a continuous version
of the original. Finally, in the following section we will assume nonnegative
external input $U_{k}\geq 0$ although many of the conclusions easily
generalize to cases with negative external input as well. Such reductions are
meaningful in engineering applications in the next section and exclude some
trivial subthreshold behaviors so that we can concentrate on the most complex
form of the context term. The simplest explicit form of (57) can then be
written as

$\displaystyle\begin{dcases}&\tau_{r}\frac{du^{i}}{dt}=-\theta(u^{i})u^{i}+\sum_{j=1}^{N}w_{j\
}^{\ i}\theta(u_{j\ }^{\ i})\theta(u_{j})u_{j\ }^{\
i}=-u^{i}+\sum_{j=1}^{N}w_{j\ }^{\ i}\theta(u_{j\ }^{\ i})u_{j\ }^{\ i}\\\
&\tau_{r}\frac{du_{j\ }^{\ i}}{dt}=-\theta(u_{j\ }^{\ i})u_{j\ }^{\
i}+\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}+\frac{u_{j\ }^{\
i}u_{j}}{\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\ m\neq
j\end{subarray}}^{N}u_{j\ }^{\ ml}}\\\ &\tau_{r}\frac{du_{k\ }^{\
ji}}{dt}=-\theta(u_{k\ }^{\ ji})\theta(u_{k})u_{k\ }^{\ ji}+\frac{u_{k\ }^{\
ji}\theta(u_{k})u_{k}}{\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\
m\neq k\end{subarray}}^{N}u_{k\ }^{\ ml}}=-\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji}+\frac{u_{k\ }^{\
ji}u_{k}}{\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\ m\neq
k\end{subarray}}^{N}u_{k\ }^{\ ml}}\\\ \end{dcases}$ (72)

where we have Lipschitz continuous flow and ignore any equations for $u_{k\
}^{\ ki}$. Notice that we may remove any remaining identifier function from
the very beginning since
$\theta(c)=\mathbf{1}_{\\{c>\alpha\\}}=\mathbf{1}_{\\{c>0\\}}$ and
$\theta(c)c=c$ for $c\geq 0$. However, we will keep it in the equations for as
long as possible in order to accommodate for generalization to cases with
$\alpha>0$.

All of the above considerations could guarantee no blow up of (72) even though
the network trajectories are not necessary bounded. The nonnegative external
input assumption in the current notation $u_{k}\geq 0$ and an initial
condition $u_{k\ }^{\ ji}(0)$ in the nonnegative region now would imply that
$u_{k\ }^{\ ji}(t)\geq 0$ for all $t>0$ and the trajectory will remain in the
nonnegative region for all times since the same will follow for $u_{j\ }^{\
i}(t)$ and $u^{i}(t)$. Assuming also that $u_{k}\leq 1$ implies $e(u_{k\ }^{\
ji},u_{k})\leq 1$ and $a(u_{k\ }^{\ ji},u_{k})\leq N$. Using these bounds and
Gronwall’s lemma with $\tau_{r}=1$, we can also obtain

$\displaystyle\frac{du_{k\ }^{\ ji}}{dt}\leq-a(u_{k\ }^{\
ji},u_{k})+1=1-\chi(u_{k\ }^{\ ji},u_{k})u_{k\ }^{\ ji}\leq 1\text{ since
}0\leq\chi(u_{k\ }^{\ ji},u_{k})\leq 1\Rightarrow$ $\displaystyle\Rightarrow
u_{k\ }^{\ ji}(t)\leq u_{k\ }^{\ ji}(0)+t$

as an upper bound on the trajectory of each metaconnection,

$\displaystyle\frac{du_{j\ }^{\ i}}{dt}\leq\sum_{\begin{subarray}{c}k=1\\\
k\neq j\end{subarray}}^{N}\chi(u_{k\ }^{\ ji},u_{k})u_{k\ }^{\ ji}+1\leq
1+\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}u_{k\ }^{\
ji}=1+\sum_{\begin{subarray}{c}k=1\\\ k\neq i\end{subarray}}^{N}u_{k\ }^{\
ji}(0)+(N-1)t\Rightarrow$ $\displaystyle\Rightarrow u_{j\ }^{\ i}(t)\leq u_{j\
}^{\ i}(0)+t(1+\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}u_{k\
}^{\ ji}(0))+\frac{N-1}{2}t^{2}$

as an upper bound on the trajectory of each connection, and

$\displaystyle\frac{du^{i}}{dt}\leq\sum_{j=1}^{N}\chi(u_{j\ }^{\
i},u_{j})u_{j\ }^{\ i}+0\leq\sum_{j=1}^{N}u_{j\ }^{\ i}\Rightarrow$
$\displaystyle\Rightarrow u^{i}(t)\leq u^{i}(0)+t\sum_{j=1}^{N}u_{j\ }^{\
i}(0)+\frac{t^{2}}{2}(N+\sum_{j=1}^{N}\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}u_{k\ }^{\ ji}(0))+\frac{N(N-1)}{6}t^{3}$

as an upper bound on the trajectory of each unit. All of this indicates that
we can rule out blow up in finite time.

The restrictions to the unit superthreshold and nonbroadcasting regions that
produced (72) also characterize the stability with all the results from the
previous section. In particular, for all units since $b_{kji}=0$ we are always
in the case b) on figure 11 with partial equilibrium state

$\bar{u}_{kji}=\frac{c_{kji}}{d_{kji}}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}+\frac{U_{kji}}{d_{kji}}$

which is nontrivial if
$c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}+U_{kji}>0$. For the connections
and metaconnections, we are once again interested in nontrivial stability
(single stable state at zero) which we get again in case b) if
$u_{klm}=0,l\in[0;N],l\neq j,m\in[1;N],m\neq i$ (considering tensor form) with
an equilibrium

$\bar{u}_{kji}=\frac{c_{kji}}{d_{kji}}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}+\frac{u_{00k}}{d_{kji}}+\frac{U_{kji}}{d_{kji}}$

but also in case e) if we satisfy the conditions
$f(u_{kji}^{1})<g(u_{kji}^{1})$ and $f(u_{kji}^{2})<g(u_{kji}^{2})$ with the
additional requirement of nonnegative stable equilibrium and case f) with the
conditions $f(u_{kji}^{1})<g(u_{kji}^{1})$ and $u_{kji}^{1}>0$. Using results
from the previous section, the main conditions of case e) can be written as

$\displaystyle f(u_{kji}^{1/2})<g(u_{kji}^{1/2})\Rightarrow-
d_{kji}r_{kji}\pm\sqrt{b_{kji}d_{kji}u_{00k}r_{kji}}<$
$\displaystyle<\mp\sqrt{b_{kji}d_{kji}u_{00k}r_{kji}}+b_{kji}u_{00k}+U_{kji}+c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}$

which after taking $d_{kji}=b_{kji}=1$ because of the previous equations
becomes

$\displaystyle-r_{kji}\pm
2\sqrt{u_{00k}r_{kji}}-u_{00k}<U_{kji}+c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}$

Since the square root is guaranteed to be real and nonnegative (due to our
previous assumptions), the inequality with a plus implies the inequality with
the minus as well and we can obtain a polynomial

$\displaystyle
r_{kji}-2\sqrt{u_{00k}r_{kji}}+u_{00k}>-U_{kji}-c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}$
$\displaystyle\bigg{(}\sqrt{r_{kji}}-\sqrt{u_{00k}}\bigg{)}^{2}>-U_{kji}-c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}$

which is always true if
$U_{kji}+c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}>0$ or otherwise true if

$\displaystyle\bigg{|}\sqrt{r_{kji}}-\sqrt{u_{00k}}\bigg{|}>\sqrt{-U_{kji}-c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}}$
(73)

Using similar approach, the main condition of case f) can be written as

$f(u_{kji}^{1})=g(u_{kji}^{1})\Rightarrow\bigg{|}\sqrt{r_{kji}}+\sqrt{u_{00k}}\bigg{|}=\sqrt{-U_{kji}-c_{kji}\sum_{l\in\mathcal{N}}w_{lki}u_{lki}}$
(74)

and the extra condition can be summarized as

$u_{kji}^{1}>0\Rightarrow-r_{kji}+\sqrt{u_{00k}r_{kji}}>0$ (75)

We will use the conditions of the three cases of nontrivial stability and all
assumptions preceding them until the end of this section. For this reason we
have to recall that $r_{kji}$ is the remainder sum defined in (68).

In comparison to the simplest form (72), the more complex form of (60) can be
written in terms of indicator functions and with all assumptions above as

$\displaystyle\begin{dcases}&\tau_{r}\frac{du^{i}}{dt}=-u^{i}+\sum_{j=1}^{N}w_{j\
}^{\ i}\theta(u_{j\ }^{\ i})u_{j\ }^{\
i}+\sum_{l=1}^{M}w^{li}\theta(u^{li})u^{li}\\\ &\tau_{r}\frac{du_{j\ }^{\
i}}{dt}=-\theta(u_{j\ }^{\ i})u_{j\ }^{\ i}+\sum_{\begin{subarray}{c}k=1\\\
k\neq j\end{subarray}}^{N}w_{k\ }^{\ ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji}+\sum_{l=1}^{M}w^{l\ }_{\ ji}\theta(u^{l\ }_{\ ji})u^{l\ }_{\
ji}+\frac{u_{j\ }^{\
i}u_{j}}{\sum\limits_{m=1}^{M}\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
j\end{subarray}}^{N}u_{j\ }^{\ nm}}\\\
&\tau_{r}\frac{du^{li}}{dt}=-\theta(u^{li})u^{li}+\frac{u^{li}u^{l}}{\sum\limits_{m=1}^{M}[u^{lm}+\sum\limits_{\begin{subarray}{c}n=0\\\
n\neq l\end{subarray}}^{N}u^{l\ }_{\ nm}]}\\\ &\tau_{r}\frac{du^{i\ }_{\
j}}{dt}=-\theta(u^{i\ }_{\ j})u^{i\ }_{\ j}+\frac{u^{i\ }_{\
j}u^{i}}{\sum\limits_{m=1}^{M}[u^{im}+\sum\limits_{\begin{subarray}{c}n=0\\\
n\neq i\end{subarray}}^{N}u^{i\ }_{\ nm}]}\\\ &\tau_{r}\frac{du_{k\ }^{\
ji}}{dt}=-\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}+\frac{u_{k\ }^{\
ji}u_{k}}{\sum\limits_{m=1}^{M}\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
k\end{subarray}}^{N}u_{k\ }^{\ nm}},\ k\neq j\\\ &\tau_{r}\frac{du^{l\ }_{\
ji}}{dt}=-\theta(u^{l\ }_{\ ji})u^{l\ }_{\ ji}+\frac{u^{l\ }_{\
ji}u^{l}}{\sum\limits_{m=1}^{M}[u^{lm}+\sum\limits_{\begin{subarray}{c}n=0\\\
n\neq l\end{subarray}}^{N}u^{l\ }_{\ nm}]}\\\
&\tau_{r}\frac{du_{j}}{dt}=-u_{j}+\sum_{i=1}^{M}w^{i\ }_{\ j}\theta(u^{i\ }_{\
j})u^{i\ }_{\ j}+U_{j}\end{dcases}$ (76)

We will use (72) and (76) as starting points for deriving smaller network
motifs involving feedforward and feedback connections.

Figure 12: Motifs of increasing complexity: a) broadcasting circuit; b)
circuit with a metaconnection; c) circuit with a feedback connection;

A study of specific network motifs can help us tie all representations
motivating such structure and dynamics with any theoretical conclusions done
so far. The simplest circuit with sufficiently complex dynamics is shown in
figure 12 case a) and has two connections coming out of one unit. Such a case
is a reduced version of (72) with only four equations

$\displaystyle\begin{dcases}&\tau_{r}\frac{du^{1}}{dt}=-u^{1}+w_{1\ }^{\
1}\theta(u_{1\ }^{\ 1})u_{1\ }^{\ 1}\\\
&\tau_{r}\frac{du^{2}}{dt}=-u^{2}+w_{1\ }^{\ 2}\theta(u_{1\ }^{\ 2})u_{1\ }^{\
2}\\\ &\tau_{r}\frac{du_{1\ }^{\ 1}}{dt}=-\theta(u_{1\ }^{\ 1})u_{1\ }^{\
1}+\frac{u_{1\ }^{\ 1}u_{1}}{u_{1\ }^{\ 1}+u_{1\ }^{\ 2}}\\\
&\tau_{r}\frac{du_{1\ }^{\ 2}}{dt}=-\theta(u_{1\ }^{\ 2})u_{1\ }^{\
2}+\frac{u_{1\ }^{\ 2}u_{1}}{u_{1\ }^{\ 1}+u_{1\ }^{\ 2}}\\\ \end{dcases}$
(77)

The partial equilibrium states for both units are therefore

$\displaystyle\bar{u}^{1}=w_{1\ }^{\ 1}u_{1\ }^{\ 1}\text{ if }u_{1\ }^{\
1}>0\text{ else }\bar{u}^{1}=0\Rightarrow\bar{u}^{1}=w_{1\ }^{\ 1}u_{1\ }^{\
1}$ $\displaystyle\bar{u}^{2}=w_{1\ }^{\ 2}u_{1\ }^{\ 2}\text{ if }u_{1\ }^{\
2}>0\text{ else }\bar{u}^{2}=0\Rightarrow\bar{u}^{2}=w_{1\ }^{\ 2}u_{1\ }^{\
2}$

and are stable. The partial equilibrium state for the first connection $u_{1\
}^{\ 1}$ in the simpler case b) of figure 11 is

$\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\ 1}=\bar{u}_{1\ }^{\
1}=u_{1}\text{ if }u_{1\ }^{\ 2}=0\text{ and }\bar{u}_{1\ }^{\ 1}\neq 0$

which also conforms to the nonbroadcasting condition $u_{1\ }^{\ 1}+u_{1\ }^{\
2}>0$. If the first connection is in the case e), the stability conditions
(73) for it are simplified to

$\displaystyle\begin{dcases}&\text{ always true if }0>0\text{ thus not
possible}\\\ &\big{|}\sqrt{u_{1\ }^{\
2}}-\sqrt{u_{1}}\big{|}>\sqrt{0-0}=0\text{ if }0\leq
0\end{dcases}\Rightarrow\big{|}\sqrt{u_{1\ }^{\ 2}}-\sqrt{u_{1}}\big{|}>0$

and the corresponding unstable and stable partial equilibrium states are

$\displaystyle 0=-\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}+\frac{\bar{u}_{1\ }^{\ 1}u_{1}}{\bar{u}_{1\ }^{\ 1}+u_{1\ }^{\
2}}\Rightarrow\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\ 1}(\bar{u}_{1\ }^{\
1}+u_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}u_{1}\Rightarrow$
$\displaystyle\bar{u}_{1\ }^{\ 1}(\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}+\theta(\bar{u}_{1\ }^{\ 1})u_{1\ }^{\ 2}-u_{1})=0\Rightarrow\bar{u}_{1\
}^{\ 1}=0\text{ or }\bar{u}_{1\ }^{\ 1}=u_{1}-u_{1\ }^{\ 2}$

The final case $\bar{u}_{1\ }^{\ 1}$ can be in is case f) where the main
condition (74) suggests

$\big{|}\sqrt{u_{1\ }^{\ 2}}-\sqrt{u_{1}}\big{|}=\pm 0\Rightarrow u_{1\ }^{\
2}=u_{1}$

and the semistable steady state $u_{kji}^{1}=-u_{1\ }^{\ 2}+\sqrt{u_{1}u_{1\
}^{\ 2}}=0$ can only be trivial since it doesn’t satisfy condition (75). We
summarize all cases in the analogical conclusions for the second connection
$u_{1\ }^{\ 2}$, namely

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 2}=u_{1}\text{ if }u_{1\ }^{\
1}=0\text{ and }\bar{u}_{1\ }^{\ 2}\neq 0\\\ &\bar{u}_{1\ }^{\ 2}=0\text{ or
}\bar{u}_{1\ }^{\ 2}=u_{1}-u_{1\ }^{\ 1}\text{ if }\big{|}\sqrt{u_{1\ }^{\
1}}-\sqrt{u_{1}}\big{|}>0\\\ &\bar{u}_{1\ }^{\ 2}=0\text{ if }u_{1\ }^{\
1}=u_{1}\end{dcases}$

where $\bar{u}_{1\ }^{\ 2}=u_{1}$ is the only stable state in the first case,
$\bar{u}_{1\ }^{\ 2}=0$ and $\bar{u}_{1\ }^{\ 2}=u_{1}-u_{1\ }^{\ 1}$ are the
unstable and stable states in the second case, and $\bar{u}_{1\ }^{\ 2}=0$ is
the semistable state in the third case. Notice that if $u_{1}-u_{1\ }^{\ 1}<0$
in the second case, the stable equilibrium $\bar{u}_{1\ }^{\ 2}$ will no
longer be admissible since it will pass on the negative side and we will be
left with stable state at zero. All cases of partial equilibria are
illustrated on figure 13. The origin is excluded by the nondistributional
condition $u_{1\ }^{\ 1}+u_{1\ }^{\ 2}\neq 0$ and we are left with one partial
stable equilibrium $\bar{u}_{1\ }^{\ 2}=u_{1}$ for fixed $u_{1\ }^{\ 2}=0$ or
vice versa. When $|\sqrt{u_{1\ }^{\ 1}}-\sqrt{u_{1}}|>0$, we have two partial
equilibria for $u_{1\ }^{\ 2}$ along the $u_{1\ }^{\ 1}$ direction with a
transcritical bifurcation when $|\sqrt{u_{1\ }^{\ 1}}-\sqrt{u_{1}}|=0$ and
therefore change of their stability after the nonzero equilibrium passes the
one at zero.

Figure 13: Phase portrait of $u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ from the
motif (77)

Some partial nontrivial stable states $(\bar{u}_{1\ }^{\ 1},\bar{u}_{1\ }^{\
2})$ on a 2-surface can be obtained by solving the algebraic polynomial system

$\displaystyle\begin{dcases}&0=-\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}+\frac{\bar{u}_{1\ }^{\ 1}u_{1}}{\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2}}\\\
&0=-\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}+\frac{\bar{u}_{1\ }^{\
2}u_{1}}{\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2}}\\\
\end{dcases}\Rightarrow\begin{dcases}&\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\
}^{\ 1}(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}u_{1}\\\
&\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}u_{1}\\\ \end{dcases}$

and hence

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 1}(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2}-u_{1})=0\\\ &\bar{u}_{1\ }^{\ 2}(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2}-u_{1})=0\\\
\end{dcases}\Rightarrow\begin{dcases}&\bar{u}_{1\ }^{\ 1}=0\text{ or
}\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2}-u_{1}=0\\\ &\bar{u}_{1\ }^{\
2}=0\text{ or }\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2}-u_{1}=0\end{dcases}$

similarly to the way it was done for the partial equilibria on 1-surface. The
desired 2-surface partial equilibrium states are then the pairs $(s,u_{1}-s)$
for $s\in[0,u_{1}]$. Examining these solutions carefully, we can observe that
in the pair $(0,u_{1})$, $\bar{u}_{1\ }^{\ 1}$ is in case f) since
$|\sqrt{u_{1}}-\sqrt{u_{1}}|=0$ and $\bar{u}_{1\ }^{\ 2}$ is in case b) while
the opposite holds for the pair $(u_{1},0)$. All remaining pairs satisfy the
conditions (73) and fall into case e) for both components. Plotting the phase
plane for ($u_{1\ }^{\ 1}$, $u_{1\ }^{\ 2}$) on figure 13, we have a line
attractor because of the coinciding nullclines $\dot{u}_{1\ }^{\
1}=\dot{u}_{1\ }^{\ 2}=0$. In the case of three output connections, such a
line would become a plane $\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2}+\bar{u}_{1\
}^{\ 3}=u_{1}$ intersecting the three axes at $(u_{1},0,0)$, $(0,u_{1},0)$,
and $(0,0,u_{1})$. Finally, the full equilibria are $(\bar{u}_{1\ }^{\
1},\bar{u}_{1\ }^{\ 2},\bar{u}^{1},\bar{u}^{2})=(s,u_{1}-s,w_{1\ }^{\
1}s,w_{1\ }^{\ 2}(u_{1}-s))$ and lie on an attractor line in a 4-dimensional
space. Conclusions on stability of the full equilibria are possible due to the
continuous dependence of the partial 1-surface equilibria of all unknowns with
respect to the two output connections. For instance, since $u_{1\ }^{\
1}\rightarrow\bar{u}_{1\ }^{\ 1}$ and $u^{1}\rightarrow\bar{u}^{1}=w_{1\ }^{\
1}u_{1\ }^{\ 1}$ in some neighborhood, we have that $(u_{1\ }^{\
1},u^{1})\rightarrow(\bar{u}_{1\ }^{\ 1},w_{1\ }^{\ 1}\bar{u}_{1\ }^{\ 1})$.

The interpretation if this motif coincides with the intuition behind it - the
system will settle down to a steady state when the potential is properly
distributed among the two outputs and any such configuration will remain
constant. At the same time if a metaconnection is depleted to zero, it will be
sustained in this way unless all remaining output connections settle to zero
as well and we obtain broadcasting. Finally, if a connection is depleted to
zero it will be sustained in this way unless there is broadcasting or it is
forced back into the positive region through excitation by its input
metaconnections.

Adding metaconnections, case b) of figure 12 is another reduced version of
(72) with five equations and one additional equation to the previous motif
(77) as

$\displaystyle\begin{dcases}&\tau_{r}\frac{du^{1}}{dt}=-u^{1}+w_{1\ }^{\
1}\theta(u_{1\ }^{\ 1})u_{1\ }^{\ 1}\\\
&\tau_{r}\frac{du^{2}}{dt}=-u^{2}+w_{1\ }^{\ 2}\theta(u_{1\ }^{\ 2})u_{1\ }^{\
2}\\\ &\tau_{r}\frac{du_{1\ }^{\ 1}}{dt}=-\theta(u_{1\ }^{\ 1})u_{1\ }^{\
1}+\frac{u_{1\ }^{\ 1}u_{1}}{u_{1\ }^{\ 1}+u_{1\ }^{\ 2}}\\\
&\tau_{r}\frac{du_{1\ }^{\ 2}}{dt}=-\theta(u_{1\ }^{\ 2})u_{1\ }^{\
2}+\frac{u_{1\ }^{\ 2}u_{1}}{u_{1\ }^{\ 1}+u_{1\ }^{\ 2}}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}\\\ &\tau_{r}\frac{du_{2\ }^{\
12}}{dt}=-\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}+\frac{u_{2\ }^{\
12}u_{2}}{u_{2\ }^{\ 12}}=-\theta(u_{2\ }^{\ 12})u_{2\ }^{\
12}+u_{2}\end{dcases}$ (78)

The partial equilibrium states for the two units as well as the metaconnection
are obtained in the same way as before, i.e. case b) of figure 11

$\bar{u}^{1}=w_{1\ }^{\ 1}u_{1\ }^{\ 1}\qquad\bar{u}^{2}=w_{1\ }^{\ 2}u_{1\
}^{\ 2}\qquad\bar{u}_{2\ }^{\ 12}=u_{2}$

The conclusions on the 1-surface equilibria of $u_{1\ }^{\ 1}$ are the same as
before

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 1}=u_{1}\text{ if }u_{1\ }^{\
2}=0\text{ and }\bar{u}_{1\ }^{\ 1}\neq 0\\\ &\bar{u}_{1\ }^{\ 1}=0\text{ or
}\bar{u}_{1\ }^{\ 1}=u_{1}-u_{1\ }^{\ 2}\text{ if }\big{|}\sqrt{u_{1\ }^{\
2}}-\sqrt{u_{1}}\big{|}>0\\\ &\bar{u}_{1\ }^{\ 1}=0\text{ if }u_{1\ }^{\
2}=u_{1}\end{dcases}$

For $u_{1\ }^{\ 2}$ we now have 1-surface equilibria given implicitly as

$\displaystyle(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})(u_{1\ }^{\ 1}+\bar{u}_{1\ }^{\
2})=\bar{u}_{1\ }^{\ 2}u_{1}\Rightarrow$ $\displaystyle\theta(\bar{u}_{1\ }^{\
2})(\bar{u}_{1\ }^{\ 2})^{2}-(w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\
12}-\theta(\bar{u}_{1\ }^{\ 2})u_{1\ }^{\ 1}+u_{1})\bar{u}_{1\ }^{\ 2}-w_{2\
}^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}u_{1\ }^{\ 1}=0$

and with modified conditions (73, 74) that split into two cases

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 2}=u_{1}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}\text{ if }u_{1\ }^{\ 1}=0\text{ and
}\bar{u}_{1\ }^{\ 2}\neq 0\\\ &\bar{u}_{1\ }^{\ 2}=u_{1/2}\text{ if
}\big{|}\sqrt{u_{1\ }^{\ 1}}-\sqrt{u_{1}}\big{|}>\sqrt{-w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}}\\\ &\bar{u}_{1\ }^{\ 2}=-u_{1\ }^{\
1}+\sqrt{u_{1}u_{1\ }^{\ 1}}\text{ if }\big{|}\sqrt{u_{1\ }^{\
1}}-\sqrt{u_{1}}\big{|}=\sqrt{-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\
12}}\end{dcases}$

for $-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}\geq 0\Rightarrow
w_{2\ }^{\ 12}\leq 0$ and

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 2}=u_{1}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}\text{ if }u_{1\ }^{\ 1}=0\text{ and
}\bar{u}_{1\ }^{\ 2}\neq 0\\\ &\bar{u}_{1\ }^{\ 2}=u_{1/2}\text{ otherwise
}\end{dcases}$

for $-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}<0\Rightarrow w_{2\
}^{\ 12}u_{2\ }^{\ 12}>0$. Here $u_{1/2}$ are the two roots of the polynomial
in $\bar{u}_{1\ }^{\ 2}$. For a positive weight, their discriminant will
always be positive so they will exist and remain distinct, while for a
negative weight the discriminant will become zero and ultimately negative so
the two roots will merge into one semistable point $\bar{u}_{1\ }^{\ 1}=-u_{1\
}^{\ 2}+\sqrt{u_{1}u_{1\ }^{\ 2}}$ and disappear before reappearing when the
conditions hold again (due to the absolute values). The second root is the
unstable partial equilibrium and is no longer the $u_{1\ }^{\ 1}$ axis
(becoming positive or negative for $u_{1\ }^{\ 1}>0$ depending on the weight)
and the the first root is the stable partial equilibrium. If $w_{2\ }^{\
12}u_{2\ }^{\ 12}<-u_{1}$, the stable root will also be on the negative side
and we will have no nontrivial stability. Also, no stability of any kind is
asserted in the range $\big{|}\sqrt{u_{1\ }^{\
2}}-\sqrt{u_{1}}\big{|}<\sqrt{-w_{2\ }^{\ 12}}$ when $w_{2\ }^{\ 12}<0$. All
cases of partial equilibria are illustrated on figure 14.

Figure 14: Phase portrait of $u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ from the
motif (78) where $w=w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}$,
$u_{1}\geq 0$, and: a) $w_{2\ }^{\ 12}>0$; b) $w_{2\ }^{\ 12}<0$;

If we consider 2-surface steady states, we need to solve the polynomial system

$\displaystyle\begin{dcases}&\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}u_{1}\\\
&(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\ 12}\theta(u_{2\
}^{\ 12})u_{2\ }^{\ 12})(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\
}^{\ 2}u_{1}\\\ \end{dcases}$

where each component must satisfy the conditions of one of the three cases
before. If $\theta(\bar{u}_{2\ }^{\ 12})=0$ we obtain the previous solution
namely the one of (77). Otherwise, we get

$\displaystyle\begin{dcases}&\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\
1}u_{1}\Rightarrow\bar{u}_{1\ }^{\ 1}=0\text{ or }\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2}=u_{1}\\\ &(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\
2}-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}u_{1}\\\ \end{dcases}$

Hence, the second equation when $\bar{u}_{1\ }^{\ 1}=0$ is

$\displaystyle(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})(0+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\
}^{\ 2}u_{1}$ $\displaystyle\bar{u}_{1\ }^{\ 2}(\theta(\bar{u}_{1\ }^{\
2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\
12}-u_{1})=0$ $\displaystyle\Rightarrow\bar{u}_{1\ }^{\ 2}=0\text{ or
}\bar{u}_{1\ }^{\ 2}=u_{1}+w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}$

and the partial 2-surface equilibria are $(0,0)$ and $(0,u_{1}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})$ where we exclude $(0,0)$ due to the
nondistributional condition $u_{1\ }^{\ 1}+u_{1\ }^{\ 2}\neq 0$.
Alternatively, the second equation when $\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\
2}=u_{1}$ is

$\displaystyle(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})u_{1}=\bar{u}_{1\ }^{\ 2}u_{1}$
$\displaystyle\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\
}^{\ 12}=\bar{u}_{1\ }^{\ 2}$ $\displaystyle\Rightarrow\forall\bar{u}_{1\ }^{\
2}\text{ if }-w_{2\ }^{\ 12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}=0$

with partial equilibria $(u_{1}-a,a),a\in[0,u_{1}]$ which exist only if $u_{2\
}^{\ 12}=0$ or $w_{2\ }^{\ 12}=0$ and the metaconnection does not have any
effect. In such a case, we will obtain the partial 1-surface equilibria of
(77) and the previous 2-surface equilibrium $(0,u_{1}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})$ will reduce to
$(0,u_{1})=(u_{1}-a,a)$ for $a=u_{1}$. If the metaconnection effect is turned
on, the line attractor is replaced by a point attractor in figure 14 a) or by
an unstable equilibrium point in b) depending on the sign of the weight $w_{2\
}^{\ 12}$. Notice that the points $(0,0)$ and $(0,u_{1}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12})$ are always points on the locus of
the roots since they solve the equation for $\bar{u}_{1\ }^{\ 2}$ which is a
polynomial of degree 2 and therefore could have at most two such intersections
with the $u_{1\ }^{\ 2}$ axis. We could use this to argue that if $w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}=w<-u_{1}$, the $(0,u_{1}+w)$ point
would lie on the negative side of the axis (now shown on the figure) and the
polynomial nullcline would only intersect the nonnegative region at $(0,0)$.
Alternatively, we could obtain the same as a condition on the derivative of
the nullcline at $(0,0)$ as

$\displaystyle(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w)(\bar{u}_{1\
}^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}u_{1}\Rightarrow\bar{u}_{1\
}^{\ 1}(\bar{u}_{1\ }^{\ 2})=-\bar{u}_{1\ }^{\ 2}+\frac{\bar{u}_{1\ }^{\
2}u_{1}}{\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w}$
$\displaystyle\frac{d\bar{u}_{1\ }^{\ 1}}{d\bar{u}_{1\ }^{\
2}}=-1-\frac{wu_{1}}{(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\
2}-w)^{2}}\Rightarrow\frac{d\bar{u}_{1\ }^{\ 1}}{d\bar{u}_{1\ }^{\
2}}(0)=-1-\frac{wu_{1}}{(0-w)^{2}}>0\Rightarrow w<-u_{1}$

The asymptotes of the hyperbola formed by the $u_{1\ }^{\ 2}$ nullcines are
$u_{1\ }^{\ 1}+u_{1\ }^{\ 2}=u_{1}$ and $u_{1\ }^{\ 2}=w$ and therefore the
second nullcline of $u_{1\ }^{\ 1}$ will coincide with the asymptote $u_{1\
}^{\ 1}+u_{1\ }^{\ 2}=u_{1}$ and never intersect these nullclines to form any
other equilibria.

If we consider the points on a 3-surface with partial equilibria $(\bar{u}_{1\
}^{\ 1},\bar{u}_{1\ }^{\ 2},\bar{u}_{2\ }^{\ 12})$ we obtain the system

$\displaystyle\begin{dcases}&\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}u_{1}\\\
&(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}\theta(\bar{u}_{2\ }^{\ 12})\bar{u}_{2\ }^{\ 12})(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}u_{1}\\\ &\bar{u}_{2\ }^{\
12}=u_{2}\end{dcases}$

which is solved analogically but with equilibria $(0,u_{1}+w_{2\ }^{\
12}\bar{u}_{2\ }^{\ 12},\bar{u}_{2\ }^{\ 12})=(0,u_{1}+w_{2\ }^{\
12}u_{2},u_{2})$ and $(u_{1}-a,a,u_{2}),a\in[0,u_{1}]$ if $u_{2}=0$ or $w_{2\
}^{\ 12}=0$. The full equilibria are similar to before but with the extra
component $\bar{u}_{2\ }^{\ 12}=u_{2}$. The interpretation is the same when
the metaconnection has no effect on the connection because of zero weight or
zero throughput - we have a line attractor and connections that settle when
the potential of the unit is fully distributed. Differently, when the
metaconnection has positive effect it will bias the distribution towards the
configuration where only the aided connection is emitted through and the
connection with no metaconnection support will converge to zero. When the
metaconnection has negative effect, the aided connection $u_{1\ }^{\ 2}$ will
settle to reduced value as long as the other connection $u_{1\ }^{\ 1}$ is
fully deactivated. Otherwise, even the slightest positive potential $u_{1\
}^{\ 1}>0$ will bring to the full depletion of $u_{1\ }^{\ 2}$ and $u_{1\ }^{\
1}=u_{1}$ as the ultimate winner.

Adding feedback connections, case c) of figure 12 is a reduced version of (76)
with eight equations

$\displaystyle\begin{dcases}&\tau_{r}\frac{du^{1}}{dt}=-u^{1}+w_{1\ }^{\
1}\theta(u_{1\ }^{\ 1})u_{1\ }^{\ 1}\\\
&\tau_{r}\frac{du^{2}}{dt}=-u^{2}+w_{1\ }^{\ 2}\theta(u_{1\ }^{\ 2})u_{1\ }^{\
2}\\\ &\tau_{r}\frac{du_{1\ }^{\ 1}}{dt}=-\theta(u_{1\ }^{\ 1})u_{1\ }^{\
1}+\frac{u_{1\ }^{\ 1}u_{1}}{u_{1\ }^{\ 1}+u_{1\ }^{\ 2}}\\\
&\tau_{r}\frac{du_{1\ }^{\ 2}}{dt}=-\theta(u_{1\ }^{\ 2})u_{1\ }^{\
2}+\frac{u_{1\ }^{\ 2}u_{1}}{u_{1\ }^{\ 1}+u_{1\ }^{\ 2}}+w_{2\ }^{\
12}\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}\\\ &\tau_{r}\frac{du^{2\ }_{\
2}}{dt}=-\theta(u^{2\ }_{\ 2})u^{2\ }_{\ 2}+\frac{u^{2\ }_{\ 2}u^{2}}{u^{2\
}_{\ 2}}=-\theta(u^{2\ }_{\ 2})u^{2\ }_{\ 2}+u^{2}\\\ &\tau_{r}\frac{du_{2\
}^{\ 12}}{dt}=-\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}+\frac{u_{2\ }^{\
12}u_{2}}{u_{2\ }^{\ 12}}=-\theta(u_{2\ }^{\ 12})u_{2\ }^{\ 12}+u_{2}\\\
&\tau_{r}\frac{du_{1}}{dt}=-u_{1}+U_{1}\\\
&\tau_{r}\frac{du_{2}}{dt}=-u_{2}+w^{2\ }_{\ 2}\theta(u^{2\ }_{\ 2})u^{2\ }_{\
2}+U_{2}\end{dcases}$ (79)

The partial equilibrium states for all units now are

$\displaystyle\bar{u}^{1}$ $\displaystyle=w_{1\ }^{\ 1}u_{1\ }^{\ 1}$
$\displaystyle\bar{u}^{2}$ $\displaystyle=w_{1\ }^{\ 2}u_{1\ }^{\ 2}$
$\displaystyle\bar{u}_{1}$ $\displaystyle=U_{1}$ $\displaystyle\bar{u}_{2}$
$\displaystyle=w^{2\ }_{\ 2}u^{2\ }_{\ 2}+U_{2}$

and are stable as usual. The single outputs $\bar{u}^{2\ }_{\ 2}$ and
$\bar{u}_{2\ }^{\ 12}$ are in the same case like all the units, namely case b)
of figure 11 with stable states

$\displaystyle\bar{u}^{2\ }_{\ 2}=u^{2}\qquad\bar{u}_{2\ }^{\ 12}=u_{2}$

The partial 1-surface and 2-surface equilibria for the competitive connections
$u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ are as in (78) but the 3-surface
equilibria and any resulting full-space equilibria differ significantly due to
the presence of feedback connections and a cycle within the motif. We can see
this by considering the full depth of the cycle in order to decouple $u_{1\
}^{\ 2}$ from everything but $u_{1\ }^{\ 1}$.

If we fix $u_{2\ }^{\ 12}$ to $\bar{u}_{2\ }^{\ 12}$ and all its dependencies
or simply consider 7-surface stable states we obtain the algebraic polynomial
system

$\displaystyle\begin{dcases}&\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}U_{1}\\\
&(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}\theta(\bar{u}_{2\ }^{\ 12})\bar{u}_{2\ }^{\ 12})(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}U_{1}\\\ &\bar{u}_{2\ }^{\
12}=\bar{u}_{2}=w^{2\ }_{\ 2}\bar{u}^{2\ }_{\ 2}+U_{2}=w^{2\ }_{\
2}\bar{u}^{2}+U_{2}=w^{2\ }_{\ 2}w_{1\ }^{\ 2}\bar{u}_{1\ }^{\
2}+U_{2}\end{dcases}$

which is solved similarly as in (78) where the second equation when
$\bar{u}_{1\ }^{\ 1}=0$ is

$\displaystyle(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}U_{2})(0+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}U_{1}$
$\displaystyle\bar{u}_{1\ }^{\ 2}(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\
2}-w_{2\ }^{\ 12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}U_{2}-U_{1})=0$ $\displaystyle\Rightarrow\bar{u}_{1\ }^{\ 2}=0\text{ or
}\bar{u}_{1\ }^{\ 2}(1-w_{2\ }^{\ 12}w^{2\ }_{\ 2}w_{1\ }^{\ 2})=w_{2\ }^{\
12}U_{2}+U_{1}$

and the partial equilibria are $(0,0)$ and $(0,(U_{1}+w_{2\ }^{\
12}U_{2})/(1-w_{2\ }^{\ 12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}))=(0,\tilde{U}_{1})$
(again excluding $(0,0)$ due to $u_{1\ }^{\ 1}+u_{1\ }^{\ 2}\neq 0$). The
second equation when $\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2}=U_{1}$ is

$\displaystyle(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}U_{2})U_{1}=\bar{u}_{1\ }^{\ 2}U_{1}$ $\displaystyle\bar{u}_{1\ }^{\
2}-w_{2\ }^{\ 12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}\bar{u}_{1\ }^{\ 2}-w_{2\ }^{\
12}U_{2}=\bar{u}_{1\ }^{\ 2}$ $\displaystyle\Rightarrow w_{2\ }^{\ 12}w^{2\
}_{\ 2}w_{1\ }^{\ 2}\bar{u}_{1\ }^{\ 2}=-w_{2\ }^{\ 12}U_{2}$

with a partial equilibrium $(U_{1}-s,s),s\in[0,u_{1}]$ if $w_{2\ }^{\ 12}=0$
or $U_{2}=w^{2\ }_{\ 2}w_{1\ }^{\ 2}=0$, only a special case thereof
$(U_{1}+U_{2}/(w^{2\ }_{\ 2}w_{1\ }^{\ 2}),-U_{2}/(w^{2\ }_{\ 2}w_{1\ }^{\
2}))=(U_{1}+\tilde{U}_{2},-\tilde{U}_{2})$ with $s=-\tilde{U}_{2}$ if $w_{2\
}^{\ 12}\neq 0$, $w^{2\ }_{\ 2}w_{1\ }^{\ 2}\neq 0$, and $U_{2}\neq 0$ and
only a special case thereof $(U_{1},0)$ when $w_{2\ }^{\ 12}\neq 0$, $w^{2\
}_{\ 2}w_{1\ }^{\ 2}\neq 0$ and $U_{2}=0$. If $w_{2\ }^{\ 12}=0$ or
$U_{2}=w^{2\ }_{\ 2}w_{1\ }^{\ 2}=0$ the first equilibrium will reduce to
another equilibrium on the line
$(0,\tilde{U}_{1})=(0,(U_{1}+0)/(1-0))=(0,U_{1})$ with $s=U_{1}$ and we can
conclude that generally in this case we have a line attractor in a
7-dimensional space. In the remaining cases we have two equilibrium points
$(\bar{u}_{1\ }^{\ 1},\bar{u}_{1\ }^{\
2},\bar{u}^{2},\bar{u}_{1},\bar{u}_{2},\bar{u}^{2\ }_{\ 2},\bar{u}_{2\ }^{\
12})$ as

$\displaystyle(0,\tilde{U}_{1},w_{1\ }^{\ 2}\tilde{U}_{1},U_{1},w^{2\ }_{\
2}w_{1\ }^{\ 2}\tilde{U}_{1}+U_{2},w_{1\ }^{\ 2}\tilde{U}_{1},w^{2\ }_{\
2}w_{1\ }^{\ 2}\tilde{U}_{1}+U_{2})=$ $\displaystyle=(0,\tilde{U}_{1},w_{1\
}^{\ 2}\tilde{U}_{1},U_{1},\frac{w^{2\ }_{\ 2}w_{1\ }^{\
2}U_{1}+U_{2}}{1-w_{2\ }^{\ 12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}},w_{1\ }^{\
2}\tilde{U}_{1},\frac{w^{2\ }_{\ 2}w_{1\ }^{\ 2}U_{1}+U_{2}}{1-w_{2\ }^{\
12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}})$

and

$\displaystyle(U_{1}+\tilde{U}_{2},-\tilde{U}_{2},-w_{1\ }^{\
2}\tilde{U}_{2},U_{1},-w^{2\ }_{\ 2}w_{1\ }^{\ 2}\tilde{U}_{2}+U_{2},-w_{1\
}^{\ 2}\tilde{U}_{2},-w^{2\ }_{\ 2}w_{1\ }^{\ 2}\tilde{U}_{2}+U_{2})=$
$\displaystyle=(U_{1}+\tilde{U}_{2},-\tilde{U}_{2},-\frac{U_{2}}{w^{2\ }_{\
2}},U_{1},0,-\frac{U_{2}}{w^{2\ }_{\ 2}},0)$

When we take a cross-section of the 7-space to observe the $(u_{1\ }^{\
1},u_{1\ }^{\ 2})$ phase plane we considered earlier, we can associate the
first point with the 2-surface equilibrium point $(0,u_{1}+w)$ and the second
with a point on the 2-surface attractor line (since $\bar{u}_{2\ }^{\ 12}=0$).

The full equilibria are then $(\bar{u}_{1\ }^{\ 1},\bar{u}_{1\ }^{\
2},\bar{u}^{1},\bar{u}^{2},\bar{u}_{1},\bar{u}_{2},\bar{u}^{2\ }_{\
2},\bar{u}_{2\ }^{\ 12})$ with the addition of an extra dimension for $u^{1}$
which is $0$ for the first attractor point and $w_{1\ }^{\
1}(U_{1}+\tilde{U}_{2})$ for the second attractor point. In the case of
attractor line in the full phase space, the infinitely many equilibrium points
are parametrized by $(U_{1}-s,s,w_{1\ }^{\ 1}(U_{1}-s),w_{1\ }^{\
2}s,U_{1},w^{2\ }_{\ 2}w_{1\ }^{\ 2}s+U_{2},w_{1\ }^{\ 2}s,w^{2\ }_{\ 2}w_{1\
}^{\ 2}s+U_{2})$. Excluding the attractor line, the interpretation for the
first equilibrium which is a feedback extension of the previous $w_{2\ }^{\
12}u_{2\ }^{\ 12}\neq 0$ equilibrium of (78) demonstrates the role of cycles
within the network. If we take $w_{2\ }^{\ 12}w^{2\ }_{\ 2}w_{1\ }^{\ 2}=1$,
the cycle will keep accumulating potential forever and the equilibrium will be
located at $\bar{u}_{1\ }^{\ 2}=\infty$. At the same time $\bar{u}_{1\ }^{\
2}$ can be finite if the weights are less than unity and therefore scale down
the accumulated potential over time. Alternatively, if we have inhibitory
feedback $w^{2\ }_{\ 2}<0$ this will scale down $\bar{u}_{1\ }^{\ 2}$ for any
choice of positive $w_{2\ }^{\ 12}$ and $w_{1\ }^{\ 2}$. If $w_{2\ }^{\
12}=0$, the steady state becomes simply $\bar{u}_{1\ }^{\ 2}=U_{1}$ and the
crucial connection for the previous behavior is removed. If $w^{2\ }_{\ 2}=0$
or $w_{1\ }^{\ 2}=0$, the steady state becomes $\bar{u}_{1\ }^{\
2}=U_{1}+w_{2\ }^{\ 12}U_{2}$ and we only have the effect of the second
external input as before (the cycle is interrupted). The second equilibrium is
in the nonnegative region and therefore admissible if $U_{2}/(w^{2\ }_{\
2}w_{1\ }^{\ 2})\leq 0$ which implies either a nonpositive weight or
nonpositive external input both of which would reduce $\bar{u}_{1\ }^{\ 1}$.
The steady state is reached since the feedback connection exactly cancels the
external input $U_{2}$ leaving the two output connections somewhere on a point
of the attractor line where this happens. This is possible in particular if we
consider inhibitory feedback connection or negative external input $U_{2}<0$
but not $w_{1\ }^{\ 2}<0$ since the resetting condition will simply set
$u^{2}$ to zero.

Figure 15: Motifs of increasing complexity: d) circuit with competing
metaconnections; e) circuit with competing feedback connections;

Extending to cases with competition while still minimizing the added
complexity will lead to motifs like the ones shown on figure 15. A competitive
reduced version of (72) adds a second metaconnection to balance the potential
among the two outputs from before. The dynamical system definition extends in
the intuitive way on top of (78) and the 1-surface equilibria of both output
connections $u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ are the same as the ones for
$u_{1\ }^{\ 2}$ in (78). However the 2-surface equilibria are different and
solve the system

$\displaystyle\begin{dcases}&(\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}-w_{2\ }^{\ 11}\theta(u_{2\ }^{\ 11})u_{2\ }^{\ 11})(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}u_{1}\\\ &(\theta(\bar{u}_{1\ }^{\
2})\bar{u}_{1\ }^{\ 2}-w_{3\ }^{\ 12}\theta(u_{3\ }^{\ 12})u_{3\ }^{\
12})(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\
2}u_{1}\end{dcases}$

In the case $w_{2\ }^{\ 11}\theta(u_{2\ }^{\ 11})u_{2\ }^{\ 11}=0$ or $w_{3\
}^{\ 12}\theta(u_{3\ }^{\ 12})u_{3\ }^{\ 12}=0$ the system and corresponding
equilibria are reduced to the previous motifs (77, 78). It remains to solve

$\displaystyle\begin{dcases}&(\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}-w_{1})(\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 1}u_{1}\\\
&(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\ 2}-w_{2})(\bar{u}_{1\ }^{\
1}+\bar{u}_{1\ }^{\ 2})=\bar{u}_{1\ }^{\ 2}u_{1}\end{dcases}$

for $w_{1}=w_{2\ }^{\ 11}\theta(u_{2\ }^{\ 11})u_{2\ }^{\ 11}\neq 0$ and
$w_{2}=w_{3\ }^{\ 12}\theta(u_{3\ }^{\ 12})u_{3\ }^{\ 12}\neq 0$. First we
obtain

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 1}+\bar{u}_{1\ }^{\
2}=\frac{\bar{u}_{1\ }^{\ 1}u_{1}}{\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}-w_{1}}\\\ &\frac{(\theta(\bar{u}_{1\ }^{\ 2})\bar{u}_{1\ }^{\
2}-w_{2})\bar{u}_{1\ }^{\ 1}u_{1}}{\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}-w_{1}}=\bar{u}_{1\ }^{\ 2}u_{1}\end{dcases}$

which after canceling any possible terms becomes

$\displaystyle\begin{dcases}&\bar{u}_{1\ }^{\ 1}+\frac{w_{2}\bar{u}_{1\ }^{\
1}}{w_{1}}=\frac{\bar{u}_{1\ }^{\ 1}u_{1}}{\theta(\bar{u}_{1\ }^{\
1})\bar{u}_{1\ }^{\ 1}-w_{1}}\\\ &w_{2}\bar{u}_{1\ }^{\ 1}=w_{1}\bar{u}_{1\
}^{\ 2}\end{dcases}$

which has nonnegative solutions $\bar{u}_{1\ }^{\ 1}$ and $\bar{u}_{1\ }^{\
2}$ and therefore an equilibrium in the nonnegative region only if $w_{1}$ and
$w_{2}$ have the same sign. Finally, assuming this we get

$\displaystyle\theta(\bar{u}_{1\ }^{\ 1})\bar{u}_{1\ }^{\
1}-w_{1}=\frac{w_{1}u_{1}}{w_{1}+w_{2}}\Rightarrow\bar{u}_{1\ }^{\
1}=\frac{w_{1}(w_{1}+w_{2})}{w_{1}+w_{2}}+\frac{w_{1}u_{1}}{w_{1}+w_{2}}$
$\displaystyle\Rightarrow(\bar{u}_{1\ }^{\ 1},\bar{u}_{1\ }^{\
2})=\bigg{(}\frac{w_{1}(u_{1}+w_{1}+w_{2})}{w_{1}+w_{2}},\frac{w_{2}(u_{1}+w_{1}+w_{2})}{w_{1}+w_{2}}\bigg{)}$

with fully explicit potentials

$\displaystyle\Rightarrow\bar{u}_{1\ }^{\ 1}=\frac{w_{2\ }^{\ 11}\theta(u_{2\
}^{\ 11})u_{2\ }^{\ 11}(u_{1}+w_{2\ }^{\ 11}\theta(u_{2\ }^{\ 11})u_{2\ }^{\
11}+w_{3\ }^{\ 12}\theta(u_{3\ }^{\ 12})u_{3\ }^{\ 12})}{w_{2\ }^{\
11}\theta(u_{2\ }^{\ 11})u_{2\ }^{\ 11}+w_{3\ }^{\ 12}\theta(u_{3\ }^{\
12})u_{3\ }^{\ 12}}$ $\displaystyle\Rightarrow\bar{u}_{1\ }^{\ 2}=\frac{w_{3\
}^{\ 12}\theta(u_{2\ }^{\ 11})u_{2\ }^{\ 11}(u_{1}+w_{2\ }^{\ 11}\theta(u_{2\
}^{\ 11})u_{2\ }^{\ 11}+w_{3\ }^{\ 12}\theta(u_{3\ }^{\ 12})u_{3\ }^{\
12})}{w_{2\ }^{\ 11}\theta(u_{2\ }^{\ 11})u_{2\ }^{\ 11}+w_{3\ }^{\
12}\theta(u_{3\ }^{\ 12})u_{3\ }^{\ 12}}$

In the case of three outputs the equilibrium would be

$(\bar{u}_{1\ }^{\ 1},\bar{u}_{1\ }^{\ 2},\bar{u}_{1\ }^{\
3})=\bigg{(}\frac{w_{1}(u_{1}+\sum_{i}^{3}w_{i})}{\sum_{i}^{3}w_{i}},\frac{w_{2}(u_{1}+\sum_{i}^{3}w_{i})}{\sum_{i}^{3}w_{i}},\frac{w_{3}(u_{1}+\sum_{i}^{3}w_{i})}{\sum_{i}^{3}w_{i}}\bigg{)}$

and analogically for more outputs. The phase plane for positive and negative
weights (with one zero weight and two zero weights represented by the previous
cases) is shown on figure 16.

Figure 16: Phase portrait of $u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ from the
competitive motif where $u_{1}\geq 0$ and: a) $w_{2\ }^{\ 11}>0$ and $w_{3\
}^{\ 12}>0$; b) $w_{2\ }^{\ 11}<0$ and $w_{3\ }^{\ 12}<0$;

In the case of positive metaconnection weights $w_{2\ }^{\ 11}$ and $w_{3\
}^{\ 12}$ and therefore contributions $w_{1}$ and $w_{2}$ (zero cases again
reducing to previous motifs) this is a stable 2-surface equilibrium where the
two competing output connections $u_{1\ }^{\ 1}$ and $u_{1\ }^{\ 2}$ balance
according to the ratio of their contribution and the total contribution of all
metaconnections. The positivity of the contributions implies positivity of the
horizontal asymptotes for each connection (the slanted asymptotes being the
same $u_{1\ }^{\ 1}+u_{1\ }^{\ 2}=u_{1}$ line) which then suggests that the
nullclines will always intersect once resulting in this equilibrium point. In
the case of negative metaconnection weights $w_{2\ }^{\ 11}$ and $w_{3\ }^{\
12}$ it is a unstable 2-surface equilibrium whose perturbation in one of the
two unbalancing zones will result in it reaching either $(0,u_{1}+w_{2})$ or
$(u_{1}+w_{1},0)$ and getting locked there due to the resetting condition. In
this case we have negative horizontal asymptotes and both nullclines will
always contain the point $(0,0)$ but for asymptotes that are negative enough
the nullclines will no longer intersect in the positive region and the system
will converge to $(0,0)$. Finally, in the cases of one positive and one
negative weight (not shown on the figure) we have no nontrivial 2-surface
equilibria in the nonnegative region since both nullclines that enter it will
be on the two sides of the $u_{1\ }^{\ 1}+u_{1\ }^{\ 2}=u_{1}$ asymptote. In
this case, the system will converge to $(u_{1}+w_{1},0)$ if $w_{1}$ is the
positive contribution and to $(0,u_{1}+w_{2})$ if $w_{2}$ is the positive
contribution.

The interpretation of a positive and a negative weight is straightforward -
the unit outputs through the excited connection and not through the inhibited
connection. The case of negative weights has a possible balance and possible
perturbations (still balanced increase or decrease in both connections) under
which this balance will be restored but other perturbations (e.g. any increase
in one connection’s potential and decrease in the other) under which the
balance will tip over to a single winner. The case of positive weights is
always balanced, never lies on the axis of any of the connections (unless we
have zero weights and reduction to previous motifs), and most interestingly
provides any connection with the ratio of its contribution to the
contributions of all. If one of the two weights approaches zero, the
connection with the other weight will approach the total of the input and its
own contribution.

A competitive reduced version of (76) adds feedback connections and therefore
competing cycles. Once again, assuming nonzero weights $w_{2\ }^{\ 11}\neq 0$
and $w_{3\ }^{\ 12}\neq 0$ (any other case reduces to the previous motifs (77,
79)), we get the balanced full equilibrium

$\displaystyle\bar{u}_{1\ }^{\ 1}=\frac{w_{2\ }^{\ 11}U_{2}(U_{1}+w_{2\ }^{\
11}U_{2}+w_{3\ }^{\ 12}U_{3})}{(1-w_{2\ }^{\ 11}w^{1\ }_{\ 2}w_{1\ }^{\
1})(w_{2\ }^{\ 11}U_{2}+w_{3\ }^{\ 12}U_{3})}$ $\displaystyle\bar{u}_{1\ }^{\
2}=\frac{w_{3\ }^{\ 12}U_{3}(U_{1}+w_{2\ }^{\ 11}U_{2}+w_{3\ }^{\
12}U_{3})}{(1-w_{3\ }^{\ 12}w^{2\ }_{\ 3}w_{1\ }^{\ 2})(w_{2\ }^{\
11}U_{2}+w_{3\ }^{\ 12}U_{3})}$

which is stable for positive weights, unstable for negative weights, and
exists only if the two weights have the same sign similarly to the previous
motif. This equilibrium corresponds to the amplified (first) equilibrium of
(79) when one of the two weights is set to zero. The inhibitory-only (second)
equilibrium of (79) is now missing since every output connection is influenced
by a metaconnection and canceling the contribution to a connection will tip
over to the one that still has a contribution. At the same time, similar
amplification rules apply as for (79) which can allow otherwise nearly
depleted connections due to high competition like for instance $(w_{2\ }^{\
11}U_{2})/(w_{2\ }^{\ 11}U_{2}+w_{3\ }^{\ 12}U_{3})\ll 1$ to still be highly
active through $1/(1-w_{3\ }^{\ 12}w^{2\ }_{\ 3}w_{1\ }^{\ 2})\gg 1$.

Looking back at the full version of (72), the partial 1-surface equilibrium
states for all units are

$\displaystyle\bar{u}^{i}=\sum_{j=1}^{N}w_{j\ }^{\ i}\theta(u_{j\ }^{\
i})u_{j\ }^{\ i}=\sum_{j=1}^{N}w_{j\ }^{\ i}u_{j\ }^{\ i}$

since $\theta(u_{j\ }^{\ i})=0$ if $u_{j\ }^{\ i}=0$ or else $\theta(u_{j\
}^{\ i})=1$. The partial 1-surface equilibrium states for all connections
satisfy

$\displaystyle(\theta(\bar{u}_{j\ }^{\ i})\bar{u}_{j\ }^{\
i}-\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji})\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\ m\neq
j\end{subarray}}^{N}u_{j\ }^{\ ml}=\bar{u}_{j\ }^{\ i}u_{j}$

and for all metaconnections satisfy

$\displaystyle\theta(\bar{u}_{k\ }^{\ ji})\bar{u}_{k\ }^{\
ji}\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\ m\neq
k\end{subarray}}^{N}u_{k\ }^{\ ml}=\bar{u}_{k\ }^{\
ji}u_{k}\Rightarrow\bar{u}_{k\ }^{\ ji}=0\text{ or
}\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\ m\neq
k\end{subarray}}^{N}u_{k\ }^{\ ml}=u_{k}$

The conditions (73,74) for all connections are

$\displaystyle\begin{dcases}&\bigg{|}\sqrt{r_{j\ }^{\
i}}-\sqrt{u_{j}}\bigg{|}\geq\sqrt{-\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k\ }^{\ ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji}}\in\mathbb{R}\text{ for }\bar{u}_{j\ }^{\ i}\\\ &\text{always true
if}\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}>0\text{ for }\bar{u}_{j\ }^{\
i}\end{dcases}$

while in the case of metaconnections, they reduce further to

$\displaystyle\bigg{|}\sqrt{r_{k\ }^{\ ji}}-\sqrt{u_{k}}\bigg{|}\geq 0\text{
for }\bar{u}_{k\ }^{\ ji}$

Thus, the partial 1-surface equilibria for all connections are

$\displaystyle\begin{dcases}&\bar{u}_{j\ }^{\
i}=u_{j}+\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}\text{ if }r_{j\ }^{\ i}=0\text{ and
}\bar{u}_{j\ }^{\ i}\neq 0\\\ &\bar{u}_{j\ }^{\ i}=u_{1/2}\text{ if
}\big{|}\sqrt{r_{j\ }^{\
i}}-\sqrt{u_{j}}\big{|}>\sqrt{-\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k\ }^{\ ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}}\\\
&\bar{u}_{j\ }^{\ i}=-r_{j\ }^{\ i}+\sqrt{u_{j}r_{j\ }^{\ i}}\\\ &\qquad\text{
if }\big{|}\sqrt{r_{j\ }^{\
i}}-\sqrt{u_{j}}\big{|}=\sqrt{-\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k\ }^{\ ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji}}\end{dcases}$

for $\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}\leq 0$ and

$\displaystyle\begin{dcases}&\bar{u}_{j\ }^{\
i}=u_{j}+\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}\text{ if }r_{j\ }^{\ i}=0\text{ and
}\bar{u}_{j\ }^{\ i}\neq 0\\\ &\bar{u}_{j\ }^{\ i}=u_{1/2}\text{ otherwise
}\end{dcases}$

for $\sum_{\begin{subarray}{c}k=1\\\ k\neq j\end{subarray}}^{N}w_{k\ }^{\
ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\ ji}>0$. The partial 1-surface equilibria
for all metaconnections are

$\displaystyle\begin{dcases}&\bar{u}_{k\ }^{\ ji}=u_{k}\text{ if }r_{k\ }^{\
ji}=0\text{ and }\bar{u}_{k\ }^{\ ji}\neq 0\\\ &\bar{u}_{k\ }^{\ ji}=0\text{
or }\bar{u}_{k\ }^{\ ji}=u_{k}-r_{k\ }^{\ ji}\text{ if }\bigg{|}\sqrt{r_{k\
}^{\ ji}}-\sqrt{u_{k}}\bigg{|}>0\\\ &\bar{u}_{k\ }^{\ ji}=0\text{ if }r_{k\
}^{\ ji}=u_{k}\end{dcases}$

If we fix $\bar{k}\in[1;N]$, i.e. we look at the emitting dynamics of a single
input unit, we can utilize the simpler structure of the metaconnections to
deduce some higher-dimensional space equilibria. In particular, for all
$i\in[1;M]$ we have

$\displaystyle(\theta(\bar{u}_{\bar{k}\ }^{\ i})\bar{u}_{\bar{k}\ }^{\
i}-\sum_{\begin{subarray}{c}m=1\\\ m\neq\bar{k}\end{subarray}}^{N}w_{m\ }^{\
\bar{k}i}\theta(u_{m\ }^{\ \bar{k}i})u_{m\ }^{\
\bar{k}i})u_{\bar{k}}=\bar{u}_{\bar{k}\ }^{\ i}u_{\bar{k}}\text{ if
}\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\
m\neq\bar{k}\end{subarray}}^{N}\bar{u}_{\bar{k}\ }^{\ ml}=u_{\bar{k}}$
$\displaystyle(\theta(\bar{u}_{\bar{k}\ }^{\ i})\bar{u}_{\bar{k}\ }^{\
i}-\sum_{\begin{subarray}{c}m=1\\\ m\neq\bar{k}\end{subarray}}^{N}w_{m\ }^{\
\bar{k}i}\theta(u_{m\ }^{\ \bar{k}i})u_{m\ }^{\
\bar{k}i})\sum\limits_{l=1}^{M}\bar{u}_{\bar{k}\ }^{\ l}=\bar{u}_{\bar{k}\
}^{\ i}u_{\bar{k}}\text{ if }\bar{u}_{\bar{k}\ }^{\ ml}=0,m\neq 0$

where the first case yields an attractor $(MN-1)$-plane of equilibria

$\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\
m\neq\bar{k}\end{subarray}}^{N}\bar{u}_{\bar{k}\ }^{\
ml}=u_{\bar{k}}\quad\text{ if }\quad\sum_{\begin{subarray}{c}m=1\\\
m\neq\bar{k}\end{subarray}}^{N}w_{m\ }^{\ \bar{k}i}\theta(u_{m\ }^{\
\bar{k}i})u_{m\ }^{\ \bar{k}i}=0$

and the second case contains a system of connection equations which is solved
in the same way as before and yields a balanced equilibrium

$\bar{u}_{\bar{k}\ }^{\ i}=\frac{\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq\bar{k}\end{subarray}}^{N}w_{m\ }^{\ \bar{k}i}\theta(u_{m\ }^{\
\bar{k}i})u_{m\ }^{\
\bar{k}i}(u_{\bar{k}}+\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq\bar{k}\end{subarray}}^{N}w_{m\ }^{\ \bar{k}l}\theta(u_{m\ }^{\
\bar{k}l})u_{m\ }^{\
\bar{k}l})}{\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq\bar{k}\end{subarray}}^{N}w_{m\ }^{\ \bar{k}l}\theta(u_{m\ }^{\
\bar{k}l})u_{m\ }^{\ \bar{k}l}}$

If we assume the input balance condition $\sum_{m\neq\bar{k}}w_{m\ }^{\
\bar{k}i}\theta(u_{m\ }^{\ \bar{k}i})u_{m\ }^{\ \bar{k}i}=0$ for all
connections or simply for all $i$, the second case will rather become a
subcase of the first

$\sum\limits_{l=1}^{M}\bar{u}_{\bar{k}\ }^{\ l}=u_{\bar{k}}\quad\text{ and
}\quad\bar{u}_{\bar{k}\ }^{\ ml}=0,m\neq 0\quad\text{ if
}\quad\sum_{\begin{subarray}{c}m=1\\\ m\neq\bar{k}\end{subarray}}^{N}w_{m\
}^{\ \bar{k}i}\theta(\bar{u}_{m\ }^{\ \bar{k}i})\bar{u}_{m\ }^{\ \bar{k}i}=0$

Instead, for our derivation of the second case we need the balancing condition
for at least one output connection to be violated or its total contribution to
be nontrivial. If the balance condition holds for a connection
$\bar{u}_{\bar{k}\ }^{\ i}$ in general, from replacements above it can be seen
that $\bar{u}_{\bar{k}\ }^{\ i}=0$ and every other competitive connection
lacks the contributions of the metaconnections of $\bar{u}_{\bar{k}\ }^{\ i}$.
In the case of negative total contribution for a connection and positive for
another, by previous conclusions we have no stable state, the first connection
will decay to zero, and the balance equilibrium will be restored among the
other output connections. This means that we can disqualify all connections
with negative total contribution in the consideration of a steady state. There
could also be other cancellations in the above fraction but if one total
contribution is guaranteed to be positive and we disqualify all negative
cases, the fraction remains well-defined.

The partial equilibria for a fixed $\bar{k}$ are equilibria on the
$MN$-dimensional manifold only. If we consider two input units $k=1,2$ we can
obtain a higher-dimensional surface equilibrium with both of them if we
satisfy the attractor $(MN-1)$-plane conditions

$\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\ m\neq
k\end{subarray}}^{N}\bar{u}_{k\ }^{\ ml}=u_{k}\quad\text{ and
}\quad\sum_{\begin{subarray}{c}l=1\\\ l\neq k\end{subarray}}^{N}w_{l\ }^{\
ki}\theta(\bar{u}_{l\ }^{\ ki})\bar{u}_{l\ }^{\ ki}=0$

which are easy to interpret: all metaconnections settle once the total
potential of a unit is distributed among its output connections or even are
permanently deactivated (once deactivated, a connection needs input from
metaconnections to get more potential from the unit) and all connections
settle to a steady state when their input metaconnections balance and settle.
However, having a $(MN-1)$-plane attractor in the output units is not always
preferred and it is not possible to instead find balanced equilibria between
two units since each one requires deactivated output metaconnections and
active input metaconnections at the same time (due to the nontrivial total
contribution requirement). In lieu of this, a balanced unit equilibrium must
be accompanied by an attractor plane of another

$\displaystyle\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=0\\\
m\neq 1\end{subarray}}^{N}\bar{u}_{1\ }^{\ ml}=u_{1}\quad\text{ and
}\quad\sum_{\begin{subarray}{c}l=1\\\ l\neq 1\end{subarray}}^{N}w_{l\ }^{\
1i}\theta(\bar{u}_{l\ }^{\ 1i})\bar{u}_{l\ }^{\ 1i}=0$ (80)
$\displaystyle\bar{u}_{2\ }^{\ i}=\frac{\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq 2\end{subarray}}^{N}w_{m\ }^{\ 2i}\theta(u_{m\ }^{\ 2i})u_{m\ }^{\
2i}(u_{2}+\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=1\\\ m\neq
2\end{subarray}}^{N}w_{m\ }^{\ 2l}\theta(u_{m\ }^{\ 2l})u_{m\ }^{\
2l})}{\sum\limits_{l=1}^{M}\sum\limits_{\begin{subarray}{c}m=1\\\ m\neq
2\end{subarray}}^{N}w_{m\ }^{\ 2l}\theta(u_{m\ }^{\ 2l})u_{m\ }^{\ 2l}}$ (81)

The interpretation of all this is important - we cannot have properly
explained and therefore concentrated inputs unless we also have some context-
providing inputs or overall input features. Notice however that all of this
must hold asymptotically and there is no such restriction imposed for any
transient behavior, i.e. both units might behave as both contextualized and
feature units before settling. This and a more involved two-unit version of
(76) will be used to shed first light on the possibility of learning in
section 3.3.3.

We should complete the section with the full version (76) which also contains
connection cycles. In short, using the simpler structure of all
metaconnections but also the lateral and feedback connections (which also
don’t have metaconnection inputs), we can obtain the $(MN-1)$-plane attractors
as (with notation abuse like $u=\bar{u}$)

$\displaystyle\begin{dcases}&u^{i}=\sum_{j=1}^{N}w_{j\ }^{\ i}\theta(u_{j\
}^{\ i})u_{j\ }^{\ i}+\sum_{l=1}^{M}w^{li}\theta(u^{li})u^{li}\\\
&\theta(u_{j\ }^{\ i})u_{j\ }^{\ i}-\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k\ }^{\ ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji}-\sum_{l=1}^{M}w^{l\ }_{\ ji}\theta(u^{l\ }_{\ ji})u^{l\ }_{\ ji}=u_{j\
}^{\ i}\\\
&\sum\limits_{m=1}^{M}[u^{lm}+\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
l\end{subarray}}^{N}u^{l\ }_{\ nm}]=u^{l}=\sum_{j=1}^{N}w_{j\ }^{\
l}\theta(u_{j\ }^{\ l})u_{j\ }^{\
l}+\sum_{i=1}^{M}w^{il}\theta(u^{il})u^{il}\\\
&\sum\limits_{m=1}^{M}[u^{im}+\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
i\end{subarray}}^{N}u^{i\ }_{\ nm}]=u^{i}=\sum_{j=1}^{N}w_{j\ }^{\
i}\theta(u_{j\ }^{\ i})u_{j\ }^{\
i}+\sum_{l=1}^{M}w^{li}\theta(u^{li})u^{li}\\\
&\sum\limits_{m=1}^{M}\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
k\end{subarray}}^{N}u_{k\ }^{\ nm}=u_{k}=\sum_{i=1}^{M}w^{i\ }_{\
k}\theta(u^{i\ }_{\ k})u^{i\ }_{\ k}+U_{k}\\\
&\sum\limits_{m=1}^{M}[u^{lm}+\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
l\end{subarray}}^{N}u^{l\ }_{\ nm}]=u^{l}=\sum_{j=1}^{N}w_{j\ }^{\
l}\theta(u_{j\ }^{\ l})u_{j\ }^{\
l}+\sum_{i=1}^{M}w^{il}\theta(u^{il})u^{il}\\\ &u_{j}=\sum_{i=1}^{M}w^{i\ }_{\
j}\theta(u^{i\ }_{\ j})u^{i\ }_{\ j}+U_{j}\end{dcases}$

which reduces to

$\displaystyle\begin{dcases}&\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}w_{k\ }^{\ ji}\theta(u_{k\ }^{\ ji})u_{k\ }^{\
ji}+\sum_{l=1}^{M}w^{l\ }_{\ ji}\theta(u^{l\ }_{\ ji})u^{l\ }_{\
ji}=0,i\in[1;M],j\in[1;N]\\\
&\sum\limits_{m=1}^{M}[u^{im}+\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
i\end{subarray}}^{N}u^{i\ }_{\ nm}]=\sum_{k=1}^{N}w_{k\ }^{\ i}\theta(u_{k\
}^{\ i})u_{k\ }^{\ i}+\sum_{l=1}^{M}w^{li}\theta(u^{li})u^{li}=u^{i}\\\
&u_{j}=\sum\limits_{m=1}^{M}\sum\limits_{\begin{subarray}{c}n=0\\\ n\neq
j\end{subarray}}^{N}u_{j\ }^{\ nm}=\sum_{l=1}^{M}w^{l\ }_{\ j}\theta(u^{l\
}_{\ j})u^{l\ }_{\ j}+U_{j}\end{dcases}$

The previous balancing condition for each feedforward connection is now
extended to include feedback metaconnections as well. There is an extra output
settling condition this time from the output to the input units and both such
conditions now also relate the inputs with the outputs of a unit. Before we
used one of the $N$ equations of the third type and then of one of the $M$
equations of the second type to close the system (79).

As before, we can also obtain some balancing equilibria but this time no unit
needs its output connections to satisfy the attractor $(NM-1)$-plane
conditions since all feedback connections don’t have input metaconnections and
no feedback connection will go to zero guaranteeing nontrivial contributions.
This can be done analogically to before assuming simple enough connection
cycles like the case of the previous two motifs of (76). Otherwise, obtaining
analytic expression is still possible tracing the various dependencies of an
unknown on itself or using symbolic solver but not practical to include here
and more useful to analyze on the numerical side.

###  3.3 Numerical implementations

Even though the emphasis of this work is on the mathematical formulation and
analysis of metanetwork models, we will briefly outline some of their
numerical realizations. In particular, we will formalize the setting for each
field of application and share observations from any relevant simulation. None
of these developed prototypes were planned to compete with specialized state-
of-the-art solutions and they serve rather an illustrative purpose for the
family of models and the motivation behind them.

####  3.3.1 Feature detector

The first numerical implementation we will present is the generation of input
selectivity in an approximation of (59) which can be considered as contrast on
a lower level in the processing chain, i.e. detection of corner and edges
features. Using a simple forward Euler method or a first order approximation
for the derivative with $\delta=\tau_{r}$ as
$\tau_{r}du/dt\approx\tau_{r}(u(t+\delta)-u(t))/\delta=u(t+\tau_{r})-u(t)$ and
normalizing the time scale, we can produce the simulation algorithm

$\displaystyle\begin{dcases}&u^{i}(t+1)=u^{i}(t)-a(u^{i}(t))+\sum_{j=1}^{N}\bar{a}(u_{j\
}^{\ i}(t),u_{j}(t))\\\ &u_{j\ }^{\ i}(t+1)=u_{j\ }^{\ i}(t)-\bar{a}(u_{j\
}^{\ i}(t),u_{j}(t))+\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}\bar{a}(u_{k\ }^{\ ji}(t),u_{k}(t))+\\\
&\qquad-\sum_{\begin{subarray}{c}k=1\\\ k\neq
j\end{subarray}}^{N}\bar{a}(v_{k\ }^{\ ji}(t),v_{k}(t))+e(u_{j\ }^{\
i}(t),u_{j}(t),u_{j\ }^{\ 0,1}(t),\dots,u_{j\ }^{\ NM}(t))\\\ &u_{k\ }^{\
ji}(t+1)=u_{k\ }^{\ ji}(t)-\bar{a}(u_{k\ }^{\ ji}(t),u_{k}(t))+e(u_{k\ }^{\
ji}(t),u_{k}(t),...),\ k\neq j\end{dcases}$ (82)

Each pixel of an input grayscale image is processed by two types of units. The
first type must process the amount of white $\omega\in[0,1]$ in the pixel and
the second the amount of black $1-\omega$. The two-layer structure of (82)
then filters this image by providing an output of the same size which contains
higher values for more important parts of the image or parts drawing the
attention of the network. The connectivity between the input and output layer
is described in figure 17 with the addition that the network is implemented as
a cellular neural network (also abbreviated as CNN). What this means is that
full interlayer connectivity is replaced with local one where a metaconnection
from a unit will excite/inhibit only the connections of its neighbors of the
same/other type within a certain range. In this way we obtain sparse
connectivity and an output unit reacts only to a region of the input image or
biologically speaking its visual field. To avoid effects of the layer
boundaries due to the sparse connectivity, the network is implemented on a
torus, i.e. the connectivity follows a periodic surface.

Figure 17: Connectivity structure used for the feature detector - the drawn
connections are replicated for each input unit of the given type.

We have studied many variations of the above filter where we also use reverse
sign interactions or even only excitations (using weak inhibition which was
defined earlier as reducing the potential of a connection by diverting the
potential of its input unit), where we output the potential only if it is
within certain range (to make the output sparse) or replace the output
potential with a measurement of the uncertainty of each unit (which was
defined earlier as broadcasting small amounts of potential through each output
connection) as an indicator of importance and greater attention of the
network. Many of them give interesting insights into possible interpretations
of features but such a discussion is too long for this presentation. The
numerical results we will report here will concern the self-excitation and
mutual inhibition of the two types of units and the uncertainty measurement of
attended input.

We ran simulations with visual inputs that range from a single contrasting
line to small resolution videos, some of which are included in the appendix on
figure 19. A sanity check with uniform white or grey (less but equal activity
in both types of units) input produced no output at all. When looking at a
contour line separating a black and a white uniform regions, the network
immediately increased its attention at the line and if the line separated a
white and a middle grey regions the same happened but it took longer. Further
decreasing contrast resulted in further decreasing speed which eventually took
too long to observe. When looking at two simultaneous contour lines of high
and low contrast, the network first noticed the first line and if provided
with sufficient time also the second. This is desired for an optimal visual
processing when there could be too much input to attend to and the
computational resources are limited for all of it. A gradient was also reduced
to simpler contrast which implies a possibility to process blurry input (one
of the most challenging type of features with relatively high error rates for
both convolutional neural networks and major supervision algorithms like
SIFT). A contour line or gradient with higher overall luminosity was
registered by the first type and vice versa which implies that units are more
sensitive to disbalance if the competing type is silent.

When looking at a triangle, the network first noticed the corners and
gradually expanded its attention to the edges and ultimately the internal
area. This implies higher feature importance of corners for the determination
of shapes and resulting objects which is a thing to expect. When challenged
with a mixture of corners with varying contrast on a cube, the network started
with the high contrast angle and moved towards the low contrast angle with
preference of contrast over corners. It did this more easily with the first
unit type and remained more reserved with the second unit type, mostly due to
the dominance of white in the input. When looking at a star filled with noise,
the network was able to first ignore part of the noise and concentrate on the
shape where it preferred the sharper angles in the outer part and then
directed its attention to the inner obtuse angles and the edges. The same was
observed for a square with different levels of noise but with parts of the
square being harder to sustain and in favor of the noise. In both these last
cases, the first unit type exhibited the described resilience to noise while
the lack of sufficient activation in the second unit type could be the
possible (and even of a necessity) reason it was more susceptible to noise.

The feature detector from (82) evolved considerably even over static input,
shifting its attention to novelty or part of the input which wasn’t processed
before due to limited attention span at all times. When looking at a rotating
triangle, the network shifted its attention in accordance with the moving
corners, tracing the most changing parts of the input image. While it
generally has time to pay attention to less important features of a static
triangle, it only has time to perceive the main features of the moving
triangle before the input changes again. When looking at static, blinking, and
moving dots, the network noticed all three although we would hope to perceive
them in a fixed order depending on complexity. Such order could be different
depending on the knowledge of the network and which input would be too simple
or too complex for it to notice first - for a sufficiently experienced network
one should expect that changes in the input larger than the previously learned
ones should get more attention. Nevertheless, we found increase of attention
also around the blinking point and along the line tracing the trajectory of
the moving point which hints towards temporal shifts of attention, i.e. shifts
of attention influenced by past locations of a visual stimulus. We have
provided visuals about this and some tested videos in the appendix figure 20.

All the results above were obtained for fixed size images of 100 by 100 pixels
with the exception of the videos which were 176x144 pixels. No scaling or
preprocessing was performed on any image or sequence frame although most of
the images were created especially for these tests. The same set of parameters
was used for all cases in order to maximize reproducibility, namely a
connection range of one (metaconnections to nearest neighbors only) and a
potential range of $[0.05,3.0]$ for output connections which if more than one
would qualify the units as broadcasting due to uncertainty. The increase or
decrease of the connectivity range changes the level of complexity of the
detected features, could account for larger (scaled and blurry) angles, and
changes the sensitivity of the overall detector by increasing the visual field
of each output unit. The increase or decrease of the potential range has
significant effect on the final number of detected features and is generally
used to obtain sparse features of fixed size from a multitude of features of
various sizes which are harder to analyze.

The sparse connectivity implementation and the symmetrical arrangement of
single or multiple cell types is an intrinsic part of many network models of
the center-surround receptive field. These models involve lateral inhibition
either as lateral connections on the output layer or as feedforward
connections from the inputs to the output units [60, 30] and also produce
forms of contrast at the output layer. There are also infinitesimal
formulations of simple enough inhibitory interactions that use convolution of
a point image with the input but the current network has additional dynamics
which violate and prevent the superposition principle and therefore at least
the conceptually straightforward possibility of using convolution.

Generally, there is no established way to compare quality of feature detection
unless it is accompanied by other layers of processing involving feature
descriptors and feature matching. What is done sometimes in computational
vision is to reproduce visual illusions like the ones described by [53, 59].
In this way we can extract particularities of the visual system which can
reveal more about the way it works and provide concrete ground for biomimic
engineering approaches and comparison. There are numerous illusions related to
contrast and explained through lateral inhibition, most known of which are the
Mach bands illusion, the Cornsweet illusion, Hermann grid illusion, the
contrast effect, etc. Testing the feature detector on these did not provide
any results, possibly due to small connection range or even some of the
choices about the network connectivity.

####  3.3.2 Object tracker

The previous feature detector can be extended to track the most interesting
features with higher resolution (density of units) near the point of interest
and lower resolution far from it. These differences in resolution are inspired
by the retinotopic mapping demonstrated experimentaly in [82] which relates
proximity in areas of the visual image to proximity on the retina. In fact,
their paper showed that the mapping is distorted in favor of the most central
area with the most peripheral area taking a rather insignificant portion of
the retina. Using a simpler version of this with just two levels of
resolution, the present object tracker can be used for moving an eye or camera
in the direction of the most interesting feature therefore optimizing the
information inflow and reducing the processing of unnecessary details.

Another significant role of allowing the network to change its perspective is
explained by [61] in their study of the fixational eye movement. As they show,
even when the eye fixates on an object, there is always a microscopic movement
which counteracts fading of the input due to adaptation. Connecting the output
of the network to a camera motor, virtual selection of a subimage in a larger
image, or any other control that would allow it to make decisions about its
orientation and resolution of the environment would prevent it from
overadapting to a constant input which in the case of our feature detector
could be represented by an overwhelming increase of broadcasting and therefore
spread of attention and decrease of the potential on the output layer.

We have included one last figure in the visual appendix namely figure 21 where
the two resolutions together with the currently observed region of the input
are shown. The information perceived in the lower resolution environment will
direct the higher resolution region towards the feature of highest interest.
In the case of a sequence of a rotating triangles, the attention and therefore
the ”gaze” of the tracker is focused on the moving angles. As a result, it
follows the rotation of the triangle. In the case of a sequence of static,
moving and blinking points, the network switches view from the moving point to
the static point until it settles for the static point. It looks only briefly
at the blinking point. In the previous case where the network couldn’t direct
its attention, even though this input was dynamic it led to overadaptation
where the activity spread around all points and their trajectories. Of course
the same happened also in the peripheral vision of the tracker but for
significantly long simulation its direction remained fixated on the static
point. We would still expect it to move towards the more dynamical points
which might be achievable with a better choice of parameters, longer
simulation or with the addition of learning which should help drive attention
away from overadapted input.

A thorough comparison of the state of the art object tracking algorithms on
various challenging datasets was performed by [85]. The two criteria for
comparison they use are tracking success rate and location error with respect
to the object center. In order to compare the object center, the present
tracker needs at least multiple layers of abstraction in order to build
objects as composite features and track those. While a tracking success rate
could be extracted, the one used by the paper requires boxed objects to
overlap and we are currently only boxing around a single feature. As all
present implementations are merely illustrative of the fields of application
and the way the family of models can be shaped into such applications, we will
not investigate this further but suggest multiple layers for the curious in
this direction.

####  3.3.3 Pattern classifier

Even though the prospect of creating effective learning rules to form the
right attractors with the right basins of attraction seems far off given the
scope of the current work, we could at least consider a possible synthesis
procedure, similarly to (36, 37) for the additive network models. Such
procedures are the perfect middle step between the better understood study of
network activity (potential phase space) and the full scale study of network
connectivity (weight phase space). First, they are a bridge from the
structurally predetermined feature detection into the unsupervised
classification by memory recall and pattern recognition but a bridge which
remains structurally predetermined. Second, they can be considered as
instantaneous learning where one doesn’t need to worry about slower weight
dynamics and any resulting structural stability as well as bifurcations of the
potential dynamics (for instance memory consolidation can take the form of
differentiation of recallable states which could be realized through pitchfork
bifurcation or appearance of two equilibrium points). All memories are usually
stored in one shot when the weights are determined so that the network has
these as attractors and the weights are fixed and static afterwards with no
learning taking place from the network activity.

The formal requirements for a synthesis procedure are two:

  * •

all stored memories must be equilibria

  * •

all stored memories must be asymptotically stable

and some known synthesis approaches satisfy one or both conditions with
additional computational drawbacks [64]. The goal then is to find weights for
all connections so that the requirements are satisfied for a desired and
externally preselected set of memories. To illustrate synthesis following
these guidelines, we will consider again the two-unit model (80) of a
feedforward network (72) and to extend on the entire idea conceptually and
numerically, we will consider a two-unit model of a reduced recurrent network
(76). For both of them we will also assume all practical conditions on
nonnegativity, continuous activation, etc. used for their derivation. In
addition to these conditions, we will require that the dynamics of the output
connections of the feature unit $u_{1}$ be restricted to the set

$\\{u_{1\ }^{\ ml}=u_{1\ }^{\ m^{\prime}l^{\prime}},\forall
m^{\prime},m\in[0;N]\backslash\\{1\\},\forall l^{\prime},l\in[1;M]\\}$ (83)

Since we are guaranteed this to hold on broadcasting with each $u_{1\ }^{\
ml}=u_{1}/MN$, the only other thing we have to do is prevent nontrivial
contributions to the output connections $\bar{u}_{1\ }^{\ ml}$ so that it
holds for all times. We can do this by setting the weights of any
metaconnections to these output connections to zero and disallowing nonzero
external inputs for them. The effect of this extra assumption is threefold:

  * •

We avoid transient behavior where a unit could switch between the roles of
feature and contextualized unit. This behavior is a lot more interesting but
is currently avoided for the sake of simplicity.

  * •

In this way the output connections of the unit can only converge to the middle
point of the attractor $(MN-1)$-plane and we won’t violate the second
condition of asymptotic stability although we have a degenerate equilibrium
point.

  * •

We will reduce the degrees of freedom of the algebraic system for the
equilibria, having fully determined values for all potentials so that we can
formulate a system for the weights.

We will call $u_{1}$ an unbiased feature unit (a unit that at no times has
metaconnection contributions to its output connections, i.e. is contextualized
or explained by others) and $u_{2}$ a contextualized unit.

Figure 18: Assumption on unbiased features forbids the first case with allows
only the second case.

The current synthesis approach given the above requirements and assumptions is
to construct an algebraic system in the weight space after first determining
all equilibrium unknowns in the potential space, all of this for a fixed input
$\tilde{u}_{k}$ and a fixed (desired for recall) output $\tilde{u}^{i}$.
Taking a 2-output version ($i=1,2$) of (80) with the extra assumption (83) on
unbiased features