# Emergent communication enhances foraging behaviour in evolved swarms
controlled by Spiking Neural Networks

ID Cristian Jimenez-Romero  Simulation and Data Lab Neuroscience, Jülich
Supercomputing Centre (JSC), Institute for Advanced Simulation, JARA,
Forschungszentrum Jülich GmbH, Jülich, Germany  ID Alper Yegenoglu  Simulation
and Data Lab Neuroscience, Jülich Supercomputing Centre (JSC), Institute for
Advanced Simulation, JARA, Forschungszentrum Jülich GmbH, Jülich, Germany
Institute of Geometry and Applied Mathematics, Department of Mathematics, RWTH
Aachen University, Aachen, Germany  ID Aaron Pérez Martin  Simulation and Data
Lab Neuroscience, Jülich Supercomputing Centre (JSC), Institute for Advanced
Simulation, JARA, Forschungszentrum Jülich GmbH, Jülich, Germany  ID Sandra
Diaz-Pier  Simulation and Data Lab Neuroscience, Jülich Supercomputing Centre
(JSC), Institute for Advanced Simulation, JARA, Forschungszentrum Jülich GmbH,
Jülich, Germany  ID Abigail Morrison  Simulation and Data Lab Neuroscience,
Jülich Supercomputing Centre (JSC), Institute for Advanced Simulation, JARA,
Forschungszentrum Jülich GmbH, Jülich, Germany  Institute of Neuroscience and
Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA BRAIN
Institute I  Computer Science 3 - Software Engineering, RWTH Aachen
University, Aachen, Germany

###### Abstract

Social insects such as ants and termites communicate via pheromones which
allows them to coordinate their activity and solve complex tasks as a swarm,
e.g. foraging for food or finding their way back to the nest. This behaviour
was shaped through evolutionary processes over millions of years. In
computational models, self-coordination in swarms has been implemented using
probabilistic or pre-defined simple action rules to shape the decision of each
agent and the collective behaviour. However, manual tuned decision rules may
limit the emergent behaviour of the swarm. In this work we investigate the
emergence of self-coordination and communication in evolved swarms without
defining any explicit rule. For this purpose, we evolve a swarm of agents
representing an ant colony. We use an evolutionary algorithm to optimize a
spiking neural network (SNN) which serves as an artificial brain to control
the behaviour of each agent. The goal of the evolved colony is to find optimal
ways to forage for food and return it to the nest in the shortest amount of
time. In the evolutionary phase, the ants are able to learn to collaborate by
depositing pheromone near food piles and near the nest to guide other ants.
The pheromone usage is not manually encoded into the network; instead, this
behaviour is established through the optimization procedure. We observe that
pheromone-based communication enables the ants to perform better in comparison
to colonies where communication via pheromone did not emerge. Furthermore, we
assess the foraging performance of the ant colonies by comparing the SNN based
model to a multi-agent rule based system. Our results show that the SNN based
model can complete the foraging task more efficiently in a shorter time. Our
approach illustrates that even in the absence of pre-defined rules, self
coordination via pheromone emerges as a result of the network optimization.
This work serves as a proof of concept for the possibility of creating complex
applications utilizing SNNs as underlying architectures for multi-agent
interactions where communication and self-coordination is desired.

##  1 Introduction

Communication using chemical signals is an efficient way to interact and
collaborate within swarm colonies. For example, social insects such as ants
and termites deposit pheromones to guide their cohorts while foraging for
food. The coordination does not follow a top-down hierarchy, i.e., there is no
directing leader. Instead, the coordination is self-organized and enables the
swarm to solve complex tasks, such as finding the shortest path between the
nest and the food source. This behaviour is shaped through evolution over
millions of years (Wilson et al., 1967; Smith et al., 2008; Wilson and Nowak,
2014; Boudinot et al., 2022).

From a computational point of view the self-organized behaviour in swarms
described in the literature has been implemented by using (simple) action
rules which allow the agents to interact with other cohorts and the
environment by depositing chemical signals. The self-coordination of the swarm
emerges as a result of these interactions, allowing the agents to solve
problems, e.g. finding the shortest path in an environment. Defining explicit
rules may help the system to solve specific tasks yielding in a foreseen
behaviour, i.e. the system is guided to find a certain solution. Although this
approach may converge into a solution it may not be the most efficient one.
Moreover, defining rules, which lead to emergent interactions and self-
coordination at the level of the swarm, may not be straightforward for complex
tasks. The pre-defined rules are biologically implausible, since they include
probabilistic actions, which may not reflect the natural behaviour of the
swarm. In this work we investigate the emergence of self-coordination and
communication in evolved swarms without defining any explicit rules which
describes the behaviour of each agent. Specifically, our research questions
are:

  * •

Does self-organized coordination emerge within the swarm, based on the
simulated physiological properties of the agents and the characteristics of
the environment?

  * •

Could the presence of a self-organized coordination between the agents enhance
the collaborative approach to solve specific problems?

To tackle these questions, we define a task and evolve an entire swarm of
agents searching for food and bringing it back to the nest, resembling the ant
colony foraging behaviour. In particular, we are interested in exploring the
role of stigmergy in the evolutionary emergence of communication within the
swarm. Stigmergy refers to indirect interactions which take place via the
environment for the purpose of navigation, communication or collaboration. Ant
colony foraging is a self-organized behaviour which is well investigated
within the literature. In these works several computational models replicate
to some extent the dynamics observed in biological colonies.

For instance, the authors of Vittori et al. (2004) and similarly of Bandeira
de Melo and Araújo (2008) compare the foraging behaviour of the Argentine ants
Linepithema with an implementation of a probabilistic model. Their models are
able to find the shortest paths in a maze to a food source relying on self-
organization based on pheromone trails. In Hecker and Moses (2015), the
authors show the result of implementing an ant swarm algorithm to control
simulated and real robots in environments, where the distribution of food
changes. They control each robot using a rule-based probabilistic model
mimicking the foraging behavior of seed-harvester ants and optimize the
performance of the collective behaviour using a genetic algorithm. They
analyze how the collective strategies and communication are evolved. The model
presented in Wilensky (1997) illustrates self-coordination and collaboration
between ants through the use of pheromone. This model is utilizing simple,
pre-defined rules to control the local decisions of the ants.

In an alternative approach, Duan and Sun (2014) propose a framework to soft
control agents, called shills, which are able to follow designed update rules.
Their task is based on an iterated prisoner’s dilemma game, where shills can
update their strategies using particle swarm optimization (PSO) mechanisms. In
their setting, shills can choose better performing strategies and provide
those to other individuals. The frequency of the communication between the
shills can be steered by different parameter configuration.

Swarms have also been combined with spiking neural networks (SNNs) in the
past. In Chevallier et al. (2010), the authors implement a spatio-temporal
model called ”SpikeAnts” and depict the emergence of synchronized activities
in an ant colony. They describe a sparsely-connected network of spiking
neurons, where each ant is represented by two interconnected neurons. The
network does not receive any external stimuli and does not incorporate
learning rules. The emergent behaviour shows a distributed learning process in
the population of agents and it is concluded that in their setting the swarms
is able to self-organize.

The aforementioned works describe and analyze emergent communication due to
chemical signals, however, these approaches are probabilistic state models or
rule based systems, which enforce a collaborative behaviour. If we want a
swarm of robots to optimally use their physiology to achieve certain tasks in
a collaborative and adaptive way, trying to formalize all the rules and
interactions which address not only individual but also collective behaviours
in a dynamic realistic environment becomes complex. This approach may also not
scale well if the complexity of the task or the number of agents increases.
Leveraging emergent behaviour could provide a reasonable and adaptable
approach to deal with this scenario. For this reason, we implemented a virtual
environment where the behaviour of swarms of agents can be optimized using
evolutionary algorithms. With these tools we study and analyze the benefit of
evolved emergent behaviour against rule based approaches in solving a task
where coordination plays a key role in the performance. In our approach agents
do not follow pre-defined rules to guide their behaviour. This is a critical
difference to previous work, as with this approach we do not steer the
learning process towards specific solutions derived from our own understanding
of how these complex systems work. Instead, we give the optimization process
full control over the solutions it will explore in order to maximize the
fitness measures. In our setting, each agent is controlled by an SNN composed
of several neurons. The agents perceive their environment through several
neurons which receive sensory information and implement a mapping from this
input to output behaviours. In general, spiking neural networks have been
proposed as alternative methods to solve machine learning tasks which offer
biologically realistic configurations in comparison to artificial neural
networks or state models. SNNs have the capability to control agents and
robots in dynamical environments (Beyeler et al., 2015; Nichols et al., 2010,
2012). We optimize the synaptic weights and spike time delays of the entire
network using genetic algorithms, so that the network is able to control the
agents to navigate in the virtual environment while searching and collecting
food and bringing it back to the nest. At the beginning of the evolutionary
process, the SNN is agnostic of the foraging task and there is no pre-defined
communication or coordination mechanism encoded into the architecture of the
SNN. There is no explicit mapping between the sensory input and the actions
taken by the ants, e.g. visual information and pheromone depositing.

Figure 1:  Screenshot of the environment. The picture shows an ongoing
simulation with an active foraging swarm. White/blue patches indicate the
presence of pheromone. The brighter the color the higher the pheromone
concentration. Food piles are depicted with green patches and leafs. The black
hexagon in the middle is the nest.

The virtual environment, in which the simulations are run, is based on the
ant-colony model by Wilensky (1997). In this model, the diffusion and
evaporation of pheromone is simulated in a 2-dimensional grid world. For an
example of a possible distribution of pheromone for an active swarm, see
Figure 1. The ants colored in red are exploring the world for food (green
leaves) and depositing pheromones near the food patches and in the vicinity of
their nest. While some ants are following the pheromone trail leading to the
food patches or to the nest, others are exploring the environment. Every ant
is steered by a spiking neural network. To optimize the network parameters and
create the environment, we connect the Learning to Learn framework (L2L;
Yegenoglu et al., 2022) with the multi-agent simulator NetLogo (Tisue and
Wilensky, 2004) and the spiking neural network simulator NEST (Gewaltig and
Diesmann, 2007).

Our manuscript is structured as follows. First, in Section 2, we introduce our
toolchain and setup of the environment. We describe the structure of the agent
network and how we employ a genetic algorithm to optimize the synaptic weights
and delays over multiple generations. Then, in Section 3 we analyze the
emergent communication via pheromones and compare our network structure to the
base model implemented in Wilensky (1997) in terms of foraging performance. We
examine how the pheromone usage and concentration is shaped by the
evolutionary process which enhances the self-coordination and communication
between the ants. Moreover, we investigate the correlation between the input
and output spike activity produced by an agent’s network. This analysis shows
us that certain input actions lead to specific output reactions. We measure
the significance and relate these actions to each other. Finally, we discuss
the results of the swarm simulations, with special interest on the emergent
communication, and we mention some avenues for future work.

As a summary, in this manuscript we show that agents without pre-defined rules
can self-organize through evolutionary optimization which allows them to solve
complex tasks as a swarm, leveraging their physiological capabilities.
Interestingly, in our specific use case of an ant colony, we observe that the
pheromone acts as an attractor rather than an event-based signalling
mechanism, which resembles the behaviour in biological colonies. This work
provides a proof of concept environment which can be used to study learning
and emergence of complex behaviour in multi-agent settings.

##  2 Material and Methods

Figure 2: Optimization workflow. In each generation $32$ individuals are
simulated. Each individual corresponds to an instance of an SNN and consists
of $15$ ants with a copy of the same network. The genetic algorithm requires
the calculated fitness of every individual to perform the optimization on the
connection weights and delays.

We carry out optimization of a spiking neural network to steer an ant colony
foraging for food using a genetic algorithm (GA). The optimization workflow is
depicted in Figure 2. There are $32$ individuals per generation. Each
individual corresponds to an instance of an SNN (see Figure 3) It is important
to note, that an individual refers to the SNN instance and not to a simulated
ant. There are $15$ ants per simulation forming the colony and every ant is
controlled by an SNN, which is a copy of the same individual (network). The GA
optimizes the connection weights and spike time delays of each network. Since
there are $32$ individuals we are running $32$ simulations in parallel. Thus,
we execute $15$ identical networks during one simulation per individual and
$480$ networks in total per generation. Each colony is simulated within a 2D
environment to perform the foraging task. After the simulation finishes, a
fitness value is calculated to evaluate the performance of the run.

###  2.1 Toolchain

Our main toolchain111Our simulation and analysis scripts are publicly
available at https://github.com/Meta-optimization/L2L/tree/neuroevolution/
consists of three applications: The meta-learning framework Learning to Learn
(L2L; Yegenoglu et al., 2022), the multi-agent simulator NetLogo (Tisue and
Wilensky, 2004) and the spiking neural network simulator NEST (Gewaltig and
Diesmann, 2007).

L2L (Version 1.0.0-beta) is the orchestrator of the workflow (see
Supplementary Material for L2L optimization details). The framework requires
the definition of the optimization algorithm, including its
(hyper-)parameters, and the simulation whose parameters are going to be
optimized. The software architecture consists of a two loop structure, the
inner loop (optimizee) and the outer loop (optimizer). In the inner loop, the
optimizee – for example a neural network with or without learning capabilities
– is trained on a family of tasks. In our setting, the SNNs in the inner loop
do not learn and only the performance on the foraging task is evaluated, the
optimization is conducted in the outer loop. The performance of the training
or simulation run is evaluated by a fitness function, yielding a fitness which
can be a scalar or a vector value. In the outer loop, the (hyper-) parameters
of the optimizee are optimized. For example, if the optimizee is a network,
the parameters could be the learning rate and the weights. The aim is to reach
an overall good performance and to enable faster learning on new, unknown
tasks. In this work we use the L2L framework’s genetic algorithm as optimizer
in the outer loop, which wraps the implementation from the DEAP (Fortin et
al., 2012) library.

NetLogo (Version 6.2.0) is a multi-agent simulator. Every object in a NetLogo
simulation is an agent and can be used as means of communication with other
agents. With NetLogo it is easy to replicate the same simulation and agent. In
our settings the simulations are spiking neural networks and the agents are
ants. Each ant is controlled by one instantiation of a common spiking neural
network. To communicate with each other the ants can deposit pheromones.
Patches are used to represent the pheromone which is dropped by the ants on
the environment. Each ant is also capable of sensing the presence of
pheromones. An important difference to notice in this implementation as
compared to the works discussed in Section 1, is that the dynamics of the
pheromone depositing and sensing are not hard-wired in the ant network model.
NetLogo allows the simulation of the pheromone dynamics through its embedded
language primitives.

The simulations uses the spiking neural simulator NEST (Version 3.0) as the
back-end. NEST is a spiking neuronal simulator for biologically plausible,
large-scale neural network simulations. Simple neuron models with biological
realistic connection structures can be simulated in an efficient way in a
variety of hardware from local machines up to High Performance Computing (HPC)
systems (Jordan et al., 2018). NEST supports multi-threading and parallel
execution with the message passage interface (MPI).

###  2.2 Ant colony foraging for food

Figure 3: Spiking network to control an ant of the colony. The topology
comprises $12$ neurons for the input, $20$ all-to-all connected neurons in the
middle and four neurons in the output.

In this section we describe the experimental setup for the ant colony. The
agents have to search for food and bring the food back to the nest, shown as a
brown/black patch in Figure 1. Each ant can only collect one food patch at a
time and must transport it back to the nest before looking for food again.
There is no pre-programmed behaviour to signal the ants that this is the
activity pattern they should follow. This pattern is enforced by not allowing
ants to carry more than one piece of food and rewarding ants when they bring
the food to the nest, at which point they also regain the capability to pick
up another piece of food. In every generation we change the position of the
nest randomly, in order to avoid overfitting the network and make the colony
more adaptable to the location of the food piles.

As depicted in Figure 1, every agent can deposit pheromone which is
represented by a white patch. The dropped pheromone spreads from the point
(patch) of release to a certain amount of neighboring patches according to the
diffusion rate, defined in the simulation parameters. Pheromone concentration
also decreases (evaporates) exponentially over time. There is no predefined
behaviour that binds depositing pheromone to any action or sensor. Other
agents can smell the pheromone and react to it. The response elicited in an
agent by the pheromone emerges from the evolutionary process carried out by
the genetic algorithm. For example, the sensing of pheromone at a certain
location could modify the trajectory of an agent looking for a food source or
the nest.

Each agent is controlled by its own network, a copy of the individual SNN. An
example of such a network is shown in Figure 3. The network consists of $36$
neurons and $720$ connections. The agents perceive their environment through
sensory stimuli from various modalities such as visual, mechanoreceptory or
chemical (pheromone) sensors while navigating. The input neurons that receive
the sensory information are connected to a second layer of neurons which are
all-to-all connected. By having an all-to-all connectivity we avoid
constraining the network to a pre-defined topology. At the same time the
second layer is connected to the output neurons which result in the agent’s
actions such as movement and pheromone laying.

The first twelve neurons correspond to the input layer. The top three neurons
in Figure 3 are receptor neurons which react to the direction of the
pheromone. The next three neurons can locate the nest. The middle of the nest
is indicated by the queen receptor. The reward and punishment for the ant are
determined by the reward receptors and nociceptors. Whenever food or the wall
is seen the green and red photoreceptors are activated correspondingly. The
heartbeat neuron stimulates the network in every timestep by supplying a small
direct current in order to maintain activity in the network. The middle layer
comprises twenty neurons, which are connected in an all-to-all manner. The
four output neurons are fully connected with the middle layer and control the
movement (rotate left, rotate right, go forward) as well the activation to
drop the pheromone.

The total fitness of the ant colony optimization is calculated by summing up
the rewards and punishments received by each of its ants in the colony during
the course of a simulation. An ant receives a small positive reward for
touching a food patch, and a large reward for returning with food to the nest.
This induces the ants to quickly return to their nest, whenever they find
food. A small punishment is given at every time step to encourage the colony
to complete the task as quickly as possible. At the same time, the ants
receive a small punishment for every selected action to avoid excessive
reactions, such as depositing too much pheromone or exhibiting too much
movement.

The simulation takes $T=2000$ steps to finish; if the ants are able to collect
all the food before the simulation ends they receive a reward as well. This
reward is the result of the difference between the total simulation time and
the time $T_{s}$ the ants require to collect the food multiplied with a scalar
$\eta$. The whole process can be formalized as:

$f_{i}=\sum_{t=1}^{T_{s}}\left(\sum_{j=1}^{J}\mathcal{N}^{(t)}_{i,j}+\mathcal{F}^{(t)}_{i,j}-\mathcal{C}^{(t)}_{i,j}\right)+\eta\left(T-T_{s}\right),$
(1)

where $t=1,\ldots,T_{s}$ is the simulation step with $T_{s}\leq T$ and $T$ the
total simulation time, $\eta$ is a scalar to weight the speed of the ants
collecting the food. Within the sum, $i$ is a specific individual,
$j=1,\ldots,J$ is an ant and $J$ is the total number of ants in a colony.
$\mathcal{N}$ is the positive reward value for coming back to the nest with
food, $\mathcal{F}$ is a positive value for touching the food and
$\mathcal{C}$ is the punishment cost. The behavioural reward and cost values
are specified in the Supplementary Material (Table S1) and the other
experimental parameters can be found in Supplementary Material (Table S2).

##  3 Results

In the following, we show the result of executing simulations of the foraging
task using the L2L framework. See Supplementary Material for further details
on the execution. We explore and compare three models:

  1. 1.

an SNN-driven model, where each ant is controlled by identical copies of a
common network,

  2. 2.

the same SNN-driven model but with the pheromone pathway deactivated (sensing
and depositing of pheromone is disabled),

  3. 3.

a rule-driven model based on the implementation in NetLogo by Wilensky (1997).
In this model each ant follows a set of predefined simple rules.

###  3.1 Fitness and emergent usage of pheromone

Figure 4:  Fitness of the ant colony of two distinct simulation runs. In the
first SNN-driven model the pheromone pathway is activated. However, the ants
of the second SNN-driven model cannot sense and deposit pheromones. While the
blue data series depicts the evolution of the first model’s fitness, the green
data series illustrates it for the second model. The dotted data series shows
the best individual found in every generation. The blue/green solid line
indicates the mean fitness and the blue/green shaded area is the standard
deviation over all individuals.

Figure 4 shows the evolution of the mean and best fitness over $1700$
generations for the SNN-driven experiments mentioned above. In the first SNN-
driven model the pheromone pathway is enabled, i.e. the capability of the ants
to deposit and sense the pheromone is active. For this model, we observe that
the mean fitness is increasing over the generations. After the $1000$th
generation the mean converges and oscillates around a value of $8100$ with a
standard deviation of $3970$. Specific individuals reach a fitness higher than
$30000$ with the highest fitness at $38004$ in generation $1420$ (see green
data series). The second SNN-driven model has no active pheromone pathway, the
ants are not able to perceive or deposit the pheromone. Here, we observe an
increasing mean, which converges after $700$ generations around a value of
$5800$ with a standard deviation of $3950$. In comparison to the first model,
specific individuals reach a high fitness in early generations (e.g.
generation $390$ with a fitness of $21716$) and the highest fitness at $24450$
in generation $1150$. The increase of performance of both models demonstrates
that the genetic algorithm in our approach is capable of evolving the SNNs to
achieve a much higher performance than that of a random network
initialization, which exhibited a negative fitness. Furthermore, the colonies
that are able to communicate via pheromones can be adapted via learning to
achieve a higher fitness and performance than the colony without the ability
to utilize pheromone.

Figure 5:  Pheromone concentration heatmap over different generations. The
brighter the color the higher the pheromone concentration. The marginal plots
at the sides are density plots, indicating the amount of the concentration
with regards to the $(x,y)$ position. The ants are colored in white, food
patches in green. The brown hexagon in the middle is the nest.

Beyond the evolution of the fitness, we examined the emergence of coordination
via pheromones. The heatmap in Figure 5 shows the pheromone concentration for
an example individual over different generations. Each simulation runs for
$2000$ steps and at step $1900$ we take a snapshot of the best individual with
the position of the ants, the pheromone and the food patches, as well as the
pheromone concentration and the food amount.

In the early generations (Figure 5a) the concentration is low, while most of
the food patches are uncollected. The ants are exploring the environment and,
from visual observation (as seen in Figure 5a), about half of the population
is close to the nest as well as to the food pile on the right side
($x:35,\,y:30$). Small pheromone patches can already be observed around the
nest and the closest food patch. This is also visible in the marginal plots
depicting the pheromone density. The peaks are at the position of the nest
($x:22,\,y:30$) and the right food pile. In generation $200$ (Figure 5b) it
can be observed that there is a correlation between the pheromone
concentration and the position of the food piles and the nest. The marginal
plots have $4$ peaks, at the nest and at the left and right food piles (bottom
left $x:5,\,y:5$, top left $x:5,\,y:52$, right $x:35,\,y:28$). In generation
$600$ a higher pheromone usage is visible (Figure 5c) in comparison to
generation $200$. The concentration is nearly equally distributed over the
environment, which is also visible in the density plots. The final snapshot
shows generation $1500$, which is similar to generation $200$, but the
pheromone is more concentrated and exhibits a clear pheromone path. In
comparison to earlier generations, the smaller number of food patches indicate
that more have been collected, thus a better task performance. The density
plots depict the highest pheromone concentration in the middle, where the nest
is positioned and at the left bottom.

###  3.2 Performance comparison

Figure 6:  Performance comparison of different ant colony models. The boxes
depict the average number of food patches collected in 2000 simulation steps
in $100$ trials. The vertical black line shows the standard deviation. In
every trial the position of the nest is changed by setting a different seed.

To assess the performance of the ant colony, we first selected the best
performing individual of the last generation after the optimization procedure.
Secondly, we calculated over $100$ simulations (trials) the average amount of
food foraged by the colony (and brought back to the nest) within the
simulation time of $2000$ steps. The trials have different seeds which
determine the initial position of the ants in each trial. There are three
piles of food positioned in different areas of the simulated 2D-world as shown
in Figure 5. Each pile contains $50$ units of food making a total of $150$
units, which is the maximum score that a colony can reach in this experimental
setup.

We tested the three models introduced in the beginning of section 3.
Furthermore, we added a forth experiment in which we impaired the pheromone
pathway for the first SNN based model. The results are shown in Figure 6. The
first bar illustrates the foraging performance of a rule based system
implemented in NetLogo (Wilensky, 1997), i.e., the ant foraging behaviour
follows pre-defined action rules. The self-organized behaviour of the ants
emerges from the interactions of individual colony members. In this setting,
the colony reaches a mean of $124.5$ foraged units of food with a standard
deviation of $19.6$. We define the rule based system as our baseline for
comparison. The second bar depicts the SNN based model 1 which reaches both a
higher mean score ($135.7$) and a lower standard deviation ($9.0$) than the
rule-based system. To measure the effect of the pheromone on the colony
coordination, we disable the pheromone sensing in the first SNN model after
evolving it. We observe that the foraging performance drops significantly and
the ants are not able to collect the food (mean: $0.6$, SD: $0.9$) as shown in
the third bar. The fourth bar shows the performance of the second SNN based
model. It reaches a mean score of $97.1$ with a standard deviation of $60.0$
and performs less efficient than the first SNN based model and the rule based
one.

Based on these results, we can conclude our evolved model with a self-
coordination system is able to complete the foraging task in a shorter amount
of time than both the rule-based system and the second SNN model which is
solely using the visual sensory pathways to navigate and forage for food. The
ants are able to learn to communicate via pheromones, which increases their
performance when foraging for food, despite the fact that there is no
hardwired mapping that defines the usage of pheromones for communication
purposes. The communication emerges within the ant colony over the generations
as a swarm strategy.

###  3.3 Input-output correlation analysis

Figure 7:  Correlation heatmap between the network input and output spike
trains. The mean Pearson product-moment correlation coefficients between the
input and output spike trains of all ants from the best individual are
calculated.

In order to better understand how the ant behaviour changes over the
generations, we correlate all input spike trains with all output spike trains
for all ants of the best individual for specific generations. The results of
this correlation analysis are limited and provide just an initial overview of
the relationships between input and output in the network. A more complete
assessment of the correlations in a recurrent network with dynamic inputs
requires a complex analysis which considers not only snapshots of inputs and
outputs but also the internal state of the model and its transitions through
time. This type of analysis is out of scope for this work and will be the
topic of further publications.

First, we create a histogram to bin the input and output spike trains using
the Electrophysiology Analysis Toolkit (Elephant222Version 0.11.1; Denker et
al., 2018). As a second step, we obtain the mean Pearson product-moment
correlation coefficients. Figure 7 depicts the correlation heatmap for the
input (x-axis) and the output (y-axis) activity (see also Figure 3 for the
network description) for four different generations. While a red color
indicates a high correlation, a blue color denotes a negative correlation. The
nomenclature follows the naming scheme in Figure 3, i.e. for the output
activity Left is rotate left, Move means move forward etc. and similarly for
the input activity.

Across all generations, Smell Middle is positively correlated with Move, with
a value of $0.265$ in generation $10$ and a reduced value of $0.116$ in
generation $1600$. One can also see that e.g. Smell Right is negatively
correlated with Move, it has a value of $-0.487$ in generation $10$, but at
generation $1600$ the strength of this anti-correlation is reduced to
$-0.182$. Interestingly, Smell Left is negatively correlated with Left in
generation $10$ ($-0.213$) but shows a positive correlation in later
generations ($0.219$ in generation $1600$). A similar observation can be made
for Smell Right and Right. We observe in early generations a negative
correlation between Move and On Nest ($-0.478$ in generation $10$), while in
later generations there is a trend towards becoming uncorrelated ($-0.0017$ in
generation $1600$). Nest Middle and Left are anti-correlated in early
generations ($-0.187$ in generation $10$) and positively correlated in later
generations ($0.135$ generation $1600$). The relationship between Visual Green
and Pheromone indicates a slight positive correlation in early generations
($0.136$ in generation $10$) but increases in later generations ($0.313$ in
generation $1600$). The fitness function punishes an extensive pheromone drop
which leads to a rather restrained usage. The input-output relationship is
changing continuously across the generations due to the evolutionary
algorithm, i.e. certain input-output relationships get enhanced or weakened.
Some of these mappings align with observations in nature, e.g. Visual Green
and Pheromone have an increasing correlation value across the generations,
which reflects the positive reinforcement when the ant perceives food. This is
also in agreement with the attractor effect of the pheromone and links to the
correlations between smelling pheromones and ant movement, serving as guiding
signals to take the ants back to the food patches.

Figure 8: Correlation coefficients of input and output spike trains for the
first ant of the best individual every ten generations. A linear regression is
calculated for the data (solid lines). A low p-value rejects the null-
hypothesis that the slope of the curve is zero. Only plots with a p-value
below $0.005$ are displayed. s is the slope of the curve and $\sigma$ the
standard deviation of the points.

Although the pheromone heatmap provides insight into ant behaviour for
specific generations, we wanted to see how the behaviour evolves over all
generations. In order to compare different input-to-output correlations and
validate our results obtained from the heatmap, we calculate the correlation
coefficients between input and output spike trains for the first ant of the
best individual every tenth generation. We then fit a linear regression on the
coefficients, which returns a p-value and the slope s; a low p-value rejects
the null-hypothesis that the slope of the curve is zero, i.e. there is not a
correlation. Additionally, we calculate the standard deviation $\sigma$ of all
points.

The visualization in Figure 8 enables us to determine trends in the ant
behavior over generations. For example, we observe that Visual Red is
initially positively correlated to Move but there is a trend towards anti-
correlation over the generations. While Visual Green and Move show an
increasing trend, Visual Red and Move become increasingly negatively
correlated over the generations. Likewise, Smell Right shows a trend towards
an increasing positive correlation with Right and a negative correlation with
Left. Smell Left is initially positively correlated to Move, but this
correlation decreases for later generations. In contrast, the correlation
between Smell Left and Nest Left increases, while it decreases for Nest Right.
The receptor for Smell Middle shows an increasing trend towards positive
correlation with Move.Nest Middle has an increasing positive correlation with
Left, while Nest Right has an increasing negative correlation with Left as the
colony evolves. Pheromone is initially dropped in the vicinity of the nest.
However, the pheromone output activity, when the ant is in the middle of the
nest, decreases over the generations. In contrast, Visual Green and Pheromone
show an increasing positive trend.

Observing the performance of the ant colony at the end of the evolutionary
process, pheromone paths seem to emerge and form attractors and trails with
higher pheromone concentration resulting from a combination of optimized
movement trends while sensing pheromone, the evaporation constant of the
pheromone, and the evolutionary pressure of the fitness metric towards the
pattern explore-get food-return food to nest. This resembles the actual
behaviour of ant colonies observed in nature.

##  4 Discussion and Outlook

In this work we discussed an implementation of a virtual environment where the
behaviour of swarms of agents controlled by SNNs could be optimized using
evolutionary algorithms. The implementation combines NEST simulations for the
SNNs, an environment described with NetLogo and an optimization with genetic
algorithms using the L2L framework. This implementation is open source (see
Supplementary Material) and can be used or extended to explore a variety of
scientific questions regarding swarm optimization as well as emergent
communication and organization. In particular, it could be of high interest to
study the coordinated emergent behaviour and communication between agents in
neurorobotics and cobotics applications, where it is essential to ensure
safety, optimal resource usage, and adaptation to dynamic environments.

We demonstrated that we could use genetic algorithms to successfully evolve
spiking neural networks to control agents (here, ants) to solve a multi-agent
food foraging task. Our approach shows emerging self coordination via
pheromone as a result of the network optimization, without defining action
rules or probabilistic state models. This work serves as a proof of concept to
illustrate multi-agent interactions with emergent communication and self-
coordination utilizing SNNs as underlying architectures. We provide a short
summary of our methods in the following:

  * •

Our SNN architecture encodes the inputs from the environment and steers the
behaviour of the ant when foraging for food.

  * •

Based on the fitness function, which evaluates the foraging performance, we
utilize a genetic algorithm to optimize the synaptic weights and delays. This
allows the network to achieve an efficient foraging behaviour.

  * •

Our analysis investigates the emergent relationship established by the SNN
between the inputs and outputs to trigger certain actions during the
evolutionary process.

By applying the methods mentioned above, we observed an emergence of
stigmergy, i.e. the communication and collaboration via pheromones. In our
evolved network we proved that communication became essential for the
efficient performance of the ants during the foraging task. Impairing the
pheromone sensing capabilities of the ants drastically hindered the
performance of the collective. This highlights the importance of stigmergic
communication, which emerged through the optimization process, for the
effective collaboration between the agents. In our experiments we showed that
agents using pheromones were able to quickly forage for food even when the the
position of the nest was changed. It is important to note that the release of
pheromone does not obey any pre-defined rule or manual synaptic pre-
configuration in the network. Instead, the stigmergy is an emergent behaviour
that is triggered under certain conditions established during the evolutionary
process.

Furthermore, we would like to highlight that the sensing of the pheromone is
not associated with any kind of pre-defined behavioural response in the
network. There is no hard-coded synaptic communication that explicitly maps
the sensory input of pheromone with a certain action. Instead, the observed
behaviour is completely based on emergent communication resulting from the
collective evolution of the colony. What we observe is that the optimization
operates directly on the structure (synapse weights) of the brain networks of
the ants, but there is also a higher level optimization in order to create a
functional network created by the collaborative behaviour of the agents. This
behaviour is achieved by the development of communication strategies between
the agents using the pheromone signals. In essence, the ants form a new
network where pheromone signals are used to coordinate the movement of the
agents as a whole to efficiently solve the task. We can see this emergent
functional effect as the correlations and anti-correlations between directed
movement and the sensing of pheromone increase with each generation (see
section 3). Based on Figure 7 it is possible to arrive at the conclusion that
the pheromone dropping is uncorrelated to actions and sensing of the
environment. However, optimized movement based on pheromone signals makes it
possible to have higher concentrations of pheromone around the areas of
interest and thus enhance the ability of the swarm to collectively solve the
task. We also confirmed the emergence of this functional network when we
disabled the pheromone receptor and the colony was not able to efficiently
solve the task anymore (see subsection 3.2).

Interestingly, the emergent communication via a simulated chemical signal
resembles the use of positive (attractor) pheromones observed in real ant
colonies. Although attractive trails of pheromone are more commonly found in
nature, it has been observed that negative (repelling) pheromones are also
used in certain species of ants (Robinson et al., 2005). This negative
pheromone also known as a “no-entry” signal is used by the ants to mark
unrewarding paths to prevent other cohorts from using them. It opens the
question of whether further experimentation simulating different evolutionary
trajectories could lead to the emergence of negative repelling pheromones
among other sophisticated coordination mechanisms in self-organized simulated
swarms. Another related question would be to study which environmental and
evolutionary requirements are necessary for different signals to emerge within
the swarm’s communication repertoire. The usage of the L2L framework allows
the modification of the outer loop optimization algorithm, opening the
possibility to explore the impact of different evolutionary optimization
features in the resulting communication patterns.

At the beginning of this work we formulated two research questions. The
results presented in the previous sections answer these. First, the usage of
pheromone as a mean of communication and self-organized coordination between
agents can emerge in the absence of predefined rules. Neither the mechanism to
trigger the release of pheromone nor the mechanism of interpreting these
signals was engineered as part of the behaviour of the agents. Through this we
can infer that the simulated embedded physiological properties of the agents
and the characteristics of the environment are exploited by the optimization
algorithm. To answer the second research question, the results in subsection
3.2 illustrate the difference in the performance of the colony in terms of
foraging efficiency. The evolved colony with stigmergic communication achieved
a higher performance than the multi-agent rule-based colony.

We would also like to address some of the main shortcomings of our work.
First, the genetic algorithm requires several hundreds of generations until it
provides a parameter setting which leads to a suitable task performance. We
want to investigate other optimization techniques which may decrease the
number of generations and result in a similar performance. Second, in this
work we use a manually defined network architecture for all individuals. This
limits the optimization process and the potential functional capabilities of
the ants. Future work optimizing the network architecture could help find new
strategies of the ants to use their physiology to efficiently perform the
foraging task. Third, even if we did not define any rule about the
relationships between input and output actions of the ants, the fitness
function used to optimize the network is static and manually defined. This
imposes evolutionary pressure on the ants towards specific solutions.
Exploring other combinations of the fitness metrics will help us, in future
experiments, understand better the strength of such pressure in the emergence
of specific behaviour and solutions by the colony. Finally, as discussed in
the results section, our analysis of the correlations between input and output
is also limited and further work needs to be done to fully understand the
relationships within the ant networks and the colony as a whole through time.

As future work we want to analyze further the emergence of the communication
features in the functional network created by the collective of ants and how
this evolves through time guided by the genetic algorithm. In particular, we
want to measure the effect of pheromone concentration and evaporation rate in
the communication strategy. A way to do this would be to represent the whole
colony as functional network where connectivity is defined by spatial
coincidence in the environment between one or more ants, signals are
represented by pheromone concentration and delays by the pheromone evaporation
rate. We also want to explore the usage of plasticity within the ant networks
and optimize the learning capabilities of the ants in the inner loop of L2L
and see if this capability could further enhance the performance of the swarm.
For example, we want to learn the foraging task utilizing synaptic plasticity
in the inner loop and optimize the model’s hyper-parameters in the outer loop.
In particular, the emergence of connectivity motifs, as emerging from the
usage of STDP in Ocker et al. (2015), could be further studied and correlated
to specific communication functions in the colony.

During pilot experiments (not shown in this manuscript) we observed that the
emergence of stigmergy could be influenced by the number of neurons in the
network of each agent. As future work we want to explore the minimal
structural requirements (number of neurons, topology, types of synapses, etc)
for the SNNs to enable the emergence of collective communication and its
effect on functional efficiency of the swarm as a whole.

The emergent foraging behaviour that has been described in this work resembles
to some extent the self-organisation observed in social insects. Self-
organisation is a process, where high order structure emerges (Kauffman et
al., 1993). The emergent behaviour allows the swarm to collectively adapt to
changing environments where the colonies become robust systems even if some
individuals fail to perform their tasks. Robustness and adaptability are
desired characteristics in the application of robotics, specially for the
deployment in changing environments with harsh or extreme conditions where
single agents are more prone to failure (e.g. space exploration, undersea
navigation, disaster areas).

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any
commercial or financial relationships that could be construed as a potential
conflict of interest.

## Author Contributions

All authors conceived of the project. CR and AY worked on the design of the
project. CR, AY and APM worked on the implementation. AY and CR produced the
results reported in the manuscript. All authors reviewed, contributed and
approved the final version of the manuscript

## Funding

The research leading to these results has received funding from the European
Union’s Horizon 2020 Framework Programme for Research and Innovation under the
Specific Grant Agreement No. 945539 (Human Brain Project SGA3). This research
has also been partially funded by the Helmholtz Association through the
Helmholtz Portfolio Theme ”Supercomputing and Modeling for the Human Brain”.
Open Access publication funded by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) - 491111487.

## Acknowledgments

CR and AY would like to thank Chiara Segala for her input and feedback
regarding the correlation analysis. We would like to thank Wouter Klijn and
Michael Herty for their ideas, support and useful input. We acknowledge the
use of Fenix Infrastructure resources, which are partially funded from the
European Union’s Horizon 2020 research and innovation programme through the
ICEI project under the grant agreement No. 800858. The authors gratefully
acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for
funding this project by providing computing time on the GCS Supercomputer
JUWELS at Jülich Supercomputing Centre (JSC).

## References

  * Wilson et al. [1967] Edward O Wilson, Frank M Carpenter, and William L Brown Jr.  The first mesozoic ants.  _Science_ , 157(3792):1038–1040, 1967.
  * Smith et al. [2008] Chris R Smith, Amy L Toth, Andrew V Suarez, and Gene E Robinson.  Genetic and genomic analyses of the division of labour in insect societies.  _Nature Reviews Genetics_ , 9(10):735–748, 2008\.
  * Wilson and Nowak [2014] Edward O Wilson and Martin A Nowak.  Natural selection drives the evolution of ant life cycles.  _Proceedings of the National Academy of Sciences_ , 111(35):12585–12590, 2014.
  * Boudinot et al. [2022] Brendon E Boudinot, Adrian Richter, Julian Katzke, Júlio CM Chaul, Roberto A Keller, Evan P Economo, Rolf Georg Beutel, and Shûhei Yamamoto.  Evidence for the evolution of eusociality in stem ants and a systematic revision of† gerontoformica (hymenoptera: Formicidae).  _Zoological Journal of the Linnean Society_ , 195(4):1355–1389, 2022.
  * Vittori et al. [2004] Karla Vittori, Jacques Gautrais, Aluizio FR Araújo, Vincent Fourcassié, and Guy Theraulaz.  Modeling ant behavior under a variable environment.  In _International Workshop on Ant Colony Optimization and Swarm Intelligence_ , pages 190–201. Springer, 2004.
  * Bandeira de Melo and Araújo [2008] Elton Bernardo Bandeira de Melo and Aluízio Fausto Ribeiro Araújo.  Modeling ant colony foraging in dynamic and confined environment.  In _Proceedings of the 10th annual conference on Genetic and evolutionary computation_ , pages 169–176, 2008.
  * Hecker and Moses [2015] Joshua P Hecker and Melanie E Moses.  Beyond pheromones: evolving error-tolerant, flexible, and scalable ant-inspired robot swarms.  _Swarm Intelligence_ , 9(1):43–70, 2015.
  * Wilensky [1997] Uri Wilensky.  Netlogo ants model.  _Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL_ , 1997.
  * Duan and Sun [2014] Haibin Duan and Changhao Sun.  Swarm intelligence inspired shills and the evolution of cooperation.  _Scientific reports_ , 4(1):1–8, 2014.
  * Chevallier et al. [2010] Sylvain Chevallier, Hélène Paugam-Moisy, and Michèle Sebag.  Spikeants, a spiking neuron network modelling the emergence of organization in a complex system.  In _NIPS’2010_ , pages 379–387, 2010.
  * Beyeler et al. [2015] Michael Beyeler, Nicolas Oros, Nikil Dutt, and Jeffrey L Krichmar.  A gpu-accelerated cortical neural network model for visually guided robot navigation.  _Neural Networks_ , 72:75–87, 2015.
  * Nichols et al. [2010] Eric Nichols, LJ McDaid, and NH Siddique.  Case study on a self-organizing spiking neural network for robot navigation.  _International Journal of Neural Systems_ , 20(06):501–508, 2010.
  * Nichols et al. [2012] Eric Nichols, Liam J McDaid, and Nazmul Siddique.  Biologically inspired snn for robot control.  _IEEE transactions on cybernetics_ , 43(1):115–128, 2012.
  * Yegenoglu et al. [2022] Alper Yegenoglu, Anand Subramoney, Thorsten Hater, Cristian Jimenez-Romero, Wouter Klijn, Aarón Pérez Martín, Michiel van der Vlag, Michael Herty, Abigail Morrison, and Sandra Diaz-Pier.  Exploring parameter and hyper-parameter spaces of neuroscience models on high performance computers with learning to learn.  _Frontiers in Computational Neuroscience_ , 16, 2022.  ISSN 1662-5188.  doi: 10.3389/fncom.2022.885207.  URL https://www.frontiersin.org/articles/10.3389/fncom.2022.885207.
  * Tisue and Wilensky [2004] Seth Tisue and Uri Wilensky.  Netlogo: A simple environment for modeling complexity.  In _International conference on complex systems_ , volume 21, pages 16–21. Boston, MA, 2004.
  * Gewaltig and Diesmann [2007] Marc-Oliver Gewaltig and Markus Diesmann.  Nest (neural simulation tool).  _Scholarpedia_ , 2(4):1430, 2007.
  * Fortin et al. [2012] Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Marc Parizeau, and Christian Gagné.  DEAP: Evolutionary algorithms made easy.  _Journal of Machine Learning Research_ , 13:2171–2175, jul 2012.
  * Jordan et al. [2018] Jakob Jordan, Tammo Ippen, Moritz Helias, Itaru Kitayama, Mitsuhisa Sato, Jun Igarashi, Markus Diesmann, and Susanne Kunkel.  Extremely scalable spiking neuronal network simulation code: from laptops to exascale computers.  _Frontiers in neuroinformatics_ , 12:2, 2018.
  * Denker et al. [2018] M. Denker, A. Yegenoglu, and S. Grün.  Collaborative HPC-enabled workflows on the HBP Collaboratory using the Elephant framework.  In _Neuroinformatics 2018_ , page P19, 2018.  doi: 10.12751/incf.ni2018.0019.  URL https://abstracts.g-node.org/conference/NI2018/abstracts#/uuid/023bec4e-0c35-4563-81ce-2c6fac282abd.
  * Robinson et al. [2005] Elva Robinson, Duncan Jackson, M. Holcombe, and Francis Ratnieks.  Insect communication: ‘no entry’ signal in ant foraging.  _Nature_ , 438:442, 12 2005.  doi: 10.1038/438442a.
  * Ocker et al. [2015] Gabriel Koch Ocker, Ashok Litwin-Kumar, and Brent Doiron.  Self-organization of microcircuits in networks of spiking neurons with plastic synapses.  _PLoS computational biology_ , 11(8):e1004458, 2015.
  * Kauffman et al. [1993] Stuart A Kauffman et al.  _The origins of order: Self-organization and selection in evolution_.  Oxford University Press, USA, 1993.