#  Eco-evolutionary Dynamics of Non-episodic Neuroevolution in Large Multi-
agent Environments

Gautier Hamon<EMAIL_ADDRESS>Flowers Team, Inria / Ensta
ParisTechBordeauxFrance ,  Eleni Nisioti<EMAIL_ADDRESS>Flowers Team,
Inria / Ensta ParisTechBordeauxFrance and  Clément Moulin-Frier
<EMAIL_ADDRESS>Flowers Team, Inria / Ensta
ParisTechBordeauxFrance

(2023)

###### Abstract.

Neuroevolution (NE) has recently proven a competitive alternative to learning
by gradient descent in reinforcement learning tasks. However, the majority of
NE methods and associated simulation environments differ crucially from
biological evolution: the environment is reset to initial conditions at the
end of each generation, whereas natural environments are continuously modified
by their inhabitants; agents reproduce based on their ability to maximize
rewards within a population, while biological organisms reproduce and die
based on internal physiological variables that depend on their resource
consumption; simulation environments are primarily single-agent while the
biological world is inherently multi-agent and evolves alongside the
population. In this work we present a method for continuously evolving
adaptive agents without any environment or population reset. The environment
is a large grid world with complex spatiotemporal resource generation,
containing many agents that are each controlled by an evolvable recurrent
neural network and locally reproduce based on their internal physiology. The
entire system is implemented in JAX, allowing very fast simulation on a GPU.
We show that NE can operate in an ecologically-valid non-episodic multi-agent
setting, finding sustainable collective foraging strategies in the presence of
a complex interplay between ecological and evolutionary dynamics.

††journalyear: 2023††copyright: rightsretained††conference: Genetic and
Evolutionary Computation Conference Companion; July 15–19, 2023; Lisbon,
Portugal††booktitle: Genetic and Evolutionary Computation Conference Companion
(GECCO ’23 Companion), July 15–19, 2023, Lisbon, Portugal††doi:
10.1145/3583133.3590703††isbn: 979-8-4007-0120-7/23/07††ccs: Computing
methodologies Artificial life Figure 1. Our simulation environment (Left) is
an extension of the Common Pool Resource (CPR) environment (Perolat et al.,
2017; Leibo et al., 2017) : a two-dimensional grid-world where some cells
contain resources (in green) that the agents (in black) can collect. Resources
grow depending on the presence of other resources around them (local growth,
Middle) with an additional very sparse spontaneous growth, which means that
over-consumption may lead to their local depletion. We introduce a latitudinal
model of resource regrowth similar to (Nisioti and Moulin-Frier, 2022) with
higher resource regrowth rate in lower latitudes (bottom of the map) than in
higher ones. We consider a very large environment of $200\times 400$ pixel
that can contain thousands of agents (in black). We prevent any environment
and population reset during a whole simulation of $1.000.000$ time steps,
enabling continual eco-evolutionary dynamics to take place. Each agent may
reproduce or die according to a physiological model modulating its energy
level as a function of life time and resource consumption (Top-Right). Agents
reproduce according to a minimal criterion (Brant and Stanley, 2017) of
maintaining energy level above a certain threshold for a certain period of
time. The population size therefore varies during the simulation according to
the current amount of available resources and the current ability of agents to
collect them. Each agent is controlled by a recurrent artificial neural
network which takes as input limited local observations and outputs the
navigation action (Bottom-Right). Evolution occurs through the mutation of a
parent’s network weights when it produces an offspring. We refer to appendix B
for more details on the environment design.

##  1\. Introduction

The main objective of this paper is to propose a method for studying large-
scale eco-evolutionary dynamics in agent-based simulations with a reasonable
level of biological and ecological plausibility. For this aim, we implement a
system with the following properties (see Fig. 1 for illustration).

##### Non-episodic simulation environment with complex intrinsic dynamics

. We model our environment after common-pool resource (CPR) appropriation
problems, where a group of agents competes for finite resources. We extend an
existing environment of CPR appropriation (Perolat et al., 2017) with the
presence of multiple niches, where resources regrow proportionally to the
density of nearby resources at different rates in different regions of the
environment (Fig 1). We prevent any environment or population reset during a
whole simulation run, enabling coupled environmental and population dynamics
leading to complex eco-evolutionary feedback effects.

##### Continuous neuroevolution in a large, size-varying agent population

The environment contains thousands of agents, each controlled by a neural
network whose weights are optimized using neuroevolution (Stanley et al.,
2019). Each network contains a memory component (LSTM), which enables
adaptation within the agent’s lifetime in the absence of weight updates. Thus
the evolutionary process can be viewed as an outer loop that optimizes the
ability of agents to adapt to different environmental conditions.

##### Physiology-driven death and reproduction

There is no notion of rewards, agents are instead equipped with a
physiological system modulating their energy level according to the resources
they consume, in a non-linear way. At the evolutionary scale, agents reproduce
as long as they are able to maintain their energy level within a reasonable
range and die if this level goes below a minimum threshold. This is departure
from the notion of fitness-based selection and more in line with a minimal
criterion selection (Brant and Stanley, 2017). Note that the population size
can vary with time.

##### Natural vs. Lab environments: Evaluation methodology

As we are interested in the system’s ability to emerge interesting behaviors
that hint to open-ended dynamics, evaluating it on pre-defined set of tasks
would defeat our purpose. For this reason we have structured our simulation
methodology as follows: we let the population of agents evolve for a long time
in a single environment and study its behavior at a large global scale and at
a smaller local scale. At the large scale, we study the dynamics of the system
in what we call the ”natural environment”, i.e. the full simulation run, by
monitoring population-wide and terrain-wide metrics. At the small scale, we
first focus on local, interesting patterns of behaviors observed in the
natural environment, such as individual agents that move in a consistent way
or collective immigration and foraging patterns. We then form specific
hypotheses about the potential drives of these behaviors and evaluate selected
agents in specific ”lab environments” that enable testing these hypotheses.
These environments differ from the one used for evolving behaviors: they are
much smaller and exhibit vastly different population and resource dynamics (we
illustrate examples of such environments in Figure 2.E).

From the perspective of neuroevolution, our empirical study aims at answering
the following questions: a) can we realistically apply neuroevolution in
multi-agent environments with thousands of agents? b) does a selection
mechanism that allows agents to reproduce locally, without requiring
generational resets, based on a minimal criterion suffice? c) does evolving
networks in a multi-agent setting lead to the emergence of adaptation
mechanisms? From the perspective of multi-agent cooperation, our study targets
the questions: a) can we simulate systems with complex eco-evo dynamics where
populations solving a CPR problem exhibit realistic behaviors? b) does
evolving under a minimal criterion enable sustainability? In the next section
we answer these questions in the affirmative.

Leveraging the GPU parallelization allowed by the JAX programming framework
(Bradbury et al., 2018), we run large-scale continual simulations in grid-
world environments with approximately $100K$ cells and thousands of agents
(notably a simulation of $1M$ time steps with such a population requires about
only 20 minutes).

More detailed background positioning our contributions with respect to the
existing literature, as well the computational details of our proposed method,
are provided in the Appendix.

##  2\. Results

We will now study the evolution of a population in our proposed system and
probe certain quantities during evolution. Note that this system required some
tuning of the hyperparameters in order to find a stable environment, as
exponential growth of both food and population can easily lead to collapse
(and even did after several generations in 3 out of the 5 seeds launched). We
will make a detailed analysis of one seed and refer to Appendix D.2 for an
analysis of another one with a different eco-evolutionary path. We provide
videos that show the real-time behavior of our system in a companion website
(https://sites.google.com/view/non-episodic-neuroevolution-in/) as well as a
repository containing code for reproducing our experiments
(https://github.com/flowersteam/EcoEvoJax).

Details on the environment and hyperparameters characterizing the natural
environment can be found in Appendix B and an explanation of how the metrics
have been implemented and how statistical significance was tested for in
Appendix C.

###  2.1. Eco-evolutionary dynamics

In this simulation, the evolution of the population is deeply interconnected
with the evolution of resources. In Fig 2.A.2, we observe that at a small
scale the population size (blue) and resources (green) present in the
environment follow a predator-prey Lokta-Volterra dynamic (Volterra, 1926).
Those oscillation are interesting from an evolutionary perspective: easier
phases with higher resources availability, in which the population grows and
where diversity can emerge, are followed by high competition phases due to an
increase in population and decrease in resources.

####  2.1.1. Coexistence of agents with different movement dynamics

Figure 2. A) 1) Amount of resources in the environment over time and 2) zoom
on a smaller timescale showing the interplay with population size (blue) B)
Percentage of individuals with different amount of movement over time C)
Greediness of a sustainable forager agent across evaluation environments that
differ in the amount of resources. D) Average efficiency across the population
in high resources task with reproduction activated and deactivated. Activating
reproduction leads to increased resource consumption.

At the beginning of evolution (steps $0-200K$, starting with random agents),
the environment has abundant resources which leads to to high-moving behaviors
as an easy first strategy in this high-resource environment (Fig 2.A,B). Then,
when the amount of resource decreases, we observe an increase in the number of
low-moving individuals (Fig 2.A,B) exploiting local resource spots (from step
$200K$). From this point, those two extreme strategies coexist in the agent
population (Fig 2.B). This differs from previous related work in a similar
environment (Aubert-Kato et al., 2015), relying on a simpler agent
architecture and a fitness-based reproduction condition, where only one
strategy ended up populating the whole environment. Those extreme behaviors
correspond to two distinct types of agents: high movement individuals are
agents that have an ”opportunistic traveler” strategy as they travel mostly in
straight line but opportunistically exploit resources spots locally
(especially from isolated resources from the sparse spontaneous growth) as
soon as they see them. On the other hand, the low movement individuals exploit
the spreading of resources by staying at the same interesting place (with
resources around) and waiting for resources to spread. We qualify this waiting
of resources as a sustainable strategy as agents do not consume resources
greedily but rather keep these resources as a reliable source of respawn for
more long term survival (for themselves but also for their offspring that will
inherit this place). We refer to video 1.a of the companion website for a
visualization of these behaviors and to the next subsection for a more
detailed and controlled analysis of the behavior (and diversity) of agents.

###  2.2. Evaluation in lab environments

How do the agents adapt their foraging behavior at an evolutionary and intra-
life timescale to maximize their reproduction rate? In the natural environment
we saw that both population size and the spontaneous regrowth of resources may
contribute to avoiding resource depletion. At an evolutionary scale the
population may adapt by regulating its size and updating its weights. But is
it possible that the agents learned to adapt to different conditions they
encounter in their lifetime in order to forage both efficiently and
sustainably? This is the question the following simulations in the lab
environments aim to address.

####  2.2.1. Does the density of resources affect agents’ greediness?

##### Set-up

There are three lab environments, with a single agent that cannot reproduce
and resource regeneration deactivated, that differ in the amount of initial
resources (see on Figure 2.E for an illustration of the low and high resources
environments). In each of these lab environment, we measure the amount of
greediness $G$, by dividing the simulation into non-overlapping fixed windows
of 20 timesteps and checking in which of these windows the agent has at least
one resource in its field of view (let’s denote this number with $T_{r}$) and
the number of these windows during which the agent consumed at least one
resource (let’s denote this number with $C_{r}$), so that $G=C_{r}/T_{r}$ . We
compute this measure on randomly sampled evolved agents from the end of the
natural environment simulation. To quantify the effect of the density of
resources we perform statistical tests comparing the greediness of each agent
in the three tasks. More information can be found in Appendix C.

Our analysis showed that agents exhibit different qualitative behaviors that
can be grouped in two types: a) agents for which no statistically significant
differences appear between tasks. These agents correspond to the opportunistic
travelers that we encountered in the natural environment and do not exhibit
resource-dependent adaptation b) agents for which there are statistically
significant differences between the low-resources and high-resources
environment, with greediness in low-resource environments being higher.
Overall, 9 out of the 50 agents exhibited this behavior (we illustrate
greediness across tasks for one of these agents in Figure 2.C), which we refer
to as sustainable foragers. These agents have learned to not over-consume
resources when these are abundant, but stay close to them to consume them
later and take advantage of the higher spread rate. On the other hand, low
resources environment means slower spread (and may mean lattitude with lower
regrow) which might exlain why even those sustainable agent prefer to take the
resource and leave.

####  2.2.2. Does peer-pressure lead to greediness?

##### Set-up

We use the high resources task but now allow agents to reproduce. This means
that, after $T_{repr}=20$ timesteps, new agents will appear, leading to
competition for resources. Our hypothesis is that this will make agents more
greedy. To test this, we measure efficiency $E$ as the average amount of
resources every individual consumes during the evaluation trial and average it
across 50 agents and 10 trials. We then compare the difference in performance
between the previous set-up (no-reproduction) with the current one, where we
average across 50 agents and 10 trials to observe whether there is a
population-wide effect.

As Figure 2.D illustrates, we observed a large change in the foraging
efficiency of the agents when reproduction was on. Efficiency increased by a
statistically significant amount, which indicates the sustainable foragers
increased their greediness under peer pressure. However, we observed that,
after an initial increase in resource consumption at the appearance of new
agents, the group slows down again and its members tend to disperse and stay
close to resources without consuming them (see companion website.B.1 for an
illustration of this behavior).

##  3\. Discussion

Our empirical study demonstrates that neuroevolution can operate in large
multi-agent environments, lead to efficient behaviors even in the absence of
episodic survival-of-the-fittest and help evolve agents that exhibits
adaptation within their lifetime without requiring weight updates.
Specifically in regards to the latter, we identified agents that change their
policy depending on resource density and presence of other agents. From an
ecological perspective, our computational study proves that agents selected
based on a minimal criterion learn sustainable behaviors and that the
population exhibits dynamics that resemble those of natural populations, such
as population size oscillations. We observed many interesting emerging
examples of collective and individual adaptation, including:  i) Population
size exhibits bursts and crashes that are correlated with the density of
resources,  ii) The system goes through phases related to the sustainability
of the agents’ foraging behavior: resources and population size initially grow
until over-population leads to near-extinction of resources which creates a
drive for agents to forage sustainably,  iii) The sustainable population
exhibits diversity in individual behaviors: some agents specialize in long-
distance travel, opportunistically consuming resources they find on their way,
while others forage locally, staying close to resources to take advantage of
the spread of resources and consuming sporadically to avoid death,  iv)
Agents’ influence each others behavior: agents that forage sustainably when
alone, temporarily increase their consumption when others enter their field of
view and then revert back to consuming less.

Interestingly, points i) and ii) above could not be observed in a standard
episodic training paradigm, where environment and population resets would
prevent any eco-evolutionary feedback. In this respect, we are considering
future experiments studying whether continual local reproduction, where
offsprings are produced next to their parent, did enable some sort of kin
selection –e.g. in the form of reducing parent’s greediness as a way to favor
the survival of their offsprings. Other future work could also focus on
studying to what extent the memory component of the agent’s cognitive
architecture contributes to intra-life adaptation.

In the past ecologists have hinted to the limitations of an anthropocentric
view on intelligence (de Waal, 2016): if we search for intelligence by looking
at performance metrics only in tasks that we excel at, then we will inevitably
miss a big part of the natural kingdom. Our study hints to a similar
conclusion for artificial agents: evolving agents in natural environments with
complex spatiotemporal dynamics in the absence of rewards and examining their
behavior in toy lab environments may bring us closer to our quest for open-end
behavior in artificial systems.

## References

  * (1)
  * Anastassacos et al. (2021) Nicolas Anastassacos, Julian García, Stephen Hailes, and Mirco Musolesi. 2021.  Cooperation and Reputation Dynamics with Reinforcement Learning.  _arXiv:2102.07523 [cs]_ (Feb. 2021).  arXiv:2102.07523 [cs]
  * Aubert-Kato et al. (2015) Nathanaël Aubert-Kato, Olaf Witkowski, and Takashi Ikegami. 2015\.  The Hunger Games: Embodied agents evolving foraging strategies on the frugal-greedy spectrum, Vol. ECAL 2015: the 13th European Conference on Artificial Life. 357–364.  https://doi.org/10.1162/978-0-262-33027-5-ch065
  * Bradbury et al. (2018) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018\.  _JAX: composable transformations of Python+NumPy programs_.  http://github.com/google/jax
  * Brant and Stanley (2017) Jonathan C. Brant and Kenneth O. Stanley. 2017.  Minimal Criterion Coevolution: A New Approach to Open-Ended Search. In _Proceedings of the Genetic and Evolutionary Computation Conference_ _(GECCO ’17)_. 67–74.
  * Brant and Stanley (2020) Jonathan C. Brant and Kenneth O. Stanley. 2020.  Diversity preservation in minimal criterion coevolution through resource limitation. In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ _(GECCO ’20)_. Association for Computing Machinery, New York, NY, USA, 58–66.  https://doi.org/10.1145/3377930.3389809
  * Chollet (2019) François Chollet. 2019\.  On the Measure of Intelligence.  https://doi.org/10.48550/arXiv.1911.01547 arXiv:1911.01547 [cs].
  * Clune (2020) Jeff Clune. 2020\.  AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence.  _arXiv:1905.10985 [cs]_ (Jan. 2020).  http://arxiv.org/abs/1905.10985 arXiv: 1905.10985.
  * Co-Reyes et al. (2020) John D. Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine. 2020.  Ecological Reinforcement Learning.  _arXiv:2006.12478 [cs, stat]_ (June 2020).  http://arxiv.org/abs/2006.12478 arXiv: 2006.12478.
  * de Waal (2016) Frans de Waal. 2016\.  _Are We Smart Enough to Know How Smart Animals Are?_ W W Norton & Co, New York, NY, US. 340 pages.
  * Grove (2014) Matt Grove. 2014\.  Evolution and Dispersal under Climatic Instability: A Simple Evolutionary Algorithm.  _Adaptive Behavior_ 22, 4 (Aug. 2014), 235–254.  https://doi.org/10.1177/1059712314533573
  * Gupta et al. (2021) Abhishek Gupta, Justin Yu, Tony Z. Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. 2021\.  Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention. In _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE Press, Xi'an, China, 6664–6671.  https://doi.org/10.1109/ICRA48506.2021.9561384
  * Hardin (1968) Garrett Hardin. 1968\.  The Tragedy of the Commons.  _Science_ 162, 3859 (1968), 1243–1248.
  * Janssen (2010) Marco A. Janssen. 2010\.  Introducing Ecological Dynamics into Common-Pool Resource Experiments.  _Ecology and Society_ 15, 2 (2010), art7.  https://doi.org/10.5751/ES-03296-150207
  * Laland et al. (2015) Kevin N. Laland, Tobias Uller, Marcus W. Feldman, Kim Sterelny, Gerd B. Müller, Armin Moczek, Eva Jablonka, and John Odling-Smee. 2015\.  The extended evolutionary synthesis: its structure, assumptions and predictions.  _Proceedings of the Royal Society B: Biological Sciences_ 282, 1813 (Aug. 2015), 20151019.  https://doi.org/10.1098/rspb.2015.1019 Publisher: Royal Society.
  * Leibo et al. (2019) Joel Z Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel. 2019.  Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research.  _arXiv preprint arXiv:1903.00742_ (2019).
  * Leibo et al. (2017) Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017.  Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. In _Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems_ (São Paulo, Brazil) _(AAMAS ’17)_. 464–473.
  * Liu et al. (2023) Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen, and Kay Chen Tan. 2023\.  A Survey on Evolutionary Neural Architecture Search.  _IEEE Transactions on Neural Networks and Learning Systems_ 34, 2 (Feb. 2023), 550–570.  https://doi.org/10.1109/TNNLS.2021.3100554
  * Marcus and Davis (2019) Gary Marcus and Ernest Davis. 2019.  _Rebooting AI: building artificial intelligence we can trust_ (first edition ed.).  Pantheon Books, New York.
  * Miikkulainen et al. (2012) Risto Miikkulainen, Eliana Feasley, Leif Johnson, Igor Karpov, Padmini Rajagopalan, Aditya Rawal, and Wesley Tansey. 2012.  Multiagent Learning through Neuroevolution.  In _Advances in Computational Intelligence_ , Jing Liu, Cesare Alippi, Bernadette Bouchon-Meunier, Garrison W. Greenwood, and Hussein A. Abbass (Eds.). Vol. 7311. Springer Berlin Heidelberg, Berlin, Heidelberg, 24–46.  https://doi.org/10.1007/978-3-642-30687-7_2
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis, Others, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015.  Human-level control through deep reinforcement learning.  _Nature_ 518, 7540 (Feb. 2015), 529–533.  https://doi.org/10.1038/nature14236 Publisher: Nature Publishing Group, a division of Macmillan Publishers Limited. All Rights Reserved..
  * Nisioti and Moulin-Frier (2022) Eleni Nisioti and Clément Moulin-Frier. 2022.  Plasticity and Evolvability under Environmental Variability: The Joint Role of Fitness-Based Selection and Niche-Limited Competition _(GECCO ’22)_. 113–121.
  * Pardo et al. (2018) Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. 2018.  Time Limits in Reinforcement Learning. In _Proceedings of the 35th International Conference on Machine Learning_. PMLR, 4045–4054.  https://proceedings.mlr.press/v80/pardo18a.html ISSN: 2640-3498.
  * Parker (2000) G A Parker. 2000\.  Scramble in Behaviour and Ecology.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 355, 1403 (Nov. 2000), 1637–1645.
  * Perolat et al. (2017) Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. 2017.  A multi-agent reinforcement learning model of common-pool resource appropriation.  _Advances in neural information processing systems_ 30 (2017).
  * Pigliucci and Muller (2010) Massimo Pigliucci and Gerd B Muller. 2010.  Evolution–the extended synthesis.  (2010).
  * Portelas et al. (2020) Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. 2020.  Automatic curriculum learning for deep RL: A short survey.  _arXiv:2003.04664 [cs, stat]_ (2020).  http://arxiv.org/abs/2003.04664
  * Post and Palkovacs (2009) David M. Post and Eric P. Palkovacs. 2009.  Eco-evolutionary feedbacks in community and ecosystem ecology: interactions between the ecological theatre and the evolutionary play.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 364, 1523 (2009), 1629–1640.  https://doi.org/10.1098/rstb.2009.0012
  * Risi et al. (2010) Sebastian Risi, Charles E Hughes, and Kenneth O Stanley. 2010\.  Evolving Plastic Neural Networks with Novelty Search.  _Adaptive Behavior_ 18, 6 (Dec. 2010), 470–491.  https://doi.org/10.1177/1059712310379923
  * Salimans et al. (2017) Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017.  Evolution Strategies as a Scalable Alternative to Reinforcement Learning.  https://doi.org/10.48550/arXiv.1703.03864 arXiv:1703.03864 [cs, stat]
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. 2017.  Mastering the game of Go without human knowledge.  _Nature_ 550, 7676 (Oct. 2017), 354–359.  https://doi.org/10.1038/nature24270 Publisher: Nature Publishing Group.
  * Silver et al. (2021) David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. 2021.  Reward is enough.  _Artificial Intelligence_ 299 (Oct. 2021), 103535\.  https://doi.org/10.1016/j.artint.2021.103535
  * Stanley et al. (2019) Kenneth O. Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen. 2019.  Designing Neural Networks through Neuroevolution.  _Nature Machine Intelligence_ 1, 1 (Jan. 2019), 24–35.  https://doi.org/10.1038/s42256-018-0006-z
  * Stanley and Lehman (2015) Kenneth O. Stanley and Joel Lehman. 2015.  _Why Greatness Cannot Be Planned: The Myth of the Objective_.  Springer Publishing Company, Incorporated.
  * Sterling (2012) Peter Sterling. 2012\.  Allostasis: A model of predictive regulation.  _Physiology and Behavior_ 106, 1 (2012), 5–15.  https://doi.org/10.1016/j.physbeh.2011.06.004 ISBN: 1873-507X (Electronic)$\$n0031-9384 (Linking).
  * Such et al. (2018) Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. 2018.  Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning.  https://doi.org/10.48550/arXiv.1712.06567 arXiv:1712.06567 [cs]
  * Volterra (1926) Vito Volterra. 1926\.  Fluctuations in the Abundance of a Species considered Mathematically.  _Nature_ 118 (1926), 558–560.
  * Wang et al. (2019) Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. 2019\.  Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions.  _arXiv:1901.01753 [cs]_ (2019).  http://arxiv.org/abs/1901.01753 arXiv: 1901.01753.
  * Woolley and Stanley (2011) Brian G. Woolley and Kenneth O. Stanley. 2011.  On the Deleterious Effects of a Priori Objectives on Evolution and Representation. In _Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation_ _(GECCO ’11)_. Association for Computing Machinery, New York, NY, USA, 957–964.  https://doi.org/10.1145/2001576.2001707

##  Appendix A Background

There are striking differences in how adaptation operates in biological versus
artificial systems. In Artificial Intelligence (AI), the most common approach
is _performance-driven_. The main assumption is that intelligence must be
implemented in a structured cognitive architecture (integrating e.g. control,
learning and memory mechanisms) which is optimized (using machine learning
methods) through pre-defined objective functions (Chollet, 2019; Marcus and
Davis, 2019; Silver et al., 2021). The proposed methods are evaluated in
benchmarks designed to capture various aspects of intelligence. For example,
Chollet (2019) defines intelligence as a measure of its skill-acquisition
efficiency over a scope of tasks, with respect to priors, experience, and
generalization difficulty and proposes a benchmark to evaluate it inspired by
psychometric intelligence tests and called the The Abstraction and Reasoning
Corpus (ARC). The rise of deep neural networks as powerful function
approximators has strongly revived this approach by allowing key advances in
e.g. representation learning and reinforcement learning in high-dimensional
spaces (Mnih et al., 2015).

In contrast, biological adaptation seems to be better characterized by the
notion of open-endedness (the continual generation of increasingly diverse
organisms) than by the notion of performance. While the popular concept of the
_survival of the fittest_ suggests that biological evolution is driven by
performance, this concept has actually little grounding in evolutionary theory
(Stanley and Lehman, 2015). An important paradigm shift is taking increasing
importance in evolutionary biology, recognizing the crucial role of eco-
evolutionary feedbacks as a main driver of evolution. The _extended
evolutionary synthesis_ (Pigliucci and Muller, 2010; Laland et al., 2015)
considers that, in the standard evolution theory as well as its modern
synthesis, too much causal significance is afforded to genes and selection,
and not enough to the developmental processes that create novel variants,
contribute to heredity, generate adaptive fit, and thereby direct the course
of evolution. It recognizes important feedback effects in terms of
_constructive development_ , i.e. the ability of an organism to shape its own
developmental trajectory by constantly responding to, and altering, internal
and external states, as well as of _reciprocal causation_ , i.e. that
developing organisms are not solely products but, by modifying their niche and
therefore its associated fitness landscape, are also causes of evolution.

Following a similar paradigm shift, a recent trend in AI is increasingly
recognizing the importance of reciprocal influence between adaptation and
environmental dynamics (Clune, 2020; Stanley and Lehman, 2015; Leibo et al.,
2019). This approach, that we can qualify as _complexity-driven_ (by
opposition to the _performance-driven_ approach mentioned above), considers
intelligence as the emergent product of adaptive systems interacting with
complex environmental dynamics. There are two main propositions here. Some
contributions study how competition and cooperation pressures in populations
of co-adapting agents can result in a behavioral arms race where each agent
has to continuously improve its skills against those of other agents, an
approach called _multi-agent autocurriculum_ (Leibo et al., 2019). Other
contributions study how learning algorithms themselves can be meta-learned for
operating in a diversity of environments. Clune (2020) calls this approach
_AI-Generating Algorithms (AI-GA)_ , with three main pillars: (1) meta-
learning architectures, (2) meta-learning the learning algorithms themselves,
and (3) generating effective learning environments. In both propositions
(autocurriculum and AI-GA), it is the complexity of the environment (either
through the presence of other co-adapting agents or through its intrinsic
diversity) that drives the ability to continuously acquire new skills and
generalize them in novel environments. In other words, if the performance-
driven approach attempts to reverse-engineer the brain (or at least its main
functions), the complexity-driven approach instead attempts to reverse-
engineer the environmental conditions that lead to intelligence.

Both performance-driven and complexity-driven approaches, however, still
differ crucially from how adaptation occurs in the natural world. Even
complexity-driven approaches are almost exclusively evaluated in terms of
their convergence toward efficient policies on evaluation tasks (e.g. their
performance in board games against humans (Silver et al., 2017), or in
generalization to novel test environments (Wang et al., 2019)). This is
especially due to the fact that they mostly rely on the reinforcement learning
(RL) framework. A central assumption in RL is that the objective of an
artificial agent must be to learn a task (or a set of tasks), that these tasks
should be defined as reward functions, and that those rewards are provided by
the environment. From a biological perspective, however, the environment does
not contain any reward whatsoever. Rewards instead, if at all, result from the
agent’s own physiology and self-regulation and have emerged from evolution as
a way to guide learning and exploration (see e.g. the metaphor of evolved
stick/carrot mechanisms in (Sterling, 2012)). Second, the standard training
paradigm in RL is episodic, i.e. the environment is regularly reset to its
initial conditions. While this procedure has the benefit to facilitate
training from a machine learning perspective (Pardo et al., 2018), it strongly
differs from natural settings where environments are persistent, i.e. where
the behavior of agents affects the environment in which the next generations
will further evolve and learn (see however (Co-Reyes et al., 2020) for a
recent attempt at studying ecologically valid non-episodic RL settings).
Episodic training in RL prevents the study of both niche construction and eco-
evolutionary feedback effects, which require that populations alter their
environment and that those changes in the environment influence the subsequent
evolution of the population (Post and Palkovacs, 2009). Contributions in
automatic curriculum learning (Portelas et al., 2020) studies such feedback
effects but focuses on how to adaptively sample novel environments of
increasing complexity, using episodic training, with the explicit objective to
improve an agent’s learning performance.

###  A.1. Neuroevolution

Neuroevolution draws inspiration from natural evolution to create agents that
learn to adapt through an evolutionary process rather than gradient-based
optimization (Stanley et al., 2019). In a surprise to many, this simple
process of selection and random mutations has recently performed competitively
with the state-of-the-art art in RL for playing Atari games (Salimans et al.,
2017; Such et al., 2018), and proven powerful in applications such as
architecture search, where the non-differentiable nature of the search space
prohibits gradient-based methods (Liu et al., 2023) and meta-learning, where
the evolutionary process is conceived as an outer optimization loops that
controls the intra-life learning plasticity of agents (Risi et al., 2010).
Multi-agent environments, which are particularly promising for neuroevolution
as they naturally entail the concept of a population, have been identified as
a frontier for this family of methods (Miikkulainen et al., 2012), arguably
due to their computational complexity and challenging multi-agent learning
dynamics .

Neuroevolution methods are classically performance-driven: solutions are
selected based on their ability to solve a pre-determined task. Complexity-
driven approaches, on the other hand, where solutions are chosen based on
criteria not directly related to performance, such as novelty, have proven
powerful in tasks for which the objective function is unknown to humans
(Woolley and Stanley, 2011). For a given criteria, neuroevolution methods can
also differ on whether solutions survive only if they are ranked high within
the population (survival of the fittest) or if their fitness is above a
threshold (minimum criterion). The latter category is the least explored
(Brant and Stanley, 2017), but has the potential of preserving a larger
phenotypic diversity within the population and is believed to be closer to
biological evolution.

Finally, neuroevolution methods almost exclusively consider discrete, over-
lapping generations, at the beginning of which solutions experience mutation
and selection simultaneously and the environment is reset to its initial
conditions. We refer to this paradigm as episodic, borrowing terminology from
RL, where recently it has been proposed to remove environmental resets, as
they may introduce the need for human supervision (Gupta et al., 2021) and are
implausible from a biological perspective (Co-Reyes et al., 2020). This
setting, termed as non-episodic or continuous in RL, is harder to envision in
evolution under survival-of-the-fittest, where dividing time into non-
overlapping generations ensures that agents compete based on the same time
budget.

###  A.2. Common-pool resource appropriation

CPR tasks abide in natural and human ecosystems: fisheries, grazing pastures
and irrigation systems are examples of multi-agent systems where self-
interested agents need to reach a sustainable resource appropriation strategy
that does not exploit the finite resources. They belong to a class of game-
theoretic tasks termed as social dilemmas, which exhibit a tension between
individual and collective motives: the optimal collective strategy is to
forage sustainably but self-interested agents will cooperate only if others
cooperate as well; otherwise they will consume resources until they deplete
them, a situation called Tragedy of the Commons (Hardin, 1968). Ecological
properties of these complex systems, such as the spatiotemporal variability of
resources and organisms are believed to play a big part in shaping solutions
to CPR problems (Janssen, 2010). From an ecological perspective, such settings
give rise to scramble competition, where organisms of the same species
appropriate resources at a rate contingent on their foraging ability, often
leading to population bursts and crashes (Parker, 2000).

With recent advances in RL, computational studies of social dilemmas have
managed to operate in simulation environments resembling the ones used in
human lab studies, where agents can navigate a grid-world consuming resources
(Perolat et al., 2017; Leibo et al., 2017). RL agents embody the self-
interested trial-and-error learning paradigm and have confirmed our intuition
that, when acting in a group, they cannot avoid a Tragedy of the Commons
unless they employ some auxiliary mechanism for guarding against exploiters,
such as learning to incur punishment (Perolat et al., 2017) and reputation
mechanisms (Anastassacos et al., 2021). These studies, however, remain far
from approaching the complexity of real ecosystems, which may comprise
thousands of organisms that do not necessarily follow the reward-maximization
paradigm.

##  Appendix B Details of the simulation

###  B.1. Environment

Our simulation environment is an extension of the CPR environment (Perolat et
al., 2017; Leibo et al., 2017) that the AI community has been using to study
the emergence of cooperation in groups of self-interested agents: a two-
dimensional grid-world where some cells contain resources (in green) that the
agents (in black) can collect. Resources grow depending of the presence of
other resources around them, which means that there is a positive feedback
loop, with reduction in resources leading to further reductions. In addition
to resources, the environment may contain walls (in blue) that kill agents
trying to traverse them (see Figure 1 for an illustration of our environment).

At each time step $t$ of the simulation a resource may grow in a pixel in a
cell of the environment with location $(x,y)$ based on the following three
processes:

  * •

a neighborhood-dependent probability $p_{I}(x,y)$ determines the probability
of regrowth in a cell based on the number of resources in its neighborhood,
$I$

  * •

a niche-dependent scaling factor $c(x)$ is used to scale $p_{I}$. We employ a
latitudinal niching model used in previous studies (Nisioti and Moulin-Frier,
2022; Grove, 2014): the world is divided into $N$ niches, each one having the
form of a horizontal stripe of pixels so that a cell’s location depends only
on its vertical position $x$. We refer to $c(x)$ as the climate value of niche
$x$.

  * •

independently of its neighbors and niche, a resource grows with a constant low
probability $c$. This is what we refer to as (sparse) spontaneous growth.

By modeling resource generation in this way we ensure that the resource
distribution follows the CPR model, that it exhibits additional spatio-
temporal variability due to the presence of niches and that resources do not
disappear too easily, which can be problematic in reset-free environments.
Thus, the combined regrowth rate for a resource $r$ is:

(1) $\displaystyle p(x,y)=p_{I}(x,y)\cdot c(x)+c$

A niche’s climate value is determined by equation:
$c(x)=(\alpha^{x}+1)/(\alpha+1)$, which returns values from 0 to 1 and allows
us to control the relationship between niche location and climate to be from
linear to exponential.

###  B.2. The agents

At each time step there is a variable number of agents $K_{t}$ in the
environment, each one characterized by its sensorimotor ability, cognitive
capacity and physiology.

##### Sensorimotor ability

An agent observes pixel values at each time step within its visual range (a
square of size [$w_{o}$, $w_{o}]$ centered around the agent, as illustrated in
the bottom right part of Figure 1). The pixel values contain information about
the resources, other agents (including their number) and walls. At each time
step an agent can choose to stay inactive or execute an action to navigate up,
down, right or left.

##### Cognitive capacity

An agent is equipped with an artificial neural network that outputs the action
to undertake based on the current observation and whose weights are
initialized randomly once at the start of the simulation. Its architecture
(illustrated in the bottom right part of Figure 1) is minimal: a convolutional
neural network, an LSTM cell that equips the agents with memory by enabling
policies conditioned on a trajectory of observatories and a linear layer that
transforms hidden states to actions.

##### Physiology

An agent is equipped with a simple physiological model modulating its level of
energy: the agent is born with an initial energy value $E_{0}$ which, at every
time step, experiences a linear decrease, and, if the agent consumes a
resource, is increased by one (see the top right part of Figure 1 for an
illustrative example of how the energy level may change within the lifetime of
a hypothetical agent). The energy is also clipped to a max value $E_{max}$.

###  B.3. Non-episodic neuroevolution

In neuroevolution (NE) a population of neural networks adapts its weights
through random mutations and a selection mechanism that promotes well-
performing policies. Under a classical NE paradigm training time is divided
into generations, at the end of which agents reproduce to form the next
generation (Stanley et al., 2019; Salimans et al., 2017).

Our proposed system deviates from this paradigm in two respects:

  * •

agents do not reproduce according to their fitness but according to a minimal
criterion (Brant and Stanley, 2017, 2020) on their energy level;

  * •

evolution is non-episodic: upon satisfying certain criteria an agent
reproduces locally (the off-spring appears on the same cell as its parent), so
that agents are added in an online fashion to the population, removing the
need for a concept of generation.

##### Reproduction

In order to reproduce an agent needs to maintain its energy level above a
threshold $E_{\text{min}}$ for at least $T_{\text{repr}}$ time steps. Once
this happens the agent produces an off-spring and is a candidate for
reproduction again. Thus, agents may have a variable number of off-spring and
do not die upon reproduction. We illustrate this relationship between energy
level and reproduction in the top right part of Figure 1. Reproduction is
asexual: an agent’s weights are mutated by adding noise sampled from
$\mathcal{N}(0,\sigma)$

##### Death

An agent dies once its energy level has been below a threshold
$E_{\text{min}}$ for at least $T_{\text{death}}$ time-steps or if its age is
bigger than a certain value $L_{\text{max}}$. Once this happens, the agent is
removed from the population forever.

###  B.4. Hyperparameters of the simulation

Below we provide the values of all hyper-parameters of the environment:

  * •

grid size : 400x200

  * •

Max population :1000 agents

  * •

Starting population : 330 agents randomly placed

  * •

Starting resources : 16 0000 randomly placed resources (there can only be at
most 1 resources at one pixel)

  * •

Field of view of agents $w_{o}$ : 15 (15x15 square with 7 in every direction)

  * •

Total number of timesteps : 1e6

  * •

Mutation variance $\sigma$ : 0.02

  * •

Energy function parameters :

    * –

Time to reproduce $T_{reproduce}$ : 140 timesteps

    * –

Time to die $T_{death}$: 200 timesteps

    * –

Max energy $E_{max}$ = Starting energy $E_{0}$ : 3

    * –

Energy death $E_{death}$ : 0

    * –

Energy decay : 0.025

    * –

Increase in energy when eating a resource : 1

    * –

Maximum age : 650

  * •

Regrowth function

    * –

$p(x,y)=p_{I}(x,y)\cdot c(x)+c$

    * –

$p_{I}(x,y)=\mathbb{1}_{I=1}*0.002$

    * –

$I$ corresponds to the 4 direct neighbors

    * –

$c(x)=\frac{(\alpha^{x}+1)}{(\alpha+1)}$ with $\alpha$ = 200

    * –

c = 0.00005

The simulation of the environment with a large number of agents and 1e6
timesteps took 20 minutes on a single GPU thanks to JAX parallelization and
speedup.

###  B.5. Details of agents architecture

The observation of the agent is fed into a 2 layer Convolutional neural
network (CNN) : First CNN has a kernel size (3,3), stride 2 and number of
features 4 followed by an average pooling of size (2,2) and stride 1 , the
second CNN has a kernel size (3,3, stride 2 and number of features 8 followed
by an average pooling of size (2,2) and stride 1. The output of the CNN is
then flattened and we concatenate to this vector the previous action of the
agent as well as a binary telling if the agent has eaten a resources or not.
This embedding is then fed into the LSTM of hidden state size 4 and we
concatenate the embedding with the output of the LSTM. The vector obtained is
then fed into a dense layer of size 8 and a tanh activation function. We
finally apply a last denser layer of size 5 (the number of actions) with
softmax to get the action probability from which the action will be sampled.
The softmax we use has a low temperature (1/50), so that the evolution can
quickly learn non random policy.

In total the agent neural network is very small with only 2445 parameters. We
chose a small neural network in order to make the evolution learning easier.

##  Appendix C Details on measures and evaluation

Figure 3. Heatmap of the amount of individuals in 17 bins of the distance
traveled during a time window in seed 1. y axis corresponds to the bins while
x axis is the timesteps of the natural env simulation. On the bin (y axis) 0
are agent that traveled a very small distance while 16 are agent that covered
a long distance (nearly the maximum amount of distance you can travel during
the 50 timesteps window)

The metrics used to characterize the system in the natural and lab environment
are:

  * •

Amount of resources in the map : sum over the whole grid of the resource
channel

  * •

Population size: Number of individual alive in the simulation

  * •

Expectancy : Average of age of agent that died during a large time window of
500 timesteps.

  * •

Percentage of the population with different amount of movement : For every
agent alive during a time window of 50, we compute the Manhattan distance
between the position at the last timestep of the window and the position at
the beginning of the window. This gives the distance traveled during this time
window for every agent alive as a number between 0 and 50. We then make 17
bins out of those 51 possible distance traveled. We then report in Figure 2.B
the percentage of the population in the two extreme bins as well as the max on
the other ones. We provide in Figure 3 a heatmap displaying the all 17 bins
during the whole natural environment simulation of seed 1.

For statistical testing we employ the ANOVA test to detect differences across
multiple conditions and Tukey’s range test for pairwise comparisons. We report
as statistically significant differences between pairs of methods with a
p-value lower than 0.05.

##  Appendix D Additional results

Figure 4. Metrics on seed 1: a) life expectancy of agents, b) total number of
resources present in the environment, c) size of the population and d)
percentage of individuals with different amount of movement

###  D.1. Large-scale trends

At the large scale several phases can be seen in the evolution of agents and
the environment. In this section, we explore those phases in seed 1 (which is
the seed described in the main paper in section 2 ) and give some new metrics
of the natural environment not mentioned in the main paper such as mean life
expectancy of the agents.

####  D.1.1. Population size and life expectancy rise and plateau

At the very beginning, in the first phase A (fig 4.A), the environment
contains plenty of resources which leads to an increase in the population. In
a second phase (fig 4.B), the population seem to start to plateau while the
amount of resources is still decreasing. This decrease in resources stops in
phase C.

During phase A, B and C, the expectancy of the agents increases (fig 4.a)
suggesting that the agents are becoming better even though the environment is
changing. The expectancy starts to plateau in phase D where it seems like the
environment reaches a more or less stable state on some metrics.

####  D.1.2. Decrease in the amount of resources: A near tragedy-of-the-
commons.

The decrease in the amount of resources in the environment at the beginning
(fig 4.b), seems to indicate that the evolving population as a whole depletes
the resources in a greedy way even though more resources means a higher spawn
of resources. The evolutionary path therefore seems to start by evolving a
population which will go towards the tragedy of the common (which is here
dampened by the fact that there are sparse spontaneous growth of resources).
This is confirmed by looking at the environment after some time (fig 6.a)
where we can see that there are only few patch of resources in some corner of
the map while the majority of the map is constantly depleted. This suggest
that at least local tragedy of the common happens in our simulation.

###  D.2.  Seed 2

Figure 5 displays all metrics we discussed in Section D.1 for seed 1, this
time measured for seed 2.

Figure 5. Metrics on seed 2: a) life expectancy of agents, b) total number of
resources present in the environment, c) size of the population and d)
percentage of individuals with different amount of movement

####  D.2.1. Diversity of eco-evolutionary path

Figure 6. Left: Diversity of environment between the 2 seeds (at timestep 600
000, zoom on the bottom right corner), here agents are in red; Right:
Percentage of the population with different amounts of movement of seed 2

We will now study some differences between seed 1 and 2.

In seed 1 sustainable and opportunistic travelers coexist during the whole
evolution (fig 4.D), while seed 2 has a majority of opportunistic travelers
and some sparse period where low moving agents emerge (fig 6.c). This may be
explained by the differences in the environment led by the agents behavior. In
fact seed 1 displays some area where there are big patches of resources
(especially in the corner) (fig 6.a)) and so where sustainable agents can
easily take advantage of. On the other end in seed 2 (fig 6.b), the map is
completely depleted of patches of resources which only allows agents to
sustain on spontaneous regrowth on random spots of the map, which might
explain why there are so much opportunistic travelers and nearly no
sustainable behavior. The sustainable behavior pic we can see in seed 2 might
be explained by timesteps where some spot of food were left for some time and
so where bigger patches of resources emerged which might have favored some
switches in behavior. See Videos 1.a and 1.b of the companion website for a
better visualization of the dynamic and behavior of agents.

The (small) diversity of evolutionary and environment path between the 2 seeds
we present are also an interesting feature of such eco-evo simulation.

###  D.3. Lab additional results

In this section, we provide additional results on the lab environment.

Fig 7 shows the average efficiency averaged on the population on different
density of resources (compared to only high resources task in fig 2.D) with
reproduction activated and deactived, we observe that on every resource
density, activating reproduction leads to increased resource consumption.

Figure 7. Average efficiency across the population across different density
levels with reproduction activated and deactivated. Activating reproduction
leads to increased resource consumption.