††thanks: To whom correspondence should be addressed.
E-mail<EMAIL_ADDRESS>
# Morphological computation and decentralized learning
in a swarm of sterically interacting robots

Matan Yah Ben Zion1,2,3 Jeremy Fersula1,2 Nicolas Bredeche2 Olivier Dauchot1
1Gulliver UMR CNRS 7083
ESPCI, PSL Research University, 75005 Paris, France  2Institut des Systèmes
Intelligents et de Robotique
Sorbonne Université, CNRS, ISIR, F-75005 Paris, France  3School of Physics and
Astronomy, and the Center for Physics and Chemistry of Living Systems
Tel Aviv University, Tel Aviv 6997801, Israel

###### Abstract

While naturally occurring swarms thrive when crowded, physical interactions in
robotic swarms are either avoided or carefully controlled, thus limiting their
operational density. In this paper, we explicitly consider dense swarms of
robots where physical interactions are inevitable. By leveraging collisions,
we offer new applications in collective self-organization using a morpho-
functional design. We demonstrate experimentally that an a priori minor
difference in the mechanical design of the robots leads to major differences
in their collective behavior when they evolve in crowded environments. We
design Morphobots, which are Kilobots augmented with a 3D-printed exoskeleton.
The exoskeleton not only significantly improves the motility and stability of
the Kilobots, it also allows to encode two contrasting dynamical behaviors in
response to an external force or a collision. This difference translates into
distinct performances at the individual and collective behaviour. These
include response to an external force, interaction with a stationary wall,
alignment with a movable obstacle, stabilization on a dynamically tilting
plane, and self-organized aggregation when addressing a phototactic task.
Enabling collisions also fluidizes the ensemble allowing the implementation of
a decentralized on-line evolutionary reinforcement learning algorithm in a
swarm of Morphobots. Finally we present a kinetic model that links the reward
function to an effective phototactic policy. Our results are of relevance for
the deployment of robust swarms of robots in a real environment, where robots
are deemed to collide, and to be exposed to external forces.

## Introduction

Figure 1: Graphical summary. Swarm robotics design space spans both morphology
and computation. Experiments with 3D printed flexible exoskeletons show they
can direct the outcome of collisions between robots. The mechanical design of
the exoskeleton, fronter versus aligner, controls whether a robot ascends or
descends a hill, which translate to pushing against or sliding along a wall.
The exoskeletons’ morphological computation extends the ability of the swarm
to execute collective tasks, including transport, phototaxis, and
decentralized-learning.

A robotic swarm is a synergetic ensemble capable of a greater task than its
constituents [1]. In a swarm, self-organization is an emergent property,
making the swarm resilient to the malfunction of an individual. Swarm
engineering poses an interdisciplinary challenge, combining electrical and
mechanical engineering, computer science, and non-equilibrium statistical
physics [2, 3]. Discovering robust design rules for a swarm offers an
opportunity to simplify the specifications of the individual robotic units,
reducing manufacturing complexity and making production at-scale accessible
[4, 5, 6, 7]. Recent development in robotic swarms demonstrated potential
applications in collective construction [8], coordinated motion for flying
UAVs [9, 10, 11, 12], patrol in open water [13] or underwater exploration [14,
15]. These applications are characterized by swarms where robots explicitly
avoid physical contact [16]. Another approach for swarm design is to keep a
cohesive swarm, where robots are constantly touching, keeping a physical
interaction that allows a continuous transmission of information. This
approach showed success in self-assembly and morphogenesis [17, 18, 19] as
well as coordination of a multicellular robotic body [20, 21, 22, 23]. To-date
artificial swarms are designed to operate exclusively in a dilute, collision-
avoided setting, or in a cohesive dense population. Resorting to in-silico
experiments to guide swarm design in a collision dominated swarm proves
limited by the ability to effectively account for the full mechanical
interaction. There is a need for physical designs rules and robust algorithms
where robots maintain a high degree of autonomy in solitude, but also retain
functionality in a crowded environment.

Unlike artificial robotic swarms, naturally occurring swarms show excellent
flexibility in both dilute and highly dense environments throughout the animal
kingdom. At the cellular scale, an individual bacterium can explore space by
performing a simple run-and-tumble [24], but can also form a dense, highly
active bacterial colony that displays internal turbulent flow [25] thereby
mixing in nutrients [26, 27]. On the millimeter scale, individual larva can
search food through chemotaxis, but self-organize into a living fountain when
a large localized food-source is present [28]. Macroscopic animals such as
fish and birds can individually nest and mate, but can come together into
giant schools and flocks to evade predation [29, 30]. Even flow of pedestrians
can emerge from simple body collisions [31]. Swarm cooperation is found on a
range of length scales, suggesting that only simple primitive behaviours are
required and physical interactions can be used for the swarm’s advantage.
Interestingly, self organization in dense assemblies is not a prerogative of
living beings. It has been shown that collective motion can emerge from
independent mindless robots physically interacting with one another,
considering a stripped-down flavour of swarm robotics as a particular kind of
active matter with no explicit decision computed at the level of the robots
[32, 33, 34, 35, 36, 37].

Here we augment robots with a flexible exoskeleton and show that their
collective task execution emerges from a single morphological parameter that
controls the outcome of collisions. We present two designs that have similar
dynamics for free individual robots, but opposite response when a robot
experiences an external force or a collision. We show that the passive
dynamics of the design allows to tune the robots response to align or oppose
an external force. Upon collision, a robot can push against, or slide along an
obstacle or another robot, based solely on its mechanical design. The emerging
collective dynamics adds a morphology layer to a robot’s computation
facilitating a combined physical and logical swarm architecture [38, 39]. We
quantitatively analyze the response to an external force and find empirically
that it is controlled by a single dynamical parameter, $\kappa$. We show
experimentally, theoretically, and numerically that a change in the sign of
$\kappa$ inverts the response to an external force, making a robot go down or
up a force field. We find $\kappa$ to be a key design parameter in the
response of the robots when encountering an obstacle (be it a wall or another
robot). Using positive and negative $\kappa$ makes two primitives that
qualitatively alter the swarms dynamics in a collision dominated, crowded
environment. We also show that $\kappa$ can be used to guide the design of
collective transport.

We then add a computational layer, where in parallel to the implicit
morphological computation, robots execute an explicit sense-act cycle. We
experimentally study phototaxis in a partially lit arena, setting a swarm of
$N=64$ robots to search the lit region. We set up the illumination field to be
unoriented (coming directly from above), and without local light gradients,
thus ensuring robots cannot use local information to guide their way to the
lit region. Moreover, the lit region is designed to be too small to host the
whole swarm. This restriction forces the robots to physically interact,
turning the phototaxis task into an exercise of collective aggregation with
sterical interactions, going from dilute (in the dark) to crowded (in the
light). Studying swarms with positive and negative $\kappa$ reveals a
qualitative and a quantitative different collective phototaxis that can be
directly explained with the dynamical nature of $\kappa$.

Finally, we show that the swarm can collectively learn to execute the
phototactic task when operating in an environment where individual robots
transition between dilute and crowded regions. We implement a decentralized
reinforcement learning algorithm inspired by social learning [40, 41, 42, 43],
and find that when collisions are allowed the swarm fluidizes, and effectively
converges to a successful phototactic strategy despite the sparse and
intermittent communication network. With each robot carrying a reward
function, an analytical link can be derived between the policy of the
individual and the ability of a the whole swarm to collectively learn. Our
results offer a balance between an ideal execution strategy, and an optimal
learning strategy. The interplay of morphology and computation is graphically
illustrated in Fig. 1, summarizing the findings of our work.

##  I The Morphobots

Figure 2: Morphobots are Kilobots augmented by an exoskeleton : A a Kilobot
with a light sensor, IR communication, and two differential drive vibration
motors, is fitted into one of two 3D printed exoskeleton designs - (i)
fronter, and (ii) aligner. B both fronter and aligner C exoskeletons are
tripods, but the restitution vector, $\hat{r}$ of the aligners (fronters) is
parallel (anti-parallel) to the orientation vector $\hat{n}$. D mean square
displacement versus time shows ballistic response for both designs ($\propto
t^{2}$), with a typical speed of 5.2 cm/s (4.8 cm/s) for the aligners
(fronters). Also shown, the mean square displacement of a bare Kilobot which
after calibration can also move ballistically, but at a speed one order of
magnitude slower (0.53cm/s). Inset: 12 seconds long trajectories of fronters
(green) and aligners (cyan), and a 36 seconds long trajectory of bare Kilobots
(gray) inside the 150 cm diameter arena. Figure 3: The orientation response of
aligners and fronters to an external force is equal and opposite. A Sketch of
the experimental setup for measuring dynamical response on an inclined plane
(see Movie 1). B Force diagram of a robot placed on an inclined plane. For
small slopes ($\varphi\leq 6^{\circ}$), robots experience an external body
force. C Trajectories of Aligners (cyan) and Fronters (green) moving on
different slopes ($\varphi$). Aligners and fronters have an opposite response
to the external force - aligners orientation turns downwards as they quickly
descend, while fronters turn upwards and slowly ascend. D Throughout its
motion, the orientation of the robot, $\hat{n}$ forms an angle $\theta$ with
the external force, $\vec{f}$, together setting the net velocity $\vec{v}$
(see Eq. 1). E Tracking the orientations shows that aligners turns downhill
($\theta\rightarrow 0$), aligning with the external force, while Fronters turn
uphill ($\theta\rightarrow\pi$), fronting the external force. Solid curves are
fits to Eq. 4, from which $\kappa$ can be extracted (see SM). F For finite
tilt angle, the response of the two designs is approximately equal and
opposite $\kappa_{aligner}=-\kappa_{fronter}\approx 0.3/d$. At a slope near
zero, response is dominated by the robots internal bias (left/right curving,
and internal noise), but averages to zero. Scale bar is 20 cm.

The Morphobots are Kilobots [17] augmented with 3D printed exoskeletons. The
Kilobot is a $3.4$ cm tall, $3.3$ cm diameter unit, standing on 3 rigid legs
made of thin metallic rods. The Kilobots are capable of differential drive
locomotion using two vibrators. When either is activated, a Kilobot turns at
about $45\deg$ per second, and when both are activated, the Kilobot moves
forward. The power source is a rechargeable lithium-ion battery that can keep
the Kilobot running for a few hours. Kilobots are equipped with an infrared
transmitter and receiver so that they can communicate with each other. The
transmitter of a robot sends light toward the surface which reflects up to the
receiver of another nearby robot. Obtaining straight motion over long
distances is difficult and requires lengthy calibration of the vibrators,
moreover, the calibration itself drifts over time.

To both improve the robot’s nominal speed, and eliminate the need for
individual calibration, we encapsulate the Kilobots within exoskeletons that
change how collision and locomotion are performed. We made two tripod based
designs (a pair of flexible legs and an opposing stiff leg), with both designs
having a circular frame (diameter $d=4.8$cm). The two designs, which we shall
call fronter versus aligner, differ in the fore-versus-aft positioning of
their flexible legs (see Fig. 2-A,B,C and Methods). The natural vibration of
the semi-flexible legs is tuned near resonance with the vibration motors, in
order to maximize the coupling with their drive. Using an anisotropic flexible
leg design (1mm thick, 6 mm wide and 20 mm long, see Supplementary Materials),
vibrations are coupled predominately to the forward stick-slip motion of the
robots. The exoskeleton allows a Morphobot to operate at an order of magnitude
higher speed than the Kilobots (see Fig. 2-D and Supporting Movie 1). More
importantly, it allows to have an explicit response to external forces and
collisions.

By tracking the motion of individual Morphobots, we find that aligners and
fronters show near identical motility. Motility is defined by the magnitude of
the displacement of an individual robot over different timescales, and
measured by tracking the time evolution of their two dimensional coordinates:
$\vec{r}\left(t\right)=\left(x\left(t\right),y\left(t\right)\right)$ when
running in the experimental arena [44]. Motility measurements were done using
a low concentration of robots (less than 1% filling fraction) excluding the
effect of collisions, and robots were tracked only when away from the walls of
the arena. The mean square displacement, $\rm{MSD}=\langle\Delta
r^{2}\left(\Delta t\right)\rangle_{N}$, of an ensemble of noisy walkers,
presents two dynamical regimes [45]. On short time intervals $\Delta t$, the
displacement of the robots is aligned with the direction of their individual
orientation and displacement is proportional to $\Delta t$. On this time scale
the motion is said to be ballistic as robots go roughly on a straight line at
a constant speed: $\langle\Delta r^{2}\left(\Delta
t\right)\rangle_{N}=v_{0}^{2}\Delta t^{2}$, where $v_{0}$ is the instantaneous
speed. On long time intervals, the noise has reoriented the robots several
times and displacement grows as $\sqrt{\Delta t}$ as robots explore the space
randomly. The motion is now said to be diffusive $\langle\Delta
r^{2}\left(\Delta t\right)\rangle_{N}=4D_{0}\Delta t$, where $D_{0}$ is called
the diffusion constant. The two regimes ($\propto\Delta t$ and
$\propto\sqrt{\Delta t}$) are separated by a crossover, which takes place on a
time scale called the persistence time, $\tau_{p}$ (see below), to which
corresponds a persistence length $\langle\Delta
r^{2}\left(\tau_{p}\right)\rangle_{N}^{1/2}$. Fig. 2-D displays the mean
square displacement measured for a set of Morphobots equipped with the two
types of exoskeleton, together with their ensemble average. One sees that the
difference in motility between the two designs is no larger than the robot-to-
robot variation within each design. The observed ballistic nature of the mean
square displacement ($\propto v_{0}^{2}\Delta t^{2}$) indicates that the
persistence length of the motion is larger than the size of the circular arena
(diameter $D=150$ cm). Fitting the ensemble mean square displacement in the
ballistic regime, we find that the mean nominal speeds are $5.2\pm 0.9$ cm/s
and $4.8\pm 0.1$ cm/s for the aligners and fronters respectively.

Altogether, the anisotropic leg design allows us to use higher motor drive
while keeping robots trajectories straight, alleviating the need for
individual robot calibration, and resulting in a nominal speed of $v_{0}\simeq
5$ cm/s, a ten-fold increase, relative to the bare Kilobot design (see
Supporting Movie 1). The notion of using passive-dynamic machines [46] allowed
to achieve a similar feat in bristle bots, using rows of soft legs [47], and
in the commercially available HEXBUG [22, 48, 23]. Using the differential
drive, robots can also pivot at $45\deg$ per second. Programming alternated
straight and pivoting motion of random duration and random direction, we can
set the Morphobots into a run-and-tumble motion [49], where, on average, the
robots run straight during a run time $\tau_{run}=2$ s, with a velocity to be
specified, and tumble during a mean tumble duration $\tau_{tumble}=4$ s. The
total duration of a run-and-tumble sequence sets
$\tau_{RT}=\tau_{run}+\tau_{tumble}$, with the resulting persistent time being
$\tau_{P}\approx 18\;\rm{s}$ (See SM).

##  II Fronters vs. Aligners response to an external force

We measure the response of the two Morphobots designs to an external body
force by placing the robots on a plane tilted by a small angle, $\varphi$. The
two designs have a strikingly different response: aligners go downhill, while
fronters climb against gravity (Fig. 3A,B and Supporting Movie 1). The effect
is consistent, regardless of the initial orientation of the robot being along
the positive $y$ direction (downhill to its right), or negative $y$ direction
(downhill to its left), excluding the effect of an internal left/right bias.
We systematically increase the slope (see Fig. 3C), and find that with
increasing $\varphi$, the trajectories turn increasingly curved, until the
robot orients downhill (aligner) or uphill (fronter).

To quantify the different responses of the robots to an external body force,
we modify a model introduced in the context of active matter to describe self-
aligning particles [48]. A Morphobot has two translational $\vec{r}=(x,y)$ and
one orientational $\hat{n}=(cos(\theta),sin(\theta))$ degree of freedom. In
the overdamped limit, the deterministic dynamics of the velocity $\vec{v}$ and
orientation $\hat{n}$ of a Morphobot with a nominal speed $v_{0}$ subjected to
a body force $\vec{f}$ (see Fig. 3D) obey the following equations :

$\displaystyle\frac{d\vec{r}}{dt}$ $\displaystyle=$
$\displaystyle\vec{v}=v_{0}\hat{n}+\mu\vec{f}$ (1)
$\displaystyle\frac{d\hat{n}}{dt}$ $\displaystyle=$
$\displaystyle\kappa\left(\hat{n}\times\vec{v}\right)\times\hat{n},$ (2)

where $\mu$ is the particle mobility and $\kappa$ is the re-orientation
strength. The first equation simply states that the velocity is the additive
combination of the propulsion velocity $v_{0}\hat{n}$ and the speed induced by
the external force, $\mu\vec{f}$. The second equation contains the key
ingredient of the model. It describes the reorientation of the Morphobots
along the direction of its motion. $\kappa$ controls the strength of this
reorientation process. It is signed and has units of curvature (1/length).
When $\kappa$ is positive, respectively negative, the Morphobots aligns
towards, respectively opposite to, the direction of the external force. For a
constant force in the $\hat{x}$ direction $\vec{f}=f\hat{x}$, with
$f=mg\rm{sin}\phi$, where $m$ is the mass of the Morphobot and $g$ is the
acceleration of gravity, the dynamical evolution of the orientation,
$\theta=\theta\left(t\right)$ relative to the $x$ axis obeys a closed form
equation :

$\frac{d\theta}{dt}=-\kappa\mu f\rm{sin}\theta.$ (3)

Eq. 3 is the equation for an overdamped simple pendulum, which can be solved
directly to give $\rm{tan}\frac{\theta}{2}=e^{-\kappa\mu{\it f}t}$ (for
initial condition $\theta\left(t=0\right)=90^{o}$). In the limit where
$v_{0}\ll\mu f$, when the orientation is expressed as a function of the
arclength, $\theta\left(s\right)$, it is found to be independent of the
external force and mobility (see Supplementary Materials),

$\theta\left(s\right)=2\rm{atan}\left(e^{-\kappa s}\right).$ (4)

To find the alignment parameter, $\kappa$, we tracked the orientation of
individual robots as they move across a fixed inclined plane (see Fig. 3E and
Methods). At the absence of an external force (zero slope, $\phi=0^{\circ}$),
both fronters and aligners move on roughly straight lines. For steeper
inclines ($\phi=3^{\circ}$ and $\phi=6^{\circ}$), both designs show similar
rate for the convergence of the orientation. Fitting the evolution of the
orientation to Eq. 4, we extract the alignment parameter, $\kappa$. The
magnitude of $\kappa$ for both designs is similar, but with opposite signs:
$-\kappa_{fronter}\approx\kappa_{aligner}\approx 0.06\;\text{cm}^{-1}$.

The reorientation of the Morphobots applies also in a dynamic environment
where the plane is tilting. Placing a Morphobot on a seesaw gives rise to
dramatically different behaviour depending on its type (see Fig. 4). An
aligner turns quickly downhill and falls off the seesaw, while a fronter turns
upwards and climbs. Once passing the center of the seesaw, the mass of the
robot suffices to tip the plane of the swinging table. The fronter is now
facing downhill, yet, again, as it moves, it turns around and recommences
climbing, until passing the center of the seesaw once more, where this
dynamics repeats, effectively tracing loops (see Fig. 4B and supporting Movie
2).

Figure 4:  Fronters and aligners respond differently to a dynamic environment.
A A Morphobot on a swinging table setup for testing morphological response to
a dynamically changing environment. B A fronter continually turns uphill,
moving in a circular motion while staying on the swinging table (see also
supporting Movie 2), whereas an aligner (C) quickly turns downhill and falls
off. Top shows schematics, bottom shows measured trajectories where scale bars
are $10\;\rm{cm}$.

##  III Wall interaction and collective transport

Figure 5: Tuning interaction with a static and moving obstacle using
morphological computation. The morphological interaction of the Morphobots
with walls can be explained using their response to an external force. When a
Morphobot collides with a wall, each impingement can be seen as a momentary
normal force. A Aligners turn to align their orientation (black arrow) with
the normal force (red). Once aligned along the wall, there is no further
rotation as there are no more collisions, and the robot slides along. B When a
fronter collides with a wall, its orientation (black) turns anti-parallel with
the normal force (red), and the robot continues to push into the wall. When
programmed to run-and-tumble, aligners and fronters interact differently with
a wall of stationary robots where aligners (A) cover much larger distance
along the wall relative to fronters (B). C Multiple collision experiments show
that aligners and fronters spend similar time near a wall ($3\tau_{p}\approx
1\;\rm{min}$) yet aligners move twice as fast. D Brownian dynamics simulations
predict that active particles with a negative alignment parameter, $\kappa<0$
(fronters), push a movable object. E With real robots: placing a round movable
obstacle shows that a swarm of fronters is capable of collective transport.
Scale bar is $10\;\rm{cm}$.

Using the response of the morphobots to an external force we can predict their
interaction with stationary walls, moveable objects and with one another.
Morphological interaction with a boundary (stationary or moveable) can be seen
as a series of collisions, at each of which the robot experiences an external
force (see Fig. 5-A). With each collision, a Morphobot slightly re-orients:
aligners turn parallel with the applied force (and away from the wall) while
fronters turn anti parallel with the applied force (and towards the wall).
This is confirmed by the direct observation of Morphobots sent to move towards
a wall of stationary robots (see Fig. 5-B). On the average, both designs spend
similar time near the wall ($\sim 3\tau_{p}\approx 1\;\rm{min}$) but aligners
travel a path along the wall which is twice longer than that of the fronters
(see Fig. 5-C and Movie 1). Using a multi-agent Brownian dynamics simulation
of active soft-discs, we were able to predict the outcome of collisions based
on the alignment parameter $\kappa$. During a collision, an agent with
$\kappa>0$ (aligner), turns towards the normal force, and away from the
obstacle, whereas an agent with $\kappa<0$ (fronter) turns against the normal
force and into the obstacle thus pushing it. Also, simulating agents with
$\kappa<0$ near a passive particle shows that the agents collectively move the
passive object (see Fig. 5-D). This has been confirmed experimentally by using
a group of fronters that work together to push a circular object (see Fig. 5-E
and Supporting Movie 1). Previous work showed that collective transport can be
implemented using global control [50]. Here we extend this notion and show
that a single, embedded parameter, $\kappa$, effectively captures the dynamics
of sterically interacting agents, reducing the required computing resources
for simulating a large swarm in a crowded environment. More importantly, it
offers a design rule: $\kappa<0$ gives rise to a desirable swarming behaviour
in many applications involving collective transport.

##  IV Collective phototaxis in the presence of steric interactions

Figure 6: The collective ability of a robotic swarm to phototaxis is
constrained by steric interactions. A a swarm of phototactic robots is placed
in an arena patterned with an un-oriented light field. As there are no local
gradients to guide a robot to the light source, robots perform a simple run-
and-tumble in the dark, and stand in the light. B When a fronter arrives to
the light spot, it collides with robots already taxied at the perimeter, but
progressively pushes its way in. Repeating this process results in the
complete coverage of the light spot (C). When an aligner arrives to the light
spot (D) it slides along the wall of robots and leaves. This leads to only
partial filling of the light spot (E), leaving an empty void in the center. F
The resulting fraction of swarm in the light as a function of time shows that
at early times ($\tau\approx 30\tau_{p}\approx 10\;\rm{mins}$), both design
show equal performance, quantitatively consistent with a diffusion limited
reaction (dashed line). at later times, fronters outperforms aligners (average
of 4 realizations with error bars being standard deviation). G Simulating
swarms with over two orders of magnitude more agents show a sharp slowdown in
the phototaxis of aligners with swarm size. H The typical time for half of the
swarm to enter the light $T_{1/2}$ grows rapidly for an increasingly larger
swarm of aligners, while having only a small effect on a swarm of fronters of
similar size. Scale bars are 20 cm.

We turn to study the ability of the swarm to execute a phototactic task,
explicitly using sensory and computational power of the Morphobots. Similar to
galvanotaxis and chemotaxis, phototaxis is a common behaviour observed in the
animal kingdom [51, 52]. Stimulus-induced locomotory movements (or taxis)
constitute an important building block in swarm robotics [53, 2, 54, 3, 55,
56], and several bio-inspired algorithmic implementations have been proposed
for phototaxis using ambient light sensors [57, 58, 59, 60, 20] and phonotaxis
using microphone and speakers [61, 62]. Here we focus on a phototactic task,
where individual robots have no global information to guide their sense-act
cycle. For this we set a swarm ($N=64\;\rm{bots}$) of either aligners
($\kappa>0$) or fronters ($\kappa<0$) to explore a circular arena (diameter
$D=150$ cm) with a lit spot that covers only $\sigma=6\%$ of the area of the
arena (diameter $b=36\;\rm{cm}$, see Fig. 6-A and S1 in SM). The light is
unoriented (coming from above), and there are no local light intensity
gradients to guide a robot to the lit region (light field is essentially
flat). This excludes the use of traditional phototactic algorithms such as
steepest decent search [63, 19] or reorientation [17]. The lit region can not
accommodate more than 80% of the robots (about 50 Morphobots), challenging the
swarm to transition between sparse and crowded.

As a first step we consider ad hoc calibrated swarms, where the Morphobots’
policies are defined as follows. Each Morphobot continuously monitors local
light intensity: when in the dark (below a predefined intensity threshold,
$P_{th}$) the robot performs a run-and-tumble motion, with a running velocity
$V_{0}$. When entering a bright region (light intensity above $P_{th}$), the
robot stops, $V_{1}=0$. At early times, when there are only a few robots in
the lit region, the two designs show similar performance as measured by the
fraction of the swarm that is in the light

$\mathcal{F}\equiv\frac{N_{1}}{N},$ (5)

where $N_{1}$ is the number of bots taxied in the light. At early times,
$\mathcal{F}$ is quantitatively captured as a diffusion limited process (see
Fig. 6-F), with the kinetic reaction constant found using Smoluchowski first
passage computation for diffusive particles with an effective diffusion
constant, $D_{eff}\approx 4.2\;cm/s$ (See Supplementary Materials) [64]. After
approximately 10 minutes, both swarms deposit a first layer of robots at the
perimeter of the lit region (Fig. 6-B,D). From this point onward, the
phototaxis rate slows down and deviates from the simple diffusion limited
reaction, as crowding of taxiing robots at the perimeter of the light spot
physically prevents further robots from entering the lit region (Fig. 6C,E).
This steric effect is quantitatively more pronounced for the aligner design,
as can be seen for the overall performance given by the fraction of the swarm
that has entered the lit region on long times (Fig. 6F).

The reason is that the collective success of each swarm depends on the
morphological interactions of a robot impinging from the dark region on the
first layer of robots at the perimeter of the lit region. Individual
Morphobots arriving at the wall of stationary robots show qualitatively
different dynamics for either design: an aligner ($\kappa>0$) that arrives at
the wall tends to reorient along the wall until it finally leaves, whilst a
fronter Morphobot ($\kappa<0$) tends to “dig in”, pushing already taxied
robots into the lit region until taxied itself (see Fig. 6B,D). These
individual events add up: fronting Morphobots progressively push the perimeter
inwards, constantly filling the bulk of the lit region with robots. By
contrast the aligning Morphobots keep reorienting away from the robots at the
perimeter, leaving a hollow in the lit region (see Fig.6 C, E and Supporting
Movie 1). This discrepancy exacerbates as the swarm size increases. By
simulating up to 8192 phototactic agents that follow Eqs. 1,2 we find that a
swarm of aligners ($\kappa>0$), becomes considerably slower to phototaxis
relative to an equivalent swarm of fronters ($\kappa<0$). The typical time,
$T_{1/2}$, to have half of the swarm in the light ($\mathcal{F}=1/2$) grows
rapidly (see Fig. 6G,H). The above findings demonstrate that the collective
ability of the swarm to perform a phototactic task at scale, hinges on a
subtle morphofunctional difference.

##  V Distributed learning of a phototactic strategy

Figure 7:  Distributed on-line reinforcement learning algorithm and deployment
of a swarm of Morphobots. A The embedded algorithm for updating the
phototactic policy of each robot, is composed of two main moduli: a sense-act
cycle and a learning cycle. B The sense-act cycle is encoded into a
Perceptron: external stimuli are weighted and summed, evaluated through a
sigmoid function, the output of which is binarized to select an action. The
weights, $w$, define the policy $\pi$ of the robot. C Illustration of a
centralized communication network where all agents (gray circles) are
connected to a central operator (red circle). D An illustration of a sparsely
connected network where agents communicate only when in close proximity. E The
learning cycle is based on the conditional diffusion of the policies across
the population, biased by a reward function $\rho$ evaluated by each robot:
when robot 2 receives a message containing the policy control parameters
$\\{w_{i}\\}_{1}$ and the reward $\rho_{1}$ of robot 1, robot 2 will inherit
the new policy, if it is associated with a higher reward $\rho_{1}>\rho_{2}$.
F-I A robot (black curve) already in the light source (F) departs G and
broadcasts its reward and policy (H) to a robot in the dark (red curve), which
inherits the new policy (I). Note that the broadcasting robot may have a wrong
policy, effectively spreading “fake news”, with neither robot knowing what is
the best policy for phototaxis. J A swarm of Morphobots with randomly
initialized light thresholds, collectively learns its environment, increasing
the swarm fraction in the light. Results are the average of 4 runs with error
bars being the standard deviation.

The swarms considered thus far had the environmental information hard-coded
into individual robots before deployment — the sense-act cycle of each and
every robot had a pre-programmed, ad hoc knowledge of the correct light
threshold for phototaxi. Concurrent challenges however require deployment of
robotic systems in unknown terrains [53, 2, 54, 3], including extra-
terrestrial exploration, underwater surveying, and search-and-rescue missions
[65, 66, 67]. Under such circumstances, robots will lack an a priori model of
the environment that can be pre-programmed. This requires an added layer to
the robots’ controller for on-line tuning of the sense-act cycle based on
continually acquired knowledge. In such conditions, a distributed robotic
system offers robustness with scale, improving both exploration and survival.
A distributed system however also poses a challenge as the swarm has to learn
collectively.

In this final section we show that in a swarm of Morphobots, both execution
and learning can be made fully distributed (see Fig. 7). Since the robots are
allowed to collide, the swarm is fluidized, sustaining information flow. An
individual collision event may be stochastic, but the mean data flow within
the swarm becomes predictable. Each robot evaluates its performance, and
compares it to other encountered robots. The selection algorithm can be made
distributed by implementing a decentralized on-line reinforcement learning
scheme, to make the swarm adaptive.

Previous work showed that a swarm can learn a phototactic strategy with a
policy defined by the weights of an Artificial Neural Network as a controller,
$\pi=\\{w_{i}\\}$ [68, 69, 70]. There, the learning was implemented using an
evolutionary algorithm to act as a direct policy search, using a reinforcement
learning method, incrementally refining behavioural policies by directly
tuning the policies’ parameters [71, 72]. The swarm is randomly initialized
with a pool of sense-act policies, and in an iterative selection process, only
the best performing policies are carried over to the next generation. In these
algorithms, the execution is indeed distributed over the swarm, but it is a
central computer performing selection and optimization (see Fig. 7C). Such
schemes are impossible to implement outside the lab if the swarm has no access
to an external operator for guidance, nor a global communication channel to
compare performances. The communication network of a swarm of Morphobots is
extremely sparse and intermittent as robots move around to change neighbors
(see Fig. 7D). Yet the communication becomes effectively connected across
times because of the swarm’s fluidity.

For a successful reinforcement learning algorithm to converge, a suitable
reward function, $\rho$, must be defined [73]. As an input, the reward
function receives the robot’s sensory information
$\rho=\rho\left(\\{i_{i}\\}\right)$ and outputs a scalar score (see Fig. 7A
and Algorithm 1 in SM). The combination of an individual robot sense-act
policy, $\pi=\pi\left(\\{w_{i}\\}\right)$, and the environment it sampled,
results in a robot’s reward at a given time. Robots can also communicate when
within communication range and exchange their policies and rewards (Fig. 7
E-I). Following such an exchange, a robot can choose to adopt the received
policy based on its associated reward, operating under the assumption that a
higher reward is the result of a superior policy. Unlike classical
evolutionary algorithm that use a centralized computer to compare
performances, the scheme presented here allows to embed both the evaluation
and the selection within the robot. In other words, robots collectively
perform some kind of social learning, where successful individual innovations
diffuse over the population [40, 41, 43]. Note that the assumption that a
higher reward is a result of a superior policy may not be always true. From
time to time robots with an incidental favorable history can have a
disproportional higher reward. Such bots will propagate a poor policy globally
to the swarm, effectively spreading “fake news”. This local-to-global gap
challenges the chosen reward function to be robust to such fluctuations. The
fluid nature of the swarm of Morphobots allows us to choose $\rho$ that
overcomes this culprit. Most swarm robotic systems employ a collision
avoidance algorithm that slows down their collective dynamics, jamming the
information flow, and creating deadlocks [8]. By contrast, the Morphobots are
allowed to collide and their swarm enjoys good mixing (see Figs. S2, S3 in
Supplementary Materials), offering a quantitative link between global and
local information.

To formulate a proper reward function we note that the collective phototaxis,
$\mathcal{F}$, as defined in Eq. 5 is an ensemble average,
$\langle\rangle_{N}$, counting the fraction of the swarm that is in the light
at a given time. $\mathcal{F}$ can not be directly optimized by an individual
robot as it stores global information. However in a system with good mixing, a
time average can approximate an ensemble average
$\phi\equiv\langle\rangle_{T}\approx\langle\rangle_{N}$. A time average
equivalent to the ensemble average given in Eq. 5 is the mean time an
individual robot spent in the light, $\Phi\equiv T_{{\rm
in\,light}}/T\approx\mathcal{F}$. In a distributed robotic system with good
mixing, a reward function that estimates the mean time the robot spent in the
light can serve as a proxy for the global performance of a swarm of its kind,
$\rho\propto\Phi\approx\mathcal{F}$. Robot $i$ can evaluate the mean time it
spent in the light through temporal average of its light sensor signal:

$\rho^{i}=\frac{1}{M}\sum^{M-1}_{k=0}P^{i}_{k},$ (6)

where $M$ is the number of averaged measurements, and $P^{i}_{k}$ is the the
$k$’th measurement taken at constant time intervals. Since the time a robot
spends in the light is inversely proportional to its speed in the light,
$\Phi\propto\rho\propto 1/V_{1}$ the above choice of reward function is
analytically linked to a given policy (see SM for detailed derivation). This
link allows us to show that when formulated as a dynamical process, the
collective learning will indeed converge. Defining the concentration of robots
with a correct policy as $c_{C}$, the concentration of robots with wrong
policy is given by $c_{W}=c_{0}-c_{C}$ (where $c_{0}$ is the constant overall
concentration of the swarm). In a system with good mixing, the encounter rate
between robots with correct and wrong policy is proportional to the product of
their concentrations, $kc_{C}c_{W}$ (to leading order), where $k$ is
proportional to mutual diffusion. When encountered, robots compare their
rewards and a robot with a wrong policy will adopt the correct policy at a
probability $f_{CW}$. The opposite is also possible (“fake news” scenario) and
a robot with a correct policy will adopt a wrong policy at a rate $f_{WC}$.
When combined, the rate equation for the growth in the population with a
correct policy becomes

$\frac{dc_{C}}{dt}=k\left(f_{CW}-f_{WC}\right)c_{C}c_{W}$ (7)

(sees Supplementary Materials for a detailed derivation). Collective learning
is achieved when the concentration of robots with correct policy grows,
implying: $dc_{C}/dt>0$. Since $k$, $c_{W}$, and $c_{C}$ are all non-negative,
collective learning is guaranteed when $f_{CW}>f_{WC}$. Given the definition
of the reward function in Eq. 6 and the selection protocol defined in Fig. 7,
robots with the correct threshold will have, on the average, a higher reward,
$\rho_{C}>\rho_{W}$, and the population with the correct policies will prevail
(see SM for a full derivation).

To test the validity of the proposed decentralized learning scheme, we
deployed a swarm of Morphobots in the same light patterned arena, but this
time with randomly initialized light thresholds (See Fig. 7, Supporting Movie
1). Weights are chosen in random from unsigned values ranging from 0-255 (see
Methods Section and Supplementary Materials for details). At very early times
only few robots slow down in the light — most robots either stand in the dark
or move quickly through the lit region. And yet the swarm fraction in the
light progressively increases with time. We tested swarms of both aligners and
fronters, and found consistent increase in the fraction of the swarm in the
light reaching approximately 0.4. At these swarm fractions in the light, the
difference between fronters and aligners is not expected to be significant as
is evident here (compare with Fig. 6F at early times). The learning swarms had
8 times more robots in the light than the case of random placement that is,
the fraction of the lit region, $\sigma=0.06$. The fact that the adaptive
swarm performed almost, but not quite as well as, the pre-calibrated swarm is
not surprising and may be attributed to the fact that dynamics of about half
of the randomly initialized population is very slow (as they stand in the
dark), and given enough time, they will also eventually learn. This may be
resolved by expanding the learning space of the sense-act cycle, to have the
robots also learn the speed in the light, $V_{1}$.

##  VI Conclusion

In this paper, we introduced a robotic swarm that can operate in both dilute
and dense settings, where collisions between robots occur frequently. We show
that the robots’ physical morphology plays an important role in terms of
collective behaviours, where a seemingly small change in the shape leads to
very different outcomes. Applying tools from statistical physics of active-
matter to swarm robotics, we show that behavior found in natural swarms (such
as ant colonies [74]) including collective self-aggregation and transport can
be attained with limited use of software-controlled behaviors by capitalizing
on the morphological properties of robots, demonstrating morphological
computation at work within the swarm.

We also showed that distributed on-line evolutionary reinforcement learning
can be implemented in a dense robot swarm, to exploit or mitigate physical
contingencies after deployment in the open. We show that this kind of
artificial social learning, when robots locally exchange information about
their behavioral strategies, can be modelled as a conditional diffusion
process. In particular, we reveal that there exists an optimal diffusion flow
that balances between exploration (diffusing winning strategies) and
exploitation (maximising efficiency), which relevance goes beyond the self-
aggregation with phototaxis task considered here.

Future collective learning implementations in swarm robotics can go beyond
diffusion of behavioral strategies. An extreme case is of sparse swarms, where
individual robots or groups may be temporarily disconnected from the whole
[75]. While dense and sparse robot swarms may be considered separately, it is
likely that a robot swarm in the open will have to switch from one
configuration to another, requiring versatile social learning capabilities. A
natural extension can be motivated by cultural evolution found in the nature
[76], which is built on the reformulation and combining previously learned
behaviours to continuously discover behaviors of growing complexity.

Future physical models of a learning swarm should go beyond the mean-field
approximation used here, and account for the spatial density fluctuations (as
is done for describing the motility induced phase separation observed in
active fluids [49, 77]). From that point of view, our work shows that building
upon the recent theoretical progress made in describing population of active
particles, one can envision a rapid development of a statistical mechanics
description of smart active matter, for future swarm engineering.

## Acknowledgements

We thank Y. Lahini and N. Oppenheimer. Funding: This work was supported by the
MSR project funded by the Agence Nationale pour la Recherche under Grant No
ANR-18-CE33-0006. Author contributions: O.D, N.B, and M.Y.B.Z conceived the
research. M.Y.B.Z Designed the experiments, theoretical models, and analysis.
M.Y.B.Z and J.F preformed the experiments and the data analysis. M.Y.B.Z, O.D,
and N.B wrote the manuscript. Competing interests: The authors declare no
competing interests. Data and materials availability: All (other) data needed
to evaluate the conclusions in the paper are presented in the paper or the
Supplementary Materials.

## Supplementary Materials

Exoskeletons CAD files
Code Phototaxis
Movies S1 and S2
Materials and Methods
Supplementary Text
Figs. S1, S2, S3
Algorithm 1

## References

  * Beni and Wang [1993] G. Beni and J. Wang, in  _Robots and biological systems: towards a new bionics?_ (Springer, 1993) pp. 703–712.
  * Brambilla  _et al._ [2013] M. Brambilla, E. Ferrante, M. Birattari, and M. Dorigo, Swarm Intelligence 7, 1 (2013).
  * Hamann [2018] H. Hamann,  _Swarm Robotics: A Formal Approach_ (Springer International Publishing, Cham, 2018) p. 215.
  * Brooks and Flynn [1989] R. A. Brooks and A. M. Flynn,  _Fast, cheap and out of control_ , Tech. Rep. (Massachusetts Inst of Tech Cambridge Artificial Intelligence Lab, 1989).
  * Dorigo  _et al._ [2020] M. Dorigo, G. Theraulaz, and V. Trianni, Science Robotics 5 (2020).
  * Floreano and Lipson [2021] D. Floreano and H. Lipson, From individual robots to robot societies (2021).
  * Dorigo  _et al._ [2021] M. Dorigo, G. Theraulaz, and V. Trianni, Proceedings of the IEEE 109, 1152 (2021).
  * Werfel  _et al._ [2014] J. Werfel, K. Petersen, and R. Nagpal, Science 343, 754 (2014).
  * Hauert  _et al._ [2008] S. Hauert, J.-C. Zufferey, and D. Floreano, Autonomous Robots 26, 21 (2008).
  * Virágh  _et al._ [2014] C. Virágh, G. Vásárhelyi, N. Tarcai, T. Szörényi, G. Somorjai, T. Nepusz, and T. Vicsek, Bioinspiration & biomimetics 9, 025012 (2014).
  * Vásárhelyi  _et al._ [2018] G. Vásárhelyi, C. Virágh, G. Somorjai, T. Nepusz, A. E. Eiben, and T. Vicsek, Science Robotics 3 (2018).
  * McGuire  _et al._ [2019] K. McGuire, C. De Wagter, K. Tuyls, H. Kappen, and G. C. de Croon, Science Robotics 4 (2019).
  * Gomes  _et al._ [2016] J. Gomes, M. Duarte, P. Mariano, and A. L. Christensen, in  _International Conference on Parallel Problem Solving from Nature_ (Springer, 2016) pp. 591–601.
  * Zahadat and Schmickl [2016] P. Zahadat and T. Schmickl, Adaptive Behavior 24, 87 (2016).
  * Berlinger  _et al._ [2021a] F. Berlinger, M. Gauci, and R. Nagpal, Science Robotics 6 (2021a).
  * Schranz  _et al._ [2020] M. Schranz, M. Umlauft, M. Sende, and W. Elmenreich, Frontiers in Robotics and AI 7, 36 (2020).
  * Rubenstein  _et al._ [2014] M. Rubenstein, A. Cornejo, and R. Nagpal, Science 345, 795 (2014).
  * Slavkov  _et al._ [2018] I. Slavkov, D. Carrillo-Zapata, N. Carranza, X. Diego, F. Jansson, J. Kaandorp, S. Hauert, and J. Sharpe, Science Robotics 3, eaau9178 (2018).
  * Wang  _et al._ [2021] G. Wang, T. V. Phan, S. Li, M. Wombacher, J. Qu, Y. Peng, G. Chen, D. I. Goldman, S. A. Levin, R. H. Austin, and L. Liu, Physical Review Letters 126, 108002 (2021).
  * Li  _et al._ [2019] S. Li, R. Batra, D. Brown, H.-D. Chang, N. Ranganathan, C. Hoberman, D. Rus, and H. Lipson, Nature 567, 361 (2019).
  * Oliveri  _et al._ [2021] G. Oliveri, L. C. van Laake, C. Carissimo, C. Miette, and J. T. Overvelde, Proceedings of the National Academy of Sciences 118 (2021).
  * Deblais  _et al._ [2018] A. Deblais, T. Barois, T. Guerin, P. H. Delville, R. Vaudaine, J. S. Lintuvuori, J. F. Boudet, J. C. Baret, and H. Kellay, Physical Review Letters 120, 188002 (2018).
  * Boudet  _et al._ [2021a] J. F. Boudet, J. Lintuvuori, C. Lacouture, T. Barois, A. Deblais, K. Xie, S. Cassagnere, B. Tregon, D. B. Brückner, J. C. Baret, and H. Kellay, Science Robotics 6, eabd0272 (2021a).
  * Berg [2000] H. C. Berg, Physics Today 53, 24 (2000).
  * Wensink  _et al._ [2012] H. H. Wensink, J. Dunkel, S. Heidenreich, K. Drescher, R. E. Goldstein, H. Löwen, and J. M. Yeomans, Proceedings of the National Academy of Sciences 109, 14308 (2012), arXiv:abs/1302.5277 [arxiv.org] .
  * Xu  _et al._ [2019] H. Xu, J. Dauparas, D. Das, E. Lauga, and Y. Wu, Nature Communications 10, 1 (2019).
  * Costerton  _et al._ [1999] J. W. Costerton, P. S. Stewart, and E. P. Greenberg, Science 284, 1318 (1999).
  * Shishkov  _et al._ [2019] O. Shishkov, M. Hu, C. Johnson, and D. L. Hu, Journal of The Royal Society Interface 16, 20180735 (2019).
  * Procaccini  _et al._ [2011] A. Procaccini, A. Orlandi, A. Cavagna, I. Giardina, F. Zoratto, D. Santucci, F. Chiarotti, C. K. Hemelrijk, E. Alleva, G. Parisi, and C. Carere, Animal Behaviour 82, 759 (2011).
  * BBC [2015] BBC, Feeding frenzy, from the hunt: Hunger at sea (oceans) (2015).
  * Moussaïd  _et al._ [2011] M. Moussaïd, D. Helbing, and G. Theraulaz, Proceedings of the National Academy of Sciences 108, 6884 (2011).
  * Deseigne  _et al._ [2010] J. Deseigne, O. Dauchot, and H. Chaté, Physical review letters 105, 098001 (2010).
  * Vicsek and Zafeiris [2012] T. Vicsek and A. Zafeiris, Physics Reports 517, 71 (2012).
  * Lam  _et al._ [2015] K.-D. N. T. Lam, M. Schindler, and O. Dauchot, New Journal of Physics 17, 113056 (2015).
  * Bechinger  _et al._ [2016] C. Bechinger, R. Di Leonardo, H. Löwen, C. Reichhardt, G. Volpe, and G. Volpe, Reviews of Modern Physics 88, 045006 (2016), arXiv:1602.00081 .
  * Bricard  _et al._ [2013] A. Bricard, J.-b. Caussin, N. Desreumaux, O. Dauchot, and D. Bartolo, Nature 503, 95 (2013).
  * Boudet  _et al._ [2021b] J. Boudet, J. Lintuvuori, C. Lacouture, T. Barois, A. Deblais, K. Xie, S. Cassagnere, B. Tregon, D. Brückner, J. Baret, _et al._ , Science Robotics 6 (2021b).
  * Pfeifer and Bongard [2006] R. Pfeifer and J. Bongard,  _How the body shapes the way we think: a new view of intelligence_ (MIT press, 2006).
  * Pfeifer and Gómez [2009] R. Pfeifer and G. Gómez, in  _Creating brain-like intelligence_ (Springer, 2009) pp. 66–83.
  * Watson  _et al._ [2002] R. A. Watson, S. G. Ficici, and J. B. Pollack, Robotics and Autonomous Systems 39, 1 (2002).
  * Bredeche  _et al._ [2018] N. Bredeche, E. Haasdijk, and A. Prieto, Frontiers in Robotics and AI 5, 12 (2018).
  * Fontbonne  _et al._ [2020] N. Fontbonne, O. Dauchot, and N. Bredeche, in  _2020 IEEE Congress on Evolutionary Computation (CEC)_ (2020) pp. 1–8.
  * Bredeche and Fontbonne [2022] N. Bredeche and N. Fontbonne, Philosophical Transactions of the Royal Society B 377, 20200309 (2022).
  * Allan  _et al._ [2019] D. Allan, C. van der Wel, N. Keim, T. A. Caswell, D. Wieker, R. Verweij, C. Reid, Thierry, L. Grueter, K. Ramos, Apiszcz, Zoeith, R. W. Perry, F. Boulogne, P. Sinha, Pfigliozzi, N. Bruot, L. Uieda, J. Katins, H. Mary, and A. Ahmadia, soft-matter/trackpy: Trackpy v0.4.2 (2019).
  * Fodor and Cristina Marchetti [2018] É. Fodor and M. Cristina Marchetti, Physica A: Statistical Mechanics and its Applications 504, 106 (2018), arXiv:1708.08652 .
  * Collins  _et al._ [2005] S. Collins, A. Ruina, R. Tedrake, and M. Wisse, Science 307, 1082 (2005).
  * Giomi  _et al._ [2013] L. Giomi, N. Hawley-Weld, and L. Mahadevan, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 469, 20120637 (2013), arXiv:1302.5952 .
  * Dauchot and Démery [2019] O. Dauchot and V. Démery, Physical Review Letters 122, 1 (2019), arXiv:1810.13303 .
  * Tailleur and Cates [2008] J. Tailleur and M. E. Cates, Physical Review Letters 100, 218103 (2008), arXiv:abs/0803.1069 [arxiv.org] .
  * Shahrokhi and Becker [2016] S. Shahrokhi and A. T. Becker, IEEE International Conference on Automation Science and Engineering 2016-November, 561 (2016).
  * Camazine  _et al._ [2003] S. Camazine, J.-L. Deneubourg, N. Franks, J. Sneyd, G. Theraulaz, and E. Bonabeau,  _Self-organization in biological systems_ (Princeton University Press, 2003).
  * Bonabeau  _et al._ [1997] E. Bonabeau, G. Theraulaz, J.-L. Deneubourg, S. Aron, and S. Camazine, Trends in ecology & evolution 5 (1997).
  * Şahin [2004] E. Şahin, in  _International workshop on swarm robotics_ (Springer, 2004) pp. 10–20.
  * Bayindir [2016] L. Bayindir, Neurocomputing 172, 292 (2016).
  * Savoie  _et al._ [2018] W. Savoie, S. Cannon, J. J. Daymude, R. Warkentin, S. Li, A. W. Richa, D. Randall, and D. I. Goldman, Artificial Life and Robotics 23, 459 (2018), arXiv:1711.01327 .
  * Savoie  _et al._ [2019] W. Savoie, T. A. Berrueta, Z. Jackson, A. Pervan, R. Warkentin, S. Li, T. D. Murphey, K. Wiesenfeld, and D. I. Goldman, Science robotics 4, 10.1126/scirobotics.aax4316 (2019).
  * Hamann  _et al._ [2008] H. Hamann, H. Worn, K. Crailsheim, and T. Schmickl, in  _2008 IEEE/RSJ International Conference on Intelligent Robots and Systems_ (IEEE, 2008) pp. 1415–1420.
  * Schmickl  _et al._ [2009] T. Schmickl, R. Thenius, C. Moeslinger, G. Radspieler, S. Kernbach, M. Szymanski, and K. Crailsheim, Autonomous Agents and Multi-Agent Systems 18, 133 (2009).
  * Kernbach  _et al._ [2009] S. Kernbach, R. Thenius, O. Kernbach, and T. Schmickl, Adaptive Behavior 17, 237 (2009).
  * Arvin  _et al._ [2011] F. Arvin, K. Samsudin, A. R. Ramli, and M. Bekravi, International Journal of Computational Intelligence Systems 4, 739 (2011).
  * Arvin  _et al._ [2014a] F. Arvin, A. E. Turgut, F. Bazyari, K. B. Arikan, N. Bellotto, and S. Yue, Adaptive Behavior 22, 189 (2014a).
  * Arvin  _et al._ [2014b] F. Arvin, A. E. Turgut, N. Bellotto, and S. Yue, in  _International Conference in Swarm Intelligence_ (Springer, 2014) pp. 1–8.
  * Schmickl and Hamann [2011] T. Schmickl and H. Hamann, Bio-inspired computing and communication networks , 95 (2011).
  * Smoluchowski [1916] M. V. Smoluchowski, Physik Zeitcher, 17, 557 (1916).
  * Uckert  _et al._ [2020] K. Uckert, A. Parness, N. Chanover, E. J. Eshelman, N. Abcouwer, J. Nash, R. Detry, C. Fuller, D. Voelz, R. Hull, D. Flannery, R. Bhartia, K. S. Manatt, W. J. Abbey, and P. Boston, Astrobiology 20, 1427 (2020).
  * Fiorello  _et al._ [2020] I. Fiorello, E. Del Dottore, F. Tramacere, and B. Mazzolai, Bioinspiration and Biomimetics 15, 10.1088/1748-3190/ab7416 (2020).
  * Berlinger  _et al._ [2021b] F. Berlinger, M. Gauci, and R. Nagpal, Science Robotics 6, 1 (2021b).
  * Trianni  _et al._ [2003] V. Trianni, R. Groß, T. H. Labella, E. Şahin, and M. Dorigo, in  _European Conference on Artificial Life_ (Springer, 2003) pp. 865–874.
  * Dorigo  _et al._ [2004] M. Dorigo, V. Trianni, E. Şahin, R. Groß, T. H. Labella, G. Baldassarre, S. Nolfi, J.-L. Deneubourg, F. Mondada, D. Floreano, _et al._ , Autonomous Robots 17, 223 (2004).
  * Soysal  _et al._ [2007] O. Soysal, E. Bahçeci, and E. Şahin, Turkish Journal of Electrical Engineering & Computer Sciences 15, 199 (2007).
  * Nolfi and Floreano [2000] S. Nolfi and D. Floreano,  _Evolutionary Robotics: the Biology, Intelligence, and Technology_ (MIT Press, Cambridge MA, USA, 2000).
  * Doncieux  _et al._ [2015] S. Doncieux, N. Bredeche, J.-B. Mouret, and A. Eiben, Frontiers in Robotics and AI 2, 1 (2015).
  * Kaelbling  _et al._ [1996] L. P. Kaelbling, M. L. Littman, and A. W. Moore, Journal of artificial intelligence research 4, 237 (1996).
  * Feinerman  _et al._ [2018] O. Feinerman, I. Pinkoviezky, A. Gelblum, E. Fonio, and N. S. Gov, Nature Physics 14, 683 (2018).
  * Tarapore  _et al._ [2020] D. Tarapore, R. Groß, and K.-P. Zauner, Frontiers in Robotics and AI 7, 83 (2020).
  * Whiten [2021] A. Whiten, Science 372 (2021).
  * Ben Zion  _et al._ [2022a] M. Y. Ben Zion, Y. Caba, A. Modin, and P. M. Chaikin, Nature Communications 13, 184 (2022a), arXiv:2012.15087 .
  * Edelstein  _et al._ [2014] A. D. Edelstein, M. A. Tsuchida, N. Amodaj, H. Pinkard, R. D. Vale, and N. Stuurman, Journal of Biological Methods 1, e10 (2014).
  * Rueden  _et al._ [2017] C. T. Rueden, J. Schindelin, M. C. Hiner, B. E. DeZonia, A. E. Walter, E. T. Arena, and K. W. Eliceiri, BMC Bioinformatics 18, 529 (2017).
  * Oppenheimer  _et al._ [2019] N. Oppenheimer, D. B. Stein, and M. J. Shelley, Physical Review Letters 123, 148101 (2019), arXiv:1903.00940 .
  * Oppenheimer  _et al._ [2022] N. Oppenheimer, D. B. Stein, M. Y. Ben Zion, and M. J. Shelley, Nature Communications 13, 804 (2022), arXiv:2103.00296 .
  * Ben Zion  _et al._ [2022b] M. Y. Ben Zion, A. Modin, and P. M. Chaikin, Arxiv cond-mat.soft doi.org/10.48550/arXiv.2203.11051 (2022b), arXiv:2203.11051 .
  * Ben Zion  _et al._ [2017] M. Y. Ben Zion, X. He, C. C. Maass, R. Sha, N. C. Seeman, and P. M. Chaikin, Science 358, 633 (2017).
  * Zhu  _et al._ [2021] G. Zhu, M. Hannel, R. Sha, F. Zhou, M. Y. Ben Zion, Y. Zhang, K. Bishop, D. Grier, N. Seeman, and P. Chaikin, Proceedings of the National Academy of Sciences 118, e2023508118 (2021).
  * Howse  _et al._ [2007] J. R. Howse, R. A. L. Jones, A. J. Ryan, T. Gough, R. Vafabakhsh, and R. Golestanian, Physical Review Letters 99, 048102 (2007), arXiv:0706.4406 .
  * Ben Zion  _et al._ [2020] M. Y. Ben Zion, Y. Caba, R. Sha, N. C. Seeman, and P. M. Chaikin, Soft Matter 16, 4358 (2020).
  * Yoones  _et al._ [2022] M. Yoones, M. Y. Ben Zion, O. Dauchot, and N. Bredeche, in  _GECCO 2022 - Proceedings of the 2022 Genetic and Evolutionary Computation Conference_ (Association for Computing Machinery, 2022) pp. 104–112.
  * Note [1] Note that using the policy directly could also be useful if two robots have similar reward but one has superior policy (say more energetically efficient). This can also be directly introduced into the reward.
  * Note [2] In our case $I$ is an array of bytes and $M=250$.
  * Note [3] On the Kilobot platform, the message size is 8 bytes.
  * Note [4] In our case, $\rho$ will be a single byte.
  * Ben Zion  _et al._ [2021] M. Y. Ben Zion, N. Bredeche, and O. Dauchot, Arxiv cond-mat.soft , 1 (2021), arXiv:2111.06953 .
  * Schnitzer [1993] M. J. Schnitzer, Physical Review E 48, 2553 (1993).

##  VII Materials and Methods

###  VII.1 Exoskeleton design and manufacturing

####  VII.1.1 Exoskeleton Manufacturing

Exoskeletons were 3D printed using either one of the following 3D printers and
materials to give similar results: startAsys Objet350 Connex3 printer, using
veroClear (modulus of elasticity $E=1-2GPa$, density $\rho=1.2g/cm^{3}$), and
3D systems projet 2500plus printer using VisiJet M2R-CL (MJP) (density
$\rho=1.14g/cm^{3}$, Elastic modulus $E\approx 1GPa$).

###  VII.2 Data acquisition and experimental setups

####  VII.2.1 Phototaxis experiments

Phototaxis experiments were performed on a $5\;\rm{mm}$ milky plexiglass sheet
placed on the floor. A circular region 6% of the arena was lit using a
projector (EPSON EB-1795F), at RGB = (10,255,255), at approximately the
spectral sensitivity peak (570nm) of the kilobot’s light sensor (TEPT5700,
Vishay Semiconductors). To reduce interference of ambient light with the light
sensor, the arena was lit using four desk lamps directed at the surrounding
walls (for a homogeneous diffusive illumination) and covered with a red
cellophane (where the robot’s light sensor’s sensitivity is at a minimum). A
red cellophane was also covering the camera used for acquisition
(PIXELINK.PL-D734MU), to reduce the saturation from the lit region, and
imaging the robots. Image acquisition was carried out using $\mu$icromanager
[78], at 2 frames per second. Robots are counted when fully in the light and
results shown in Figs. 6F, 7J are the average of four realizations with error
bars being the standard deviation.

####  VII.2.2 Code used

Robots were programmed to phototaxis using variations of the following two
codes: 1. phototaxisAdhoc.c — robots stand when their photoperceptron is
activated given a predefined threshold. 2. phototaxisLearning.c — Robot stand
when photoperceptron is activated but light threshold are randomly
intitialized and evolve through the learning algorithm.

####  VII.2.3 Inclined plane experiments

Coupling between external force and robot’s orientation was measured by
letting robots run on a plane made of $5\;\rm{mm}$ thick plexiglass placed at
an incline, and imaged using SonyAlphaS7 camera, with video rate image
acquisition at 30 frames per second.

####  VII.2.4 Data analysis

Raw images were preprocess using ImageJ [79], or ffmpeg, followed by particle
locating using trackpy package [44], or custom code, and trajectories were
linked using trackpy [44].

####  VII.2.5 Simulating morphobots

Computer simulations were carried out by adding a stochastic term to the Eqs.
1, 2 and using using 5th order Runge-Kutta for the time propagation of
Brownian dynamics of active soft-discs by adopting code from previous
publications [80, 81, 82].

##  VIII Supplementary Materials

###  VIII.1 Exoskeleton design

Exoskeletons were designed as a round chassis (diameter $d=4.8\;\rm{cm}$) with
an off-center cavity where the kilobot is pressed in. The exoskeleton stands
on three, $L=2\;\rm{cm}$ tall legs — one leg is stiff and round, and two
opposing legs that are flexible and with a rectangular cross-section
(thickness $T=1\;\rm{mm}$, and width $W=0.6\;\rm{cm}$, see Fig. 2 in main text
as well as previous work on elastic beams [83, 84]). The flexible legs were
designed to have their first natural vibration near resonance with the
vibration motors frequency (Pololu $10X2.0\;\rm{mm}$ vibration motors,
$f_{motor}\approx 250\;\rm{Hz}$). This was done by estimating the legs to be
elastic beams which are described dynamically using the Euler–Bernoulli beam
theory for a homogeneous thin beam

$EI\frac{\partial^{4}h}{\partial x^{4}}=-\mu\frac{\partial^{2}h}{\partial
t^{2}},$ (8)

where $h$ is the deflection, $E$ is the elastic modulus, $\mu$ is the linear
mass density (given by $\mu=TW\rho$), and $I$ is the beam moment of inertia
(given by $I=\frac{W\times T^{3}}{12}$). In Fourier space, Eq. 8 becomes

$\omega=\sqrt{\frac{EI}{\mu}}k^{2},$ (9)

where $\omega$ is the angular frequency, and $k$ the wave number. The
wavelength of the first harmonic $\lambda_{1}$ of a beam of length $L$ fixed
at one end is $\lambda_{1}=4L$, along with Eq. 9 allows us to find that the
natural first harmonic of the leg is to leading order

$f_{1}=\frac{\pi}{16\sqrt{6}}\sqrt{\frac{E}{\rho}}\frac{T}{L^{2}}.$ (10)

Note the Eq. 10 is independent of the width. Given the materials and geometry
used (see previous section), $f_{1}\approx 250\;\rm{Hz}\approx f_{motor}$.

###  VIII.2 Individual robot dynamics

####  VIII.2.1 Speed of an individual persistent particle

The effective diffusion constant of a run-and-tumble particle, that covers a
distance $l_{p}$, during the time interval $\tau_{RT}$ is
$D_{eff}\approx\frac{1}{4}l_{p}^{2}/\tau_{RT}$ [85]. In the run and tumble
mode, each Morphobot is programmed to run for $\tau_{run}=2\;\rm{s}$ at its
nominal speed of $v=5\;\rm{cm}/\rm{s}$, then perform a tumble where it pivots
on the average for $\tau_{pivot}=4\;\rm{s}$. The step size is thus
$l_{p}=v\cdot\tau_{run}\approx 10\;\rm{cm}$, and the total duration is
$\tau_{RT}=6\;\rm{s}$. The resulting effective diffusion constant is
$D_{eff}\approx\frac{1}{4}\left(10\;\rm{cm}\right)^{2}/6\;\rm{s}=4\;\rm{cm}^{2}/\rm{s}$.

####  VIII.2.2 Particle orientation as a function of trajectory length under
constant force

The dynamics of the orientation, $\theta$, relative to the positive $\hat{x}$
direction, for an active particle in the over-damped limit at the absence of
noise is found to be that of a simple overdamped pendulum (Eq. 3 in main text)
[86]:

$\dot{\theta}=-\kappa\mu f\rm{sin}\theta.$ (11)

It is helpful to formulate Eq. 11 as a function of $s$, the trajectory
traveled. In 2D, $s$ is defined locally by the Euclidean distance:

$ds^{2}=dx^{2}+dy^{2}$ (12)

Dividing Eq. 12 by $dt^{2}$ reminds us that

$\left(\frac{ds}{dt}\right)^{2}=\left(\frac{dx}{dt}\right)^{2}+\left(\frac{dy}{dt}\right)^{2}=v^{2}$
(13)

We can find $v^{2}$ by multiplying Eq. 1 (main text) by itself:

$v^{2}=v_{0}^{2}+\left(\mu f\right)^{2}+v_{0}\mu f\rm{cos}\theta.$ (14)

Using the chain rule, $\frac{d\theta}{dt}=\frac{d\theta}{ds}\frac{ds}{dt}$,
and plugging into Eqs. 13 and 14, we get an ODE describing $\theta$ as a
function of $s$, the arclength:

$\frac{d\theta}{ds}=\frac{\kappa
f\rm{cos}\theta}{\sqrt{v_{0}^{2}+f^{2}+v_{0}f\rm{cos}\theta}}.$ (15)

Equation 15 can be solved exactly, however it is more instructive to examine
the limit of $\frac{v_{0}}{\mu f}\ll 1$ which then simplifies to

$\frac{d\theta}{ds}=\kappa\rm{cos}\theta.$ (16)

For $\theta\left(s=0\right)=\frac{\pi}{2}$, Eq. 16 gives Eq. 4 in the main
text:

$\theta\left(s\right)=2\rm{atan}\left(e^{-\kappa s}\right).$ (17)

###  VIII.3 Phototaxis without learning

####  VIII.3.1 Phototaxis Arena

Fig. S1: Phototaxis arena.

####  VIII.3.2 Photoperceptron

Each robot has a photoperceptron encoded to measure environmental cues, and
take a specific action given that the perceptron fired or not. The minimal
architecture for the photoperceptron implemented has a neural network of a
single neuron (perceptron), with two inputs, bias $b=x_{0}$, and the locally
measure light power $p=x_{1}$, and their associated weights $w_{0}$ and
$w_{1}$, which together effectively define the threshold for firing. The
perceptron output is $p$, a logistic function of the form

$p=\frac{1}{1+e^{-h}}.$ (18)

where $h=\sum_{i=0}^{1}w_{i}x_{i}$

####  VIII.3.3 Sense-act cycle

The sense-act cycle uses the robot’s local light intensity measurement, where
the robot chooses between two meta-behaviours: run and tumble or walk and
tumble, given the photoperceptron is below threshold $P<0.5$, or above
threshold $P\geq 0.5$, respectively. In both cases the motion is a sequence of
a moving phase (run/walk) and tumble phase. In the tumble phase, the robot
randomly chooses a direction (clockwise or counterclock wise) and a random
duration with a mean of 6 seconds. Randomizing the duration compensates for
the robots’ internal dynamical variability, and environmental variability. For
example, robots with stronger/weaker motors will take less/more time to tumble
and complete the same reorientation. Similarly, a robot near a wall (or
another robot) or in solitude. When in run mode, the robots sets both motor
values to high, approximately moving at a straight line (persistence length
much larger than the robot’s size), at its nominal speed $V_{0}=v_{0}$. When
walking, the robot moves for a duration $\tau_{\rm{walk}}$, and then stands
for a duration $\tau_{\rm{stand}}$. The ratio between the two controls the
mean speed of the robot when walking
$V_{1}=\frac{\tau_{\rm{walk}}}{\tau_{\rm{walk}}+\tau_{\rm{stand}}}V_{0}$. When
$\tau_{\rm{walk}}=0$ the robot stands when photoperceptron fires.

####  VIII.3.4 Phototaxis rate for low concentration non interacting robots
estimated using first arrival time

The rate of robot arrival to the light spot at early times (where the lit
region is mostly empty), can be found from the reaction rate constant, $k$,
using Smoluchowski first arrival given a number concentration concentration
$c$. In 2D the mean first arrival time, $\tau_{FA}=1/kc$, of a Brownian
particles with diffusion constant $D$, arriving at a target of size $b$ in an
arena of diameter $d$ is given by

$k=\frac{2\pi D}{\rm{log}\frac{d}{b}}.$ (19)

The rate at which robot are subtracted from the dark region
$\frac{dN_{bulk}}{dt}$, is given from a first order reaction rate

$\frac{dN_{bulk}}{dt}=-kN_{bulk},$ (20)

which solves readily to $N_{bulk}=A\rm{exp}\\{-kt\\}$. Given an initial number
of robots, $N_{0}$, the number of robots in the light, $N$, is given from the
robot conservation $N_{0}=N+N_{bulk}$,

$N\left(t\right)=N_{0}\left(1-e^{-kt}\right),$ (21)

which is plotted in Fig. 6.

###  VIII.4 Collective learning in a sterically interacting swarm

In this final section we derive an analytic link between the behaviour of
individuals in a swarm and the convergence of a decentralized learning scheme.
We first establish empirically that a swarm where robots are allowed to
collide has good mixing, a property required to establish the individual to
collective correspondence. We then cast the collective learning as a dynamical
process where robots in the swarm can exchange their state upon encounter. And
finally, we show that given the reward function, $\rho$, as defined in the
main text (Eq. 6), a randomly initialized swarm will converge on a successful
phototaxis strategy.

####  VIII.4.1 Individual dynamics of pre-calibrated robots

The mean time a robot spends in the light spot, $\Phi$, depends on three
quantities: 1. The ratio between the speed of the robot in the light $V_{1}$
to its speed outside the light, $V_{0}$; 2. The area fraction of the light
spot, $\sigma$; 3. Whether or not the robot has the correct threshold for
phototaxis. Allowing individual, pre-set robots in the light patterned arena
to perform phototaxis shows two distinct behaviours: robots with a correct
light threshold for phototaxis spend more time in the lit region than robots
with wrong threshold (either too high or too low, see Fig. S2B) for which time
in light is as good as random $\Phi_{W}\approx\sigma$.

Fig. S2: Dynamics and performance of individual pre-programmed robots. A Mean
square displacement of the two meta behaviours: the fast run-and-tumble
(pink), or the much slower walk-and-tumble (red), with a fixed speed ratio
between run and walk. The run-and-tumble behaviour transitions from ballistic
($\propto t^{2}$ to linear $\propto t^{1}$ at approximately one persistent
time $\tau\equiv t/\tau_{p}\approx 1$. Inset shows typical, 20 minutes long
trajectories of both run-and tumble (pink) and walk-and-tumble (red) inside
the circular 150 cm wide arena. B The time fraction in the light $\Phi$ of
individual robot shows that a robot with too high or too low light threshold
performs as good as random, while a robot that has a correct policy, is
responsive to the light source. The policy of each robot is determined by
weights of the photo-perceptron, setting the light threshold to activate the
photo-perceptron (solid curves).

####  VIII.4.2 Collective dynamics of a pre-calibrated swarm

To test how the speed ratio $V_{1}/V_{0}$ effects the number of bots in the
light, phototaxis experiments were done by increasing the size of the lit
region to 25% of the arena. Experiments were performed with either $64$ bots
or $32$ bots (see inset in Fig. S3B). The speed ratio was set by controlling
the $\tau_{\rm{stand}}$ as described in Section VIII.3.3, changing the speed
from standing in light $V_{1}/V_{0}=0$ to effectively being agnostic to the
light $V_{1}/V_{0}=1$ (see Fig. S3B). The relative number of robots in the
light was tracked over the course of up to 2 hours, and the mean and standard
deviation of the last $10\%$ of the experiment duration were used to evaluate
the steady state fraction of the swarm in the light, $\mathcal{F}_{ss}$.

Fig. S3: Collective performances of pre-programmed robots. A Swarm fraction in
the light for a fixed speeds ratio ($V_{1}/V_{0}=1/15$) different
initializations: correctly calibrated swarm (black), low (red) high (pink)
threshold, and randomly initialised swarm (gray). Insets show typical
trajectories showing both low and high are agnostic to the light spot, while
the correctly, ad hoc, initialized robot, dwells longer in the light spot,
increasing its reward $\rho$. B The run to walk speed ratios determine the
collective ability of a swarm to phototaxis, and a steady-state flux balance
model (see Eq. 27) quantitatively predicts the percentage of the swarm in the
light, with no fitting parameters.

####  VIII.4.3 Collective decentralized learning

The collective learning was implemented by deploying a randomly initialized
swarm of Morphobots that follow the algorithm detailed below (see Algorithm 1)
to give the results presented in Fig.7 in the main text.

Data:

$i$ : the unique identifier of the current robot,

$T$ : evaluation time,

$\pi$ : Policy function,

$\bm{w^{i}}$ : Random uniform initialisation of policy parameters

$R[T]$ : Empty reward buffer of size $T$,

$r$ : Current light intensity,

$\rho^{i}$ : performance self-assessment during $T$,

$\bm{\mathrm{a}}$ : Null action vector,

$\bm{\mathrm{o}}$ : Null observation vector

1 begin

2 $t=0$

3 while  _True_ do

4 $\bm{\mathrm{o}},r$ = sense()

5 $R[t\mod T]=r$

6 $\bm{\mathrm{a}}=\pi(\bm{\mathrm{o}}|\bm{w^{i}}$)

7 act($\bm{\mathrm{a}}$)

8 if  _$t >T$_ then

9 $\rho^{i}=\sum_{k=0}^{T-1}R[k]$

10 broadcast($\bm{w^{i}}$,$\rho^{i}$)

11 if  _$new\\_message$_ then

12 $\bm{w^{j}}$,$\rho^{j}$ = $decode\\_message$

13 if  _$\rho^{j} >\rho^{i}$_ then

14 $w^{i}=w^{j}$

15 end if

16

17 end if

18

19 end if

20 $t=t+1$

21 end while

22

23 end

24

Algorithm 1 The phototactic-HIT algorithm

Each robot in the population runs Algorithm 1. This Algorithm builds from the
HIT algorithm originally presented in [42, 87], with two important
modifications. Firstly, the algorithm is designed with phototaxis as an
objective (the $\rho$ objective function on line 9 measures the amount of
light measured in the last $T$ time steps). Secondly, transfer of all control
parameters is performed, which implies that whenever a robot receives a
parameter set which is deemed to be performing better than its own, this
parameter set is fully copied and overwrites existing parameter values.
Mutation is not used in this implementation, as the number of robots used
($64$) and the low dimensionality of the search space ($\mathbb{R}^{2}$)
allows for a sufficient amount of behavioral diversity in the initial
population of robots (see [42] for a comprehensive study of transfer and
mutation rate and operators).

It should also be noted that performance assessment ($\rho^{i}$) is not reset
to zero after a parameter update, which implies that performance assessment of
a newly updated policy is under-estimated for $T$ time steps. While this may
temporarily lead to theoretically best-performing control parameters being
overwritten before $T$ steps elapsed, the elitist selection scheme ensures
that the original best-performing individual is never lost.

Finally, note that in the algorithm the observable $o$ contains information
about the current light intensity $r$. They are separated for semantic reasons
as $o$ is used as an input value for the policy $\pi$, while $r$ is used to
update the performance self-assessment $\rho^{i}$.

####  VIII.4.4 Collective learning as a dynamical process

The goal is to link policy and environment to find a route for a successful
decentralized learning scheme on the average. The time evolution of the
probability $p^{i}_{C}$, respectively $p^{i}_{W}$, of finding a robot $i$ with
a correct, respectively wrong, policy is described by the following master
equation :

$\frac{dp^{i}_{C}}{dt}=\omega^{i}_{C,W}p^{i}_{W}-\omega^{i}_{W,C}p^{i}_{C},$
(22)

where $p^{i}_{W}=1-p^{i}_{C}$ and $\omega^{i}_{C,W}$, respectively
$\omega^{i}_{W,C}$, are the transition rates at which the robot $i$ with a
wrong policy turns into a correct policy and conversely. We assume space is
homogeneous and identify the probability of a given policy for a given robot
with the concentration of robots, $c_{C,W}$, having that policy. Since
transition between states happens only through local communication, the
transition rates are proportional to the robot encounter rate constant, the
latter being independent of the policies of the encountering robots. Under the
above hypothesis, the transition rates are proportional to the concentrations
and the transition elements read $\omega_{C,W}\approx kf_{CW}c_{C}$, and
symmetrically $\omega_{W,C}\approx kf_{WC}c_{W}$, where $k$ is the encounter
rate, and $f_{CW}$, respectively $f_{WC}$, is the transfer probability of $W$,
respectively $C$, becoming $C$, respectively $W$, upon encounter. As a result,

$\frac{dc_{C}}{dt}=k\left(f_{CW}-f_{WC}\right)c_{C}(c_{0}-c_{C}),$ (23)

where $c_{0}$ is the overall concentration of robots in the arena (leading to
Eq. 7 in the main text).

To achieve positive mean collective learning, $dc_{C}/dt>0$, the factors of
the product in Eq. 23 has to be non negative, which amounts to

$f_{WC}>f_{CW}$

As learning is decentralized, $f$ is an embedded function, and can depend only
on information locally accessible to the robot. This includes the robot’s
policy (here, the weights of its neural network, $\\{w_{i}\\}$), its
measurement history, and the message received. Here we exclude a direct
functional dependence of $f$ upon the policy as it will artificially bias the
learning to a given trait (say turn left) instead of a desired fitness (say
measure light). 111Note that using the policy directly could also be useful if
two robots have similar reward but one has superior policy (say more
energetically efficient). This can also be directly introduced into the
reward. Therefore $f$ will depend on some function of the history of its own
sensor input and the history of an encountered robot. In our minimal system
the measurement history contains the $M$ most recent light intensity
measurements, $I_{i\in 1..M}$ (stored as a rolling buffer, i.e FIFO).222In our
case $I$ is an array of bytes and $M=250$. Since the robots have a finite
bandwidth, in order to allow asynchronous performance, where communication
does not slow down dynamics, message should be kept succinct. This means
robots should only exchange a reduced version of their memory that can be
contained within a message.333On the Kilobot platform, the message size is 8
bytes. This will be a function of the memory, and will be called the reward
function, $\rho=\rho\left(\\{I_{i}\\}\right)$.444In our case, $\rho$ will be a
single byte. In our implementation scheme, robots continuously broadcast a
message with their reward and policy; a robot receiving a message compares the
$B$roadcasted reward, $\rho_{B}$, to its self-reward, $\rho_{S}$, and if
$\rho_{B}>\rho_{S}$ the policy will be deterministically adopted. The question
then becomes what functional form does the local $\rho$ take, to satisfy
global positive learning on the average:
$\langle\rho_{C}\rangle>\langle\rho_{W}\rangle$.

Empirically we found that an ad hoc calibrated swarm at steady state,
individual robots with a correct policy spend more time in the light
$\Phi_{C}>\Phi_{W}$ (see Fig.S2), where the mean time in the light of an
individual robot is measured by
$\Phi=\frac{1}{T}\int_{-T}^{0}dtI\left(t\right)$. Identifying the reward
function as proportional to the mean time in the light, $\rho\propto\Phi$,
allows us to bias the swarm towards learning the correct policy. We do this by
computing the reward as the average of the light intensity buffer,
$\rho=\frac{1}{M}\sum_{i=1}^{M}I_{i}$ (Eq. 6 in the main text).

Recalling that the transfer probability function depends only on the received
and embedded reward, the condition for learning simply becomes that on the
average, correct robots have a greater reward:

$\rho_{C}>\rho_{W}$ (24)

####  VIII.4.5 Mean reward in a well mixed swarm

To satisfy the learning condition presented in the previous section (Eq. 24)
we recall the good mixing property found empirically in previous sections
wherein the locally computed reward can approximate an individual’s time
average, which is a proxy to the global performance
$\rho\propto\Phi\approx\mathcal{F}$ [92]. A swarm with good mixing can be
modeled as a fluid that obeys continuity equation:

$\frac{dC}{dt}=-\nabla J$ (25)

where $C$ is concentration and $J$ is the flux. At steady state the
concentration does not change in time, and so the left hand side of Eq. 25
vanishes $\frac{dC}{dt}=0$. In a system made of a collective with spatially
varying speed, it has been shown that concentrations are inversely
proportional to speeds (see Schnitzer who studied field taxi in bacteria
[93]). For the simple case discussed here, space is divided in two regions,
lit and dark, where robots have speeds $V_{1}$ and $V_{0}$ respectively. This
results in the following relation of concentration of robots in the light
given their speed in the light,

$C_{0}V_{0}=C_{1}V_{1},$ (26)

where $C_{1}$ and $C_{0}$ are the concentrations of the robots in the light
and in the dark respectively. We next derive a link between Eq. 26 and the
definition of the swarm fraction in the light, $\mathcal{F}$ as defined in Eq.
27 the main text. The concentration of robots in the light, $C_{1}$ is the
number of robots in the light over the area of the lit region
$C_{1}=N_{1}/\sigma A$. The concentration of robots in the dark is the
remaining number of robots in the swarm, $N_{0}=N-N_{1}$ over the remaining
area of the arena: $N_{0}/(1-\sigma)A$. Plugging these into Eq. 26 and
rearranging gives,

$\mathcal{F}_{ss}=\frac{1}{\frac{V_{1}}{V_{0}}\left(\frac{1}{\sigma}-1\right)+1},$
(27)

connecting the collective performance (fraction of the swarm in the light), to
individual policies (speed ratio), given the environmental condition (fraction
of the arena that is lit). For the case of an agnostic swarm, $V_{0}=V_{1}$,
the swarm does not perform better than random $\mathcal{F}\approx\sigma$
(consistent with measured performance of individuals Fig. S2 and collectives
Fig. S3). When the robots are responsive to the light, $V_{1}<V_{0}$, they
accumulate in the light spot, and both the individual dynamics (Fig. S2B) and
the collective dynamics (Fig. S3B), are also consistent with the above
derivation with Eq. 27 setting the high bound for the swarm’s performance.
When robots are responsive, the swarm accumulates in the light, and individual
robots spend more time in the light. Since the reward function as defined in
Eq. 6 is proportional to the time a robot spends in the light
$\rho\propto\Phi$, robots that slow down in the light will satisfy the
learning condition (Eq. 24), and their superior policy will spread throughout
the swarm.