\NAT@parfalse

# Purely vision-based collective movement of robots

David Mezey${}^{1,2^{*}}$ Renaud Bastien3 Yating Zheng1,2 Neal McKee5 David
Stoll2
Heiko Hamann${}^{2,4^{**}}$ & Pawel Romanczuk1,

###### Abstract

Collective movement inspired by animal groups promises inherited benefits for
robot swarms, such as enhanced sensing and efficiency. However, while animals
move in groups using only their local senses, robots often obey central
control or use direct communication, introducing systemic weaknesses to the
swarm. In the hope of addressing such vulnerabilities, developing bio-inspired
decentralized swarms has been a major focus in recent decades. Yet, creating
robots that move efficiently together using only local sensory information
remains an extraordinary challenge. In this work, we present a decentralized,
purely vision-based swarm of terrestrial robots. Within this novel framework
robots achieve collisionless, polarized motion exclusively through minimal
visual interactions, computing everything on board based on their individual
camera streams, making central processing or direct communication obsolete.
With agent-based simulations, we further show that using this model, even with
a strictly limited field of view and within confined spaces, ordered group
motion can emerge, while also highlighting key limitations. Our results offer
a multitude of practical applications from hybrid societies coordinating
collective movement without any common communication protocol, to advanced,
decentralized vision-based robot swarms capable of diverse tasks in ever-
changing environments.

## One-Sentence Summary

We present a purely vision-based robot swarm achieving ordered group motion
even without central information or communication.

## Keywords

swarm robotics ,  visual perception ,  collective movement ,  flocking ,
decentralized design

## Introduction

Collective movement is universally present in groups. It is observed in
animals across different species and sizes showing fascinating patterns in
space and time with remarkable precision—from fish swirling in schools 1,
starlings flying in synchronised murmurations 2 or insects marching in
organized bands 3. Collective movement offers a wide range of benefits for
animal groups, such as improved sensing 4, higher foraging efficiency 5, 6 or
reduced risk of predation 7 among others 8, 9. In promise of inheriting its
advantages in engineered swarms of robots, flocking remains a target behavior
of high interest for bio-inspired swarm robotic applications including
environmental monitoring 10, 11, underwater missions 12, 13, automation of
agricultural tasks 14, 15, search and rescue in emergency scenarios 16, or
even enhanced exploration of the surface of Mars 17.

Swarm robotic systems often adopt heuristic rules—designed with a bird’s-eye
perspective—to foster emergent collective movement in mobile robots. These
include aligning with nearby robots, following zonal rules based on distance,
or moving towards the group’s center of mass. Even though these strategies
might not centralize control over the swarm, they often rely on globally or
locally shared information, which is frequently collected, processed or
distributed from a central node 18, 19, 20, 21, 22. Examples include distances
between robots, relative positions, absolute GPS coordinates, and directional
data. Moreover, even when information is gathered locally, it is often
communicated between robots through radio, infrared (IR) signals or other
channels 23, 24, 25, 26, 27, 28. Relying on global knowledge or on direct
communication between robots comes with unavoidable costs: firstly, the swarm
can solely operate in environments where centrally processed knowledge is
reliably accessible at any time, hence becomes vulnerable to tampered or
missing information; secondly, the group becomes sensitive to jammed
communication channels 29 and to communication delays 30 or outages 31.
Furthermore, direct information sharing between robots is only possible with
compatible communication systems.

Social animals, in contrast to artificial systems, robustly realize collective
motion across taxa 32. While doing so, they only use locally available sensory
information without direct access to positions or heading directions of
others. However, reproducing bio-inspired, decentralized collective motion
within robotic swarms has proven to be a complex challenge. Firstly, these
challenges arise due to classical bio-inspired flocking algorithms being
designed from a global perspective, making central control a straightforward
choice 33. Secondly, engineering robot hardware with reliable relative
localization and communication techniques capable of replacing global
information or planning presents its own set of challenges 34. Furthermore,
mapping local interactions to global emergent behavior is a complex task in
itself 35, 36, 37.

The first simple flocking algorithm inspired by social insects that does not
rely on communication, memory, or global information was proposed over a
decade ago 38. This decentralized method, relying exclusively on short-range
(local) sensing, is particularly suitable for small robots that have strictly
limited capacity in perception, communication, or computation. However, its
practical applicability remains limited to small groups.

For larger robots with enhanced capabilities, vision-based flocking has also
been suggested as a viable alternative among other sensor-based solutions 39.
In such an approach, robots would primarily rely on local visual information,
similar to fish 1 and birds 40, where vision is the principal modality
supporting collective movement.

To deploy the first autonomous aerial swarm without central control or mutual
communication, Petráček et al. 41 utilized a novel relative localization
technique 42 based on ultraviolet markers and light sensors, laying the
foundation for vision-based decentralized swarms.

Only recently, Schilling et al. 43 presented the first vision-based drone
swarm capable of collective motion using only regular onboard RGB cameras.
This study shows that with advancements in computer vision, drones can now
detect each other during operation without additional visual markers, even in
outdoor settings. This approach also demonstrates that purely visual
information can drive the collective migration of up to three aerial robots.
Although this algorithm relies solely on visual cues, it does so only to
estimate the relative distance between drones. This data is fed into a
simplified flocking module based on the traditional Reynolds flocking model
44, originally developed for computer graphics. Since estimating relative
heading angles and alignment rules from camera images is not trivial,
migratory goals were introduced to achieve group motion in common directions.
To provide accurate input for the flocking module, the tracking algorithm also
uses general assumptions about the environment and the size of other robots.
The performance of this algorithm under various sensory or environmental
constraints, in scenarios with agents unaware of migratory goals, or in groups
larger than three, remains to be evaluated.

A purely vision-based model of collective motion, first introduced by Bastien
and Romanczuk 45, is a novel concept. This model, based on minimal visual
interactions, has been shown to yield collisionless, polarized group motion in
simulated agents. By using only low-level sensory information, it fosters an
inherently decentralized design and has been suggested as a promising
alternative to current flocking controllers in swarm robotics. However,
constraints typical for both natural and robotic collectives are yet to be
explored. Examples include the effects of a limited field of view and a
confined physical environment in which agents are embodied.

Using this simple vision-based model, we demonstrate stable flocking in a
swarm of ten differential drive terrestrial robots and lay the theoretical
foundation of the model under sensory and environmental limitations using
agent-based simulations. While our simulations provide general theoretical
insights into collective dynamics in realistic scenarios, our robot
experiments validate these findings under the limitations of real-world
settings. To our knowledge, this is currently the only robot swarm operating
solely based on individual vision, without any centrally computed or
explicitly shared information.

By first implementing the model in an agent-based simulation framework, we
show that a collisionless and polarized collective motion is achievable, even
with a limited field of view (FOV) and a confined arena, that generally reduce
group order due to out-of-sight group members. To validate our results, we
transfer the model to our purely vision-based robots operating in a confined
space. We then show that robots guided by this algorithm can maintain ordered
collective movement (1) only using their single, limited FOV camera stream as
individual input for flocking, (2) without explicitly estimating the positions
or orientations of others, (3) without storing any state information about the
environment in memory, (4) without directly communicating their state with
others or with any central node, and (5) running all necessary computations
onboard. To conclude our study, we discuss potential future research towards a
reliable, purely vision-based swarm of mixed entities.

## Vision-based model of collective motion

Figure 1:  Vision-based Model, A: A focal (red) agent with orientation $\psi$
and with 5 numbered visible neighbors (grey). The focal agent has a full,
$2\pi$ FOV (shaded with pink) and an unlimited visual range. The projection of
the visual field of the focal agent (represented on a dotted circle) is 1
where other agents are visible (thicker dark arcs), 0 otherwise. B: The
resulting unfolded 1-dimensional visual projection field $V(\phi,t)$ of the
focal agent. Projection blobs are numbered according to the corresponding
visible agents in panel A. When agents partially occlude each other, visual
blobs merge (e.g.: agent 4 and 5). C: The front-back (left) and left-right
(right) social force fields shaping agent velocity and turning rate
respectively. The map depicting $F_{v_{i}}^{soc}$ displays grey (or purple) at
specific relative positions, indicating how a focal agent at the origin would
accelerate (or decelerate) in response to the presence of another agent of
identical body size at those positions. The map showing $F_{v_{i}}^{soc}$ is
grey (or red) at those relative positions where these responses would be
turning right (or left). Accurate scales in both maps depend on model
parameters with $\alpha_{0}$ and $\beta_{0}$ controlling response amplitudes
while $\alpha_{1}$ and $\beta_{1}$ controlling equilibrium distances from the
origin in which no response is given. D: Emergent movement patterns of the
vision-based model in toroidal space with full field of view (as in Fig. 3).
These are Stuck-in-place (SIP), Flocking (F), Swarming (S), Unordered (X),
Milling (M), Moving swarms (pS). Colors along the agents’ trajectories
represent their orientations $\psi$ (See color wheel on the right).

The purely vision-based model of collective motion 45 in 2D considers agent
$i$ as a disc with diameter $L_{0}$ moving within a global coordinate system
in the direction $\psi_{i}(t)\in[0,2\pi)$ with velocity $v_{i}(t)$ (see Fig.
1A). The model dynamically changes the state of each agent ($v_{i}(t)$,
$\psi_{i}(t)$) based on its visual input, represented as a visual projection
field $V_{i}(\phi,t)$ (VPF). This input is a 1-dimensional binary
representation of the surrounding visual information with the focal agent
itself at the center. It is ‘1’ for relative angles around the focal agent
($\phi_{i}\in[-\pi,\pi]$) where another agent is visible, and ‘0’ where no
other agents are detected. As a result, $V_{i}(\phi,t)$ acts as a “visual
radar”, where every other agent within sight translates into a perceptual
representation or visual “blob” (see Fig. 1A-B). This binary input of the
vision-based model does not distinguish between surrounding peers and the
visual representation of multiple agents can overlap, forming larger visual
“blobs” than what individual agents would produce alone (see Fig. 1B, agent 4
and 5). In this way, the model’s input implicitly accounts for the visual
occlusions of visible peers. Furthermore, a strict identification of other
agents is not necessary for the resulting applications. This flexibility
allows the use of a wide range of detection methods, including LiDAR, IR
sensors, CNN-based peer detection, and simple color or other image feature-
based segmentation methods.

The state update is described by a set of differential equations using only
the visual input and the actual state of the agent (see Eq. 1-6 and Sec.
Calculating Visual Social Interactions for further details). Agents move with
their preferred speed until others become visible. Upon detecting others,
social interactions ($F_{v_{i}}^{soc}$ and $F_{\psi_{i}}^{soc}$) are added to
the agent’s self-propelled movement, adjusting its velocity and turning rate
accounting for the presence of visible peers, i.e.

$\displaystyle\frac{\partial{v_{i}(t)}}{\partial t}$ $\displaystyle=$
$\displaystyle F^{ind}(v_{i}(t))+F_{v_{i}}^{soc}(V_{i}(\phi_{i},t))$ (1)
$\displaystyle\frac{\partial{\psi_{i}(t)}}{\partial t}$ $\displaystyle=$
$\displaystyle F_{\psi_{i}}^{soc}(V_{i}(\phi_{i},t))$ (2)

where the first (self-propelled) velocity term in Eq. 1 accounts for agents
preference to their individual speed $v_{0}$ in absence of others, i.e.

$F^{ind}(v_{i}(t))=\gamma(v_{0}-v_{i}(t))$ (3)

with parameter $\gamma$ effectively tuning the strength of this preference.

The model generates a social force on each agent emerging from the interplay
of attraction and repulsion, influencing both movement velocity
($F_{v_{i}}^{soc}$) and turning rate ($F_{\psi_{i}}^{soc}$) as illustrated in
Fig. 1C. The equations governing these social forces read as follows:

$\displaystyle
F_{v_{i}}^{soc}(V_{i}(\phi_{i},t))=\int\limits_{-\pi}^{\pi}\alpha_{0}A(\phi_{i},t)\cos\phi_{i}d\phi_{i}$
(4)

$\displaystyle
A(\phi_{i},t)=-V_{i}(\phi_{i},t)+\alpha_{1}\left(\frac{\partial{V_{i}(\phi_{i},t)}}{\partial\phi_{i}}\right)^{2}$
(5)

$\displaystyle
F_{\psi_{i}}^{soc}(V_{i}(\phi_{i},t))=\int\limits_{-\pi}^{\pi}\beta_{0}B(\phi_{i},t)\sin\phi_{i}d\phi_{i}$
(6)

$\displaystyle
B(\phi_{i},t)=-V_{i}(\phi_{i},t)+\beta_{1}\left(\frac{\partial{V_{i}(\phi_{i},t)}}{\partial\phi_{i}}\right)^{2}$
(7)

The total extent of the exerted social force is calculated from the width and
position of all visual projection “blobs”, indicating the presence of other
agents, by integrating the visual input over the focal agent’s retina (see Eq.
4-7 and Sec. Calculating Visual Social Interactions). The model yields
reflexive behavior of agents without relying on any memory of past events,
i.e. only the current visual input is considered.

The model has four principal parameters in 2-dimensional space: $\alpha_{0}$
controls the intensity of acceleration responses to visible others, while
$\beta_{0}$ controls the amplitude of social turning responses. $\alpha_{1}$
and $\beta_{1}$ determine the front-back ($L^{fb}_{eq}$) and left-right
($L^{lr}_{eq}$) equilibrium distances between individuals. When a visible peer
approaches the focal agent and crosses these equilibrium distances, the
overall social force turns from attraction into repulsion in both acceleration
and turning respectively.

The visuo-motor system’s symmetries, distinguishing front from back and left
from right, are modeled through cos and sin masks across the visual field (see
Fig. S1 and Fig. 2 in 45). These masks modulate visual input, enabling
direction-specific responses in acceleration and turning respectively. For
instance, the cos mask changes signs from the front to the rear of the agent
(from $\phi_{i}=0$ towards $\phi_{i}=\pm\pi$). As a result, the same
attraction to a social cue yields positive acceleration when the cue is ahead
and negative deceleration when the cue is behind. Note, that in the original
model, agents have an omnidirectional FOV, allowing them to perceive others in
any direction.

We emphasize that there is no direct alignment rule present in this model.
Instead, an effective alignment of agents emerges as a collective behavioral
property from the interplay of lower-level local attraction-repulsion
interactions, adaptive agent speed, and the symmetries of the agents’ visuo-
motor systems. Our approach is supported by recent studies showing that
explicit alignment rules, while used widely 44, 46, 47, are not a prerequisite
to achieve polarized collective movement 30, 48, 49, 50, 51.

The vision-based model can produce a wide range of collective motion patterns
only using simple visual cues (see 45 and Fig. 1D). However, it was previously
assumed that agents have access to omnidirectional visual information and move
in an infinite space—idealized conditions compared to real-world scenarios.
The effect of typical constraints characteristic to real-world systems, such
as a limited FOV and spaces constrained by their walls, which significantly
influence collective behavior in zonal models 52, was not previously
investigated. These constraints apply to most embodied groups including
robotic swarms limited by their sensory capabilities and the physical space
they move in. Therefore, our work is twofold: First, we lay the theoretical
foundation by exploring the effects of a limited field of view on the vision-
based model and consider the implications of constrained spaces. Such
investigation is crucial to assess the applicability of the vision-based model
for controlling robots;

Figure 2:  Limiting the active FOV, A: A focal (red) agent with orientation
$\psi$, with a limited FOV between relative angles $[-\phi_{L},\phi_{L}]$
highlighted with pink between pink dotted lines and with three other agents
(grey), either fully visible (2), partially visible (3) or not visible (1).
In-sight parts of other agents are colored darker grey. B: The original
1-dimensional visual projection field $V(\phi,t)$ with black visual blobs for
in-sight, and light grey for out-of-sight visual information. The active field
of view is highlighted with pink between pink dotted lines. C: The resulting
1-dimensional limited projection field $V_{i}^{\phi_{L}}(\phi,t)$. Partial
visual blobs (3) are fully recovered to avoid boundary effects coming from a
limited FOV (See Sec. Limited FOV). D: Emergent movement patterns (As in Fig.
3) of $N_{A}=10$ agents on a torus arena and with limited FOV. These are Lines
(L), fragmented Leader-Follower (frLeFo), fragmented Flocking (frF), Unordered
(X), Leader-Follower (LeFo), Flocking (F). Colors along the agents’
trajectories represent their orientations $\psi$ (See color wheel on the
right).

Second, we implement the model as a robot controller and transfer it to ten
terrestrial robots forming a vision-based swarm. Our approach is truly
decentralized from robot perception to every computational step executed
onboard. We then demonstrate the collective movement of this swarm in real-
world conditions, explore the effect of key model parameters on movement
order, and showcase some of the primary benefits of the vision-based swarm
through four exemplary videos53.

### Quantifying Collective Behavior

During our analyses, we focus on five key metrics described in detail in Sec.
Metrics. Here we give a short overview of these: We quantified group
polarization ($P$), mean inter-individual distance ($\tilde{D}$), the overlap
or collision avoidance time ratio ($R_{o}^{sim}$, $R_{o}^{exp}$), group
circularity ($RCA$), and the number of largest subgroup ($N_{clus}^{max}$)
after hierarchical clustering of the agents to quantify group fragmentation.
In short, polarization ($P$) indicates the uniformity of movement direction
within the group, achieving high values close to 1 when all agents move in
similar directions while low values close to zero for unorganized movements
where agents or subgroups move in different directions. $R_{o}^{sim}$ in
simulated agents shows what proportion of simulation time (in percent) agents
spent overlapping and is directly related to agent-agent collision probability
in embodied groups. As overlaps are not possible in a physical space, during
robot experiments $R_{o}^{exp}$ shows what proportion of experiment time (in
percent) robots spent with avoiding collisions with each other (see Sec.
Collision Avoidance). $R_{o}^{sim}$ reaches 100 if all agents overlapped
through the whole simulation time and 0 if no agent-agent overlaps were
observed. Similarly, $R_{o}^{exp}$ reaches 100 if every robot was in collision
avoidance during the whole experiment time, and 0 if no collision avoidance
event has been triggered. The circularity of groups ($RCA$) describes the
shape of the group. An $RCA$ value of 1 signifies a perfectly circular group,
whereas values approaching 0 indicate elongated formations. Furthermore, we
clustered agents according to their movement similarity (i.e. similarity based
on heading angle and relative distance) and measured the size of the largest
cohesive polarized subgroup ($N_{clus}^{max}$) to quantify group fragmentation
(for further details see Sec. Metrics). For fully fragmented groups this value
is one, while in fully cohesive group this value reaches the number of agents
in the group ($N_{A}=10$).

### Field of View

#### Model Modifications

We started our investigation by implementing the vision-based model in an
agent-based simulation framework 54. In the first set of simulations,
$N_{A}=10$ disc-shaped agents were uniformly distributed across a toroidal
arena, i.e. across an arena with periodic boundaries on opposite sides (see
also Sec. Simulations On a Torus). The width and height of the arena were set
to 900 pixels, which is approximately 164 times the radius of agents
($R_{A}=5.5$ pixels) similar to experimental conditions (see Sec. Mobile Robot
Experiments). The agents’ FOV varied from 0 (equivalent to blind agents) to
$2\pi$ (representing unobstructed vision), allowing us to systematically
explore the impact of FOV restrictions on the vision-based collective
behavior.

The choice of a toroidal arena serves a dual purpose: (1) It allows us to
directly compare our results with prior studies that assume agents operate in
infinite space with unrestricted FOV, while (2) preventing catastrophic group
fragmentation—a possible issue in infinite environments where agents with
limited FOV could permanently lose visual contact with each other. The
periodic boundaries circumvent this problem by ensuring that individuals
exiting one side of the arena reappear on the opposite side, sustaining visual
contact and group integrity even with a limited FOV.

The agents’ FOV was restricted by setting their visual projection fields
$V_{i}(\phi,t)$ to zero for all relative angles $\phi$ exceeding the FOV
limits $\pm\phi_{L}$, thereby creating a limited visual field
$V_{i}^{\phi_{L}}(\phi,t)$. If another agent is only partially visible at the
peripheries of the active FOV, the entire visual projection blob corresponding
to that agent is reconstructed (see Sec. Limited FOV and Fig.2A-C). Limiting
the FOV this way effectively introduces a blind spot behind each agent, such
that they perceive information only around the primary direction of movement
(front).

By default, visual range of agents is technically unlimited but only visual
blobs wider than a single retinal pixel are considered during the composition
of the visual projection fields. This way the resolution of the projection
field ($N_{ret}$) ultimately determines the maximum range of vision similarly
to actual visual sensors. Lower resolutions result in coarser visual fields,
which may fail to detect distant agents if their visual representation is too
small. The resolution of the simulated agents’ retinas is aligned with what is
achievable on our robot platform (see Table 1 and 2).

#### Results

To study the influence of a limited FOV on the collective movement of
simulated agents, we conducted a series of simulations where we varied the
model’s parameters $\alpha_{0}$, $\beta_{0}$ and the agents’ FOV, while other
parameters remained constant as detailed in Table 1. These simulations were
repeated 20 times, exploring FOVs from 25% to 100% of the complete range of
$2\pi$. We quantified the collective behavior through key metrics described in
Sec. Quantifying Collective Behavior.

Our findings, aligning with Bastien and Romanczuk 45, show that groups with
unobstructed FOV exhibit behaviors from flocking, with high group polarization
and cohesion, to milling and swarming under varied $\alpha_{0}$ and
$\beta_{0}$ conditions (see Fig. 3 column A). Flocking (F) is observed where
agents can sufficiently modulate their speed ($\alpha_{0}$ is large enough)
but they are not overly maneuverable ($\beta_{0}$ is relatively small),
leading to cohesive and circular group formations with minimal overlaps (Fig.
3 column A, pattern F).

Figure 3:  Effects of a limited FOV on a torus. Emergent collective movement
patterns as depicted in Fig. 2 (top row) and summary metrics (rows) of the
resulting collective movement with different fields of view (columns) over
different $\alpha_{0}$ (y axis) and $\beta_{0}$ (x axis) parameters.
Simulations with full FOV align with the results of Bastien and Romanczuk 45.
Observed movement patterns are Unordered (X), Leader-Follower (LeFo),
fragmented Leader-Follower (frLeFo), Flocking (F), fragmented Flocking (frF),
Lines (L). Reducing agents’ FOV facilitates polarized movement via leader-
follower (LeFo, frLeFo) dynamics. Due to a blind spot behind agents, groups
with lower FOVs are less cohesive than those with a full field of view.
Collisionless polarized flocking behavior was observed for all FOV values
larger than 25%.

Lower speed modulation ($\alpha_{0}$) gives rise to either milling behavior
(M) with small $\beta_{0}$ conditions or swarming (pS, S) behavior with large
$\beta_{0}$ conditions. During milling agents follow circular paths around the
group’s center of mass. Swarming is characterized by continuous turning in
response to visible agents. This can lead to cohesive movement as a swarm (pS)
or to the fixation of the group’s center of mass in a stationary position (S).
When $\beta_{0}$ is too small agent’s do not turn sufficiently in the presence
of others to implement organized collective motion, hence with low speed
modulation ($\alpha_{0}$) they are unorganized (X), or when speed modulation
is high enough they remain stuck in place (SIP).

Reducing the FOV to 75% (seeFig. 3, column B) reveals a new parameter range
with high group polarization at low $\alpha_{0}$ and high $\beta_{0}$. In such
settings, agents with a full FOV typically exhibit swarming behavior (S). With
a limited FOV in this regime, polarized collective movement can still emerge:
as agents with restricted FOVs may break away from the group when they lose
sight of others due to the blind spot behind them, others will follow the
“blind” leader causing an overall polarized movement of the group. This state
of polarized movement, resulting from the underlying leader-follower dynamics
(LeFo), shows lower circularity than flocking behavior. It also tends to
oscillate between polarized and non-polarized states, reflecting the remains
of true swarming states (S) in case of a full FOV. Similarly, when both
turning and speed modulation are high, swarming states (pS, S) at a full FOV
break down into intermittent polarized and depolarized states. As the group
begins to swarm it becomes fragmented as one or more agents break away. These
newly emerged ”blind” leaders are soon followed by others, leading to a
fragmented leader-follower pattern (frLeFo). In a periodic environment, such
as a torus, subgroups eventually merge, sustaining an endless cycle of fission
and fusion. Furthermore, we observed an elongated movement pattern (L) due to
a combination of the periodic environment and the limited FOV, for both low
$\alpha_{0}$ and $\beta_{0}$ conditions. These patterns show agents in single
or multiple lines moving in the same (fully polarized) or opposite directions
(anti-polarized), and they were only occasionally observed with a 75% FOV.

Reducing the FOV further to 50% (see Fig.3 column C) decreases polarization
for two main reasons. Firstly, a narrower active visual field increases the
likelihood of agents losing sight of each other, resulting in the group
splitting into smaller subgroups. While these subgroups might be polarized
within themselves, their diverging movement directions diminish overall group
polarization. Secondly, a narrow field of view complicates rejoining into a
single polarized group when the group becomes fragmented. This challenge
arises because agents lack visual information about subgroups, some typically
being ahead while others behind. Consequently, with lower $\beta_{0}$ values,
we observe periodic fragmentation into flocking subgroups that attract each
other from opposite directions but pass through each other without being able
to merge into a single cohesive group (frF). We noted elongated polarized or
antipolarized lines (L) appearing more frequently and across a broader range
of model parameters at a 50% FOV than a 75% FOV. Such movement patterns
coexisted with other, usually leader-follower patterns.

Further reducing the FOV to 25% (see Fig. 3 column D) intensifies the
challenges to group cohesion seen with a 50% FOV. Agents break away easier
from others and rejoining to a cohesive group is difficult as agents are
deprived of most of their visual input. The narrow frontal field of view
further stabilizes elongated line patterns (L).

In sum, limiting agents’ active visual field, while fostering group
polarization through temporary “blind” leaders, poses fundamental challenges
to group cohesion. The larger the blind spot, the easier it is for agents to
lose track of each other eventually leading to group fragmentation. Subgroups
with limited FOV often show difficulty in rejoining to a single cohesive group
after fragmentation. Nonetheless, with periodic boundary conditions, we
observed a part of the parameter space where polarized cohesive flocking
motion persisted for all FOVs. Our results highlight the resilience of the
vision-based model and the resulting collective behavior despite significant
visual limitations and suggest a wide applicability in the design of robotic
swarms even in environments where sensory input is restricted.

### Confined Spaces

#### Model Modifications

In realistic scenarios, agents often have to navigate in structured
environments constrained by their physical boundaries. Repeatedly interacting
with walls might fundamentally change collective behavior, potentially
preventing movement coordination or facilitating transitions between
behavioral states reported in fish, computational models, and drone swarms 55,
56, 19. To study the influence of space confinement on the vision-based model,
we implemented reflective boundary conditions in the simulations. When agents
reach these boundaries they must turn back to the arena orthogonal to their
incidence angles, i.e. their new heading angle $\psi_{i}(t+1)$ is chosen as

$\displaystyle\psi_{i}(t+1)$
$\displaystyle\in\left\\{(\psi_{i}(t)\pm\frac{\pi}{2})\mod 2\pi\right\\}$ (8)

such that:

$\displaystyle\quad x_{i}(t+1)$
$\displaystyle\in[x_{\min},x_{\max}]\quad\text{where}$ (9) $\displaystyle\quad
x_{i}(t+1)$ $\displaystyle=x_{i}(t)+v_{i}(t)\cos(\psi_{i}(t+1))$ (10)

and

$\displaystyle\quad y_{i}(t+1)$
$\displaystyle\in[y_{\min},y_{\max}]\quad\text{where}$ (11)
$\displaystyle\quad y_{i}(t+1)$
$\displaystyle=y_{i}(t)+v_{i}(t)\sin(\psi_{i}(t+1))$ (12)

where $v_{i}(t)$ is the velocity of agent $i$ in time $t$ at position
$(x_{i}(t),y_{i}(t))$ with heading angle $\psi_{i}(t)$ and vertical walls are
at $x_{min}$ and $x_{max}$, while horizontal walls are at $y_{min}$ and
$y_{max}$.

Interactions with boundaries precede any vision-based behavior, meaning agents
reflect off walls even if their vision-based model suggests otherwise. We
assume agents can detect walls at short range, which can be implemented for
example via IR sensors (see Sec. Collision Avoidance).

Figure 4:  Effects of a confined arena, Summary metrics (rows) of the vision-
based collective movement with different FOVs (columns) over different
$\alpha_{0}$ (y axis) and $\beta_{0}$ (x axis) parameters in a confined arena
with reflective walls. Introducing boundaries generally decreases polarization
and cohesion of the groups and makes it challenging to identify stable
movement patterns.

#### Results

We simulated the movement of $N_{A}=10$ agents with the same parameters as in
the previous section (see Table 1), and systematically compared different
environmental conditions to set the stage for experimentation with robots in
real-world environments. We described the resulting collective movement
through the same summary metrics described in Sec. Quantifying Collective
Behavior. The introduction of walls imposed significant and regular
perturbation on movement, making it challenging to identify stable movement
patterns, such as flocking, milling or swarming states. Instead, we identified
a specific set of parameters that produced collective movement which was
comparatively cohesive, collisionless, and highly polarized, even within
reflective walls. This suggests the possibility of a stable flocking state and
holds promise for developing controllers for embodied agents.

In confined spaces, agents with full FOV remain cohesive and, after a
transition period, resume their original movement patterns despite the
perturbation caused by the walls. Due to these temporary unorganized states,
polarized groups in a limited space show on average lower polarization, less
cohesion and lower circularity than those on a torus, especially with lower
FOVs. On the other hand, the introduction of parallel walls causes agents to
move in similar directions once reflected. This increases the polarization of
otherwise unorganized groups with low polarization. These groups, which cannot
modulate their turning sufficiently without walls (i.e., with low
$\beta_{0}$), show higher polarization than those in toroidal arenas.

When agents’ FOV is limited, walls become a significant disruptive factor on
both group polarization and group cohesion. By repeatedly perturbing the
group, walls can suddenly change agents visual input depending on which
direction they turn back towards the arena. As a result, agents might easily
break away from the group due to their lack of memory of past positions and
their blind spot.

In conditions where leader-follower dynamics resulted in stable polarized
movement on a torus (see Fig. 3, LeFo, frLeFo), walls fully deteriorate
polarization, as these states are particularly fragile to external
perturbations. If a temporarily “blind” leader is forced by a wall to turn
back towards the group that follows it, the group loses polarization until a
new leader can emerge. This effect was consistent across all parameters for
limited FOVs.

Walls also reduce group cohesion resulting in higher inter-individual
distances and less circular group formations than in toroidal environments. In
terms of collisions, groups in confined spaces performed similarly to those on
a torus if both control parameters $\alpha_{0}$ and $\beta_{0}$ were large
enough, such that agents were both sensitive and maneuverable enough to avoid
dangerously short distances with others.

The parameter regime yielding the most polarized movement was consistent with
those corresponding to flocking states (F) in periodic environments, although
polarization notably decreases when reflective boundaries are introduced.
These “islands” of polarized states show comparably smaller inter-individual
distances and elongation than other parameter combinations suggesting more
cohesive groups. Also, with FOVs higher than 25%, we observed close to zero
overlap ratios in at least parts of these regimes, indicating effective
collision avoidance. Due to these considerations they are the best candidates
to use for controllers realizing group migration in robotic applications.

Our results show that confining the arena with reflective walls reduces group
cohesion with limited FOVs. Reflections from walls can lead agents to lose
sight of each other, promoting group fragmentation—a fundamental challenge in
groups with limited perceptual capabilities moving in confined spaces, without
the possibility of relying on memory or an explicit model of the surrounding
environment. The interplay between the effects of sensory and environmental
constraints we presented clearly shows the complexity of achieving effective
collective movement in real-world scenarios, and the necessity of further
exploration through embodied systems such as a physical robot platform.

Figure 5:  Robot Platform, A: Close up image of three vision-based robots. B:
Computational steps of the robot controller implementing vision-based
collective movement. 1.: A camera image is acquired, and unwarped from fisheye
lense distortion. 2.: Other robots are detected on the image on board using a
tensor processor unit (TPU). 3.: Detection boxes (yellow) are projected to the
horizontal axis to create the limited projection field. 4.: Partial projection
blobs are recovered using the height of the detection box to create the full
visual projection field (VPF). 5.: The VPF is fed into the vision-based model
to calculate the desired robot state. 6.: Motor commands are calculated and
executed by the robot. C: Hardware components of a vision-based robot: 1:
Camera module, 2: Battery, 3: 3D-printed structural scaffolding, 4: Socket for
Raspberry Pi 4B and Google Coral EdgeTPU, 5: Thymio II base robot. D: Swarm of
ten vision-based robots

.

## Vision-based decentralized flocking of robots

### Robot Platform

After exploring the effects of realistic limitations on the vision-based
model, we transferred it to terrestrial robots through a controller software
57. We use the Thymio II 58 educational robot platform extended with a
Raspberry Pi 4B development board and an RGB fisheye camera module, having a
roughly half horizontal FOV (approximately 175°). Robots perceive their
environment via the camera stream and detect other robots in the registered
image using a convolutional neural network (CNN) based object detector (see
Sec. From Detection to Robot Movement and Fig. 5B, steps 1-2). Detection runs
onboard using a Google Coral tensor processing unit (TPU), generating a 1D
binary visual projection field (VPF) as the input for the controller algorithm
(See Fig. 5B, step 3). If other robots are only partially visible on the
peripheries of the camera image, the height of the detection box is used to
recover the entire projection ”blob” in the resulting projection field. This
is done similarly to our ”blob”-recovery method during simulations (see Fig.2,
B-C) to avoid boundary effects caused by a limited FOV (see Sec. Limited FOV
and Fig.5B, step 4). Detection of other robots runs at 5 Hz on an input
resolution of 320 $\times$ 200 pixels. Note that any method capable of
selectively detecting other robots around the focal one would suffice to
generate the visual projection field, e.g., other selective segmentation
techniques or LiDar-based detection methods 59, 60. We used CNN-based object
detection due to its widely explored applicability, relatively low price, the
recent improvements in inference speed on edge-devices, and its relatively
high reliability in changing light conditions.

Figure 6:  Experimental setup for vision-based robot flocking A Example data
sources from a robot experiment. These are as follows: (1) The top view
reconstruction of tracked robot positions and orientations (left), with the
arena wall depicted with dotted lines and the positions of the two RGB cameras
shown with colorful arcs. Reconstructed robots are color-coded according to
their IDs. (2) The onboard RGB camera streams of individual robots with
detection boxes (top right). The individual camera streams are color-coded
according to the robot IDs in the top view reconstruction. (3) RGB camera
images with tracking overlay (bottom right). The positions and angles of the
RGB cameras are shown on the top view reconstruction with corresponding pink
(RGB #1) and purple (RGB #2) arcs. B Reconstructed top view frames of an
experiment showing stable flocking behavior of ten robots. The frame order
from one to six is in the top left, and the corresponding experiment time is
in the top right corner of each chosen frame. The same experiment is presented
in Video 1 61

After calculating the visual input for the controller, velocity and turning
rate values are determined according to the vision-based model. These are then
translated into motor commands for the robot’s 2-wheel differential motor
system, resulting in movement (see Fig. 5B, steps 5-6). To guard the integrity
of the robots in experimental scenarios we used the seven built-in IR
proximity sensors of the Thymio II robot platform. We used these sensors to
(1) avoid possible robot-robot collisions and (2) to turn robots away from
walls of the arena similarly to wall-reflections in simulation experiments.
When close to an object (whether a wall or another robot), the robot turns
away from the object until the path ahead is clear, irrespective of the visual
input (see Sec. Collision Avoidance).

We built a swarm of ten robots (see Fig. 5D). For our analysis, we collected
the position and orientation data of the swarm using an OptiTrack motion
capture system with 20 infrared tracking cameras, which detect reflecting
markers attached to the robots 62. Additionally, we collected individual
camera streams from the robots, published along with calculated detection
boxes and the resulting projection fields via the local network, as well as
two external RGB streams covering the whole arena (see Fig. 6A and Video 161).
The collected tracking data was used solely for later analysis and was not
distributed to the robots during experiments.

We conducted all robot experiments in a 9m$\times$6m arena with flexible grey
plastic walls of 8cm height. To minimize false visual detections of external
objects outside the arena, we installed an additional wall with a height of
40cm. Being approximately twice the height of the robots, this wall blocks the
robots’ view once they are near the arena edge.

Robots were initialized in a polarized configuration centered around a
randomly selected position near the arena wall. They were started and stopped
sequentially from a central computer but did not receive any information from
this central node or explicitly communicate their internal states with each
other during experiments. In other words, robots operated in a fully
decentralized manner. They acquired local visual information using their
camera stream, detected other robots using onboard detection, and calculated
and executed resulting motor commands onboard as well.

### Mobile Robot Experiments

Based on our simulated groups, we chose a range of model parameters where
cohesive, polarized motion can emerge. We varied the two main parameters in
this parameter range and conducted two sets of about 60-minute recordings for
each combination. By that we produced more than 30 hours of experimental data
of collective movement of ten robots 63. We either fixed the maneuverability
parameter ($\beta_{0}$) while adjusting the social acceleration value
($\alpha_{0}$), or vice versa. As before, we quantified the resulting
collective movement through the metrics described in Sec. Quantifying
Collective Behavior. We filtered the recorded data by only including data
points where none of the robots were executing collision avoidance maneuvers
due to walls, i.e. where all robots acted according to their vision-based
controller.

Supporting our results with simulations, we found that polarized cohesive
motion stably arise in our robot swarm on a wide range of model parameters.
Polarization of the group was highest when both $\alpha_{0}$ and $\beta_{0}$
were large enough, allowing robots to sufficiently modulate both their
velocity and turning rate in response to visible others. In these conditions
robot collision events were unlikely and the group remained polarized for long
timescales aligning with the walls of the arena. Group fragmentation was
possible due to a limited FOV but rare.

To study the effect of maneuverability parameter ($\beta_{0}$) on the robot
swarm, we first fixed $\alpha_{0}$ to a value sufficiently large for agents to
modulate their speed in the presence of others. We then varied $\beta_{0}$
between 0, where agents do not modulate their turning rate in response to
others, to 4, where agents react with a high degree of turning modulation to
visual cues (see Fig. 7A). We found that with sufficiently large parameter
$\beta_{0}$ groups achieved highly polarized movement with reduced temporal
variance and inter-individual distance as they could sufficiently modulate
their heading angle to remain in a single polarized group for extended
periods. In this parameter regime group fragmentation was also less common,
though still possible due to a limited FOV. The resulting collective motion
was collisionless or collision avoidance events were rarely triggered by other
robots due to the restricting effects of the arena walls.

As expected, when $\beta_{0}$ was too low, agents could not modulate their
turning rate enough to achieve high polarization or cohesive movement captured
by low polarization and cluster size values. Furthermore, robots in this
regime collided more often as they could not adjust their paths fast enough to
avoid dangerously low inter-individual distances from group members increasing
the risk of collisions.

Conversely, when $\beta_{0}$ was too high, polarization also decreased
compared to mid-range $\beta_{0}$ values due to an observed over-reactive
turning behavior. In these regimes agents over-react to visible robots to the
extent that they may easily lose sight of others resulting in group
fragmentation. This is characterized by lower polarization and polarized
cluster size and higher inter-individual distances due to diverging subgroups.

Interestingly, although we found the same trend in all metrics with increasing
$\beta_{0}$ values in simulations and robots, we found a mismatch between the
optimal $\beta_{0}$ value between the two systems: in the robot swarm
significantly larger $\beta_{0}$ values yield optimal performance compared to
simulations. This mismatch might be caused by key differences resulting from
the embodiment of agents in the collective. Next to other factors arising from
friction and body weight, physical robots simply can not turn with arbitrary
turning rates the same way as simulated agents can. Such differences highlight
the importance of validating model results on embodied systems to assess the
practicability of the model in real-world scenarios.

Figure 7:  Effect of model parameters on robot flocking. A: The effect of
varying $\beta_{0}$ (x-axes) with fixed $\alpha_{0}=0.9$ (red) on summary
metrics (columns), i.e. group polarization ($P$), mean distance between robots
($\tilde{D}$), size of the largest cohesive, polarized subgroup
($N_{clus}^{max}$), and time ratio spent with robot collision avoidance
($R_{o}^{exp}$). B: The effect of varying $\alpha_{0}$ (x-axes) with fixed
$\beta_{0}=1.25$ (orange) on the same metrics from left to right respectively.
Thick colored lines in both panel A and B depict temporal means over two
repetitions of 60 minutes, dotted grey lines the means of individual
repetitions, shaded areas are the standard deviations (STD) over time.
Collision avoidance events are binary, hence STD is not provided in this case.
In both cases, either when $\alpha_{0}$ or $\beta_{0}$ was fixed, robots were
able to (1) maximize group polarization and (2) cohesion and (3) minimize
collisions with some mid-range optimal parameter combinations.

To study the effect of the velocity modulation parameter $\alpha_{0}$ we then
fixed $\beta_{0}$ to a value large enough for robots to sufficiently modulate
their turning rate. We changed $\alpha_{0}$ between 0, where agents do not
modulate their speed, and 2, where agents react with large
acceleration/deceleration to other visible robots (see Fig. 7B).

Aligning well to our previous results with changing $\beta_{0}$, changing
$\alpha_{0}$ revealed similar insights. With minimal $\alpha_{0}$ values
robots couldn’t sufficiently accelerate to remain cohesive and polarized and
they could not decelerate to avoid physical collisions with each other.
Increasing the parameter to an optimal mid-value showed emergent collective
motion that is highly polarized, cohesive with little to no collisions
avoidance events. Too large $\alpha_{0}$ values introduce over-reactive
acceleration behavior and oscillations between forward and backward movement.
Note that the vision-based model allows deceleration to negative velocities,
i.e. robots can move backwards with respect to their heading, while their
visual FOV is oriented forward. This might increases the chance of robot
collisions with large $\alpha_{0}$ values as agents move faster and more
frequently in a direction from which they can not gather visual information.

We demonstrate the behavior of the vision-based swarm by four representative
videos53. First, we select one of the best achieving parameter combination,
where both $\alpha_{0}$ and $\beta_{0}$ are from the optimal middle parameter
ranges. We then recorded the resulting collective movement and collected the
camera stream of agents and compiled these into a single video file (see Video
1 61). During this experiment robots aligned with the arena wall and moved as
a cohesive polarized group. We did not observe robot collisions or group
fragmentation, although these both are possible due to a limited FOV and a
confined space robots have to navigate in. We observed rare false positive
detections (see Video 1 61, Robot 1 from experiment time 01:03:30 to 01:05:00)
of other visible objects outside the arena (such as visible curtains or
occasional light reflections) mistakenly identified as robots that temporarily
biased the resulting robot movement–an example of type of noise that can not
fully taken into consideration with physics-based simulations. False negative
detections of other robots were also rare but possible when others were too
close to a focal robot (See Video 1 61, Robot 2, from experiment time 11:52:00
to 12:21:00), temporarily increasing the probability of robot collisions.

Due to the limited FOV of robots, they can break away from the group
(fission). But when these robots detect others again, they will rejoin the
group (fusion). Although with a limited FOV rejoining events are more
challenging than with a full FOV, we demonstrate that this is possible with
the minimal vision-based model we use even with a half FOV. To present group
fragmentation and rejoining events we recorded two videos. Video 2 64 shows a
single robot breaking away from the swarm due to the the group’s direction
when arrived to the wall. As the group was almost orthogonal to the wall, the
chance of some agents turning away from others was high. Once the robot,
having previously split from the others, can see the group again, it can
approach and rejoin it. Furthermore, in Video 3 65 we show similar joining
behavior with two agents first breaking away from the group forming their own
subgroup until others become visible again.

One of the main advantages of decentralized design is that no central data
management or path calculation is necessary to achieve global order. As a
result robots can be dynamically added to or removed from the group without
disturbing the collective behavior. We show how the group can be extended on
the fly by adding robots to the group one by one on Video 4 66. Newly
introduced robots speed up and join to the group ahead once others are in-
sight. Removing robots on the fly is similarly unproblematic. Doing so, the
group quickly recovers from missing members. When a robot is removed,
remaining robots will modulate their speed to establish the preferred distance
from others, keeping the collective movement intact. With this we showcase
stable and flexible flocking behavior of robots achieved through minimal,
purely visual interactions and we underscore the promising practicality of the
presented algorithm.

## Discussion

In this work, we implemented a purely vision-based model of collective
movement on a swarm of ten terrestrial robots with a limited FOV operating in
confined spaces. Our preliminary analysis with agent-based simulations further
supports that the model is resilient to perceptual and environmental
constraints and can facilitate polarized, cohesive, and collisionless
collective movement in simulated agents and robots. Our robot swarm is fully
decentralized, with robots acting solely based on their camera streams without
using central information, sharing information directly, or having memory of
past events. Using this minimal vision-based controller, the swarm achieved
highly polarized collective motion and maintained group cohesion. Most
efficient collective motion was obtained especially when robots could modulate
their speed and turning rate sufficiently to react but not overly react to
others. These results show that our purely vision-based and decentralized
robot control approach is feasible and effective in achieving robust
collective movement of embodied agents.

Robots had a limited FOV, which could eventually lead to losing sight of each
other and breaking away from the swarm causing group fragmentation especially
in confined arenas. Our simulation results also support our observations in
physical robots. Once the group is fragmented, it is more challenging to fuse
into a single collective with a limited FOV. Future work could improve
practical applicability by implementing a full, omnidirectional vision for
robots, allowing fully collisionless and cohesive movement even in strictly
limited spaces. As highlighted by our simulation results, such an improvement
would also enable robots to show complex movement patterns, such as swarming
or milling around a target point.

Along with a limited FOV, visual occlusion shapes information transfer in
visual networks 67. As a result, visual occlusion has a crucial impact on
collective behavior arising from visual interactions. These include the
flocking behavior of pedestrian crowds 68, the spatio-temporal characteristics
69 and collective detection capabilities of fish schools 70, social
information use in collective foraging 71, or the performance of drone swarms
using zonal flocking models 72, among others. In the simplest case, without
strict identification of peers, the input of the vision-based model implicitly
accounts for visual occlusion, allowing overlapping projection “blobs” when
robots occlude each other. In this case, the model considers the visual
representation of each visible peer to construct the final binary input. On
the other hand, using detection methods such as CNN-based object detection
allows assigning individual robots to their bounding boxes on the registered
camera image. This enables selecting only a few robots to interact with and
naturally raises the question of neighbor selection: which visible neighbors
should be considered during collective motion? Similar to fish paying
attention only to a few visible peers 73, 74, or to those that are the fastest
75, selecting which in-sight neighbors to follow might largely influence the
vision-based model presented in this paper.

Robots in our framework detected others utilizing their camera streams, with
an onboard CNN-based object detection network. Although recent advancements in
edge computing allow near real-time inference with neural networks on edge
devices, achieving practical usability for real-time motion control remains a
challenge. The slower the detection of others, the slower the speed at which
robots can reliably operate. Furthermore, false detections can influence the
model by introducing noise to the visual input, resulting in movement
perturbations that can only be partially modeled in physics-based robot
simulators. While false positive detections can erroneously attract robots to
unwanted targets, false negative detections might increase the probability
that robots collide. Therefore, reducing the false detection rate and
increasing inference speed are crucial in practical vision-based applications
relying on such detection methods. In future work, improved object detection
techniques could improve the visual input of the model leading to faster, more
reliable, and robust collective movement in ever-changing environments.

A possible approach to improve detection accuracy is to increase the input
resolution of the camera stream. However, this would drastically decrease
inference speed with the current technology. Another option is to extend the
current system with event-cameras, which register slight changes in brightness
with high temporal resolution and low resource usage. Combining event-based
movement detection with traditional CNN-based techniques can overcome the
drawbacks of frame-based methods and has already proven useful for long-term,
high-resolution monitoring of penguin behavior 76. Such a composite system
could provide a truly real-time movement control while maintaining robot-
detection accuracy.

Our results also generalize to systems capable of producing similar projection
fields around focal robots as what we presented, including advanced
segmentation techniques or selective LiDAR mapping. The primary challenge
remains developing a robot-specific detection system that effectively filters
out objects irrelevant to collective movement. In shoals of fish, for
instance, social recognition of conspecifics is based on a wide range of cues
depending on the species. These can be olfactory or acoustic signals, electric
“fingerprints” in weakly electric fish, or general visual cues such as
coloration, body shape, or even UV markings 77, 78, 79.

In our study, we used the 2-dimensional version of the purely vision-based
model presented by Bastien and Romanczuk 45, which allowed us to provide
vision-based decentralized control for terrestrial robots moving on a plane.
Transferring our results to aerial robotics, implementing 3-dimensional motion
modulation for unmanned aerial vehicles (UAVs), is an interesting future
direction.

Regarding scalability, a key question is how our results generalize to large
robot swarms. As the number of robots and group density increases, the
likelihood of collisions inherently rises, too. Here, the question of neighbor
selection is especially crucial as the visual fields of robots become
saturated in large groups. Therefore, generating reliable real-time input is
key to achieve collisionless movement also in large groups.

The vision-based model we used integrates simple visual cues over the visual
field of the focal agent to determine the direction and velocity of the
agent’s motion. Although Bastien and Romanczuk 45 propose to optionally
include optic flow in the state update rule, the effect of including such
temporal derivatives of the visual cues is yet to be explored with this model.
At the same time, on the other hand, it was shown in simulated particles that
optic flow alone can account for similar patterns of collective motion 80 as
those yielded by minimal visual cues presented in this paper. It is an
interesting future direction to study how the combination of simple visual
cues and optic flow co-shape collective behavior. Especially so, as the exact
neural encoding of optic flow and its impact on collective animal behavior is
still not clear 81, 82. Taking a biorobotic perspective 83, 84, 85, 74, the
impact of optic flow on collective behavior could be systematically mapped in
both robots and animals. In this context, mobile robot swarms would serve as
embodied models of collective motion, to study the combined impact of optic
flow and simple visual cues in collective systems. This approach would allow
for the rigorous identification of scenarios where optic flow enhances or
stabilizes collective performance. Forming a loop between natural and
artificial systems, these findings could then generate hypotheses for further
research in biology to better understand animal collectives and improve bio-
inspired, vision-based robot swarms. On the other hand, although there are
bio-inspired optic flow-based sensors, robot guidance methods 86 and
collective movement models 80, 87, optic flow-based coordination of a
collective in embodied swarm robot systems remains an unprecedented challenge.

Our framework allows to introduce heterogeneity in collectively moving swarms.
Firstly, being the presented model purely vision-based, if robots could
identify a variety of distinct entities—from other robots or inanimate objects
to animals, including humans—they could adapt their movement without the
necessity for direct communication. By that, one could engineer composite
movement strategies within mixed-entity swarms and hybrid societies 88 where
direct communication is impossible, either due to different communication
protocols (for example, differing robot-to-robot communication standards) or
the complete absence of standard communication channels (e.g. between robots
and animals). Such an approach has great potential in many applications 89. It
could help us designing adaptive robot swarms behaving according to the
perceived human user’s needs 90 during human-swarm interaction 91, or in
wildlife rescue, pest control, in understanding animal collectives 92 or
improving robot swarms.

Secondly, as the vision-based model and every resulting application use a
spatial scale defined by the body size of conspecifics, it is possible to
flexibly introduce agents of different sizes while keeping the collective
movement of the group intact—fundamentally different from classical models of
collective motion. As an agent gets larger, its visual projection “blob” on
others’ retinas also grows, making others stay farther away. On the other
hand, the stability of such mixed-sized robot swarms is yet to be explored.
Furthermore, in its simplest form, the vision-based model does not distinguish
between peers far from the focal agent and peers that are smaller than the
focal agent. In future research, addressing this size-distance ambiguity in
practical settings is key to adapting equilibrium distances for stable
collective movement, for example by mapping detectable objects to their
typical physical sizes.

Integrating computer vision datasets (e.g.93) with various labelled objects
can further enhance adaptability. By mapping detected objects to model
parameters, robots could interact with a wide range of inanimate objects
through avoidance or following behaviors. Additionally, a dictionary of
observed objects would enable robots to dynamically adjust their movement
parameters based on their surroundings. For instance, following an object
might be beneficial in some contexts but not in others. This context-specific
bias in movement patterns would allow for nuanced navigation in cluttered or
dynamically changing environments. This approach could be extended with
machine learning techniques to optimize collective movement according to
arbitrary goals by tuning the parameter mapping. Using the vision-based model
with such contextual tuning opens new directions for swarms operating in real-
world scenarios.

Our results offer promising future perspectives in vision-based swarm
robotics. By decentralizing the control of individual robots, swarms can
operate in challenging environments where centralized solutions break down
either because of the absence of central knowledge or because of communication
failures. With further refinements, our results have the potential to
generalize to other input techniques and robot (or other agent) types in
dynamic environments paving the way for diverse swarms including many,
fundamentally different physical entities.

## Acknowledgements

### Contribution Statement

The initial concept was conceived by DM, RB, HH, and PR. The first draft was
written by DM. All authors reviewed and contributed to the final version. DM
implemented the agent-based simulations, analyzed the corresponding results,
built the robot platform, and developed the vision-based robot controller.
Supplementary videos, code, and data were deposited by DM, who also produced
the supplementary videos and their components, as well as the main figures. RB
created supplementary figure S1. All these materials were reviewed by all
coauthors. Preliminary experiments for data collection were carried out by DM
and YZ. The experimental infrastructure was prepared for final data
collection, and the robot experiments were collaboratively planned and
conducted by DM, DS, and YZ. The analysis pipeline for collected robot data
was established and corresponding results were analyzed by DM and NM. NM
deposited the corresponding code.

### Code and Data Accessibility

All relevant code and datasets used for our findings have been deposited: See
94 for agent-based simulation framework and simulation data, 63 for
experimental data, 95 for replay and analysis tool for experimental data and
57 for the vision-based robot controller. Supplementary Videos are available
on the TIB-AV portal of the Leibniz Information Centre for Science and
Technology University Library as a video series 61, 64, 65, 66 and via Zenodo
in 4K resolution53.

### Disclosure Statement

During the preparation of this work, DM used the large language model (LLM)
GPT-4/4o to improve various parts of the text for (1) readability, flow and
clarity, (2) grammatical correctness, and (3) simplicity. The LLM was prompted
to rate snippets of the draft according to a list of criteria corresponding to
these three points and to provide detailed reasoning for the ratings.
Grammatical improvements were then added to the text, after which single
synonyms, alternative word compositions, and in some cases sentence-level
reformulations were introduced. No changes other than stylistic or grammatical
were introduced during this process. After using this tool, all authors
reviewed and edited the content as needed and take full responsibility for the
content of the publication.

### Competing Interests

The authors declare no competing interests.

### Funding

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC 2002/1 “Science of
Intelligence” – project number 390523135.

## Supplementary Notes

### Calculating Visual Social Interactions

Social interactions are calculated from (1) the area of visual projection
“blobs” ($V_{i}(\phi_{i},t)$) and (2) their edges
($(\frac{\partial{V_{i}(\phi_{i},t)}}{\partial\phi_{i}})^{2}$). These are
modulated by a trigonometric mask allowing differentiating between (1) front
and back ($\cos(\phi_{i})$) or (2) between left and right ($\sin(\phi_{i})$).
These trigonometric masks model the symmetry of the agents’ visuo-motor
system. For instance, an attraction force in the turning response of the focal
agent (turning towards a conspecific) will depend on which side the focal
agent perceives the corresponding visual blob. The same attraction force shall
yield turning right when the other agent is on the right but turning left when
the other agent is on the left. Similarly, an attraction force in the focal
agent’s velocity (accelerating towards a conspecific) shall yield acceleration
when the other agent is in front of, but deceleration when the other agent is
behind the focal agent. Note that any odd function (changing sign at 0) would
suffice to model left-right symmetry and any even function (changing sign at
$\pm\frac{\pi}{2}$) to model front-back symmetry of this kind.

The modulated area term (e.g. $\int\cos{\phi}V_{i}(\phi,t)d\phi$) increases
with larger visual blobs i.e. closer visible agents, hence it corresponds to
short range interactions. The modulated edge term (e.g.
$\int\cos{\phi}\frac{V_{i}(\phi,t)}{d\phi}d\phi$) increases with narrower
blobs, i.e. farther visible agents, hence it corresponds to long-range
interactions. Summed with their opposite signs these terms generate long-range
attraction and short-range repulsion forces on agents purely arising from
visual perception (see Fig. 1C).

To give a detailed example for acceleration response we refer to Bastien and
Romanczuk 45 Fig. 2 and Fig. S1: When a another agent is far in front of the
focal agent it’s corresponding visual projection blob is central and narrow.
As a result, its’ small overlapping area with the cosine mask causes a small
short-range repulsion. At the same time its’ high edges near to each other
around $\phi_{i}=0$ where $\cos(\phi_{i})$ has a maximum, cause a large
positive long-range attraction. Summed together these two cause an overall
positive change in the agent’s velocity (acceleration). In contrast, when
another agent is too close in front of the focal one, the corresponding visual
blob is wide. Therefore when modulated with $\cos(\phi)$, edges become lower
as they get more peripheral. This results in smaller long-range attraction
than in the previous case. Due to the large modulated area of the blob now a
large short-range repulsion arises. The sum of these two will result in an
overall negative change in velocity (deceleration).

Similarly, the turning rate of the agent is modulated via long-range
attraction and short-range repulsion forces calculated from projection blob
area and edge size. Note, that turning behavior is shaped by the $\sin$ mask
instead of $\cos$ that was used for acceleration. When an agents is far ahead,
the focal agent turns towards it according to the overall attraction coming
from the corresponding small blob area and it’s close edges. On the other
hand, when the visible individual is near, the focal agent turns away due to
the overall repulsion coming from the large blob area and small modulated
edges.

Figure S1:  Calculation of social acceleration and turning response The visual
input of a focal agent (white disc) with another agent directly ahead (striped
disc) is modulated through cos (left) and sin (right) masks to calculate the
social acceleration and turning response respectively. The modualted area
terms (i.e. $\int\cos{\phi}V_{i}(\phi,t)d\phi$ and
$\int\sin{\phi}V_{i}(\phi,t)d\phi$) are shown with orange highlight while the
modulated edge terms (i.e. $\int\cos{\phi}\frac{V_{i}(\phi,t)}{d\phi}d\phi$
and $\int\sin{\phi}\frac{V_{i}(\phi,t)}{d\phi}d\phi$) with dotted lines.

### Limited FOV

When limiting agents’ FOV, partial blobs might occur on the peripheries of the
active visual field (i.e. where $|\phi|\leq\phi_{L}$). Note that even if a
partial projection blob is small — and small visual blobs correspond to far
away agents — it might actually correspond to a nearby agent that would yield
large blobs when fully visible otherwise (See also Fig. 2A). This mismatch
between partial detection blob size and distance information might perturb
collective movement as it introduces unwanted attraction on the peripheries of
the visual field (boundary effects). To overcome such corrupted sensory
information we used heuristics in both simulations and in robots. In
simulations, when a detection blob is only partially in the active visual
field (i.e. $V_{i}(\phi_{L}+1,t)=1$ or $V_{i}(-\phi_{L}-1,t)=1$), we assume
that it’s a partial projection blob and we recover the whole blob even if that
extends over the limits of the active vision. In robot agents we use a similar
method. Here, we have a 2D visual perception where bounding boxes have not
only widths, but heights as well, and we can use this information to recover
the full detection box by assuming detection boxes of other robots to be
square. If this is not the case and the detection box touches the left- or
rightmost pixel of the robot’s camera stream, we assume that it is a partial
detection and we extend the resulting projection blob according to the height
of the detection box instead of using only its partial width. Our heuristic
recovery of peripheral blobs is based on the assumption that partially seeing
a nearby conspecific can be cognitively distinguished from fully seeing a far
away conspecific in natural agents. This is a reasonable assumption, if agents
can detect and recognize each other as conspecifics, or they have an internal
model about the typical size of group members.

### Simulations On a Torus

To allow direct comparison of collective movement of simulated groups with
limited FOV with those presented by Bastien and Romanczuk 45 we used an arena
with periodic boundaries. The width and height of the arena was set to 900
pixels, which is approximately 164 times the radius agents ($R_{A}=5.5$
pixels). An arena with periodic boundaries allows agents leaving on one side
to reappear instantaneously on the other side. This solution is equivalent to
simulating collective movement on the surface of a torus. Simulating
collective movement in such an arena mimics the properties of an unlimited
environment when the group is cohesive. When due to a limited FOV the group
fractures, on the other hand, agents previously broken away will reappear in
the arena allowing fusion of the group, that an unlimited environment would
not support resulting in catastrophic group fragmentation.

Visual interactions in such a toroidal environment can carry some ambiguities,
as with an unlimited visual range a focal agent could theoretically have
access to multiple visual representations of the same agent. For example if a
visible agent is straight ahead of the focal one, due to the periodic boundary
conditions the visual representation of this agent could be accessed both from
a relative direction of 0 (from the front) and from a relative angle of
$\pm\pi$ (from the back through the periodic boundaries). In other words,
multiple visual “copies” of the same agent would be accessible in different
relative directions and to different distances.

To account for such ambiguities on a torus, we always take only a single
visual representation of a visible agent into account when calculating the
visual input of the focal individual. To choose which “copy” of the visible
agent shall be used to create the visual projection field of the focal agent,
we use a simple tiling method of the environment. First we generate eight
copies of the visible agent $i$ with position $(x_{i},y_{i})$ by adding or
subtracting the arena width ($W$) or height ($H$) from its first and second
coordinates. This yields a set of agent copies $C_{i}$ including the original
visible agent coordinates as follows:

$C_{i}=\\{\vec{x}_{j}=(x_{j},y_{j})\ |\ x_{j}\in[x_{i}-W,x_{i},x_{i}+W],\
y_{j}\in[y_{i}-H,y_{i},y_{i}+H]\\}$ (13)

From these we then choose the agent copy with position $\vec{x}_{k}$ that
minimizes euclidean distance to the focal agent, i.e.

$\vec{x}_{k}\in C_{i}\ |\
k=\operatorname*{arg\,min}_{j}\sqrt{(x_{j}-x_{i})^{2}+(y_{j}-y_{i})^{2}}$ (14)

The visual projection corresponding to the selected copy is then added to the
visual projection field of the focal agent.

### Metrics

During this study we used five key metrics: group polarization ($P$), mean
inter-individual distance ($\tilde{D}$), the ratio of simulation/experiment
time agents spent in overlap or in collision avoidance mode for actual robots
($R_{o}$), group circularity ($RCA$), and group fragmentation through
hierarchical clustering of the agents.

Being $\vec{x}_{i}(t)$ the position vector and $\vec{n}_{i}(t)$ the unit
heading vector of agent $i$ at time t and, $N_{A}$ is the total number of
agents, $\vec{h}_{i}(t)$ is the position vector of the vertex $i$ of the
convex hull surrounding the group (ordered by adjacency) with $N_{h}$ vertices
in total the above metrics are formalized as follows:

Polarization order ($P$) is defined as the length of the mean of the agents’
unit heading vectors and it indicates the uniformity of movement direction
within the group. It reaches high values close to 1 when all agents move in
similar directions while low values close to zero for either unorganized
movements where all agents move in different directions or when the group is
fragmented into subgroups.

$P(t)=\frac{1}{N_{A}}\left|\left|\sum_{i=1}^{N_{A}}\vec{n}_{i}(t)\right|\right|$
(15)

The mean inter-individual distance ($D$) is defined as the average distance
between agents.

$\tilde{D}(t)=\frac{2}{N_{A}(N_{A}-1)}\sum_{i=1}^{N_{A}}\sum_{0<j<i}\left\|\vec{x}_{i}(t)-\vec{x}_{j}(t)\right\|$
(16)

Given the diameter of the surrounding convex hull defined by the agents in the
group (i.e. the distance between the farthest two non-adjacent vertices):

$d(t)=\begin{cases}\max_{i,j}\left|\left|\vec{h}_{i}(t)-\vec{h}_{j}(t)\right|\right|\quad\text{such
that }1<|i-j|<(N_{A}-1)\quad&\text{if }N_{h}>3\\\
\max_{i,j}\left|\left|\vec{h}_{i}(t)-\vec{h}_{j}(t)\right|\right|\quad&\text{if
}N_{h}\leq 3\end{cases}$ (17)

And using the circular convention of $\vec{h}_{N_{h}+1}=\vec{h}_{1}$, the
convex hull’s area as:

$A(t)=\frac{1}{2}\sum_{i}\left(h_{i,1}(t)h_{i+1,2}(t)-h_{i+1,1}(t)h_{i,2}(t)\right)$
(18)

The circularity of groups ($RCA$) can be defined by comparing the convex
hull’s area with that of a circle with the same diameter ($A_{C}(d(t))$). An
$RCA$ value close to 1 signifies a perfectly circular group, whereas values
approaching 0 indicate elongated formations.

$RCA(t)=\frac{A(t)}{A_{C}(d(t))}=\frac{A(t)d(t)^{2}\pi}{4}$ (19)

In simulated agents $R_{o}^{sim}$ shows what proportion of the average
simulation time (in percent) agents spent overlapping and is directly related
to agent-agent collision probability in embodied groups. Consequently,
$R_{o}^{sim}$ reaches 100 if all agents overlapped through the whole
simulation time and 0 if no agent-agent overlaps were observed.

$D_{ij}$ being the distance between agent $i$ and $j$ and $R_{A}$ the radius
of agents, $R_{o}^{sim}$ can be formulated as follows:

$R_{o}^{sim}=\frac{1}{2N_{A}}\sum_{i=1}^{N_{A}}\frac{T_{o}}{T}100=\frac{1}{2N_{A}}\sum_{i=1}^{N_{A}}\frac{\\#\\{t:\exists
j|D_{ij}(t)<2R_{A},j\in\\{1,...,N_{A}\\},j\neq i\\}}{T}100$ (20)

In robot experiments overlaps are not possible, hence $R_{o}^{exp}$ is defined
by the proportion of the total experiment time (in percent) robots spent in
collision avoidance mode due to other robots. As a result, $R_{o}^{exp}$
reaches 100 if all robots spent the whole experiment in obstacle avoidance
mode due to other robots and 0 if no robot avoidance protocol has been
activated at all.

$C_{i}(t)=1$ denoting robot $i$ being in robot avoidance mode at time $t$
while this value being 0 denoting normal operation of robot $i$, $R_{o}^{exp}$
can be formulated as:

$R_{o}^{exp}=\frac{1}{N_{A}}\sum_{i=1}^{N_{A}}\frac{T_{c}}{T}100=\frac{1}{N_{A}}\sum_{i=1}^{N_{A}}\frac{\\#\\{t:C_{i}(t)=1\\}}{T}100$
(21)

Furthermore, we clustered agents according to their movement similarity (i.e.
similarity based on their heading angle and distance from others) to quantify
group cohesion. This way we could define polarized and cohesive subgroups even
if these were otherwise not forming a single group. This is possible, as
agents’ can lose sight of each other due to their limited FOV leading to
multiple leaders and subgroups following them. We then calculated the size of
the largest polarized, cohesive subgroup $N_{clus}^{max}$. This is
$N_{clus}^{max}=N_{A}=10$ if all agents moving together in close vicinity
towards the same direction and 1 if each agent is relatively far from others
while they move in different directions. We chose the threshold of the
clustering method heuristically as 0.275, such that it yields clustering
comparable with our judgment for polarized subgroups.

For hierarchical clustering in simulations we used our custom simulation
framework 54 and the fastcluster 96 python package with Ward clustering
scheme, with dissimilarity metric defined as follows. Given the inter-
individual distance matrix in time $t$ as

$\bar{\bar{\mathbf{D}}}(t):D_{i,j}(t)=||\vec{x}_{i}(t)-\vec{x}_{j}(t)||$ (22)

The normalized distance matrix is defined as

$\bar{\bar{\mathbf{D^{N}}}}(t):D^{N}_{i,j}(t)=\frac{|\operatorname*{median}(\bar{\bar{\mathbf{D}}}(t))-D_{i,j}(t)|}{\max(\bar{\bar{\mathbf{D}}}(t))}$
(23)

And the final dissimilarity matrix is defined as:

$\bar{\bar{\mathbf{M}}}(t):M_{i,j}(t)=\frac{(1-P_{i,j}(t)+D^{N}_{i,j}(t)}{2}$
(24)

where $P_{i,j}(t)$ is the polarization value between agent $i$ and $j$ in time
$t$, i.e.

$P_{i,j}(t)=\frac{||\vec{n}_{i}(t)+\vec{n}_{j}(t)||}{2}$ (25)

Similarly, for robot data we used hierarchical clustering taking the robots’
orientation and distance into consideration using our custom visualization and
analysis tool developed in Julia 97, 95. We used the clustering threshold of
0.1653 and dissimilarity matrix:

$M_{i,j}(t)=1-\sqrt{\left(1-\frac{\left|\left|\vec{x}_{i}(t)-\vec{x}_{j}(t)\right|\right|}{r_{\text{max}}}\right)\frac{\vec{n}_{i}(t)\cdot\vec{n}_{j}(t)+1}{2}}$
(26)

where

$r_{max}=\max_{i,j,k,l,t_{*}}\sqrt{\left(x_{i,1}(t_{1})-x_{j,1}(t_{2})\right)^{2}+\left(x_{k,2}(t_{3})-x_{l,2}(t_{4})\right)^{2}}$
(27)

### Collision Avoidance

During experiments, depending on the selected model parameters, it can happen
that robots move too close to each other. To avoid possible robot-robot
collisions and to reflect robots from arena walls we implemented a custom
obstacle avoidance process running parallel with the robot controller. When an
obstacle ahead is detected with one of the five frontal proximity sensors of
the Thymio II base robot, a recursive emergency protocol takes over, stops the
robot and rotates it to the direction where the obstacle signal is lower
(turning away from the obstacle) until the path ahead is clear. Symmetric
obstacles are avoided with a fixed-direction, 90 degree turn. In case the
proximity sensors in the back of the robots are activated, robots stop and
retry to act according to their vision-based controller in every two seconds.
This is practical when robots would go backwards but the arena walls do not
allow them. Robot collision avoidance events are not recorded, but given the
fixed, relatively high rotational speed of the robots during collision
avoidance, it is easily detectable from robot tracking data using the angular
velocity of robots and their distance from other robots or from walls.

### From Detection to Robot Movement

To generate model input from the agents’ raw camera streams robots have to
detect each other in their environment. In our framework, robots use the “SSD
MobileNet V2 FPNLite 320$\times$320” object detector, a freely available
convolutional neural network-based solution 98, 99 with a feature-pyramid
(FPN) feature extractor network 100. Such an object detector can provide
bounding box coordinates of detected robots on the camera frames after
training. We trained the network with ca. 2000 original images taken by robots
in realistic scenarios using tensorflow 2101. The dataset was labeled and
augmented using Roboflow 102. The trained model was full-integer quantized and
compiled such that inference could be delegated to Google Coral Edge TPU USB
accelerator devices 103 onboard. This enabled a stable 5 Hz object detection
rate on the edge devices with a 320 by 200 pixel (stretched to 320 $\times$
320 pixel) image resolution during experiments. Bounding boxes that overlapped
more than 50% or that was smaller than 3 pixels wide have been filtered during
experiments. After detecting robots within the FOV, the resulting bounding
boxes are projected to the horizontal axis to produce a visual projection
field (See Fig. 5B, steps 2-3). As attraction-repulsion forces depend on the
width of the resulting blobs and blobs are truncated on the peripheries due to
a limited FOV, we recovered partial peripheral detections as described in
section Limited FOV to avoid boundary effects. The resulting visual projection
field is transformed to a pair of theoretical acceleration and turning rate
values using the vision-based model. The output is then translated to motor
velocity values via a differential steering algorithm 104. The absolute motor
velocity of robots has been limited to ca. 75% of the maximum motor velocities
of the ThymioII base robots.

Parameter |  Description | Value | Unit | Variable Name
---|---|---|---|---
$W$ |  Arena width | 900 | px | ENV_WIDTH
$H$ |  Arena height | 900 | px | ENV_HEIGHT
$N_{ret}$ |  Resolution of the visual projection input of agents | 320 | px | VISUAL_FIELD_RESOLUTION
$R_{A}$ |  Half bodylength of agents | 5.5 | px | RADIUS_AGENT
$\gamma$ |  Self-propulsion preference factor | 0.1 | - | VF_GAM
$v_{0}$ |  Preferred individual speed | 1 | px/ts | VF_V0
$\alpha_{1}$ |  Parameter controlling front-back equilibrium distance $L^{fb}_{eq}$ | 0.09 | - | VF_ALP1
$\beta_{1}$ |  Parameter controlling left-right equilibrium distance $L^{lr}_{eq}$ | 0.09 | - | VF_BET1
$\alpha_{0}$ |  Parameter controlling social acceleration/deceleration | var | - | VF_ALP0
$\beta_{0}$ |  Parameter controlling social turning response | var | - | VF_BET0
$2\phi_{L}$ |  Field of view | var | - | AGENT_FOV
$T$ |  Simulation time | 20000 | ts | T
$N$ |  Number of agents | 10 | - | N
- |  Arena type | torus/walls | - | BOUNDARY
- |  Vision range |  2000 (i.e. $\inf$) | px | VISION_RANGE
Table 1: Fixed and varied (var) parameters of the simulation framework used for Fig. 2 and 4. px denotes pixels, and ts simulation timesteps. Parameter |  Description | Value | Unit
---|---|---|---
$W$ |  Arena width | 900 | cm
$H$ |  Arena height | 600 | cm
$N_{ret}$ |  Resolution of the visual projection input of agents | 320 | px
$R_{A}$ |  Half bodylength of agents | 5.5 | cm
$\gamma$ |  Self-propulsion preference factor | 0.1 | -
$v_{0}$ |  Preferred individual speed | 90 | arbitrary motor unit
$\alpha_{1}$ |  Parameter controlling front-back equilibrium distance $L^{fb}_{eq}$ | 0.09 | -
$\beta_{1}$ |  Parameter controlling left-right equilibrium distance $L^{lr}_{eq}$ | 0.09 | -
$\alpha_{0}$ |  Parameter controlling social acceleration/deceleration | var | -
$\beta_{0}$ |  Parameter controlling social turning response | var | -
$2\phi_{L}$ |  Field of view | ca. 175 | deg
$T$ |  Experiment time | ca. 60 | min
$N$ |  Number of agents | 10 | -
- |  Arena type | walls | -
- |  Vision range | 3-6 | m
Table 2: Parameters used during robot experiments for Fig. 7.

## References

  * Pavlov et al. [2000] DS Pavlov, AO Kasumyan, et al.  Patterns and mechanisms of schooling behavior in fish: a review.  _Journal of Ichthyology_ , 40(2):S163, 2000.
  * King and Sumpter [2012] Andrew J King and David JT Sumpter.  Murmurations.  _Current Biology_ , 22(4):R112–R114, 2012.
  * Buhl et al. [2006] Jerome Buhl, David JT Sumpter, Iain D Couzin, Joe J Hale, Emma Despland, Edgar R Miller, and Steve J Simpson.  From disorder to order in marching locusts.  _Science_ , 312(5778):1402–1406, 2006.
  * Ioannou et al. [2012] Christos C Ioannou, Vishwesha Guttal, and Iain D Couzin.  Predatory fish select for coordinated collective motion in virtual prey.  _Science_ , 337(6099):1212–1215, 2012.
  * Clark and Mangel [1986] Colin W Clark and Marc Mangel.  The evolutionary advantages of group foraging.  _Theoretical population biology_ , 30(1):45–75, 1986.
  * Pitcher et al. [1982] TJ Pitcher, AE Magurran, and IJ Winfield.  Fish in larger shoals find food faster.  _Behavioral Ecology and Sociobiology_ , 10:149–151, 1982.
  * Sword et al. [2005] Gregory A Sword, Patrick D Lorch, and Darryl T Gwynne.  Migratory bands give crickets protection.  _Nature_ , 433(7027):703–703, 2005.
  * Beauchamp [2021] Guy Beauchamp.  Flocking in birds increases annual adult survival in a global analysis.  _Oecologia_ , 197(2):387–394, 2021.
  * Morelli et al. [2019] Federico Morelli, Yanina Benedetti, Mario Díaz, Tomas Grim, Juan Diego Ibáñez-Álamo, Jukka Jokimäki, Marja-Liisa Kaisanlahti-Jokimäki, Kunter Tätte, Gábor Markó, Yiting Jiang, et al.  Contagious fear: Escape behavior increases with flock size in european gregarious birds.  _Ecology and evolution_ , 9(10):6096–6104, 2019.
  * Elwin et al. [2019] Matthew L Elwin, Randy A Freeman, and Kevin M Lynch.  Distributed environmental monitoring with finite element robots.  _IEEE Transactions on Robotics_ , 36(2):380–398, 2019.
  * Schmickl et al. [2011] Thomas Schmickl, Ronald Thenius, Christoph Moslinger, Jon Timmis, Andy Tyrrell, Mark Read, James Hilder, Jose Halloy, Alexandre Campo, Cesare Stefanini, et al.  Cocoro–the self-aware underwater swarm.  In _2011 Fifth IEEE Conference on Self-Adaptive and Self-Organizing Systems Workshops_ , pages 120–126. IEEE, 2011.
  * Leonard et al. [2010] Naomi E Leonard, Derek A Paley, Russ E Davis, David M Fratantoni, Francois Lekien, and Fumin Zhang.  Coordinated control of an underwater glider fleet in an adaptive ocean sampling field experiment in monterey bay.  _Journal of Field Robotics_ , 27(6):718–740, 2010.
  * Luvisutto et al. [2022] Alberto Luvisutto, Aaesha Al Shehhi, Nikita Mankovskii, Federico Renda, Cesare Stefanini, and Giulia De Masi.  Robotic swarm for marine and submarine missions: Challenges and perspectives.  In _2022 IEEE/OES Autonomous Underwater Vehicles Symposium (AUV)_ , pages 1–8. IEEE, 2022.
  * Albani et al. [2017] Dario Albani, Joris IJsselmuiden, Ramon Haken, and Vito Trianni.  Monitoring and mapping with robot swarms for agricultural applications.  In _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pages 1–6. IEEE, 2017.
  * Stachniss et al. [2022] Cyrill Stachniss, Gert Kootstra, Daniele Nardi, and Vito Trianni.  Monitoring and mapping of crop fields with uav swarms based on information gain.  In _Distributed Autonomous Robotic Systems: 15th International Symposium_ , volume 22, page 306. Springer Nature, 2022.
  * Arnold et al. [2018] Ross D Arnold, Hiroyuki Yamaguchi, and Toshiyuki Tanaka.  Search and rescue with autonomous flying robots through behavior-based cooperative intelligence.  _Journal of International Humanitarian Action_ , 3(1):1–18, 2018.
  * Kang et al. [2019] Chang-kwon Kang, Farbod Fahimi, Rob Griffin, D Brian Landrum, Bryan Mesmer, Guangsheng Zhang, Taeyoung Lee, Hikaru Aono, Jeremy Pohly, Jesse McCain, et al.  Marsbee-swarm of flapping wing flyers for enhanced mars exploration.  Technical report, 2019.
  * Yasuda et al. [2014] Toshiyuki Yasuda, Akitoshi Adachi, and Kazuhiro Ohkura.  Self-organized flocking of a mobile robot swarm by topological distance-based interactions.  In _2014 IEEE/SICE International Symposium on System Integration_ , pages 106–111. IEEE, 2014.
  * Vásárhelyi et al. [2018] Gábor Vásárhelyi, Csaba Virágh, Gergő Somorjai, Tamás Nepusz, Agoston E Eiben, and Tamás Vicsek.  Optimized flocking of autonomous drones in confined environments.  _Science Robotics_ , 3(20):eaat3536, 2018.
  * Cheraghi et al. [2020] Ahmad Reza Cheraghi, Asma Ben Janete, and Kalman Graffi.  Robot swarm flocking on a 2d triangular graph.  In _2020 5th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS)_ , pages 154–162. IEEE, 2020.
  * Ban et al. [2021] Zhe Ban, Junyan Hu, Barry Lennox, and Farshad Arvin.  Self-organised collision-free flocking mechanism in heterogeneous robot swarms.  _Mobile Networks and Applications_ , 26(6):2461–2471, 2021.
  * Bahaidarah et al. [2024] Mazen Bahaidarah, Fatemeh Rekabi-Bana, Ognjen Marjanovic, and Farshad Arvin.  Swarm flocking using optimisation for a self-organised collective motion.  _Swarm and Evolutionary Computation_ , page 101491, 2024.
  * Turgut et al. [2008] Ali E Turgut, Hande Çelikkanat, Fatih Gökçe, and Erol Şahin.  Self-organized flocking in mobile robot swarms.  _Swarm Intelligence_ , 2:97–120, 2008.
  * Ducatelle et al. [2011] Frederick Ducatelle, Gianni A Di Caro, Carlo Pinciroli, Francesco Mondada, and Luca Gambardella.  Communication assisted navigation in robotic swarms: self-organization and cooperation.  In _2011 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 4981–4988. IEEE, 2011.
  * Ferrante et al. [2014] Eliseo Ferrante, Ali Emre Turgut, Alessandro Stranieri, Carlo Pinciroli, Mauro Birattari, and Marco Dorigo.  A self-adaptive communication strategy for flocking in stationary and non-stationary environments.  _Natural Computing_ , 13:225–245, 2014.
  * Weinstein et al. [2018] Aaron Weinstein, Adam Cho, Giuseppe Loianno, and Vijay Kumar.  Visual inertial odometry swarm: An autonomous swarm of vision-based quadrotors.  _IEEE Robotics and Automation Letters_ , 3(3):1801–1807, 2018.
  * Bennis et al. [2019] Ismail Bennis, Alain Gaugue, and Michel Menard.  Short-range and long-range cooperative communication for littoral environment monitoring.  In _2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC)_, pages 1958–1963. IEEE, 2019.
  * Zhao et al. [2020] Taifei Zhao, Ying Xie, Shan Xu, Jing Wang, and Hua Wang.  Flocking of uav formation with wireless ultraviolet communication.  _Wireless Personal Communications_ , 114:2551–2568, 2020.
  * Higgins et al. [2009] Fiona Higgins, Allan Tomlinson, and Keith M Martin.  Survey on security challenges for swarm robotics.  In _2009 Fifth International Conference on Autonomic and Autonomous Systems_ , pages 307–312. IEEE, 2009.
  * Zheng et al. [2022] Yating Zheng, Cristián Huepe, and Zhangang Han.  Experimental capabilities and limitations of a position-based control algorithm for swarm robotics.  _Adaptive Behavior_ , 30(1):19–35, 2022.
  * Casas and Mitschele-Thiel [2018] Victor Casas and Andreas Mitschele-Thiel.  On the impact of communication delays on uavs flocking behavior.  In _2018 IEEE Wireless Communications and Networking Conference Workshops (WCNCW)_ , pages 67–72. IEEE, 2018.
  * Moussaid et al. [2009] Mehdi Moussaid, Simon Garnier, Guy Theraulaz, and Dirk Helbing.  Collective information processing and pattern formation in swarms, flocks, and crowds.  _Topics in Cognitive Science_ , 1(3):469–497, 2009.
  * Wiandt et al. [2015] Bernat Wiandt, Andras Kokuti, and Vilmos Simon.  Application of collective movement in real life scenarios: Overview of current flocking solutions.  _Scalable Computing: Practice and Experience_ , 16(3):233–248, 2015.
  * Wang et al. [2023] Shuo Wang, Yongcai Wang, Deying Li, and Qianchuan Zhao.  Distributed relative localization algorithms for multi-robot networks: A survey.  _Sensors_ , 23(5):2399, 2023.
  * Hamann et al. [2014] Heiko Hamann, Gabriele Valentini, Yara Khaluf, and Marco Dorigo.  Derivation of a micro-macro link for collective decision-making systems: Uncover network features based on drift measurements.  In _Parallel Problem Solving from Nature–PPSN XIII: 13th International Conference, Ljubljana, Slovenia, September 13-17, 2014. Proceedings 13_ , pages 181–190. Springer, 2014.
  * Hamann et al. [2016a] Heiko Hamann, Gabriele Valentini, and Marco Dorigo.  Population coding: A new design paradigm for embodied distributed systems.  In _Swarm Intelligence: 10th International Conference, ANTS 2016, Brussels, Belgium, September 7-9, 2016, Proceedings 10_ , pages 173–184. Springer, 2016a.
  * Hamann [2018] Heiko Hamann.  _Swarm robotics: A formal approach_ , volume 221.  Springer, 2018.
  * Moeslinger et al. [2011] Christoph Moeslinger, Thomas Schmickl, and Karl Crailsheim.  A minimalist flocking algorithm for swarm robots.  In _Advances in Artificial Life. Darwin Meets von Neumann: 10th European Conference, ECAL 2009, Budapest, Hungary, September 13-16, 2009, Revised Selected Papers, Part II 10_ , pages 375–382. Springer, 2011.
  * Chen et al. [2022] Siyuan Chen, Dong Yin, and Yifeng Niu.  A survey of robot swarms’ relative localization method.  _Sensors_ , 22(12):4424, 2022.
  * Bajec and Heppner [2009] Iztok Lebar Bajec and Frank H Heppner.  Organized flight in birds.  _Animal Behaviour_ , 78(4):777–789, 2009.
  * Petráček et al. [2020] Pavel Petráček, Viktor Walter, Tomáš Báča, and Martin Saska.  Bio-inspired compact swarms of unmanned aerial vehicles without communication and external localization.  _Bioinspiration & Biomimetics_, 16(2):026009, 2020.
  * Walter et al. [2018] Viktor Walter, Nicolas Staub, Martin Saska, and Antonio Franchi.  Mutual localization of uavs based on blinking ultraviolet markers and 3d time-position hough transform.  In _2018 IEEE 14th International Conference on Automation Science and Engineering (CASE)_ , pages 298–303. IEEE, 2018.
  * Schilling et al. [2021] Fabian Schilling, Fabrizio Schiano, and Dario Floreano.  Vision-based drone flocking in outdoor environments.  _IEEE Robotics and Automation Letters_ , 6(2):2954–2961, 2021.
  * Reynolds [1987] Craig W Reynolds.  Flocks, herds and schools: A distributed behavioral model.  In _Proceedings of the 14th annual conference on Computer graphics and interactive techniques_ , pages 25–34, 1987.
  * Bastien and Romanczuk [2020] Renaud Bastien and Pawel Romanczuk.  A model of collective behavior based purely on vision.  _Science advances_ , 6(6):eaay0792, 2020.
  * Vicsek et al. [1995] Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet.  Novel type of phase transition in a system of self-driven particles.  _Physical review letters_ , 75(6):1226, 1995.
  * Couzin et al. [2002] Iain D Couzin, Jens Krause, Richard James, Graeme D Ruxton, and Nigel R Franks.  Collective memory and spatial sorting in animal groups.  _Journal of theoretical biology_ , 218(1):1–11, 2002.
  * Romanczuk et al. [2009] Pawel Romanczuk, Iain D Couzin, and Lutz Schimansky-Geier.  Collective motion due to individual escape and pursuit response.  _Physical Review Letters_ , 102(1):010602, 2009.
  * Ferrante et al. [2012] Eliseo Ferrante, Ali Emre Turgut, Cristián Huepe, Alessandro Stranieri, Carlo Pinciroli, and Marco Dorigo.  Self-organized flocking with a mobile robot swarm: a novel motion control method.  _Adaptive Behavior_ , 20(6):460–477, 2012.
  * Murakami et al. [2017] Hisashi Murakami, Takayuki Niizato, and Yukio-Pegio Gunji.  Emergence of a coherent and cohesive swarm based on mutual anticipation.  _Scientific reports_ , 7(1):46447, 2017.
  * Strömbom and Tulevech [2022] Daniel Strömbom and Grace Tulevech.  Attraction vs. alignment as drivers of collective motion.  _Frontiers in Applied Mathematics and Statistics_ , 7:717523, 2022.
  * Soria et al. [2019] Enrica Soria, Fabrizio Schiano, and Dario Floreano.  The Influence of Limited Visual Sensing on the Reynolds Flocking Algorithm.  In _2019 Third IEEE International Conference on Robotic Computing (IRC)_ , pages 138–145, February 2019.  doi: 10.1109/IRC.2019.00028.  URL https://ieeexplore.ieee.org/document/8675652.
  * Mezey et al. [2024a] David Mezey, Renaud Bastien, Zheng Yating, McKee Neal, David Stoll, Heiko Hamann, and Pawel Romanczuk.  Video Data (4K): Purely vision-based collective movement of robots, June 2024a.  URL https://zenodo.org/doi/10.5281/zenodo.12515912.
  * [54] David Mezey.  P34ABM: A novel vision-based mechanistic model wrapped in a pygame-based simulation framework to study social information use in collective foraging.  URL https://github.com/scioip34/ABM.
  * Hemelrijk et al. [2010] Charlotte K. Hemelrijk, Hanno Hildenbrandt, José Reinders, and Eize J. Stamhuis.  Emergence of Oblong School Shape: Models and Empirical Data of Fish.  _Ethology_ , 116(11):1099–1112, 2010.  ISSN 1439-0310.  doi: 10.1111/j.1439-0310.2010.01818.x.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1439-0310.2010.01818.x.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1439-0310.2010.01818.x.
  * Tunstrøm et al. [2013] Kolbjørn Tunstrøm, Yael Katz, Christos C. Ioannou, Cristián Huepe, Matthew J. Lutz, and Iain D. Couzin.  Collective States, Multistability and Transitional Behavior in Schooling Fish.  _PLOS Computational Biology_ , 9(2):e1002915, February 2013.  ISSN 1553-7358.  doi: 10.1371/journal.pcbi.1002915.  URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002915.  Publisher: Public Library of Science.
  * Mezey [2024a] David Mezey.  VSWRM : Vision-based decentralized controller for swarm robotics, June 2024a.  URL https://github.com/mezdahun/VisualSwarm.
  * Mondada et al. [2017] Francesco Mondada, Michael Bonani, Fanny Riedo, Manon Briod, Lea Pereyre, Philippe Retornaz, and Stephane Magnenat.  Bringing Robotics to Formal Education: The Thymio Open-Source Hardware Robot.  _IEEE Robotics & Automation Magazine_, PP:1–1, February 2017.  doi: 10.1109/MRA.2016.2636372.
  * Yilmaz and Bayindir [2022] Zahir Yilmaz and Levent Bayindir.  Lidar-based robot detection and positioning using machine learning methods.  _Balkan Journal of Electrical and Computer Engineering_ , 10(2):214–223, 2022.
  * Dietrich and Dörr [2019] Robin Dietrich and Stefan Dörr.  Deep learning-based mutual detection and collaborative localization for mobile robot fleets using solely 2d lidar sensors.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6706–6713. IEEE, 2019.
  * Mezey [2024b] David Mezey.  Video 1: Purely vision-based flocking of ten robots.  Mezey, David, 2024b.  URL https://doi.org/10.5446/68205.  https://doi.org/10.5446/68205.
  * Furtado et al. [2019] Joshua S Furtado, Hugh HT Liu, Gilbert Lai, Herve Lacheray, and Jason Desouza-Coelho.  Comparative analysis of optitrack motion capture systems.  In _Advances in Motion Sensing and Control for Robotic Applications: Selected Papers from the Symposium on Mechatronics, Robotics, and Control (SMRC’18)-CSME International Congress 2018, May 27-30, 2018 Toronto, Canada_ , pages 15–31. Springer, 2019.
  * Mezey et al. [2024b] David Mezey, Renaud Bastien, Yating Zheng, Neal McKee, David Stoll, Heiko Hamann, and Pawel Romanczuk.  Dataset: Purely vision-based collective movement of robots, 2024b.  URL https://depositonce.tu-berlin.de/handle/11303/21882.
  * Mezey [2024c] David Mezey.  Video 2: Fission/fusion of a single robot.  Mezey, David, 2024c.  URL https://doi.org/10.5446/68204.  https://doi.org/10.5446/68204.
  * Mezey [2024d] David Mezey.  Video 3: Fission/fusion of two robots.  Mezey, David, 2024d.  URL https://doi.org/10.5446/68203.  https://doi.org/10.5446/68203.
  * Mezey [2024e] David Mezey.  Video 4: Adding/removing robots on the fly.  Mezey, David, 2024e.  URL https://doi.org/10.5446/68201.  https://doi.org/10.5446/68201.
  * Poel et al. [2021] Winnie Poel, Claudia Winklmayr, and Pawel Romanczuk.  Spatial structure and information transfer in visual networks.  _Frontiers in Physics_ , 9:716576, 2021.
  * Dachner et al. [2022] Gregory C Dachner, Trenton D Wirth, Emily Richmond, and William H Warren.  The visual coupling between neighbours explains local interactions underlying human ‘flocking’.  _Proceedings of the Royal Society B_ , 289(1970):20212089, 2022.
  * Kunz and Hemelrijk [2012] Hanspeter Kunz and Charlotte K Hemelrijk.  Simulations of the social organization of large schools of fish whose perception is obstructed.  _Applied Animal Behaviour Science_ , 138(3-4):142–151, 2012.
  * Davidson et al. [2021] Jacob D Davidson, Matthew MG Sosna, Colin R Twomey, Vivek H Sridhar, Simon P Leblanc, and Iain D Couzin.  Collective detection based on visual information in animal groups.  _Journal of the Royal Society Interface_ , 18(180):20210142, 2021.
  * Mezey et al. [2024c] David Mezey, Dominik Deffner, Ralf HJM Kurvers, and Pawel Romanczuk.  Visual social information use in collective foraging.  _PLOS Computational Biology_ , 20(5):e1012087, 2024c.
  * Schilling et al. [2022] Fabian Schilling, Enrica Soria, and Dario Floreano.  On the scalability of vision-based drone swarms in the presence of occlusions.  _IEEE Access_ , 10:28133–28146, 2022.
  * Jiang et al. [2017] Li Jiang, Luca Giuggioli, Andrea Perna, Ramón Escobedo, Valentin Lecheval, Clément Sire, Zhangang Han, and Guy Theraulaz.  Identifying influential neighbors in animal flocking.  _PLoS computational biology_ , 13(11):e1005822, 2017.
  * Lei et al. [2020] Liu Lei, Ramón Escobedo, Clément Sire, and Guy Theraulaz.  Computational and robotic modeling reveal parsimonious combinations of interactions between individuals in schooling fish.  _PLoS computational biology_ , 16(3):e1007194, 2020.
  * Puy et al. [2024] Andreu Puy, Elisabet Gimeno, Jordi Torrents, Palina Bartashevich, M Carmen Miguel, Romualdo Pastor-Satorras, and Pawel Romanczuk.  Selective social interactions and speed-induced leadership in schooling fish.  _Proceedings of the National Academy of Sciences_ , 121(18):e2309733121, 2024.
  * Hamann et al. [2024] Friedhelm Hamann, Suman Ghosh, Ignacio Juarez Martinez, Tom Hart, Alex Kacelnik, and Guillermo Gallego.  Low-power, continuous remote behavioral localization with event cameras.  In _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2024.
  * Saverino and Gerlai [2008] Cristina Saverino and Robert Gerlai.  The social zebrafish: behavioral responses to conspecific, heterospecific, and computer animated fish.  _Behavioural brain research_ , 191(1):77–87, 2008.
  * Griffiths and Ward [2011] Siân W. Griffiths and Ashley Ward.  _Social Recognition of Conspecifics_ , chapter 9, pages 186–216.  John Wiley & Sons, Ltd, 2011.  ISBN 9781444342536.  doi: https://doi.org/10.1002/9781444342536.ch9.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/9781444342536.ch9.
  * Ward et al. [2020] Ashley JW Ward, Maud IA Kent, and Michael M Webster.  Social recognition and social attraction in group-living fishes.  _Frontiers in Ecology and Evolution_ , 8:15, 2020.
  * Castro et al. [2024] Diego Castro, Franck Ruffier, and Christophe Eloy.  Modeling collective behaviors from optic flow and retinal cues.  _Physical Review Research_ , 6(2):023016, 2024.
  * Cronin et al. [2014] Thomas W Cronin, Sönke Johnsen, N Justin Marshall, and Eric J Warrant.  _Visual ecology_.  Princeton University Press, 2014.
  * Horrocks et al. [2023] Edward AB Horrocks, Isabelle Mareschal, and Aman B Saleem.  Walking humans and running mice: perception and neural encoding of optic flow during self-motion.  _Philosophical Transactions of the Royal Society B_ , 378(1869):20210450, 2023.
  * Franceschini [2014] Nicolas Franceschini.  Small brains, smart machines: from fly vision to robot vision and back again.  _Proceedings of the IEEE_ , 102(5):751–781, 2014.
  * Serres and Ruffier [2017] Julien R Serres and Franck Ruffier.  Optic flow-based collision-free strategies: From insects to robots.  _Arthropod structure & development_, 46(5):703–717, 2017.
  * Horsevad et al. [2022] Nikolaj Horsevad, Hian Lee Kwa, and Roland Bouffanais.  Beyond bio-inspired robotics: how multi-robot systems can support research on collective animal behavior.  _Frontiers in Robotics and AI_ , 9:865414, 2022.
  * Franceschini [2004] Nicolas Franceschini.  Visual guidance based on optic flow: a biorobotic approach.  _Journal of Physiology-Paris_ , 98(1-3):281–292, 2004.
  * Krongauz et al. [2024] David L Krongauz, Amir Ayali, and Gal A Kaminka.  Vision-based collective motion: a locust-inspired reductionist model.  _PLOS Computational Biology_ , 20(1):e1011796, 2024.
  * Hamann et al. [2016b] Heiko Hamann, Yara Khaluf, Jean Botev, Mohammad Divband Soorati, Eliseo Ferrante, Oliver Kosak, Jean-Marc Montanier, Sanaz Mostaghim, Richard Redpath, Jon Timmis, et al.  Hybrid societies: challenges and perspectives in the design of collective behavior in self-organizing systems.  _Frontiers in Robotics and AI_ , 3:14, 2016b.
  * De Schutter et al. [2001] Geoffroy De Schutter, Guy Theraulaz, and Jean-Louis Deneubourg.  Animal–robots collective intelligence.  _annals of mathematics and artificial intelligence_ , 31:223–238, 2001.
  * Kaduk et al. [2023] Julian Kaduk, Müge Cavdan, Knut Drewing, Argiro Vatakis, and Heiko Hamann.  Effects of human-swarm interaction on subjective time perception: Swarm size and speed.  In _HRI_ , pages 456–465, 2023.
  * Hasbach and Bennewitz [2022] Jonas D Hasbach and Maren Bennewitz.  The design of self-organizing human–swarm intelligence.  _Adaptive Behavior_ , 30(4):361–386, 2022.
  * Landgraf et al. [2021] Tim Landgraf, Gregor HW Gebhardt, David Bierbach, Pawel Romanczuk, Lea Musiolek, Verena V Hafner, and Jens Krause.  Animal-in-the-loop: using interactive robotic conspecifics to study social behavior in animal groups.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 4:487–507, 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Mezey et al. [2024d] David Mezey, Bastien Renaud, Yating Zheng, Neal McKee, David Stoll, Heiko Hamann, and Pawel Romanczuk.  Simulation dataset: Purely vision-based collective movement of robots, 2024d.  URL https://depositonce.tu-berlin.de/handle/11303/22024.
  * McKee [2024] Neal McKee.  SwarmViz, March 2024.  URL https://depositonce.tu-berlin.de/handle/11303/21201.
  * Müllner [2013] Daniel Müllner.  fastcluster: Fast hierarchical, agglomerative clustering routines for r and python.  _Journal of Statistical Software_ , 53(9):1–18, 2013.  doi: 10.18637/jss.v053.i09.  URL https://www.jstatsoft.org/index.php/jss/article/view/v053i09.
  * [97] Neal McKee.  Swarmviz: Graphical visualisation and analysis tool for motion tracking data of robotic swarms.  URL https://github.com/nealmckee/swarmviz/tree/main.
  * Huang et al. [2017] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al.  Speed/accuracy trade-offs for modern convolutional object detectors.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7310–7311, 2017.
  * [99] Tensorflow.  Tensorflow 2 object detection zoo, tensorflow/models.  URL https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * Developers [2022] TensorFlow Developers.  Tensorflow.  _Zenodo_ , 2022.
  * Dwyer et al. [2022] B Dwyer, J Nelson, J Solawetz, et al.  Roboflow (version 1.0)[software].  _URL: https://roboflow. com. computer vision_ , 2022.
  * [103] Google.  Google coral usb accelerator.  URL https://coral.ai/products/accelerator.
  * [104] G. W. Lucas.  A tutorial and elementary trajectory model for the differential steering system of robot wheel actuators.  URL https://rossum.sourceforge.net/papers/DiffSteer/.