# ZeroCAP: Zero-Shot Multi-Robot Context Aware Pattern Formation via Large
Language Models

Vishnunandan L. N. Venkatesh, and Byung-Cheol Min This material is based upon
work supported by the National Science Foundation under Grant No. IIS-1846221.
Vishnunandan L. N. Venkatesh and Byung-Cheol Min are with the SMART Lab,
Department of Computer and Information Technology, Purdue University, West
Lafayette, IN 47907, USA<EMAIL_ADDRESS>
###### Abstract

Incorporating language comprehension into robotic operations unlocks
significant advancements in robotics, but also presents distinct challenges,
particularly in executing spatially oriented tasks like pattern formation.
This paper introduces ZeroCAP, a novel system that integrates large language
models with multi-robot systems for zero-shot context aware pattern formation.
Grounded in the principles of language-conditioned robotics, ZeroCAP leverages
the interpretative power of language models to translate natural language
instructions into actionable robotic configurations. This approach combines
the synergy of vision-language models, cutting-edge segmentation techniques
and shape descriptors, enabling the realization of complex, context-driven
pattern formations in the realm of multi robot coordination. Through extensive
experiments, we demonstrate the systems proficiency in executing complex
context aware pattern formations across a spectrum of tasks, from surrounding
and caging objects to infilling regions. This not only validates the system’s
capability to interpret and implement intricate context-driven tasks but also
underscores its adaptability and effectiveness across varied environments and
scenarios.

##  I Introduction

Pattern formation, a key idea in the organization of multi-robot systems [1],
involves arranging robots in a specific way to reach a common objective. This
skill is essential for various purposes such as environmental surveillance,
precise farming, cooperative transportation, and emergency management. The
significance of pattern formation lies in its capacity to utilize the combined
actions of robots to carry out tasks that go beyond what individual robots can
achieve, thus improving effectiveness, adaptability, and robustness. Central
to this endeavor is not just the geometric arrangement of robots, but the
evolution towards purposeful pattern formations—where the design and execution
of patterns are intrinsically linked to the context and objects within the
environment. This nuanced approach marks a significant departure from
traditional methodologies, steering the field towards more interactive and
object-centric tasks that can be applicable to real world scenarios as shown
in Fig 1.

Despite its significance, traditional approaches to pattern formation in
robotics have been predominantly constrained by model/rule-based methods and
machine learning paradigms. These methodologies, while foundational, have
their limitations: model rule-based strategies [1, 2, 3, 4], are often
tailored to specific scenarios/patterns and lack the flexibility to adapt to
new or unmodeled patterns. Reinforcement learning (RL) approaches, on the
other hand, require training time and extensive reward design, making them
impractical for scenarios requiring immediate deployment. Moreover, machine
learning (ML) based methods [5, 6, 7] demand large datasets of annotated or
labeled data for training, a requirement that poses substantial challenges in
dynamic or novel environments. These limitations across all the methodologies
are particularly pronounced in scenarios where pattern formations must be
conditioned with respect to specific objects of interest, underscoring the
need for a more adaptable and intuitive methodology.

Figure 1: A conceptual image illustrating the proposed Zero-Shot multi-robot
Context Aware Pattern (ZeroCAP) formation system, beginning with an initial
environment where a car is parked incorrectly. Upon receiving the command from
human operator to “Surround the incorrectly parked car at the corners!”, the
system identifies the target vehicle and autonomously positions the robot
cones at strategic corner locations, showcasing the successful execution of a
zero-shot context aware pattern formation task.

Integrating language comprehension into robotic systems heralds considerable
progress [8, 9], yet it poses unique challenges, especially in spatially-
focused tasks such as pattern formation. While language models have shown a
remarkable ability as general pattern machines [10] and offer a robust
framework for parsing and generating human-like text, Vision Language Models
(VLMs) falter in accurately localizing spatial tasks, accentuating the need
for techniques that adeptly marry abstract instruction understanding with
their tangible implementation.

Our contribution to this evolving field addresses the challenge previously
identified by [3], focusing on the dynamic assignment of a variable number of
robotic agents to specific target positions for intricate pattern formations.
By combining vision-based processing with the sophisticated interpretative
capabilities of Large Language Models (LLMs), we introduce a Zero-Shot multi-
robot Context Aware Pattern (ZeroCAP) formation system that facilitates zero-
shot, language, and context-conditioned pattern formation. It processes both
images and natural language instructions to identify objects of interest and
generate precise coordinates for robot deployment, enabling immediate and
purposeful pattern formation directly influenced by user instructions.

We distinguish our approach by categorizing pattern formations into three
purposeful types: 1) General pattern formation, where robots arrange
themselves around the object of interest based on user specification; 2)
Infill pattern formation, where robots occupy the interior space of a region
based on user specification; and 3) Caging pattern formation, creating a
boundary around the object. This categorization increases the flexibility of
deployment and signifies a shift from standard geometric patterns to
contextually relevant formations tailored to the specific user context.
Introducing purposeful pattern formation, our system adapts to environmental
cues through natural language instructions, demonstrating language and
context-driven pattern formation’s viability.

The following sections will explore the technical challenges addressed,
methodologies applied, and the significant impact our work has on advancing
robotics towards intuitive, flexible, and context-aware multi-robot systems.

##  II Related Works

Pattern Formation. Multi-robot pattern formation research has explored
centralized control [11], [12] and distributed approaches [1], [13].
Centralized systems face scalability and adaptability challenges, while
distributed methods, including consensus-based [14], leader-follower [15], and
bio-inspired [16], improve scalability but lack robustness due to pattern-
specific, rule-based strategies. Advancements to the feild include potential
field-based methods[17], [4] and artistic formations [3], plus novel
algorithms such as event-triggered mechanisms [2]. However, model/rule-based
strategies typically are designed for specific scenarios, lacking the
adaptability required for new or unseen patterns or even handling user
specific instructions. Conversely, learning-based strategies like Q-learning
[7], [6]while offering more generalizability are often time intensive with
their training and are limited to the fixed patterns that the system was
trained for. Despite the above progress over the years, significant challenges
persist in creating flexible solutions that quickly adapt to contextual user
preferences and situations without heavy pre-configuration, training, or
extensive information exchange. These issues highlight a broader need in the
feild for moving from traditional geometric patterns to contextually relevant
formations tailored to context specific user tasks and environments.

Our approach diverges from these traditional paradigms by adopting zero-shot
learning using a language model for robotics, sidestepping the limitations of
both rule-based and machine learning methodologies. By leveraging natural
language instructions, our system achieves a flexible, intuitive means of
directing multi-robot pattern formation, capable of adaptation to novel
scenarios without the need for extensive dataset training or predefined rules.
This represents an advancement over existing methods centralized or
decentralized.

LLM for Robotics. LLMs are pivotal in robotics for task planning through zero-
shot learning [18, 19, 10], excelling in diverse plan generation [20, 9, 8]
and performance refinement [21]. In the realm of multi-agent systems, LLMs
have been instrumental in fostering agent cooperation, equipping individual
agents with the capabilities to improve interaction dynamics and collaborative
efficacy [22, 23, 24]. Despite their growing utility in robotics, none of
these noteable methodologies cover pattern formation among multi-robot
systems. Vision Language Models, while useful in many applications involving
robot navigation[25] and high-level action control[26, 27], are limited by
their spatial reasoning capabilities [28], impacting their effectiveness in
robotics applications that require precise spatial coordination. Our research
leverages VLMs and LLMs for multi-robot pattern formation, addressing the
spatial task limitations of VLMs and pioneering a linguistically-driven
approach to complex robotic coordination aided by representing shapes using a
node graph.

##  III Problem Description

Our research delves into the nuanced task of deploying robotic agents to form
patterns around or inside a designated object of interest $O$, located within
an environment, $e$. This environment is depicted through an image ($I_{E}$),
paired with a natural language instruction ($Ins$) that outlines the desired
pattern formation task. Operating under a centralized control system, robots,
simplified to points of mass, are coordinated to ensure precise pattern
formation in the context of $O$ and $Ins$.

The procedure begins with the identification and segmentation of $O$ from
$I_{E}$, leading to a mask image that segregates $O$ from the rest of $e$.
This step is crucial to accurately define the geometric shape of $O$. The
shape of $O$ is mathematically modeled as a connected graph $G=(V,E)$ [29],
where $V$ signifies the set of vertices or nodes, outlining discrete points
that mark the shape’s corners or distinct features, and $E$ represents the set
of edges connecting these vertices, thus framing the pattern without self-
connectivity.

Let $R=\\{r_{1},r_{2},\ldots,r_{N}\\}$ represent the list of robotic agents
available, where $N$ is the total number of robots. Given $Ins$, and $I_{E}$,
the system, through a series of steps that include object detection,
segmentation, and shape determination resulting in $G$, leverages an LLM to
interpret and reason natural language instructions in light of $G$. The output
of the system is a list of $(x,y)$ coordinates,
$C_{xy}=\\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{N},y_{N})\\}$, which assigns
each robot $r_{i}$ in $R$ to a specific coordinate $(x_{i},y_{i})$ for
deployment. This assignment facilitates the formation of the specified pattern
around/inside $O$, in accordance with $Ins$.

Formally, this process can be encapsulated as a function $F$, where
$F(I_{E},Ins)\rightarrow C_{xy}$, mapping the inputs to the deployment
coordinates for each robot.

Figure 2: An overview of the ZeroCAP system. It traces the workflow from the
initial natural language instruction and input image of the environment
through to the final deployment of robots, illustrating the sequence of
processing stages—including context identification using Vision Language Model
(VLM), object segmentation, shape description, and Large Language Model (LLM)
coordination for precise robot placement in the environment. Three key stages
are highlighted and explained in Section IV.

##  IV ZeroCAP Formation System

This section delineates the comprehensive methodology of our ZeroCAP formation
system, which introduces a novel approach for zero-shot multi-robot context
aware pattern formation. Leveraging advanced VLMs, the methodology integrates
linguistic instructions with visual perception, enabling precise spatial
reasoning and object identification in multi-robot systems. Our approach
harnesses the synergy of cutting-edge segmentation techniques, and utilizes
standard computer vision algorithms for obtaining shape description. The
decision-making process is empowered by a specialized LLM, which synthesizes
the spatial description of object and linguistic context to generate target
deployment coordinates for the robotic agents. The multi-robot control is
centrally orchestrated, ensuring efficient navigation. The following
subsections provide a detailed exposition of each stage, from initial input
processing to the final deployment of robots.The overall structure of the
proposed ZeroCAP framework is shown in Fig. 2.

###  IV-A Context Identification using VLM

The initiation of our methodology engages a VLM to interpret an environmental
image $I_{E}$ and a natural language instruction $Ins$. The architecture of
this model excels at correlating visual stimuli with corresponding linguistic
descriptors, emulating human-like proficiency in image-language tasks.
Provided with $I_{E}$ and $Ins$, the VLM processes these inputs in tandem,
producing two key outputs: the Object of Interest $O$ and the Pattern
Formation Instruction $PI$.

The Object of Interest $O$ is identified utilizing the VLM’s zero-shot
detection capabilities, circumventing the need for prior object-specific
training and ensuring versatility within varying environments. Conventional
object detection methods may localize objects but lack the sophisticated
reasoning to discern context as VLMs do. Given an instruction akin to that of
Fig. 3, standard natural language-aided object detectors [5] would falter in
reasoning with which object is odd. In contrast, the advanced reasoning
capabilities of the VLM, contextualized by the instruction, render it superior
for identifying objects of interest within complex scenarios. This nuanced
understanding of context allows the VLM not only to detect but also to reason
about objects in a manner aligned with the given linguistic directives.
Concurrently, the Pattern Formation Instruction $PI$ is extrapolated from the
natural language directive enriched by contextual nuances in $I_{E}$. Here,
the VLM generates $PI$, inspired by chain-of-thought reasoning [30], to
discern spatial and context-sensitive task directives. In Algorithm 1, Line 1
corresponds to estimating the context using the VLM.

One notable strength of our system is our approach to mitigating the common
issue of precise spatial localization, a challenge in standard VLMs. The
inherent limitations of VLMs in tasks that require exact positional outputs
make it impractical to use these models to directly provide the deployment
coordinates $C_{xy}$. Instead, the VLM is optimized to identify $O$ with
necessary precision and to extract a context-rich $PI$. These are
indispensable for subsequent segmentation and precise pattern formation stages
conducted by more specialized subsystems. By honing in on general features and
attributes discernible in the environment and the language input, the VLM
identifies $O$ with the necessary precision for subsequent segmentation and
pattern formation stages.

The VLM’s output, consisting of $O$ and $PI$, thus propels the ZeroCAP system
into the segmentation phase, where $O$ is visually distinguished within
$I_{E}$, and $PI$ informs the strategic pattern formation instruction to be
implemented by the robotic entities.

Figure 3: An illustration of the VLM processing a natural language instruction
to identify the object of interest and generate a pattern formation
instruction, which guides the deployment of robots in a specified arrangement
around the object. The object is not explicitly mentioned in the instruction
and must be reasoned by the VLM within the context of image of the
environment.

###  IV-B Object Segmentation and Shape Description

Following the identification of $O$ and the acquisition of the $PI$ from the
VLM, the system transitions into the crucial stages of segmentation and shape
description. These stages leverage sophisticated language conditioned
segmentation techniques [5, 31], to perform image segmentation, followed by
shape description stage using standard computer vision methods.

####  IV-B1 Object Segmentation

The segmentation process begins with language-prompt-based object
segmentation, which employs LangSAM, an open-source language-conditioned
segmentation model built upon segment-anything approach and the GroundingDINO
detection model [31, 32]. Provided with the Object of Interest $O$ and the
Input Image $I_{E}$, these models generate a binary segmentation map in which
the object is highlighted in white against a black background.

Upon completion, the segmentation yields a Masked Image $I_{M}$ and an
accuracy metric $\alpha$, compared against a predefined threshold $t$. If
$\alpha$ meets or exceeds this threshold, the process proceeds, confirming the
segmentation’s quality for subsequent tasks. This segmentation approach
ensures the precise extraction of $O$ from $I_{E}$, encapsulated within
$I_{M}$ for the next processing stage. Lines 2 to 4 of the Algorithm 1 expound
the segmentation process.

####  IV-B2 Shape Description

With a high quality $I_{M}$ at hand, the ZeroCAP system advances to the shape
description phase. Here, standard computer vision techniques preprocess
$I_{M}$, including filtering and enhancement operations to sharpen feature
delineation. Through OpenCV’s edge detection algorithms and contour detection
methods, $I_{M}$ is analyzed to determine $O$ geometric form. This analysis
yields the object’s edges $E$ and vertices $V$, forming a connected graph
$G=(V,E)$ that mathematically represents the object’s structure. Algorithm 1
delineates this process in Lines 10 – 13, with a method call in Line 5.

Integrating these segmentation techniques with precise shape description, the
ZeroCAP system ensures accurate recognition, segmentation, and modeling of the
Object of Interest. This forms a solid foundation for the ensuing methodology
phase: LLM-based Pattern Former.

Algorithm 1 ZeroCAP System Pipeline

1:Image of the Environment $I_{E}$, Natural Language Instruction Ins

2:List of coordinates for robot deployment $C_{xy}$

3:$(O,PI)\leftarrow\textsc{VLM.Process}(I_{E},\textit{Ins})$

4:$(I_{M},\alpha)\leftarrow\textsc{Segment}(O,I_{E})$

5:if $\alpha<t$ then

6: return “Insufficient segmentation accuracy.”

7:$(V,E)\leftarrow\textsc{ShapeDescriptor}(I_{M})$

8:$C_{xy}\leftarrow\textsc{LLM.GenerateCoordinates}(PI,V,E,R)$

9:for each coordinate $(x_{i},y_{i})$ in $C_{xy}$ do

10: $r_{i}\leftarrow R[i]$

11: DeployRobots($r_{i},(x_{i},y_{i})$)

12:procedure ShapeDescriptor($I_{M}$)

13: Preprocess the masked image $I_{M}$

14: Extract edges $E$ and vertices $V$

15: return $(V,E)$ from $I_{M}$

16:procedure DeployRobots($r,(x,y)$)

17: Command robot $r$ to move to coordinates $(x,y)$

18: Verify that robot $r$ has reached $(x,y)$

###  IV-C LLM-based Pattern Former

Upon accurately delineating the geometry of $O$, the context-aware multi-robot
deployment system advances to determining goal position coordinates using a
LLM. This stage involves the LLM being provided with the number of available
robots $R$, interpreting the Pattern Formation Instruction $PI$ alongside the
object’s spatial characteristics, represented through edges $E$ and vertices
$V$ to provide an output representing positions for robots.

The LLM’s task is to integrate the $PI$ with the spatial data of $E$ and $V$
to determine the optimal pattern formation. This involves comprehensive
reasoning to align the robots according to $PI$, while considering the shape
of the object and the available robotic agents $R$.

In particular, our methodology eschews the direct use of the entire Masked
Image $I_{m}$ as input to the LLM, a decision underscored by considerations of
computational efficiency and token costs. Direct image is resource-intensive,
whereas leveraging $E$ and $V$ markedly reduces computational load especially
when accounting for tokens in the context of LLM’s. This choice highlights our
system’s capacity for real-time processing and deployment efficiency while
also remaining inexpensive in computation and practicality.

The culmination of this process is the production of a coordinate list
$C_{xy}$ as seen in Line 6 of Algorithm 1, which prescribes exact deployment
coordinates for each robot.

###  IV-D Robot Deployment

Upon generating the deployment coordinates $C_{xy}$ with the LLM, the ZeroCAP
system initiates robot deployment, translating algorithmic output into
coordinated physical actions.

Each robot $r_{i},i\in N$ is directed toward its assigned location
$(x_{i},y_{i})$ under centralized control, ensuring precise navigation. Upon
arrival, the system verifies each robot’s position against $C_{xy}$, adjusting
as necessary to rectify any positional inaccuracies, thereby ensuring the
formation aligns perfectly with the pattern instructions. This is delineated
in Lines 7 to 9 and 14 to 16 in Algorithm 1.

The successful deployment of robots in the context of the Object of Interest
$O$ marks the completion of the pattern formation task, embodying the ZeroCAP
system’s capability to execute context-driven, instruction-based patterns
through autonomous robotic agents.

##  V Experiment

(a) General pattern formation tasks: robots arrange themselves around the
object of interest based on user specification

(b) Infill pattern formation tasks: robots occupy the interior space of a
region based on user specification

(c) Caging formation tasks: creating a cage like boundary around the object

(d) Application oriented task: surrounding the incorrectly parked car with
four robots.

Figure 4: Illustrations of the three types of pattern formation tasks and a
task demonstration of Fig 1, showcasing the system’s adaptability to various
instruction types and object configurations. In all tasks, robots are
represented as black points initially distributed at the bottom of the scene
in the initial environment, transitioning to form user-directed patterns in
the final environment. This illustrates the dynamic capability of our system
to interpret instructions and spatially reorganize the robots accordingly.
Please zoom in on the images for details.

Our experimental evaluation investigates the ZeroCAP system’s zero-shot
pattern formation capabilities across ten tasks as shown in Fig. 4, delineated
into three categories as prefaced in Section I: general pattern formations as
shown in subfigure 4(a), infill pattern formations as shown in subfigure 4(b),
and caging formation tasks as shown in subfigure 4(c) and the task referred to
in Fig 1 as shown in subfigure 4(d).

Tasks are diversified by object setup to evaluate the system’s adaptability
and discernment capabilities under varying environmental complexities. In the
Single Object setup (ENV a, b, d, g), the environment contains a solitary
object, simplifying the identification process. The Multi-Object setup (ENV c,
e, h) introduces additional objects as distractors, challenging the system to
isolate the target object based on explicit user instructions. Lastly, the
Hidden Object setup (ENV f, i, park) presents a more complex scenario where
the natural language instruction implies the object of interest without
explicit mention, compelling the system to employ reasoning to deduce the
correct object.

###  V-A Baseline Comparisons

Our system’s performance was benchmarked against three baselines. The first is
an RL-based simplified approach with hand-crafted rewards, serving primarily
as a pattern-formation benchmark without natural language comprehension. The
second and third baselines use GPT4 Vision [33] and llava-v1.6 [34],
respectively, relying solely on vision-language models to complete the tasks.
The LLMs employed in our experiments are GPT-4 [35], llama70b [36], and Claude
[37], contrasting their performance against each other and the baselines. For
ablation studies, the default configuration of the ZeroCAP system utilizes
GPT-4 as the primary LLM.

###  V-B Environment

Experiments were conducted in a 2D simulation using Pygame, with robots as
point-mass entities capable of precise movements to target coordinates. The
setup includes multi-object tasks with infill and caging requirements,
omitting obstacle avoidance behaviors to focus on the pattern formation tasks.
A subset of tasks was also executed on a real testbed, featuring Hamster
mobile robots [38] under an overhead camera setup. Fig. 5 showcases two tasks
executed in a real-world setting, serving as a tangible proof of concept that
corroborates our simulation results. To facilitate these tasks, the
environment is simplified into a grid-based image, with each grid box
approximately matching the size of a robot. This adaptation allows the robots
to retain their point mass entity characterization within the practical
constraints of the real-world experiments, ensuring consistency in their
deployment and movement. For demonstrations of simulator and real-world tasks,
please refer to the supplementary video.

###  V-C Evaluation Metrics

We employed two main evaluation metrics: Success Rates (SR) and Goal Condition
Recall (GCR). SR indicates whether the robots formed the pattern successfully,
verified by handcrafted ground truth labels, while GCR quantifies the
proportion of robots correctly positioned within the pattern. A GCR of 1
implies a perfect match with the ground truth and, consequently, an SR of 1.
Each task underwent 10 trials, with averages reported for SR and GCR, chosen
to account for the inherent variability in LLM performance.

##  VI Results and Analysis

TABLE I: Comparative analysis of pattern formation methodologies across
General, Infill, and Caging tasks, evaluating Success Rates (SR) and Goal
Condition Recall (GCR) metrics across different methods.

Methods | General | Infill | Caging | Total
---|---|---|---|---
SR | GCR | SR | GCR | SR | GCR | SR | GCR
GPT-4 Vision (Only) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
LLaVA v1.6 (Only) | 0 | 0 | 0.13 | 0.1 | 0 | 0.07 | 0 | 0.07
RL Baseline | 0.35 | 0.53 | 0.28 | 0.41 | 0.88 | 0.9 | 0.5 | 0.61
ZeroCAP (GPT-4) | 0.71 | 0.85 | 0.86 | 0.89 | 0.74 | 0.82 | 0.77 | 0.85
ZeroCAP (lama-2-70b) | 0.2 | 0.47 | 0.39 | 0.71 | 0.32 | 0.67 | 0.3 | 0.61
ZeroCAP (claude-3-opus) | 0.57 | 0.78 | 0.72 | 0.88 | 0.64 | 0.73 | 0.64 | 0.79

Table I shows a comprehensive evaluation of different methodologies tested in
this paper, revealing the effectiveness of the ZeroCAP system in comparison to
baselines. The initial baselines, GPT-4 Vision and LLava v1.6, were employed
to evaluate the feasibility of using VLMs directly for zero-shot context-aware
pattern formation. The performance of these baselines was significantly poor.
GPT-4 Vision generally failed to generate any coordinates, reflecting its
known limitations with spatial localization tasks. LLava, while capable of
producing coordinates, generated positions that were largely inaccurate and
contextually irrelevant. These outcomes effectively demonstrate that direct
application of VLMs for context-aware pattern formation is non-trivial, and
without additional processing or reasoning mechanisms, VLMs fall short of
providing viable solutions. The RL baseline exhibited a mixed performance
across the task spectrum. It notably struggled with the general pattern
formation tasks, where user specifications varied greatly and required a high
degree of interpretation and flexibility. The RL method’s rigidity and lack of
language comprehension rendered it unable to adapt to the nuanced demands of
these tasks. On the other hand, the RL approach showed considerable success in
Caging tasks, aligning with the more straightforward objective of these tasks
where the overarching goal is consistently to form a barrier around the
object.

Figure 5: Two real-world scenarios demonstrating ZeroCAP system’s ability to
execute context-aware pattern formations.

The ZeroCAP methods, incorporating LLMs like GPT-4, llama-2-70b, and
claude-3-opus, present a contrasting picture. The GPT-4 based ZeroCAP achieves
the highest overall SR and GCR across all types of tasks, indicating its
superior pattern recognition and execution in zero-shot conditions. This
suggests that GPT-4’s advanced natural language processing abilities are
effectively leveraged by the ZeroCAP system to interpret instructions and
generate appropriate robotic behaviors. The performance of ZeroCAP using
llama-2-70b and claude-3-opus varies. While the llama-2-70b variant has some
success, its inconsistency across tasks points to possible limitations in its
ability to handle complex spatial reasoning as compared to GPT-4.
Claude-3-opus shows a relatively more consistent performance. Across different
pattern formation types, ZeroCAP methods behave in a manner that suggests
certain patterns may be more conducive to LLM-driven execution. The General
pattern formation tasks appear to be the most challenging, possibly due to the
abstract nature of the instructions and the need for more nuanced
interpretation. Infill tasks, which involve a more defined area and clear
objectives being in most cases to deploy robots inside a region, show the
highest SR and GCR for GPT-4, indicating a strong alignment between the
method’s capabilities and the task requirements. Caging tasks, while more
structured, still present a challenge, but ZeroCAP methods demonstrate
reasonable effectiveness.

###  VI-A Ablation: Segmentation Methods

The ablation study presented in Table II examines the efficacy of CLIPSeg [5]
and LangSAM as object segmentation methods within the context of language-
prompted object identification. These methods, while capable of segmenting
objects based on language instructions, do not possess the same reasoning
abilities provided by full-fledged VLMs. The table compares the performance of
these segmenters used in isolation (denoted by ‘Only’) to their performance
when integrated into the ZeroCAP framework.

TABLE II: Ablation studies comparing the success rates (SR) of standalone segmentation methods and their integration within the ZeroCAP system. Method | SR
---|---
CLIPSeg (Only) | 0.51
LangSAM (Only) | 0.43
ZeroCAP with CLIPSeg | 0.78
ZeroCAP with LangSAM | 0.77

It is evident that using CLIPSeg and LangSAM alone for object identification
and segmentation results in modest success rates of 0.51 and 0.43,
respectively. This suggests that while they can interpret natural language to
some extent and perform segmentation, the lack of deeper contextual
understanding and spatial reasoning affects their overall effectiveness in
pattern formation tasks. This is especially a factor in case of hidden object
task scenarios.

When these segmentation methods are incorporated into the ZeroCAP system, the
success rates improve significantly to 0.78 for CLIPSeg and 0.77 for LangSAM.
The integration of these segmenters within ZeroCAP likely benefits from
additional contextual processing and spatial reasoning capabilities provided
by the LLMs in the system, leading to more successful task outcomes.

###  VI-B Ablation: Shape Descriptors

The ablation study presented in Table III evaluates the influence of different
shape descriptors on the performance of the ZeroCAP system. ‘Increase in
Tokens’ indicates the relative percentage increase in the number of tokens
used by the LLM compared to using only ‘Edges’ as the shape descriptor, which
serves as the baseline in this context.

TABLE III: Ablation studies on the impact of different shape descriptors on Success Rates (SR) and their relative increase in tokens required for LLM processing. Shape Descriptor | SR |  | Increase
---
in Tokens
Edges | 0.77 | -
Edges and Vertices | 0.48 | 2%
Binary Image Matrix | 0.71 | 400%

Analyzing the data, it is evident that using only ‘Edges’ as a shape
descriptor achieves the highest SR of 0.77. This suggests that ‘Edges’ alone
provide ample information for the LLM to accurately generate the desired
patterns without requiring additional token input. In contrast, when ‘Edges
and Vertices’ are combined, there is a modest increase in tokens by 2%, but
the SR significantly decreases to 0.48. This decrease could imply that
including vertices introduces a level of complexity that the LLM does not
effectively translate into better pattern formation. Moreover, vertices as a
descriptor could potentially be misleading for non-polygonal shapes where the
definition of vertices is less clear, potentially leading to inaccurate
pattern formation by the LLM.

The binary image matrix descriptor is obtained by converting the segmented
object mask into a binary matrix, which is then resized to a 100x100 grid to
fit within the token constraints of the LLM. This matrix serves as a input for
the LLM, representing the shape in a form that can be directly processed. The
LLM utilizes this grid-based representation to understand the spatial
arrangement and generate target coordinates for the robots. The binary image
matrix descriptor, incurs a 400% increase in token usage. This upsurge poses
practicality concerns due to token limits and practical costs. Additionally,
resizing may obscure vital shape details, affecting pattern accuracy. Despite
a reasonable SR of 0.71, the trade-off between detail richness and token
efficiency is crucial for the ZeroCAP system’s viability.

##  VII Discussion

The research presented in this paper marks a significant stride in the domain
of multi-robot systems, particularly in the application of LLMs for zero-shot
context aware pattern formation. Nonetheless, the current scope is confined to
static patterns, leaving the exploration of dynamic patterns - those that
evolve over time in response to environmental changes or ongoing tasks - as a
promising avenue for future research. Dynamic pattern formation would not only
extend the flexibility and adaptability of multi-robot systems but also
increase their applicability in real-world scenarios such as search and rescue
operations or environmental monitoring, where conditions are continually
shifting.

Our current method’s reliance on centralized processing poses potential
bottlenecks and failure points in scenarios requiring quick robot
adaptability. Exploring decentralized or hybrid control systems could improve
robustness and scalability by merging centralized planning with decentralized
execution. Additionally, incorporating sensory feedback mechanisms may enhance
decision-making, maintaining system effectiveness amid real-time environmental
changes.

Our research lays a foundation for future advancements in robotic pattern
formation and practical applications, extending from complex coordination
tasks in surveillance, logistics, and warehousing to context-aware strategies.
Progressing beyond current limitations and enhancing our foundational
framework is vital for fully realizing robotics’ capabilities in intricate,
context-sensitive environments.

##  VIII Conclusion

Our research has successfully introduced ZeroCAP, an innovative framework that
harmonizes the advanced linguistic comprehension of large language models with
the physical capabilities of robotic systems to achieve zero-shot pattern
formation. The system’s adeptness at deciphering natural language instructions
and materializing these into precise spatial arrangements has been
demonstrated across various contexts, laying the groundwork for future
advancements in multi-robot coordination. While our study focused on static
patterns and centralized control, the insights gained provide a stepping stone
for exploring dynamic pattern formations. ZeroCAP’s performance in context-
aware task execution signal a promising direction for the integration of AI
and robotics, propelling us towards more intelligent, adaptable, and efficient
multi-robot applications in diverse real-world settings.

## References

  * [1] K.-K. Oh, M.-C. Park, and H.-S. Ahn, “A survey of multi-agent formation control,” _Automatica_ , vol. 53, pp. 424–440, 2015.
  * [2] W. Xu and D. W. Ho, “Clustered event-triggered consensus analysis: An impulsive framework,” _IEEE Transactions on Industrial Electronics_ , vol. 63, no. 11, pp. 7133–7143, 2016.
  * [3] J. Alonso-Mora, A. Breitenmoser, M. Rufli, R. Siegwart, and P. Beardsley, “Multi-robot system for artistic pattern formation,” in _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 4512–4517.
  * [4] R. Gayle, W. Moss, M. C. Lin, and D. Manocha, “Multi-robot coordination using generalized social potential fields,” in _2009 IEEE International Conference on Robotics and Automation_. IEEE, 2009, pp. 106–113.
  * [5] Y. Lin, M. Chen, W. Wang, B. Wu, K. Li, B. Lin, H. Liu, and X. He, “Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation,” 2023.
  * [6] J. Wang, J. Cao, M. Stojmenovic, M. Zhao, J. Chen, and S. Jiang, “Pattern-rl: Multi-robot cooperative pattern formation via deep reinforcement learning,” in _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , 2019, pp. 210–215.
  * [7] B. S. Prasad, A. G. Manjunath, and H. Ramasangu, “Multi-agent polygon formation using reinforcement learning.” in _ICAART (1)_ , 2017, pp. 159–165.
  * [8] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personalized robot assistance with large language models,” _Autonomous Robots_ , vol. 47, no. 8, pp. 1087–1102, 2023.
  * [9] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023, pp. 11 523–11 530.
  * [10] S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng, “Large language models as general pattern machines,” _arXiv preprint arXiv:2307.04721_ , 2023.
  * [11] C. Belta and V. Kumar, “Trajectory design for formations of robots by kinetic energy shaping,” in _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292)_ , vol. 3. IEEE, 2002, pp. 2593–2598.
  * [12] V. Indelman, “Towards cooperative multi-robot belief space planning in unknown environments,” _Robotics Research: Volume 1_ , pp. 441–457, 2018.
  * [13] Y. Cao, W. Yu, W. Ren, and G. Chen, “An overview of recent progress in the study of distributed multi-agent coordination,” _IEEE Transactions on Industrial informatics_ , vol. 9, no. 1, pp. 427–438, 2012.
  * [14] R. Olfati-Saber, J. A. Fax, and R. M. Murray, “Consensus and cooperation in networked multi-agent systems,” _Proceedings of the IEEE_ , vol. 95, no. 1, pp. 215–233, 2007.
  * [15] Y. Hong, G. Chen, and L. Bushnell, “Distributed observers design for leader-following control of multi-agent networks,” _Automatica_ , vol. 44, no. 3, pp. 846–850, 2008.
  * [16] J. Liu, P. Li, W. Chen, K. Qin, and L. Qi, “Distributed formation control of fractional-order multi-agent systems with relative damping and nonuniform time-delays,” _ISA transactions_ , vol. 93, pp. 189–198, 2019.
  * [17] C. Belta and V. Kumar, “Abstraction and control for groups of robots,” _IEEE Transactions on robotics_ , vol. 20, no. 5, pp. 865–875, 2004.
  * [18] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [19] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig, “Language models of code are few-shot commonsense learners,” _arXiv preprint arXiv:2210.07128_ , 2022.
  * [20] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2motion: From natural language instructions to feasible plans,” _Autonomous Robots_ , vol. 47, no. 8, pp. 1345–1365, 2023.
  * [21] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,” 2023.
  * [22] Z. Liu, W. Yao, J. Zhang, L. Xue, S. Heinecke, R. Murthy, Y. Feng, Z. Chen, J. C. Niebles, D. Arpit, R. Xu, P. Mui, H. Wang, C. Xiong, and S. Savarese, “Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents,” 2023.
  * [23] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “Metagpt: Meta programming for a multi-agent collaborative framework,” 2023.
  * [24] S. S. Kannan, V. L. N. Venkatesh, and B.-C. Min, “Smart-llm: Smart multi-agent robot task planning using large language models,” 2023.
  * [25] D. Shah, B. Osiński, b. ichter, and S. Levine, “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” in _Proceedings of The 6th Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 14–18 Dec 2023, pp. 492–504. [Online]. Available: https://proceedings.mlr.press/v205/shah23b.html
  * [26] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, _et al._ , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” _arXiv preprint arXiv:2307.15818_ , 2023.
  * [27] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh, “Physically grounded vision-language models for robotic manipulation,” _arXiv preprint arXiv:2309.02561_ , 2023.
  * [28] J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [29] R. J. Qureshi, J.-Y. Ramel, and H. Cardot, “Graph based shapes representation and recognition,” in _Graph-Based Representations in Pattern Recognition: 6th IAPR-TC-15 International Workshop, GbRPR 2007, Alicante, Spain, June 11-13, 2007. Proceedings 6_. Springer, 2007, pp. 49–60.
  * [30] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023.
  * [31] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” _arXiv:2304.02643_ , 2023.
  * [32] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, _et al._ , “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” _arXiv preprint arXiv:2303.05499_ , 2023.
  * [33] OpenAI, “GPT-4V(ision) System Card.” 2023.
  * [34] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in _NeurIPS_ , 2023.
  * [35] OpenAI, “GPT-4 Technical Report,” 2023.
  * [36] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, _et al._ , “Llama 2: Open Foundation and Fine-Tuned Chat Models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [37] “Anthropic Claude 3.” [Online]. Available: https://www.anthropic.com/news/claude-3-family
  * [38] A. Lee, W. Jo, S. S. Kannan, and B.-C. Min, “Investigating the effect of deictic movements of a multi-robot,” _International Journal of Human–Computer Interaction_ , vol. 37, no. 3, pp. 197–210, 2021.