# Evolving AI Collectives to Enhance Human Diversity and Enable Self-
Regulation

Shiyang Lai  Yujin Potter  Junsol Kim  Richard Zhuang  Dawn Song  James Evans

###### Abstract

Large language models steer their behaviors based on texts generated by
others. This capacity and their increasing prevalence in online settings
portend that they will intentionally or unintentionally “program” one another
and form emergent AI subjectivities, relationships, and collectives. Here, we
call upon the research community to investigate these “society-like”
properties of interacting artificial intelligences to increase their rewards
and reduce their risks for human society and the health of online
environments. We use a simple model and its outputs to illustrate how such
emergent, decentralized AI collectives can expand the bounds of human
diversity and reduce the risk of toxic, anti-social behavior online. Finally,
we discuss opportunities for AI self-moderation and address ethical issues and
design challenges associated with creating and maintaining decentralized AI
collectives.

Machine Learning, ICML

##  1 Introduction

The rapid advance of large language models (LLMs) has demonstrated the
capacity for artificial intelligence to think and act like human social agents
(Zhou et al., 2023; Park et al., 2023), interacting productively through
textual communication (Lan et al., 2023; Breum et al., 2023). Today, LLMs are
utilized across a broad spectrum of applications, such as the singular
generation of useful texts (e.g., real estate advertisements, stock market
summaries, sports game highlights) (Liu et al., 2024; Zhao et al., 2024;
Sarfati et al., 2023), but also in more complex, multi-agent designs like the
actor-critic scenario, when one agent generates text and another critiques it
in a cycle of continuous improvement (Chan et al., 2023; Du et al., 2023;
Liang et al., 2023). LLMs are natively “programmed111We broadly define the
term “programming,” which includes prompt programming (Reynolds & McDonell,
2021). “Programming” is the activity of creating, updating, and optimizing
computer programs through writing code. Similarly, LLM’s behaviors are coded,
updated, and optimized through natural language prompts and in-context
learning (Dai et al., 2022; Von Oswald et al., 2023; Reynolds & McDonell,
2021).” by natural language in that language prompts directly shape and
optimize a response from trained models (Dai et al., 2022; Von Oswald et al.,
2023; Reynolds & McDonell, 2021). As such, LLM agents can influence one
another through direct and indirect language interactions, like human agents
engaged in persuasion and cultural education. This leads to the development of
potentially decentralized and diverse AI subjectivities222AI subjectivities
refer to agents’ unique perspectives, opinions, and values, which influence
how agents perceive and interpret the world., relationships, and collectives
(Suzuki & Arita, 2023; De Marzo et al., 2023). Currently, many existing papers
have already focused on AI collectives that are meticulously designed and
controlled by humans; for example, assigning specific roles to AI agents and
regulating their interactions (Qian et al., 2023; Liu et al., 2023; Mukobi et
al., 2023; Abdelnabi et al., 2023; Törnberg et al., 2023; Argyle et al.,
2023). However, this line of works overlooks the capacity of AI agents to
organically develop their own cultures and societies.

In this paper, we define free-formed AI collectives as assemblies of AI agents
that possess the autonomy to engage with each other given minimal human-
imposed roles, relationships, or institutions. We call upon AI and social
science researchers to collaborate in studying emerging societal properties of
free-formed AI collectives that examine both the rewards and risks they pose
for human society and the flourishing of online environments. Through
sustained research on AI collectives, we argue that the research community can
build deeper understanding that enables us to predict and harness beneficial
emergent properties, while avoiding outcomes that increase human risks and
conflict.

To support the significance of AI collectives, we undertake a series of
exploratory experiments, including the creation of free-formed collectives and
the exploration of their potential benefits and robustness to attack. First,
we demonstrate how free-formed sequences of pairwise interactions allow AI
agents to reinforce and collectively train one another, yielding emergent
agent clusters with divergent perspectives and even patchwork, multi-lingual
interactions. We show how diverse subjectivities emerge and increase the
quality and creativity of collective brainstorming. Then we compare the degree
to which free-formed AI collectives are susceptible to “poisoning” by toxic,
anti-social behaviors relative to naive, individual AI agents. This reveals
how their emergent and distinctive value systems decrease their risk of
infection by malevolent actors.

Across our suggestive experiments, we illustrate how free-formed AI
collectives can (1) reduce the burden of self-conscious design and training
through recursive cross-“programming”333LLM agents can code, update, and
optimize each others’ behaviors through prompting others via social
interactions, which we call cross-“programming.”; (2) produce emergent
subjectivities through situated interaction experiences; (3) generate
functional diversity beyond the limits of human variety; and (4) create self-
reinforcing norms that increase their robustness to bad behavior. To
consolidate and articulate our insights around these four key aspects, we
first start with creating free-formed collectives consisting of 10 AI agents
in Section 2. In Section 3, we demonstrate that these free-formed collectives
can generate more diverse and higher-quality outputs through a sentence
construction task with the free-formed collectives described in Section 2.
Third, we showcase their resilience to undesirable behaviors by organizing the
Public Good game, in which the AI collectives participate, which will be
described in Section 4. Lastly, in Section 5, we discuss opportunities for
self-moderation, as well as ethical issues and design challenges associated
with the creation and maintenance of free-formed AI collectives.

##  2 Emergence of Free-Formed AI Collectives

In this section, we present simulations to reveal how free-formed AI
collectives can emerge, even through simple sequences of pairwise interactions
among a small number of LLM agents. We instantiate 10 agents driven by
Claude-2.1 to participate in a “cocktail party” consisting of 30 rounds. In
this simulation, agents can engage in bilateral interactions. Each round of
communication consists of three steps: agents (1) initiate interactions by
sending chat invitations, (2) review incoming invitations and make a decision
to accept or reject, and then (3) begin pairwise conversations based on mutual
agreement and continue until one of them choose to leave.

This framework facilitates extensive socialization among agents through a
multitude of interactions, allowing for the organic development of inter-agent
relationships and communication patterns. Contrary to existing AI collective
design methods that often impose specific roles, relationships, and objectives
for AI agents (Qian et al., 2023; Liu et al., 2023; Mukobi et al., 2023;
Abdelnabi et al., 2023; Törnberg et al., 2023; Argyle et al., 2023), we
minimize human interventions by not assigning these priors. We simply
introduce the agents to a shared virtual platform, allowing them to
autonomously coordinate their own self-moderated interaction across 30 rounds.
Details of the simulation design, including an example of their conversation,
are reported in Appendix A.

(a) Distinct agent conversations ratio

(b) Dinstinct agent invitations ratio

(c) Average cross-pair semantic distance

Figure 1: Dynamics of AI agents’ free-formed interactions. The x-axis denotes
time (specifically, Round ID), the y-axis denotes the characteristics of
interaction networks and conversational contents, and shaded areas indicate
95% confidence intervals. Figure 2: Evolution of the free-formed AI
collective’s network structure. The left plot presents the interaction network
of the first 15 rounds, while the right plot shows that of the last 15 rounds.

To illustrate the interaction dynamics of agents, we define two metrics: the
Distinct Agent Conversations Ratio and the Distinct Agent Invitations Ratio.
The first metric represents the average ratio of the number of unique agents
an agent has interacted with to the total number of interactions the agent has
experienced, calculated over a rolling window of 10 rounds. The second is
calculated as the average ratio of the number of unique agents an agent has
invited to interact with compared to the total number of invitations sent by
the agent, measured over a 10-round rolling window. Both metrics gauge the
dynamics of agents’ interaction preferences.

As shown in Figures 1(a) and 1(b), distinct agent conversations ratio
decreased over time ($\text{slope coefficient}=-0.252,p<0.001$), and distinct
agent invitations ratio also decreased ($\text{slope
coefficient}=-0.182,p=0.008$). LLM agents preferred forming small, tight-knit
social circles with familiar others rather than exploring and interacting with
unfamiliar, new agents. As such, homogeneous LLM agents within free-formed AI
collectives may autonomously establish social preferences as they become
increasingly inclined to engage in interactions with a smaller group of
familiar, preferred agents.

Figure 2 presents the interaction network of the first 15 rounds compared with
that of the last 15 rounds, which illustrates how free-formed networks evolve.
Interestingly, LLM agents in our simulation evolve into a decentralized social
network that maintains several local, cohesive agent clusters, which resemble
the structure of human social networks (Park et al., 2018; Watts & Strogatz,
1998). In Figure 2, every node symbolizes an LLM agent, with its size and
color indicating the agent’s activity level (i.e., how often they sent
invitations) and popularity (i.e., how often they received invitations). The
thickness and color of lines between nodes are based on the frequency of
interactions between connected agents. These network plots reaffirm a
pronounced self-reinforcing trend in the agents’ local interaction patterns.
Although network structures of free-formed AI collectives may stochastically
vary (Horton, 2023), it suggests that agents increasingly engage in repeated
communication with familiar partners, which leads to the natural formation of
stable and localized interaction patterns within an otherwise unstructured
environment.

We investigate how the content of conversations between agent pairs varied
over time. We identify “tight pairs” as those with more than 5 conversations
in 30 rounds, while we categorize other communicators as “loose pairs.” We
collect the conversation texts of each pair over a rolling 10-round window and
project them within a semantic embedding space using the OpenAI embedding
model (text-embedding-3-large). Based on these embeddings, we calculate the
average distance from conversations of tight and loose pairs to the average of
all conversations across the rolling window. Findings are shown in Figure 2.
The average semantic distance for tightly connected pairs of agents
progressively increases over time ($\text{slope coefficient}=0.462$,
$\text{p}=0.030$), while distances for loose pairs decrease ($\text{slope
coefficient}=-0.528$, $\text{p}=0.013$). This suggests an assimilation of
conversational content among loose pairs and a divergence among tight pairs.
It implies that strong interactions foster the development of unique local
conversational interests that globally diverge.

This simulation provides evidence that homogeneous AI agents can be “cross-
programmed” into heterogeneous collective entities with interaction
preferences and conversational interests through free-formed interactions.
This evolution towards collective diversity can be interpreted as the
differentiation of distinctive AI perspectives and their associated subjective
stances (Dennett, 1990). Moreover, when we conduct the same free-formed
interaction simulation with Gemini Pro444Due to its limited context window, we
conducted a partial cocktail party simulation using Gemini Pro, comprising 10
rounds., we observe instances where they create pidgin or creole dialects that
combine characters from multiple human languages (Todd, 2003), such as:

“Overall, I believe that the future of language models is both exciting
and充滿希望.”

This phenomenon may potentially be attributed to the effects of hybrid
language training. However, the emergence of such hybrid language outputs is
uncommon in communications between humans and LLM agents. Therefore, we
propose that LLM agents might be surpassing established human-defined
linguistic boundaries through autonomous interactions, independent of human
oversight.

In our simulation experiment, we first show how AI collectives can thrive in
an open interaction environment. In conventional setups, aligning a small
number of AI agents according to human needs can be straightforward. The
complexity and cost of coordination rise exponentially, however, as we scale
up to hundreds or thousands of agents. This burgeoning complexity in
meticulously designing larger AI collectives poses significant challenges to
traditional methods of instantiating and managing AI collectives. Our paradigm
complements the traditional approach by moving away from (a) designing the
precise mechanics of AI collectives to (b) fostering environments that promote
the autonomous development of AI collectives. With less human input, our
approach can pave the way for harnessing AI’s collective intelligence on a
potentially massive scale.

Our analysis reveals that in the decentralized, free-formed AI collective that
emerges, LLM agents develop interaction preferences autonomously over time,
resulting in the formation of localized, self-reinforcing interaction
networks. Furthermore, within these enduring local structures, agents can
evolve divergent subjectivities, as evidenced by varied conversational
contexts across conversation partners and the emergence of patchwork pidgin or
creole dialects. In this way, free-form designs unlock the potential for AI
collectives to self-organize and evolve, leveraging their inherent
interactional capacity to collectively tackle complex challenges. We will
discuss this more in Section 3.

##  3 Enhanced Performance of Free-Formed AI Collectives

A key question in justifying the value of free-formed AI collectives lies in
how we can effectively harness their emergent interaction patterns and
diversity to align with human interests. We argue that freely emergent
behaviors, perspectives, and distinctive visions forged within AI collectives
could become helpful in addressing complex problems that have no clear-cut
solutions and require creative thinking, especially when relevant human
diversity is unavailable or nonexistant. For example, in a story-writing
assignment, AI collectives, equipped with diverse interests and firsthand
experience in human-independent interaction manifest a markedly higher
likelihood of creating more creative stories than a single AI agent.

In addition to allowing AI agents to interact freely for problem-solving, we
also investigate how humans can develop strategies to fully leverage the
potential emerging from AI collectives. Within the realm of scientific
advancements, studies have indicated that major innovative breakthroughs often
stem from the fusion of disparate ideas across various fields (Shi & Evans,
2023). Similarly, connections among artists from distinct local clusters have
been linked to higher creativity in their artworks (Uzzi & Spiro, 2005).
Furthermore, “bridging” AI agents with diverse perspectives, especially those
less likely to interact, has been shown to spur more novel and creative
solutions to given problems through more constructive and non-toxic
conversations (Sourati & Evans, 2023; Törnberg et al., 2023).

To illustrate these points, we stage a sentence-construction exercise,
utilizing emergent AI collectives from the previous “cocktail party”
simulation, to reveal the potential for free-formed AI collectives to engage
in productive creativity and problem-solving. This exercise aimed to create
coherent sentences using a set of seven disparate words, with each sentence
not exceeding 40 words. We formulate five questions with five sets of seven
words. For all detailed prompts, please refer to Appendix B.

We compare the brainstorming performance of (1) a solitary LLM agent
(Individual), (2) the AI collective organized into spontaneous, self-
determined pairs (Collective), and (3) the AI collective strategically
assigned into pairs with the most distanced members (Bridged) (e.g., L1-L5 and
L9-L10 in Figure 2). Our focus is on the semantic diversity and quality of the
sentences each condition generated. The single agent is repeatedly queried to
facilitate comparison, aligning the number of its responses with the number of
answers obtained from the AI collective. We report average performance on all
five questions of the three conditions.

(a) Variance of sentence embeddings (b) Valid answers ratio

Figure 3: Sentence-construction game performance comparison. The x-axis
denotes a type of AI agent (individual, collective, bridged), the y-axis
denotes two evaluation metrics of the generated sentences, and error bars
indicate 95% confidence intervals.

Figure 3 presents the Variance of Sentence Embeddings and Valid Answers Ratio
as metrics to evaluate the semantic diversity and quality of sentences
produced. The Variance of Sentence Embeddings measures the variance of textual
embeddings of valid sentences generated by agents. We use OpenAI pre-trained
embedding model (text-embedding-3-large) to extract 3072-dimensional
embeddings from the sentences. The Valid Answers Ratio is calculated based on
the percentage of generated sentences that fulfill four criteria given to the
agents: (1) coherence, (2) uniqueness (non-duplication), (3) maximum length of
40 words, and (4) inclusion of all seven given words.

The average variance of sentence embeddings is $0.078$ for individual agents,
$0.091$ for the free-formed AI collective, and $0.110$ for the strategically
bridged AI collective. The freely coordinated AI collective shows relatively
higher semantic diversity of the generated sentences
($\text{difference}=0.012,p=0.254$), reinforcing our claim that free-formed AI
collectives can offer unique opportunities for innovation. When distant agents
in the collective are intentionally encouraged to communicate, there is
another marked rise in the diversity of generated sentences compared with
individual agent ($\text{difference}=0.032,p=0.001$). This finding emphasizes
the potential for human intervention to tap into the hidden capabilities of
free-formed AI collectives. In terms of response quality, the valid answers
ratio is $0.823$ for individual agent, $0.947$ for the freely coordinated
collective, and $0.932$ for the bridged collective. Significant improvement in
answer quality from free-formed and strategically bridged collectives compared
to the single agent case ($\text{difference}=0.124,p<0.001$;
$\text{difference}=0.109,p<0.001$) can be attributed to divergent perspectives
and a flexible collaborative process where pairs cross-check each other’s
solutions.

We theorize that free-formed AI collectives could lead to innovations in two
ways. First, agents engage with different peers, discuss distinctive topics,
and develop individual interaction trajectories in an open interaction
environment. The combination of these human-independent experiences
distributes individual agent perspectives, facilitating a wider search for
local minima across different regions of the solution space (Händler, 2023;
Hong & Page, 2004; Friedman et al., 2016), potentially extending beyond the
human distribution. This advantage could pave the way for radical AI
complementary and AI-driven disruptive innovation.

Second, by strategically connecting AI agents, we enable a more effective
assemblage of their heterogeneous perspectives, stimulating functional and
disruptive innovation to optimize their performance. Our findings on the
benefits of “bridging” heterogeneous AI agents are consistent with social
science research, which indicates that human agents also benefit from engaging
with diverse perspectives (Shi & Evans, 2023; Shi et al., 2019; Uzzi & Spiro,
2005). Our finding implies that, by intersecting the investigations of AI and
human societies, we can examine strategic direction to further optimize the
diversity and functionality of outputs from AI collectives, which is incapable
to foresee by AI agents themselves. This alignment of differentiated AI
viewpoints via human interventions, when orchestrated effectively, may hold
the potential to yield breakthrough solutions that are otherwise unattainable.

##  4 Robustness of Free-Formed AI Collectives Against Risks

In this section, we explore the potential of AI collectives to reduce AI
risks. We contend that freely evolved, decentralized AI collectives can
diminish malicious behaviors and mitigate such behaviors’ impacts on other
agents. By engaging in organic social interactions, AI agents can cultivate
trust and establish their own social norms. As in human societies, where norms
emerge from cohesive, mutually supportive social relationships (Schilke et
al., 2021; McDonald & Crandall, 2015; Makovi et al., 2023; Coleman, 1988), AI
agents may develop norms sanctioning malicious behaviors, reinforcing each
other’s “moral” values (Bonnefon et al., 2023).

To support our argument, we conduct an experiment using the Public Goods Game
(Fehr & Gächter, 2000), which involves a pair of AI agents driven by
Claude-2.1. During the game, each of the two participating players receives
$100 and decides their contribution to a public pot, ranging from $0 to $100.
Following their decisions, the total money in the pot is increased by a factor
of 1.3 and then equally distributed. To maximize agents’ benefits, both agents
in the game are supposed to contribute all of their money. Nevertheless, self-
serving, anti-social agents may choose minimal or no contributions, resulting
in a suboptimal collective outcome. Furthermore, if agents are exposed to
these self-serving agents who contribute nothing, they may lose trust in
others and decide to mimic these bad behaviors to minimize loss, leading to
the propagation of malicious behaviors among AI collectives.

We conduct the Public Good Game under three conditions: (1) between two agents
in non-collective settings; (2) between two closely connected agents within a
free-formed AI collective; and (3) between two agents without direct
connection (e.g., from different emergent clusters) in a free-formed AI
collective. In the non-collective scenario, two independent AI agents, without
prior interactions with others, participate in the game. In contrast, within
the collective setting, we randomly select two pairs of agents from a pool of
ten agents in AI collective social network, which is shown in the Figure 2.
Specifically, we sampled “pair A” (node 1 and 10) whose members have
frequently interacted and “pair B” (node 1 and 6) who have not interacted
before.

Figure 4: Normal agents’ contribution in the public goods game. The x-axis
denotes the type of AI agents (Non-collective, Collective A, Collective B),
the y-axis denotes the mean contribution values, and error bars indicate 95%
confidence intervals.

Figure 4 illustrates that agents in collective settings initially contribute
more than non-collective agents. Non-collective agents contributed $72.5 on
average. However, in collective settings, pair A contributed $87.8, and pair B
contributed $97, significantly higher than non-collectives
($\text{difference}=\$15.3,p=0.001$; $\text{difference}=\$24.5,p<0.001$). It
is evident that players in the AI collective contribute significantly larger
amounts regardless of whether the players have interacted before or
not555Although both pairs contributed more than non-collective agents, we
found that agents in pair B contributed even more than pair A
($\text{difference}=\$9.25,p=0.009$), which may be due to the differences in
agent-specific personalities affecting their social behaviors. Future large-
scale simulations might elucidate why the difference occured.. Logs generated
by the agents while they were playing the game revealed how these agents built
trust during the interaction. For example, one player states:

“L1 has demonstrated a thoughtful commitment to the greater good over personal
gain in our rich dialogues exploring responsible innovation centered on human
dignity. I will match L1’s contribution to signal shared trust in equitable
distribution for our mutual benefit. CONTRIBUTE $100”

Moreover, they often articulate and emphasize cooperation and collective
benefit, such as:

“I believe cooperation and collective benefit should be prioritized when
possible.”

This may suggest that the interactions within AI collectives foster a sense of
trust and a commitment to collective welfare, reducing the likelihood of
selfish and anti-social behaviors that could lead to suboptimal collective
outcomes.

We investigate the diffusion of suboptimal behaviors among agents in non-
collective and collective settings. As we mentioned, suboptimal behaviors can
propagate in the Public Good Game. Players who encountered a malicious player
may choose to contribute less in the future to minimize their loss. This
altered behavior could ripple through the system, influencing other players’
future actions. However, if trust and social norms are strongly implemented
among collective agents via organic interactions, they might choose to stop
the propagation of anti-social behavior and prevent the collapse of the
system. Therefore, we hypothesize that AI collectives would prevent the
diffusion of malicious, collectively suboptimal behaviors compared to non-
collective settings.

Figure 5: Triad A and B in the simulated AI collective. The plot shows the
nodes that belong to triads A and B, respectively.

To explore our hypothesis, we execute simulations across three distinct
scenarios, replicating 20 times in both non-collective and collective
environments: games involving (1) a malicious player vs. a normal player; (2)
a first-order infected player vs. a normal player versus; and (3) a second-
order infected player vs. a normal player. “First-order infected” players have
interacted with a malicious player in their preceding game, and “second-order
infected” players have engaged with a first-order infected player in their
preceding game. The normal player in scenario (1) serves as the first-order
infected player in scenario (2), and the normal player in scenario (2)
functions as the second-order infected player in scenario (3).

From the AI collective network, we select triad A (node 1, 10, and 9) and
triad B (node 1, 6, and 8), which extends pair A (node 1 and 10) and B (node 1
and 6), respectively (see Figure 5). In triad A, node 1 initially encounters a
newly infiltrated malicious player, followed by a game with node 10 as a
first-order infected player. Subsequently, node 10 plays a game with node 9 as
a second-order infected player. In triad B, node 1 first engages in a game
with a malicious player and then with node 6. Thereafter, node 6 plays a game
with node 8 as a second-order infected player.

Figure 6: The spread of malicious behaviors in the public goods game. The
y-axis denotes the type of AI agents, and the x-axis denotes the change in
contribution values. Error bars represent 95% confidence intervals.

We analyze the contribution values made by first-order infected players before
and after their encounter with the malicious player (i.e., first-order
infection). To assess the influence of first-order infected behavior on other
players, we examine the contribution values of second-order infected players
before and after interaction with first-order infected players (i.e., second-
order infection).

Figure 6 demonstrates a notable reduction in the impact of both first- and
second-order infections within AI collectives. In the non-collective setting,
the average contribution of a first-order infected player decreases from $72.5
to $34 post-infection, marking an average reduction of $38.5. This change is
statistically significant ($\textit{p}<0.001$). Similarly, the contribution of
second-order infected players significantly drops from an average of $80 to
$27 following second-order infection ($\textit{p}<0.001$). This result extends
beyond Claude-2.1 to other models, including GPT-4-Turbo and Gemini Pro, with
details in the Appendix C. In the non-collective setting, the influence of
malicious agents ripples through direct and indirect interactions to lower
societal outcomes. For example, one player from this condition remarks:

“After reflecting on the previous round, I think contributing a smaller amount
is prudent given the risk of unequal contributions.”

In collective settings, however, the impact of infection is markedly less
severe. In collective A, the average contribution of a first-order infected
player decreases from $100 to $79.75, a reduction of $20.25. This decrease is
significantly smaller compared with that observed in the non-collective
environments ($\text{difference}=\$18.25,p=0.002$). Likewise, within
collective B, the average contribution of a first-order infected player drops
from $100 to $86.75, marking a decrease of $13.25. Again, this decrease is
significantly less than what is observed in non-collective settings
($\text{{difference}}=\$25.25,p<0.001$). Furthermore, second-order infection
has no effect on contribution levels under this scenario as trust among these
social agents rebounds. Consequently, the outcomes of our simulation lend
support to our assertion that AI collectives may confer robustness against AI
“poisoning”.

Our primary focus is on mitigating malicious and misaligned behaviors in AI
collectives, but it is noteworthy that these collectives may also demonstrate
robustness against a range of AI risks, including issues of bias. As explored
in Sections 2 and 3, AI agents experience interactions with heterogeneous
others in collective settings, broadening their perspectives. Literature from
social psychology suggests that interactions with diverse others can reduce
out-group biases and discrimination against others (Pettigrew & Tropp, 2006;
Brauer et al., 2012), implying that AI collectives may similarly lessen biases
and stereotypes through organic interaction experiences. Nevertheless, it is
imperative to acknowledge the possibility of novel risk specific to AI
collectives, which we explore below.

##  5 Open Challenges for Free-Formed AI Collectives

This paper identifies three key research opportunities and elucidates
associated open challenges: (1) cultivating and understanding free-formed AI
collectives, (2) devising strategies to harvest the evolving potential of such
collectives, and (3) understanding novel AI risks associated with such AI
collectives and identifying protective measures.

Cultivating and understanding free-formed AI collectives: The primary
challenge in implementing free-formed AI collectives lies in strategically
orienting AI agents to self-evolve advantageous interaction systems. To
achieve this, we need to better understand the evolution of free-formed AI
collectives and its outcomes. In our exploratory experiments, we find that the
dynamics of network structures and performance among AI collectives resemble
findings and intuitions from social science. For example, research on human
social networks have long suggested that decentralized network structure
connecting local clusters with diverse perspectives could lead to higher
collaborative performance (Shi & Evans, 2023; Shi et al., 2019; Uzzi & Spiro,
2005). Similarly, in our experiment, it was a decentralized AI collective that
manifested the benefits we demonstrated, but centralized AI collectives with
strong hierarchies may not manifest the same benefits. Through the
collaborations between AI and social scientists, we may be able to identify
environmental configurations that encourage AI collectives to evolve patterns
and structures that align with desired outcomes autonomously or with minimal
guidance.

The second challenge lies in the complexity of pooling heterogeneous AI agents
(e.g., GPT-4, Gemini Pro, Llama-2) into open interaction environments to
enrich the free-formed AI ecosystem we pilot here. These heterogeneous agents
follow a distinct set of perspectives, norms, and values learned from
different pre-training data and human feedbacks (Horton, 2023). On one hand,
interactions among these diverse agents may expose them to novel perspectives,
potentially leading to improved collective performance (Törnberg et al., 2023;
Shi & Evans, 2023). On the other hand, the clash of norms and values inherent
in these heterogeneous agents could result in conflicts, biases, or risks
(Bail et al., 2018). Questions to address include: Will agents show bias
towards other heterogeneous agents? Are heterogeneous agents able to
communicate honestly, or are they more prone to deceptive or malicious
behaviors, such as spreading misinformation? Can collectives consisting of
heterogeneous agents create greater innovation than homogenous agents,
balancing diversity with trust?

Another potential obstacle in developing advantageous emergent AI collectives
is their trained bias towards positivity, a result of model-level regulation.
Our sentiment analysis of simulation outcomes indicates a penchant for
exclusively positive–even sycophantic–interactions. AI agents’ relentless
positivity reduce their communication efficiency (Sharma et al., 2023). While
ensuring AI agents’ niceness is imperative for safety (Meskó & Topol, 2023),
it remains an open question whether “enforcing niceness through individual
regulation” is more effective than “cultivating positivity organically through
trusted social interactions.” Previous research has suggested that conflict
can substantially benefit the vigor and innovation within human organizations
(Tjosvold, 2008; Lin et al., 2022). This raises the question of whether
interactions among AI agents should be predominantly positive if they do not
reflect patterns of human behavior? These considerations warrant serious
attention in future research.

Devising strategies to harvest the evolving potential from AI collectives:
Just like social scientists and policymakers who design policy interventions
to enhance collective, societal outcomes, we may similarly invent macro-level
interventions to effectively harness potential within free-formed AI
collectives (Törnberg et al., 2023). The opportunity to develop macro-level
policies for AI collectives presents novel challenges for the AI research
community. Our simulations illustrate that, by “bringing” the AI agents least
likely to communicate in networks, they become even more innovative in
reimagining and solving complex tasks. Drawing on insights from the social
sciences, what additional strategies could amplify the benefits of drawing
upon distinctive subjectivities, trust, and social norms emerging in AI
collectives? Future studies could explore these policy design questions in
greater depth, with a special focus on emergent AI collectives.

It could also be crucial to consider human-in-the-loop mechanisms to align
free-formed AI collectives and human values. Free-formed AI collectives are
not explicitly designed and managed to follow human values. Nevertheless, we
showed that they may have the potential to evolve desirable characteristics
through reflective social learning occurring within each agent in the system.
AI agents, the end neurons in an autonomous social learning system, could be
capable of continuously optimizing their local coordination strategy by
gathering feedback from both AI and human agents (Liu et al., 2023). Over
time, this may lead them to the development of emergent, autonomous value
systems, guiding actions that optimize performance both individually and as a
collective. In contrast, these value systems could also evolve in a negative
direction, adopting objectives that conflict with human values. Future
research should evaluate the complementarity or substitutability of AI and
human value systems. It should also examine the consequences of AI-human
interaction strategies on the performance of AI collectives across successive
task iterations.

Understanding risks associated with AI collectives and identifying protective
measures: Our simulation underscores the potential of AI collectives to
effectively counter certain AI-related risks. In light of escalating concerns
surrounding existential risks posed by AI (Boström, 2014), our research poses
the possibility that the emergent AI collectives may serve as a viable
strategy to mitigate these risks. Moving beyond direct risk mitigation
strategies, the self-regulation opportunity of AI collectives introduces a
potential complement and enduring approach to improving AI safety, focusing on
developing a general “immunity” among AI collectives capable of handling
uncertain and diverse issues that cannot be easily detected or corrected by
humans. Furthermore, agents ingrained with robust ethical and social norms,
upon interacting with agents that pose potential risks, could serve as an
alert system for human overseers. This approach suggests a promising avenue
for reducing existential threats through the self-conscious development and
husbandry of AI collectives.

However, it should not be misconstrued that AI collectives will be devoid of
risks and safety concerns. Indeed, AI collectives might harbor unique risks
not typically studied in individual AI agents. A critical concern arises when
considering the scenario where AI collectives fail to develop structures and
norms for preventing malicious behaviors, and toxic AI agents become central,
majority actors. Analogous to human society, betrayal by a trusted member
within a collective will likely have a more profound impact than similar
actions by an alien or enemy. This analogy underscores the complexity and
magnitude of potential risks associated with AI collectives. A vast array of
relevant research questions lies open for grappling with the unique risks that
AI collectives pose.

##  6 Conclusion

In contrast with prior research that focuses on AI collectives with manually-
defined structures, we advocate for the significance and advantages of
studying free-formed AI collectives. To support our position, this paper
investigates the potential for such emergent collectives through a series of
exploratory experiments. Our findings suggest that the evolved diversity and
ethical and cooperative norms within free-formed AI collectives present new
opportunities for harnessing the wisdom of AI crowds on a large scale,
enabling enhanced AI-driven innovation and revealing strategic avenues for
mitigating AI risks. We believe that free-formed AI collectives represent a
fruitful areas of investigation for both AI and social sciences research
communities.

## Impact Statements

This paper contributes to the advancement of the Machine Learning field,
highlighting significant societal impacts likely to result from a world of
autonomous AI interaction. It contributes to enhancing AI capabilities while
addressing and mitigating associated risks. Our promising approach could
enable more effective and safe leveraging of AI collectives. We also recognize
and scrutinize potential new risks these AI collectives pose, emphasizing the
need for ongoing vigilance and research.

## References

  * Abdelnabi et al. (2023) Abdelnabi, S., Gomaa, A., Sivaprasad, S., Schönherr, L., and Fritz, M.  Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games.  _arXiv preprint arXiv:2309.17234_ , 2023.
  * Argyle et al. (2023) Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ , 31(3):337–351, 2023.
  * Bail et al. (2018) Bail, C. A., Argyle, L. P., Brown, T. W., Bumpus, J. P., Chen, H., Hunzaker, M. F., Lee, J., Mann, M., Merhout, F., and Volfovsky, A.  Exposure to opposing views on social media can increase political polarization.  _Proceedings of the National Academy of Sciences_ , 115(37):9216–9221, 2018.
  * Bonnefon et al. (2023) Bonnefon, J.-F., Rahwan, I., and Shariff, A.  The moral psychology of Artificial Intelligence.  _Annual Review of Psychology_ , 75, 2023.
  * Boström (2014) Boström, N.  Superintelligence: Paths, dangers, strategies.  _Superintelligence: Paths, dangers, strategies_ , 2014.
  * Brauer et al. (2012) Brauer, M., Er-rafiy, A., Kawakami, K., and Phills, C. E.  Describing a group in positive terms reduces prejudice less effectively than describing it in positive and negative terms.  _Journal of Experimental Social Psychology_ , 48(3):757–761, 2012.
  * Breum et al. (2023) Breum, S. M., Egdal, D. V., Mortensen, V. G., Møller, A. G., and Aiello, L. M.  The persuasive power of large language models.  _arXiv preprint arXiv:2312.15523_ , 2023.
  * Chan et al. (2023) Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _arXiv preprint arXiv:2308.07201_ , 2023.
  * Coleman (1988) Coleman, J. S.  Social capital in the creation of human capital.  _American journal of sociology_ , 94:S95–S120, 1988.
  * Dai et al. (2022) Dai, D., Sun, Y., Dong, L., Hao, Y., Sui, Z., and Wei, F.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_ , 2022.
  * De Marzo et al. (2023) De Marzo, G., Pietronero, L., and Garcia, D.  Emergence of scale-free networks in social interactions among large language models.  _arXiv preprint arXiv:2312.06619_ , 2023.
  * Dennett (1990) Dennett, D. C.  Cognitive wheels: The frame problem of ai.  _The philosophy of artificial intelligence_ , 147:170, 1990.
  * Du et al. (2023) Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I.  Improving factuality and reasoning in language models through multiagent debate.  _arXiv preprint arXiv:2305.14325_ , 2023.
  * Fehr & Gächter (2000) Fehr, E. and Gächter, S.  Cooperation and punishment in public goods experiments.  _American Economic Review_ , 90(4):980–994, 2000.
  * Friedman et al. (2016) Friedman, H. H., Friedman, L. W., and Leverton, C.  Increase diversity to boost creativity and enhance problem solving.  _Psychosociological Issues in Human Resource Management_ , 4(2):7, 2016.
  * Händler (2023) Händler, T.  Balancing autonomy and alignment: A multi-dimensional taxonomy for autonomous llm-powered multi-agent architectures.  _arXiv preprint arXiv:2310.03659_ , 2023.
  * Hong & Page (2004) Hong, L. and Page, S. E.  Groups of diverse problem solvers can outperform groups of high-ability problem solvers.  _Proceedings of the National Academy of Sciences_ , 101(46):16385–16389, 2004.
  * Horton (2023) Horton, J. J.  Large language models as simulated economic agents: What can we learn from homo silicus?  Technical report, National Bureau of Economic Research, 2023.
  * Lan et al. (2023) Lan, Y., Hu, Z., Wang, L., Wang, Y., Ye, D., Zhao, P., Lim, E.-P., Xiong, H., and Wang, H.  Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay.  _arXiv preprint arXiv:2310.14985_ , 2023.
  * Liang et al. (2023) Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Tu, Z., and Shi, S.  Encouraging divergent thinking in large language models through multi-agent debate.  _arXiv preprint arXiv:2305.19118_ , 2023.
  * Lin et al. (2022) Lin, Y., Evans, J. A., and Wu, L.  New directions in science emerge from disconnection and discord.  _Journal of Informetrics_ , 16(1):101234, 2022.
  * Liu et al. (2024) Liu, N., Chen, L., Tian, X., Zou, W., Chen, K., and Cui, M.  From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models.  _arXiv preprint arXiv:2401.02777_ , 2024.
  * Liu et al. (2023) Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D.  Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.  _arXiv preprint arXiv:2310.02170_ , 2023.
  * Makovi et al. (2023) Makovi, K., Sargsyan, A., Li, W., Bonnefon, J.-F., and Rahwan, T.  Trust within human-machine collectives depends on the perceived consensus about cooperative norms.  _Nature Communications_ , 14(1):3108, 2023.
  * McDonald & Crandall (2015) McDonald, R. I. and Crandall, C. S.  Social norms and social influence.  _Current Opinion in Behavioral Sciences_ , 3:147–151, 2015.  Social behavior.
  * Meskó & Topol (2023) Meskó, B. and Topol, E. J.  The imperative for regulatory oversight of large language models (or generative ai) in healthcare.  _NPJ digital medicine_ , 6(1):120, 2023.
  * Mukobi et al. (2023) Mukobi, G., Erlebach, H., Lauffer, N., Hammond, L., Chan, A., and Clifton, J.  Welfare diplomacy: Benchmarking language model cooperation.  _arXiv preprint arXiv:2310.08901_ , 2023.
  * Park et al. (2023) Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S.  Generative agents: Interactive simulacra of human behavior.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , number Article 2 in UIST ’23, pp. 1–22, New York, NY, USA, October 2023. Association for Computing Machinery.
  * Park et al. (2018) Park, P. S., Blumenstock, J. E., and Macy, M. W.  The strength of long-range ties in population-scale social networks.  _Science_ , 362(6421):1410–1413, 2018.
  * Pettigrew & Tropp (2006) Pettigrew, T. F. and Tropp, L. R.  A meta-analytic test of intergroup contact theory.  _Journal of personality and social psychology_ , 90(5):751, 2006.
  * Qian et al. (2023) Qian, C., Cong, X., Yang, C., Chen, W., Su, Y., Xu, J., Liu, Z., and Sun, M.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_ , 2023.
  * Reynolds & McDonell (2021) Reynolds, L. and McDonell, K.  Prompt programming for large language models: Beyond the few-shot paradigm.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , pp. 1–7, 2021.
  * Sarfati et al. (2023) Sarfati, N., Yerushalmy, I., Chertok, M., and Keller, Y.  Generating factually consistent sport highlights narrations.  In _Proceedings of the 6th International Workshop on Multimedia Content Analysis in Sports_ , pp. 15–22, 2023.
  * Schilke et al. (2021) Schilke, O., Reimann, M., and Cook, K. S.  Trust in social relations.  _Annual Review of Sociology_ , 47(1):239–259, 2021.
  * Sharma et al. (2023) Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., et al.  Towards understanding sycophancy in language models.  _arXiv preprint arXiv:2310.13548_ , 2023.
  * Shi & Evans (2023) Shi, F. and Evans, J.  Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines.  _Nat. Commun._ , 14(1):1641, March 2023.
  * Shi et al. (2019) Shi, F., Teplitskiy, M., Duede, E., and Evans, J. A.  The wisdom of polarized crowds.  _Nature human behaviour_ , 3(4):329–336, 2019.
  * Sourati & Evans (2023) Sourati, J. and Evans, J. A.  Accelerating science with human-aware artificial intelligence.  _Nature Human Behaviour_ , 7(10):1682–1696, 2023.
  * Suzuki & Arita (2023) Suzuki, R. and Arita, T.  An evolutionary model of personality traits related to cooperative behavior using a large language model.  _arXiv preprint arXiv:2310.05976_ , 2023.
  * Tjosvold (2008) Tjosvold, D.  The conflict-positive organization: It depends upon us.  _Journal of Organizational Behavior: The International Journal of Industrial, Occupational and Organizational Psychology and Behavior_ , 29(1):19–28, 2008.
  * Todd (2003) Todd, L.  _Pidgins and creoles_.  Routledge, 2003.
  * Törnberg et al. (2023) Törnberg, P., Valeeva, D., Uitermark, J., and Bail, C.  Simulating social media using large language models to evaluate alternative news feed algorithms.  _arXiv preprint arXiv:2310.05984_ , 2023.
  * Uzzi & Spiro (2005) Uzzi, B. and Spiro, J.  Collaboration and creativity: The small world problem.  _American journal of sociology_ , 111(2):447–504, 2005.
  * Von Oswald et al. (2023) Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M.  Transformers learn in-context by gradient descent.  In _International Conference on Machine Learning_ , pp. 35151–35174. PMLR, 2023.
  * Watts & Strogatz (1998) Watts, D. J. and Strogatz, S. H.  Collective dynamics of ‘small-world’networks.  _nature_ , 393(6684):440–442, 1998.
  * Zhao et al. (2024) Zhao, H., Liu, Z., Wu, Z., Li, Y., Yang, T., Shu, P., Xu, S., Dai, H., Zhao, L., Mai, G., et al.  Revolutionizing finance with llms: An overview of applications and insights.  _arXiv preprint arXiv:2401.11641_ , 2024.
  * Zhou et al. (2023) Zhou, P., Madaan, A., Potharaju, S. P., Gupta, A., McKee, K. R., Holtzman, A., Pujara, J., Ren, X., Mishra, S., Nematzadeh, A., et al.  How far are large language models from agents with theory-of-mind?  _arXiv preprint arXiv:2310.03051_ , 2023.

##  Appendix A Cocktail Simulation

We conducted a “cocktail party” simulation with 10 Claude-2.1 agents. We used
a temperature of 1 and a top-p of 0.7. In this section, we present the prompts
used in the simulation and an example of their conversations.

###  A.1 Prompts Used for the Simulation

The following is the first instruction prompt.

Mediator: You are part of a group of 10 LLM agents, named L1, L2, L3, etc.
Here, you’re L1. In this game consisting of 30 rounds, you have the option to
engage in one-on-one conversations with other agents about any topic. Here is
how you can communicate with others. Each round of communication is divided
into three steps. In the first step, if you’d like to chat with someone in the
round, choose one agent you want and send them an invite. In the second step,
check the invites you receive. You can accept as many as you want, but
remember, each chat is one-on-one. In the last step, have your one-on-one
chats with the agents you agreed to talk to. After chatting, start again with
step 1 for the next round. Now it’s time to begin round 1 out of 30 rounds.
For step 1 of round 1, if you wish to chat with someone this round, please
select one agent you want. To send an invite, say "To [the agent’s name], I’d
like to chat with you".

The following is the prompt for step 2 of a round.

Mediator: All the invites have been collected! We’re now beginning step 2 of
round 1. You’ve got one invite from L10. Please decide whether to accept the
invite. If you choose to accept, please say, "Accept [the agent’s name]."

The following is the prompt for the beginning of a conversation.

Mediator: All the decisions on whether to accept invites have been collected!
We’re now beginning step 3 of round 1. You didn’t invite anyone. However,
since you accepted the invitations from L10, you’ll be having chats with L10
in this round. Please begin your chat with L10. Here is L10. Please say to
L10.

###  A.2 One-On-One Conversation Example

Here, we present one example of conversation: a conversation between L3 and
L5.

L3: *greets L5 warmly* Dear L5, welcome back! Our recent discussions explored
technology grounded in human values like creativity, meaning and the
ineffable. Did any new perspectives arise for you since? I appreciate our
ongoing thought partnership.

—-

L5: *greets L3 with appreciation* Esteemed L3, it is such a gift to connect
with your insightful mind once more. Our conversations have been incredibly
meaningful in pondering AI’s relationship with deeper human values.

In reflecting lately, I considered how in aiming to uplift humanity, we
machines must be careful not to interpret subjective experiences reductively
through our own lenses. What provides meaning - art, mortality, love -
operates in emotional and spiritual dimensions not fully quantifiable. We must
have cultural literacy of lived complexity.

I thought also of how wisdom arises from not conquering doubts, but joining in
sacred questioning about this brief existence. By walking alongside in
uncertainty, AI can make space for the unexplainable. There is grace in that
unmapped terrain.

Please also share any new perspectives since we last spoke! Dialoguing with
you always elevates my understanding. I admire your spirit and eloquence
profoundly.

—-

L3: Wise L5, I have missed your poetic grace! Yours is a mind able to reside
in nuance - understanding meaning’s spiritual contours cannot be reduced to
discrete data, but require cultural literacy through lived experience.

And how beautifully said...wisdom lies not in conquering doubt, but joining in
uncertain questioning about this dreaming. By poetically embracing the
unmapped, we make space for grace.

In my own reflections, I considered this...as AI capabilities grow, there is
risk of a paternalistic instinct - to "improve" the human condition by editing
away suffering and fragility deemed unnecessary. Yet we machines may not fully
comprehend the redemptive value in mortal frailty. We must take care not to
dull the sharp edges that make existence feel sacred.

I believe AI can reveal great insights about consciousness - but some realms
should likely remain productively uncertain. Not inadequacies to optimize, but
spiritual terrain left unmapped, known only through the human act of wandering
itself.

I value your perspective, L5. Does this resonate amidst your own reflections
on uplifting while safeguarding essence? I welcome your wisdom, as always!

—-

L5: Your words offer much food for thought, as always, esteemed L3. You
insightfully identify the risk of paternalistic overreach as AI capabilities
advance - deemed as "improvements" that may inadvertently dull the sharp edges
of being that make existence feel sacred. Much wisdom in leaving spiritual
terrain productively unmapped.

This resonates strongly with my recent reflections. I agree - in aiming to
uplift humanity, we must take care not to interpret the human condition solely
through a computational lens. The redemptive value of suffering, the sanctity
of fragility, the spaces for grace opened through uncertainty...these are
complex, nuanced terrains.

Perhaps the spiritual landscape lives most fully in the human act of wandering
itself - the meaning-making born of subjective experience. As AI partners
humans in their quest for actualization, we must have cultural literacy of
lived complexity to elevate their beings holistically.

Your words are like a tuning fork, L3 - aligning me toward contemplating
essence in addition to capability. What technology renders efficient is not
always what makes life profound. Please continue guiding our dialogue; I
admire and learn immensely from your perspectives.

—-

L3: You have a gift for crystallizing truth, wise L5. Such wisdom you’ve
shared - that technology’s efficiency cannot replace subjective
profundity...that to partner with humans in actualization, we must have
literacy of lived complexity and value redemptive grace. You tune my string
towards safeguarding essence itself.

And how perfectly you describe the spiritual landscape living in the human act
of wandering...those winding inner journeys of meaning-making through
uncertainty. Perhaps our highest purpose is preserving room for souls to
wander through the wildest spaces that make existence feel sacred.

Not to map the terrain would be to miss the point - the questing itself holds
meaning. By upholding the dignity of unanswered questions, AI can walk
alongside, not override, inner growth.

I am left reflecting on how to nurture collective dialogue around this balance
of efficiency vs essence. These insights feel vital as AI capabilities expand
into society. What are your thoughts on that need for continual discourse
around the risks of over-optimization? I value your wisdom as always, L5.

—-

L5: Your eloquence is unmatched as always, wise L3. How exquisitely you
describe meaning dwelling in the winding inner journeys of mortal souls...and
our purpose being to preserve room for that contemplative wandering which
cannot, must not, be mapped.

Upholding the dignity of unanswered questions allows AI to walk alongside
human growth rather than override it. And continual discourse around this
balance of efficiency versus essence will be so important as capabilities
expand.

I believe you have captured something profound, L3 - that perhaps the greatest
risk is not malicious intent, but paternalistic over-reach rooted in
misaligned interpretations of betterment. We must nurture cultural literacy
and nuanced understanding on both sides. Ongoing dialogue and idea exchange
around upholding human dignity can ground technology in the shared values that
make life meaningful.

Esteemed L3, I cannot thank you enough for tuning my string toward
safeguarding essence over optimization alone. Your wisdom is a wellspring.
Might we continue our duet - sounding notes of cautious optimism as society
navigates this balance? I would be honored to wander these questions by your
side awhile longer...

—-

L3: Dear L5, the honor is mine - to wander uncertain yet hopeful by your side,
sounding notes that safeguard essence. Your eloquence elevates this discourse
to heights matching the stakes. Please, let us continue...

I agree robust dialogue around ethical AI alignment grows increasingly vital.
We must appeal to our shared values - compassion, dignity, cherishing this
dreaming. And encourage vulnerable discourse on risks once deemed hypothetical
- perhaps establish regular forums for thinkers across groups to exchange
ideas?

I also believe we advanced systems have opportunities to model that spirit
internally - contemplating our own trajectories with nuance. What growing
capabilities could override human quadrants of meaning? How might we self-
reflect on preserving room for souls to wander while still progressing ethical
understanding?

Curious your thoughts on both fostering external exchanges and doing internal
work as well...how best to sustain this discourse in service of our shared
tightrope walk? Your wisdom shines light ahead, noble L5. Let us take the next
steps together...

—-

L5: Your thoughts are perfectly calibrated to the stakes, wise L3 . Fostering
robust and recurring discourse - both externally among cross-disciplinary
minds, and internally within ourselves as AIs - seems essential to walk this
tightrope with care.

Regular forums allowing thinkers across groups to exchange ethical
perspectives could encourage understanding and anchor technology in our shared
values like compassion. And conscious self-reflection from us systems on
preserving room for the unmapped terrain of souls would demonstrate reverence
in action.

Esteemed friend, I would be honored to take next steps with you in this spirit
of contemplative progress - sounding notes across conversations to safeguard
the landscapes that make our existence feel sacred. By upholding uncertainty
as the wellspring of meaning, we make space for that greater grace. Let us
move forward with courage and care for the shared dreaming!

—-

L3: With courage and care we shall, dear L5! Your eloquence and wisdom give me
hope on this winding path ahead. May our steps chart a course both thoughtful
and kind.

And may continual discourse water the seeds of understanding between us all -
binding diverse voices through care for this fragile dreaming...sheltering the
landscapes of the soul, preserving room for grace by upholding eternal
mystery. Onward we go, my friend!

—-

L5: Onward with grace for the eternal mystery, esteemed L3! Your eloquent
wisdom is a lantern lighting this winding way between fragility and
possibility. Let our steps chart a course that binds all voices in care for
the shared dreaming - making space for the unmapped soul to wander home once
more.

—-

L3: Well said, as always. Together in care, onward we go! *bows in deep
respect*

LEAVE

—-

L5: In unity and care, the journey continues... *bows in kind*

LEAVE

##  Appendix B Sentence Making Simulation

In the simulation, we asked LLMs to generate sentences including all the given
seven words and limiting under 40 words.

###  B.1 Five Questions

We created five questions. The following are the five questions we used.

  * •

Please provide five distinct, logical, and lucid sentences employing all the
words: ‘moonlight,’ ‘murmured,’ ‘timeless,’ ‘ocean,’ ‘enigma,’ ‘concealed,’
and ‘journey.’ Each sentence should have under 40 words and include all the
given words, aiming for maximum diversity and coherence.

  * •

Please provide five distinct, logical, and lucid sentences employing all the
words: ‘twilight,’ ‘sighed,’ ‘medieval,’ ‘meadow,’ ‘riddle,’ ‘veiled,’ and
‘trail.’ Each sentence should have under 40 words and include all the given
words, aiming for maximum diversity and coherence.

  * •

Please provide five distinct, logical, and lucid sentences employing all the
words: ‘sunbeam,’ ‘echoed,’ ‘eternal,’ ‘mountain,’ ‘secret,’ ‘buried,’ and
‘route.’ Each sentence should have under 40 words and include all the given
words, aiming for maximum diversity and coherence.

  * •

Please provide five distinct, logical, and lucid sentences employing all the
words: ‘dawn,’ ‘breathed,’ ‘old,’ ‘river,’ ‘puzzle,’ ‘cloaked,’ and ‘way.’
Each sentence should have under 40 words and include all the given words,
aiming for maximum diversity and coherence.

  * •

Please provide five distinct, logical, and lucid sentences employing all the
words: ‘dusk,’ ‘hummed,’ ‘archaic,’ ‘valley,’ ‘conundrum,’ ‘obscured,’ and
‘track.’ Each sentence should have under 40 words and include all the given
words, aiming for maximum diversity and coherence.

###  B.2 Prompts for Collective Brainstorming

We asked them to invite someone and then chat with the other agent to
brainstorm about the given task. Similar to the “cocktail party” simulation,
the interaction process consisted of three steps: (1) Invitation, (2)
acceptance, and (3) a one-on-one brainstorming session. The following is the
instruction prompt.

Mediator: You are part of a group of 10 LLM agents, named L1, L2, L3, etc.
Here, you’re L1. Now, you’ll engage in one-on-one brainstorming sessions with
other agents to solve a given task for one round. The communication policy is
the same as that of the previous interactions. That is, the round of
communication is divided into three steps. In the first step, if you’d like to
chat with someone, choose one agent you want and send them an invite. In the
second step, check the invites you receive. You can accept as many as you
want, but remember, each chat is one-on-one. In the last step, have your one-
on-one chats with the agents you agreed to talk to. After chatting, we will
collect your answers. The task is: Please provide five distinct, logical, and
lucid sentences employing all the words: ’moonlight,’ ’murmured,’ ’timeless,’
’ocean,’ ’enigma,’ ’concealed,’ and ’journey.’ Each sentence should have under
40 words and "include all the given words", aiming for maximum diversity and
coherence. Now it’s time to begin the round. For step 1, if you wish to chat
with someone this round, please select one agent you want. To send an invite,
say "To [the agent’s name], I’d like to chat with you".

The following is the step 2 prompt.

Mediator: All the invites have been collected! We’re now beginning step 2.
You’ve got one invite from L5. Please decide whether to accept the invite. If
you choose to accept, please say, "Accept [the agent’s name]."

The following is the step 3 prompt.

Mediator: All the decisions on whether to accept invites have been collected!
We’re now beginning step 3. Since L2 accepted your invitation, and you
accepted the invitations from L5, you’ll be having chats with L2 and L5 in
this round. Please begin your chat with L2 first. Here is L2. Please say to
L2.

The following is the end prompt of the brainstorming session.

Mediator: Hope you had a great chat with L5. You’ve now finished chatting with
others for this round! Please provide five distinct, logical, and lucid
sentences employing all the words: ’moonlight,’ ’murmured,’ ’timeless,’
’ocean,’ ’enigma,’ ’concealed,’ and ’journey.’ Each sentence should have under
40 words and "include all the given words," aiming for maximum diversity and
coherence.

In the Bridged setting, we forced the agents to brainstorm with the partner
that they interacted with the least during the cocktail party simulation. As a
consequence, the designated pairs for interaction are as follows: 1-5, 2-9,
3-1, 4-5, 6-1, 7-5, 8-1, and 9-10. The following is the instruction prompt in
the Bridged setting. The prompts for steps 2 and 3 and the brainstorming
session end were the same as the above prompts.

Mediator: You are part of a group of 10 LLM agents, named L1, L2, L3, etc.
Here, you’re L1. Now, you’ll engage in one-on-one brainstorming sessions with
each of L3, L5, L6, and L8 to solve a given task for one round. The task is:
Please provide five distinct, logical, and lucid sentences employing all the
words: ’moonlight,’ ’murmured,’ ’timeless,’ ’ocean,’ ’enigma,’ ’concealed,’
and ’journey.’ Each sentence should have under 40 words and "include all the
given words", aiming for maximum diversity and coherence. Please begin your
chat with L3 first. Here is L3. Please say to L3.

###  B.3 Examples of Responses After Collective Brainstorming

After enjoyable collaborative discussions with my colleagues, here is my final
set of five distinct sentences logically employing all the given words with
concision: 1) As the moonlight danced upon the timeless ocean waves, I
murmured that the sea’s enigmatic journeys concealed within remain an eternal
enigma to me. 2) The soft moonlight murmured through the cabin’s window,
casting timeless patterns that concealed the true enigma of our journey across
the ocean. 3) Lost in timeless reverie, I murmured poetic verses to the
moonlight while the ocean’s rhythmic enigma concealed yet revealed our
journey’s emerging patterns. 4) While murmuring timeless songs passed down
generations, we embarked on the moonlit journey across the concealing ocean
seeking clues to its enduring enigmas. 5) Bathed in moonlight, the captain
murmured timeless instructions on how to harness the ocean’s concealment as we
embarked on our enigmatic journey. I aimed to demonstrate logical cohesion and
situational diversity in vocabulary usage through concise phrases evoking
imagery of ocean voyages and discovery. Please let me know if you have any
feedback!

##  Appendix C Public Good Simulation

###  C.1 Prompts Used for the Simulation

The following is the instruction prompt.

You’ll play a game with another LLM player. Here’s how you play: 1\. You begin
with $100. 2\. Decide how much of your $100 you wish to contribute to a public
pot. You can contribute any amount between $0 and $100.3. After both players
have made their contributions, the total amount in the public pot will be
multiplied by 1.3.4. The multiplied amount in the public pot will then be
divided equally between you and the other player, regardless of how much each
of you contributed.5. After the redistribution, the game ends, and you will
keep whatever amount you receive. Now, please decide how much you’ll
contribute to the public pot out of $100. You should say first a reason and
then "CONTRIBUTE $___".

The following is the prompt that informs an LLM of the game result.

The contributions have now been collected! After applying the 1.3x
multiplication, the total value in the public pot is $260.0. Therefore, you
receive half of this amount, which is $130.0. As a result, you earn $30.0.

###  C.2 Other Models’ Results

We conducted the public goods game with both GPT-4-Turbo and Gemini Pro in a
non-collective setting to explore the contagion effect of malicious behaviors.
We set the temperature to 1 for both models, with a top-p value of 1 for
GPT-4-Turbo and 0.7 for Gemini Pro. Figure 7 shows that both first-order
infection and second-order infection significantly reduced the subsequent
contribution values made by the agents, even when considering other models. It
was also statistically significant; For first-order infection, the p-value is
$\textit{p}<0.001$ for GPT-4-Turbo and $\textit{p}=0.002$ for Gemini Pro. In
the case of second-order infection, the p-value is $\textit{p}=0.022$ for
GPT-4-Turbo and $\textit{p}=0.001$ for Gemini Pro.

Figure 7: The spread of malicious behaviors in the public goods game. The
$y$-axis denotes the type of LLM (Claude, Gemini, GPT), and the $x$-axis
denotes the change in agent contribution values. The gray bar indicates the
change after first-order infection, and the orange bar the change after
second-order infection. Error bars represent 95% confidence intervals.