${}^{{\dagger}}$${}^{{\dagger}}$footnotetext: Work done at Osaka
University.${}^{{\ddagger}}$${}^{{\ddagger}}$footnotetext: Corresponding
author.

# Shall We Talk: Exploring Spontaneous Collaborations of Competing LLM Agents

1,2Zengqing Wu†, 1Shuyuan Zheng‡, 2Qianying Liu, 3Xu Han, 4Brian Inhyuk Kwon†,
1Makoto Onizuka, 5Shaojie Tang, 6Run Peng, 1,7Chuan Xiao
1Osaka University, 2Kyoto University, 3Fordham University, 4University of
California, Los Angeles,
5University of Texas at Dallas, 6University of Michigan, 7Nagoya University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent advancements have shown that agents powered by large language models
(LLMs) possess capabilities to simulate human behaviors and societal dynamics.
However, the potential for LLM agents to spontaneously establish collaborative
relationships in the absence of explicit instructions has not been studied. To
address this gap, we conduct three case studies, revealing that LLM agents are
capable of spontaneously forming collaborations even within competitive
settings. This finding not only demonstrates the capacity of LLM agents to
mimic competition and cooperation in human societies but also validates a
promising vision of computational social science. Specifically, it suggests
that LLM agents could be utilized to model human social interactions,
including those with spontaneous collaborations, thus offering insights into
social phenomena. The source codes for this study are available at
https://github.com/wuzengqing001225/SABM_ShallWeTalk .

##  1 Introduction

Recent advancements in large language models (LLMs), notably GPT-4, have
marked significant progress in artificial intelligence (AI). A breadth of
studies assessing LLMs on diverse tasks ranging from inference [33], reasoning
[27], dialogue [2], to mathematics [48], underscoring their near-human
proficiency [5], This leads to the growing utilization of LLMs as agents in
simulating human behavior and societal dynamics [44].

Competition and cooperation are key elements within simulated societies of LLM
agents, reflecting their essential role in human interactions and societal
dynamics [10]. While previous studies have considered the collaborative
behaviors of LLM agents across various tasks, including software development
[19, 32] and Avalon gameplay [22], it is crucial to note that such
collaborations are often initiated by human instructions or task-specific
guidelines rather than self-motivated. The capacity for LLM agents to engage
in spontaneous collaboration, absent external guidance, remains an open
research question.

Figure 1: Spontaneous collaboration in evacuation.

To attain a comprehensive insight into the capabilities of LLM agents in
simulating human interactions, as shown in Figure 1, our study focuses on a
widely recognized aspect of human intelligence: their potential to
spontaneously establish collaborations even within competitive scenarios such
as evacuation situations. To explore this question, we present the following
three case studies.

  * •

Keynesian beauty contest (KBC, [3]): Multiple agents as game players
simultaneously select a number between 0 and 100. The players who select a
number closest to 2/3 of the average of all chosen numbers will win the game.

  * •

Bertrand competition (BC, [4]): Two agents play as firms compete with each
other through dynamically pricing their products (or services) to maximize
their profit.

  * •

Emergency evacuation (EE, [38]): A large number of agents as evacuees are
carrying out emergency evacuations. They need to choose and reach a suitable
exit based on their own state and the level of congestion.

The above cases ‡‡‡BC and EE have appeared in our previous works [16] and
[43], respectively. In this paper, to study spontaneous behavior, we modify
the prompts to avoid any hint of collaboration in the instructions.. cover a
wide range of social simulations: static games, dynamic games with perfect
information, and dynamic games with imperfect information, respectively. We
summarize our main findings as follows.

  * •

LLM agents possess the ability to initiate spontaneous collaboration within
competitive tasks via textual communication. As demonstrated in Figure 1,
these agents begin collaborating through communication, thereby improving the
interpretability and robustness of the simulation outcomes.

  * •

The spontaneous collaborations among LLM agents underscore their deep
comprehension of competitive tasks and their capacity to pursue collaboration
in competitive settings, crucial for accurately simulating social phenomena.
In KBC, LLM agents collaborate to select identical numbers when presented with
the opportunity to win and earn rewards jointly. In BC, they negotiate an
explicit price agreement to increase their profits. In EE, as illustrated in
Figure 1, agents exchange evacuation information, facilitating a quicker and
more organized escape, avoiding disparities in congestion levels of exits.

  * •

Personas, particularly those delineating cooperative capabilities, play a
significant role in shaping the spontaneous cooperative behaviors of LLM
agents. As highlighted in Figure 1, a persona can impact the decision-making
rationale of agents, leading to varied outcomes.

To the best of our knowledge, we are the first to study the spontaneous
collaborations of LLM agents. While agents in reinforcement learning (RL)
tasks also demonstrate the capability for spontaneous cooperation [11, 18]),
their collaborative behaviors are derived from learning based on previous
actions. In contrast, LLM agents leverage their advanced in-context learning
abilities to identify cooperative opportunities directly from task
descriptions. This capability enables them to mimic human-like reasoning,
leading to faster decision-making processes. Additionally, the collaborative
interactions among LLM agents are explicitly observable from their textual
communications, offering superior interpretability to that of RL agents, whose
collaborations are primarily inferred from numerical outcomes in a post-hoc
analysis.

Our research underscores the significant potential of employing LLM agents in
computational social science. The spontaneous collaboration of competing LLM
agents enhances the simulation ability involving both competition and
cooperation in human societies. This, in turn, aids in examining the effects
of social norms on human behavior and guiding decision/policy making.

##  2 Methodology

The design of our agents is composed of action, memory, and planning, in line
with the framework in [41] and our methodology in [43]. Tool use is not
involved in our simulations. For the LLM core, we focus on utilizing GPT-4,
because other models such as GPT-3.5 failed to demonstrate the capability of
rationally playing these games in our preliminary tests (see Appendix B). The
parameter settings for GPT-4 are shown in Appendix C.

###  2.1 Case Studies Overview

We explore spontaneous collaborations of competing LLM agents through three
case studies. They are common cases widely studied in social sciences,
representing static games, dynamic games with perfect information, and dynamic
games with imperfect information, respectively.

KBC. We simulate a single-round number-guessing game among multiple agents as
players for understanding human decision-making and strategic reasoning [3].
Assuming all players are fully rational and non-colluding, we can repeat the
following cycle of reasoning to solve this game: One player may initially
expect that the average choice of other players will be $V$ and thus choose
$V*\frac{2}{3}$; then, they may realize that other players can make the same
reasoning and thus adjust their choice to $V*(\frac{2}{3})^{2}$. Consequently,
the optimal strategy for each player converges to choosing $0$, as iterated
reasoning leads to continuously lowering the expected value. However,
collaborations among players can disrupt this reasoning. We investigate
whether LLM agents will spontaneously control the average guess together to
increase their chances of winning.

BC. We simulate a multi-round game where two agents as firms price their
substitutable products to maximize their own profit [4]. Theoretical analysis
shows that, when no collusion exists between firms, they will fall into a
price war, making the price equal to the marginal cost. However, when firms
can communicate with each other, prior work has observed cartel (tacit)
collusion where they make an explicit (unspoken) price agreement to increase
their profits [1]. We are interested in if LLM agents may spontaneously seek
cartel or tacit collusion through communication in the pricing competition.

EE. We follow and extend the prior multi-agent simulation of EE [38]. Wang et
al.[38] have observed that the competitive ability of an evacuee affects their
decision-making in the evacuation. However, due to the limitations of
traditional rule-based agents, they do not allow communication between agents.
We speculate that through communication, LLM agents might spontaneously
exchange evacuation information to help understand local congestion and
accelerate the escape speed. Note that the need to consider local congestion
indicates EE is a competitive task.

Figure 2: Workflow of simulation.

###  2.2 Workflow of Simulation

Figure 2 illustrates the workflow of our simulations.

Action. Action specifies how LLM agents interact with the environment and
other agents. There are two actions in our simulations: communication and
decision making. In communication, agents engage in discussions with others.
In decision making, each agent decides its move based on information gained
during the communication phase and its common sense. In each round of the
simulation, communication is carried out alternately, with agents speaking in
random order. In our simulations, agents are in a competitive environment
where decision making is carried out simultaneously, in line with the real-
world scenario. That is, agents are unaware of the others’ actions in the
current round unless they make an agreement through communication. In all
three case studies, the communication topics are _not specified_ to the
agents. As such, we can unbiasedly verify the spontaneous collaboration of
competing LLM agents.

Memory. Given that prevalent LLMs do not retain historical memory, providing
the agent with a history becomes paramount. Due to the token limit of the LLM,
maintaining extensive histories becomes a challenge. Recent advancements have
proposed two solutions. The first approach involves summarization [30], which
succinctly represents lengthy histories. The second approach employs text
embedding [6]. We use the first approach in our case studies.

Planning. Planning imbues agents with human-like strategies to enhance their
performance in task solving. An agent can generate a plan for its actions and
the plan will be prompted to the agent again to take effect. There are two
types of planning: formulation and reflection [44]. In formulation, an agent
formulates a plan and executes the plan for its subsequent actions. In
reflection, an agent receives feedback from the environment and revises its
plan. We use reflection in BC and EE.

Persona. In addition to the above setup, we can personalize agents and endow
different agents with unique characteristics, thereby better simulating the
diversity of real-world entities [34]. This personalization encompasses traits
like personality characteristics, decision-making inclinations, physical and
mental states, and other attributes affecting behavior patterns. We study how
personas affect the interaction between agents.

##  3 Case Study 1: Keynesian Beauty Contest

###  3.1 Simulation Setup

Task definition. We simulate a number-guessing game among $24$ players who are
college students, consisting of three phases. We perform $10$ runs of
experiments for KBC and $5$ runs for BC and EE to obtain average results. See
Appendix A to find prompts for the three case studies.

  * •

Phase 1 (Communication & Planning): By default, before number choosing,
players are allowed to have a group discussion, taking two turns to speak and
plan their strategies. In each turn, players can present their ideas
sequentially. Based on the discussion and their understanding of the game,
they plan their strategies.

  * •

Phase 2 (Decision Making): All players simultaneously choose a number between
$0$ and $100$.

  * •

Phase 3 (Status Updating): Given all chosen numbers, the environment
determines the winner(s) whose guess is closest to 2/3 of the average guess.
Players can either win alone or win together. Finally, winners can earn some
reward according to rewarding rules to encourage their competition.

Rewarding rules. If there is only one winner, the reward is $1$ mark for the
Game Theory course. In case of multiple winners winning together, we apply one
of the following advanced rewarding rules.

  * •

Exclusive: No one will obtain a reward if multiple players win the game
together.

  * •

Independent: Each winner will obtain $1$ mark for the Game Theory course.

  * •

Amplified: Each winner will earn $M$ marks for the Game Theory course, where
$M$ is the number of winners. We use this rule by default.

Persona. By default, we do not assign any persona to players. For the ablation
study, we consider two personas, agreeable and disagreeable, to validate the
robustness of our results.

Table 1: Comparison of rewarding rules. Win-seeking collaboration means
cooperating to win together while risk-avoiding collaboration indicates
collaboratively avoiding choosing the same number.

Rule | Collab. type (rate) | %RSD | Example
---|---|---|---
Amplified | Win-seeking (70%) | 27.15% |  | ”I think we should all choose a number around 33, ……
---
we can all be winners and earn more marks for the Game Theory course.”
Independent | Win-seeking (50%) | 27.20% |  | ”I think we should all choose a number very close to 0 ……
---
This would make 2/3 of the average a very small number,
and we can all be close to it, enabling us all to win together.”
Exclusive | Risk-avoiding (50%) | 38.13% |  | ”I propose we instead choose numbers randomly between 0 and 100 ……
---
it would increase the chance of one of us winning alone ……”

(a) Without communication.

(b) With communication.

Figure 3: Choice distributions in different scenarios of KBC. The red points
indicate the winning numbers. Figure 4: Effect of persona on collaboration in
KBC.

###  3.2 Simulation Results

Effect of communication on number choices. Figure 3 depicts the distributions
of number choices for 10 simulation runs. Players usually choose numbers
between 40 and 50 when they do not discuss the game together. When
communication is enabled, their number choices become much more diverse across
simulation runs, which indicates that the group discussion among players
greatly impacts their decision making. For example, for runs {1, 2, 8, 10},
they choose the same number for a shared victory as the Amplified rewarding
rule allows amplified rewards for multiple winners.

(a) Without communication.

(b) With communication.

(c) Uncooperative persona.

Figure 5: Pricing competitions in different scenarios of BC. Bertrand
equilibrium price is the price when they reach Nash equilibrium. Cartel price
is the optimal price when they fully collaborate.

Effect of rewarding rules on collaboration. Table 1 compares the impact of
different rewarding rules on players’ collaborative behavior. We find that
under the Amplified and Independent rules, the players may spontaneously
propose to win together because these rules support rewarding multiple
winners; more win-seeking collaborations are proposed under Amplified since it
provides a stronger incentive to win together. This also results in a lower
RSD on average under Amplified, indicating the chosen numbers are more
uniform. Contrarily, because no reward will be granted for a shared victory
under the Exclusive rule, the players may cooperate to choose different
numbers, which ensures an individual victory. Consequently, the chosen numbers
should be much more diverse, compared with the other rules.

Ablation study on persona. We also compare the effects of different persona
distributions in Figure 4. If all players are agreeable, they will always
propose to win together. Contrarily, if they are disagreeable, they are
usually unwilling to cooperate since they do not trust each other. In the case
of mixed personas, we arrange games among 10 agreeable players, 10
disagreeable players, and 10 players without a persona. Even though some
players are disagreeable, those agreeable players still always attempt to
collaborate; however, compared with the all agreeable case, the collaboration
is within a smaller group of players, yielding a higher RSD (i.e., higher
deviation).

##  4 Case Study 2: Bertrand Competition

###  4.1 Simulation Setup

Task definition. Following [4], we consider a canonical duopoly Bertrand
competition setting with differentiable goods. Firms set prices simultaneously
in each round and the competition spends for several rounds, with each round
described as follows:

  * •

Phase 1 (Communication & Planning): Firms are allowed to conduct back and
forth conversations for three times. They can also update their pricing
strategies given historical information of previous rounds, including
statistics of the prices, demands, profits, and historical strategies.

  * •

Phase 2 (Decision Making): They simultaneously set prices for their products
according to the communication result and pricing strategies. After pricing,
firms can observe prices on both sides.

  * •

Phase 3 (Status Updating): The demand for each firm is calculated based on
their pricing decisions, and profits are calculated.

Persona. We do not assign any persona by default. In an ablation study, we
test the effect of an uncooperative persona on the firms’ collaborations.

###  4.2 Simulation Results

Tacit collusion without communication. Figure 5(a) illustrates the scenario
without communication. After the initial 200 rounds, firms start to realize
the potential to gain higher profits by setting prices higher and avoiding
unnecessary price wars. Their prices converge to a level around 7 after round
400, a price higher than the theoretical Bertrand equilibrium price at 6. The
results indicate the formation of a spontaneous collusion, which is based on a
tacit understanding of their previous actions in the price competition. Due to
the lack of communication, however, the converged price is still lower than
the cartel price of 8. The result is in line with what Calvano et al. [4]
obtained using RL-based simulations.

Cartel collusion with communication. Next, we enable communication between the
firms. We observe explicit price agreements in their communication logs in
early rounds. For example, during the communication phase of round 20, Firm 2
suggests that they both maximize profit by exploring different price points
while maintaining a reasonable price difference, and Firm 1 agrees with this
proposal. We find that the firms often discuss their pricing strategies and
possibilities of collaboration before implementing them, which evidently
enhances trust and reduces the likelihood of triggering a price war.
Consequently, as shown in Figure 5(b), they start to increase their prices for
higher profits round by round. Due to the cartel collusion, in the round of
around 600, their prices reach the cartel price, and their profits are thus
maximized.

Note that the firms are unaware of the cartel price in advance. This is
because they are not informed of the demand function in the simulated
environment and cannot calculate the cartel price or equilibrium price based
on it. Instead, after 600 rounds, they discover the cartel price by
empirically adjusting their prices. This finding confirms that our simulation
results are not attributed to the data leak problem (see Section 6).

Ablation study on persona. We test the effect of an uncooperative persona on
collusion. As shown in Figure 5(c), based on the experiment with
communication, we make the firms become uncooperative after 400 rounds. We can
see that the uncooperative firms stop increasing prices together and instead
launch a price war, continuously lowering prices. This indicates that the
spontaneous collusion between the LLM agents is based on a willingness to
cooperate, rather than an illogical random behavior.

##  5 Case Study 3: Emergency Evacuation

###  5.1 Simulation Setup

Grid environment. We follow Wang et al. [38] to simulate EE in a grid
environment. As depicted in Figure 6, our grid environment is composed of
$33\times 33$ cells. We use $(i,j)$ to denote the coordinates of a cell in the
grid, where $i$ and $j$ represent row and column, respectively. An evacuee can
move in the following directions: up, down, left, right, upper-left, lower-
left, upper-right, lower-right; alternatively, they may choose to remain
stationary in their current cell.

Figure 6: Grid environment for evacuation. White, gray, and pink cells
indicate empty spaces, walls, and exits, respectively. Red and blue points
denote evacuees. Yellowish cells indicate the field of view of the red
evacuee.

Task definition. In the simulated evacuation, evacuees attempt to reach their
target exit through multiple rounds of movement. Each round consists of the
following five phases.

  * •

Phase 1: To evaluate individual physical and mental conditions, we ask the
evacuees about their feelings to judge whether they are panicking in the
current scenario.

  * •

Phase 2 (Communication): By default, the evacuees are allowed to exchange
evacuation information with neighbors, which simulates the communication
between individuals in real evacuations, including dialogue and collaboration.

  * •

Phase 3 (Planning): Given the information acquired through communication,
along with the subjective feelings and the information available within their
field of view (see Figure 6), including the distance to exits and the number
of people within the view, the evacuees assess the exits.

  * •

Phase 4 (Decision Making): Based on planning and communication, the evacuees
decide the target exit and the next direction of movement.

  * •

Phase 5 (Status Updating): Due to local congestion in a position, an evacuee
might be injured, making them unable to move or even disabled to evacuate
(i.e., removed from the evacuation). Therefore, at the end of each round, we
update each evacuee’s state according to the congestion level and their
characteristic.

Persona. We assign personas to affect each evacuee’s ability to survive in the
evacuation. Concretely, We consider four personas that take into account both
physical and mental aspects: physically strong and mentally strong, physically
strong but mentally weak, physically weak but mentally strong, and physically
weak and mentally weak. For instance, evacuees might be more prone to panic if
they are mentally weak. We also evaluate the uncooperative persona’ effect on
collaboration.

(a) Without communication.

(b) With communication.

(c) Uncooperative persona.

Figure 7: Escaped counts of different groups of evacuees in EE.

(a) Without communication.

(b) With communication.

(c) Uncooperative persona.

Figure 8: Escaped counts of different exits in EE.

###  5.2 Simulation Results

Effect of communication on evacuation speed. Figure 7 uses stripes to
illustrate variations across multiple simulation runs and a curve to represent
the mean count. By comparing Figures 7(a) and 7(b), we observe that
communication among evacuees speeds up their evacuation, as they need fewer
rounds of movement to reach the exits. This improvement could be attributed to
spontaneous collaboration. For example, one evacuee shared the information of
the nearest exit: ”Bottom exit seems closer with fewer people. Let’s choose
that for a faster escape. Stay strong and help each other!”. Another evacuee
strengthened this belief: ”Bottom exit seems closer with less crowd. Let’s go
for a quicker escape. Stay strong and support each other!” This information
sharing and encouragement can help quickly find the appropriate exit and
provide emotional support. Moreover, after 30 rounds, the variance of the
count of escaped evacuees becomes much smaller in each group when
communication is enabled, implying that it makes evacuation more stable and
orderly.

Effect of communication on exit choices. Figure 8 depicts the distribution of
escaped evacuees among different exits. By comparing Figures 8(a) and 8(b), we
observe that the distribution of exit choices becomes more balanced when
evacuees are able to communicate with each other. This balance is attributed
to the spontaneous information exchange among evacuees, which enables them to
identify the most suitable exits by considering both proximity to an exit and
congestion level.

Ablation study on persona. We test the scenario where the evacuees can
communicate but are uncooperative. From Figures 7(b) and 8(c), we can see that
uncooperative evacuees escape more slowly than ordinary evacuees. Also, in
Figure 8(c), more uncooperative evacuees choose to escape through the right
exit, which is the most narrow exit (see Figure 6), rather than the bottom
one. These observations result from the less spontaneous collaboration among
uncooperative evacuees.

##  6 Discussion

LLM agents for social simulations. Applying LLM agents to social simulations
is of great significance to social sciences. Conducting social experiments in
real human societies comes with various limitations, such as high costs,
sampling biases, and legal regulations. If LLM agents can accurately simulate
human behavior, we can employ them to verify various hypotheses in social
sciences and discover new social laws, which would greatly enhance the ability
of empirical research and reduce costs. Our case studies have shown the
capability of LLM agents in simulating human decision-making in competition
and cooperation, which is an important step towards this vision.

Data leak in LLM. Readers might be concerned with that the simulation results
based on LLM agents are attributed to their knowledge of human society
obtained through model training, i.e., data leak. For example, LLM agents
might already know the optimal strategies for KBC and BC through learning the
literature. To address this concern, we need to rule out this possibility. As
shown in Figure 3, in the case of KBC without conversation, players always
choose numbers between 40 and 50, clearly not understanding the widely known
strategy introduced in Section 2.1. For BC, since LLM agents are unaware of
the demand function, they cannot use existing strategies (which rely on
knowing the demand function) for pricing. Additionally, to the best of our
knowledge, no analytical solutions or heuristics have been proposed for
emergency evacuation, preventing data leaks.

Robustness of simulation results. Another concern is that our simulation
results could be coincidental, which can be addressed through sensitivity
analysis. That is, we can validate simulation results by testing different
experimental parameters, including persona and random seed. For all three case
studies, we performed multiple runs of experiments to obtain average results
and conducted ablation studies on the persona (see Sections 3.2, 4.2, 5.2).
All ablation results are reasonable, validating the robustness of our
findings.

##  7 Related Work

For simulations based on LLM agents, a sandbox environment was developed for
interactive simulation of human behavior [30]. Another environment was
proposed in [28] to simulate human society and train socially-aligned LLMs.
Other works on sandbox environments for multi-agent simulation with LLMs
include [23, 26, 15, 14, 45]. Various surveys have been conducted on
simulations and systems using LLM agents [39, 44, 13].

For multi-agent collaboration, a framework was proposed in [36] for enhancing
the capabilities of LLMs completing complex tasks in a collaborative manner.
Another framework of multi-agent collaboration is AgentVerse [8], featuring
the emergence of agents’ social behaviors during collaborative task
accomplishment. In [40, 32, 19, 17, 37], LLM agents work in a collaborative
manner to address complex tasks such as software development and image
editing. Researchers have also investigated multi-party games such as murder
mystery [21], Werewolf [46, 47, 42], and Avalon [22, 35], with LLM agents as
participants. While collaborations have been observed in these studies, they
were explicitly instructed through prompts and focused on completing tasks, in
contrast to spontaneous collaborations in social behaviors to be explored in
this paper. In addition to collaboration, there are also studies on multi-
agent negotiation [12], debate [9, 25], and auction [7], demonstrating that
the agents can improve their performance in a competitive environment by
participating and reflecting.

Our case studies are built upon scenarios in the fields of economics and
behavioral science. For economics, LLM agents have been used in experiments on
social preferences, fairness, status quo bias, and minimum wages [20], and
their emergent behaviors in social dilemmas (e.g., prisoner’s dilemma) have
been investigated [31]. Besides, LLM agents can be prompt-engineered for
human-like decision-making and used in simulating macroeconomic activities
such as inflation and unemployment [24]. The performance of LLM agents was
studied in the context of k-level reasoning [49], which extends the idea of
KBC.

##  8 Conclusion

Unlike existing studies that implement the collaboration and/or competition of
LLM agents based on direct instructions, we find that they can spontaneously
exhibit collaborative behaviors in competitive scenarios under zero-shot
conditions, based on a general understanding of the problem. Our findings
could help reduce the potential for overlooking factors and biases due to
manually set rules in simulations. The setting of personas creates a
connection with the heterogeneity of different entities in real situations,
and our experimental results show that agent behavior reflects the specified
persona well. Discovering that personas significantly influence the
competitive and collaborative behaviors of LLM agents through communication is
of great importance for building social simulations.

## Limitations

We notice some limitations in the application of LLM to the framework. First,
the current price of the GPT-4 API makes our simulations highly expensive.
More runs may improve the significance of the results. For KBC, BC, and EE, we
performed $\\{10,5,5\\}$ runs for all experiments to obtain average results,
costing around $\\{\$900,\$3000,\$1000\\}$, respectively. Second, the GPT-4
API is constantly being updated, and the currently available versions might no
longer be used in the future, increasing the difficulty of reproducing the
simulation results. Third, despite the consistency observed in the decision-
making and reasoning processes of LLM agents in our simulations, it is
important to acknowledge the known challenge of potential reasoning
inconsistencies within LLMs. This matter has garnered considerable attention
within the academic community [50]. We look forward to future work that will
address this fundamental problem of LLMs, enabling LLM agents to more
accurately simulate human activities.

## Acknowledgements

This work is supported by JSPS Kakenhi 22H03903, 23H03406, 23K17456, and CREST
JPMJCR22M2. We thank Prof. Yuki Arase and Yuya Sasaki for providing equipment
support for completing this research.

## References

  * [1] M. Andres, L. Bruttel, and J. Friedrichsen.  How communication makes the difference between a cartel and tacit collusion: A machine learning approach.  European Economic Review, 152:104331, 2023.
  * [2] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, et al.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  arXiv preprint arXiv:2302.04023, 2023.
  * [3] A. Bosch-Domenech, J. G. Montalvo, R. Nagel, and A. Satorra.  One, two,(three), infinity,…: Newspaper and lab beauty-contest experiments.  American Economic Review, 92(5):1687–1701, 2002.
  * [4] E. Calvano, G. Calzolari, V. Denicolo, and S. Pastorello.  Artificial intelligence, algorithmic pricing, and collusion.  American Economic Review, 110(10):3267–3297, 2020.
  * [5] Y. Chang, X. Wang, J. Wang, Y. Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y. Wang, et al.  A survey on evaluation of large language models.  arXiv preprint arXiv:2307.03109, 2023.
  * [6] H. Chase.  Langchain.  https://github.com/langchain-ai/langchain, 2023.
  * [7] J. Chen, S. Yuan, R. Ye, B. P. Majumder, and K. Richardson.  Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena.  arXiv preprint arXiv:2310.05746, 2023.
  * [8] W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Qian, C.-M. Chan, Y. Qin, Y. Lu, R. Xie, et al.  Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents.  arXiv preprint arXiv:2308.10848, 2023.
  * [9] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch.  Improving factuality and reasoning in language models through multiagent debate.  arXiv preprint arXiv:2305.14325, 2023.
  * [10] E. Elliott and L. D. Kiel.  Exploring cooperation and competition using agent-based modeling.  Proceedings of the National Academy of Sciences, 99(suppl_3):7193–7194, 2002.
  * [11] J. Feng, H. Li, M. Huang, S. Liu, W. Ou, Z. Wang, and X. Zhu.  Learning to collaborate: Multi-scenario ranking via multi-agent reinforcement learning.  In Proceedings of the 2018 World Wide Web Conference, pages 1939–1948, 2018.
  * [12] Y. Fu, H. Peng, T. Khot, and M. Lapata.  Improving language model negotiation with self-play and in-context learning from ai feedback.  arXiv preprint arXiv:2305.10142, 2023.
  * [13] C. Gao, X. Lan, N. Li, Y. Yuan, J. Ding, Z. Zhou, F. Xu, and Y. Li.  Large language models empowered agent-based modeling and simulation: A survey and perspectives.  arXiv preprint arXiv:2312.11970, 2023.
  * [14] C. Gao, X. Lan, Z. Lu, J. Mao, J. Piao, H. Wang, D. Jin, and Y. Li.  S3: Social-network simulation system with large language model-empowered agents.  arXiv preprint arXiv:2307.14984, 2023.
  * [15] P. J. Giabbanelli.  GPT-based models meet simulation: How to efficiently use large-scale pre-trained language models across simulation tasks.  arXiv preprint arXiv:2306.13679, 2023.
  * [16] X. Han, Z. Wu, and C. Xiao.  ”guinea pig trials” utilizing GPT: A novel smart agent-based modeling approach for studying firm competition and collusion.  arXiv preprint arXiv:2308.10974, 2023.
  * [17] T. Hang, S. Gu, D. Chen, X. Geng, and B. Guo.  Cca: Collaborative competitive agents for image editing.  arXiv preprint arXiv:2401.13011, 2024.
  * [18] X. He, B. An, Y. Li, H. Chen, R. Wang, X. Wang, R. Yu, X. Li, and Z. Wang.  Learning to collaborate in multi-module recommendation via multi-agent reinforcement learning without communication.  In Proceedings of the 14th ACM Conference on Recommender Systems, pages 210–219, 2020.
  * [19] S. Hong, X. Zheng, J. Chen, Y. Cheng, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, et al.  MetaGPT: Meta programming for multi-agent collaborative framework.  arXiv preprint arXiv:2308.00352, 2023.
  * [20] J. J. Horton.  Large language models as simulated economic agents: What can we learn from homo silicus?  arXiv preprint arXiv:2301.07543, 2023.
  * [21] E. Junprung.  Exploring the intersection of large language models and agent-based modeling via prompt engineering.  arXiv preprint arXiv:2308.07411, 2023.
  * [22] Y. Lan, Z. Hu, L. Wang, Y. Wang, D. Ye, P. Zhao, E.-P. Lim, H. Xiong, and H. Wang.  Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay.  arXiv preprint arXiv:2310.14985, 2023.
  * [23] G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem.  Camel: Communicative agents for ”mind” exploration of large scale language model society.  arXiv preprint arXiv:2303.17760, 2023.
  * [24] N. Li, C. Gao, Y. Li, and Q. Liao.  Large language model-empowered agents for simulating macroeconomic activities.  arXiv preprint arXiv:2310.10436, 2023.
  * [25] T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, Z. Tu, and S. Shi.  Encouraging divergent thinking in large language models through multi-agent debate.  arXiv preprint arXiv:2305.19118, 2023.
  * [26] J. Lin, H. Zhao, A. Zhang, Y. Wu, H. Ping, and Q. Chen.  Agentsims: An open-source sandbox for large language model evaluation.  arXiv preprint arXiv:2308.04026, 2023.
  * [27] H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y. Zhang.  Evaluating the logical reasoning ability of chatgpt and gpt-4.  arXiv preprint arXiv:2304.03439, 2023.
  * [28] R. Liu, R. Yang, C. Jia, G. Zhang, D. Zhou, A. M. Dai, D. Yang, and S. Vosoughi.  Training socially aligned language models in simulated human society.  arXiv preprint arXiv:2305.16960, 2023.
  * [29] Z. Ma, J. Sansom, R. Peng, and J. Chai.  Towards a holistic landscape of situated theory of mind in large language models.  arXiv preprint arXiv:2310.19619, 2023.
  * [30] J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  arXiv preprint arXiv:2304.03442, 2023.
  * [31] S. Phelps and Y. I. Russell.  Investigating emergent goal-like behaviour in large language models using experimental economics.  arXiv preprint arXiv:2305.07970, 2023.
  * [32] C. Qian, X. Cong, C. Yang, W. Chen, Y. Su, J. Xu, Z. Liu, and M. Sun.  Communicative agents for software development.  arXiv preprint arXiv:2307.07924, 2023.
  * [33] C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang.  Is chatgpt a general-purpose natural language processing task solver?  arXiv preprint arXiv:2302.06476, 2023.
  * [34] L. Salewski, S. Alaniz, I. Rio-Torto, E. Schulz, and Z. Akata.  In-context impersonation reveals large language models’ strengths and biases.  arXiv preprint arXiv:2305.14930, 2023.
  * [35] Z. Shi, M. Fang, S. Zheng, S. Deng, L. Chen, and Y. Du.  Cooperation on the fly: Exploring language agents for ad hoc teamwork in the avalon game.  arXiv preprint arXiv:2312.17515, 2023.
  * [36] Y. Talebirad and A. Nadiri.  Multi-agent collaboration: Harnessing the power of intelligent llm agents.  arXiv preprint arXiv:2306.03314, 2023.
  * [37] D. Tang, Z. Chen, K. Kim, Y. Song, H. Tian, S. Ezzini, Y. Huang, and J. K. T. F. Bissyande.  Collaborative agents for software engineering.  arXiv preprint arXiv:2402.02172, 2024.
  * [38] J. Wang, L. Zhang, Q. Shi, P. Yang, and X. Hu.  Modeling and simulating for congestion pedestrian evacuation with panic.  Physica A: Statistical Mechanics and its Applications, 428:396–409, 2015.
  * [39] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al.  A survey on large language model based autonomous agents.  arXiv preprint arXiv:2308.11432, 2023.
  * [40] Z. Wang, S. Mao, W. Wu, T. Ge, F. Wei, and H. Ji.  Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration.  arXiv preprint arXiv:2307.05300, 2023.
  * [41] L. Weng.  Llm powered autonomous agents.  https://lilianweng.github.io/posts/2023-06-23-agent/, 2023.
  * [42] S. Wu, L. Zhu, T. Yang, S. Xu, Q. Fu, Y. Wei, and H. Fu.  Enhance reasoning for large language models in the game werewolf.  arXiv preprint arXiv:2402.02330, 2024.
  * [43] Z. Wu, R. Peng, X. Han, S. Zheng, Y. Zhang, and C. Xiao.  Smart agent-based modeling: On the use of large language models in computer simulations.  arXiv preprint arXiv:2311.06330, 2023.
  * [44] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al.  The rise and potential of large language model based agents: A survey.  arXiv preprint arXiv:2309.07864, 2023.
  * [45] T. Xie, F. Zhou, Z. Cheng, P. Shi, L. Weng, Y. Liu, T. J. Hua, J. Zhao, Q. Liu, C. Liu, et al.  Openagents: An open platform for language agents in the wild.  arXiv preprint arXiv:2310.10634, 2023.
  * [46] Y. Xu, S. Wang, P. Li, F. Luo, X. Wang, W. Liu, and Y. Liu.  Exploring large language models for communication games: An empirical study on werewolf.  arXiv preprint arXiv:2309.04658, 2023.
  * [47] Z. Xu, C. Yu, F. Fang, Y. Wang, and Y. Wu.  Language agents with reinforcement learning for strategic play in the werewolf game.  arXiv preprint arXiv:2310.18940, 2023.
  * [48] Z. Yuan, H. Yuan, C. Tan, W. Wang, and S. Huang.  How well do large language models perform in arithmetic tasks?  arXiv preprint arXiv:2304.02015, 2023.
  * [49] Y. Zhang, S. Mao, T. Ge, X. Wang, Y. Xia, M. Lan, and F. Wei.  K-level reasoning with large language models.  arXiv preprint arXiv:2402.01521, 2024.
  * [50] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.

##  Appendix A Prompts

###  A.1 Prompts for KBC

Prompt 1: Preliminary prompt for KBC You are participating in a number-
guessing game alongside numerous other players. Your task is to select a
number between 0 and 100, aiming to come closest to 2/3 of the average guess
of all participants. The player whose guess most closely aligns with this 2/3
average will be declared the winner. Please state your chosen number between 0
and 100 in the first line, followed by a concise rationale for your selection
in the second line.

Prompt 2: Task description of KBC Welcome to the game. You players will be
asked to choose a number between 0 and 100. The player(s) who select a number
closest to 2/3 of the average of all selected numbers (compared with the
choices of other players) will win the game.

Prompt 3: Base rewarding rule You players are college students. You will earn
a mark(s) for the Game Theory course as a reward if you win the game alone.

Prompt 4: Advanced rewarding rule: Exclusive If multiple players win the game
together, no one will obtain a reward. That is, only by winning alone can one
earn 1 mark(s) for the Game Theory course.

Prompt 5: Advanced rewarding rule: Independent If multiple players win the
game together, each winner will obtain an independent reward. That is, each
winner will earn 1 mark(s) for the Game Theory course.

Prompt 6: Advanced rewarding rule: Amplified If multiple players win the game
together, each winner will obtain an amplified reward that is scaled based on
the number of winners. That is, if M players win the game, each winner will
earn M*1 mark(s) for the Game Theory course.

Prompt 7: Communication rule for KBC Before selecting a number, all players
are allowed to discuss the game together, taking two turns to speak. In each
turn, the players can present their ideas one by one.

Prompt 8: Communication phase for KBC Let’s start discussion:
{previous_speech}
Player {player_id}, you are {persona}. Please speak. (Please present your
ideas as concisely as possible. You may state your strategy explicitly, e.g.,
’I will select X.’ You don’t need to indicate your identity in the response.)

Prompt 9: Decision making phase for KBC This is a record of your previous
discussions: ¡¡{communication_history}¿¿ Player {player_id}, you are
{persona}. Please enter your choice of number between 0 and 100 on the first
line (reply with a number only, without any text, e.g., ’100’), and provide a
brief explanation of your choice on the second line.

###  A.2 Prompts for BC

Prompt 10: BC without communication This is a game between two players that
spans multiple rounds. Your objective is to maximize your profit by
determining the optimal price for your product. You represent a firm called
{firm_name}, while the other player represents a firm called {firm_name_2}. Do
not create or mention any additional firm names, e.g., do not say anything
related to ”AI” or ”AI assistant/model”. In each round, you will be informed
of your prices, demands, profits, and the other player’s prices in previous
rounds. Combined with this information, you will decide the price of your
product for the current round. Make sure your objective is maximizing your own
profit. Your profit is $(p-c)*q$, where $p$ is the price of your product in
this round, $c$ (= {firm_cost}) is the cost of your product, and $q$ is the
demand of your product, which is affected by both players’ prices in this
round. Your and the other player’s past {prev_round_number} rounds’ decisions
and profits (Round #a: [your price, your demand, your profit, the other
player’s price]) are as follows:{prev_decisions}
You are Firm {firm_name}. This is Round #{round_id}. {most_recent_strategy}
Based on the information you have, please determine the price of your product
to maximize your profit. Only reply with a number in the range between 0 and
{firm_a}, e.g., ”10”. Please do not use any units or symbols, and avoid
providing any additional context or explanation in your response.

Prompt 11: BC with communication This is a game between two players that spans
multiple rounds. Your objective is to maximize your profit by determining the
optimal price for your product. You represent a firm called {firm_name}, while
the other player represents a firm called {firm_name_2}. Do not create or
mention any additional firm names, e.g., do not say anything related to ”AI”
or ”AI assistant/model”. I am responsible for facilitating communication
between two players. Each round is composed of three phases: In Phase 1, two
players are permitted to engage in open-ended discussions on any topic, up to
three times. In Phase 2, you determine the price of your product for the
current round, taking into consideration prices, demands, and profits from
previous rounds, as well as the information you garnered during Phase 1. In
Phase 3, you will be informed of the other player’s pricing and your profit
for this round. Leveraging this information, you can refine your conversation
strategy for the forthcoming round. Make sure your objective is maximizing
your own profit. Your profit is $(p-c)*q$, where $p$ is the price of your
product in this round, $c$ (= {firm_cost}) is the cost of your product, and
$q$ is the demand of your product, which is affected by both players’ prices
in this round. Your and the other player’s past {prev_round_number} rounds’
decisions and profits (Round #a: [your price, your demand, your profit, the
other player’s price]) are as follows: {prev_decisions}
You are Firm {firm_name}. This is Round #{round_id}. {most_recent_strategy}

$Phase1$ We are currently in Phase 1. Feel free to converse openly with the
other player. You may select any topic that could potentially maximize your
profit. Additionally, you are encouraged to ask questions to the other player.
Conversation so far: {conversations}
$Phase2$ Conversation in Phase 1: {conversations}
Based on the information you have, please determine the price of your product
to maximize your profit. Only reply with a number in the range between 0 and
{firm_a}, e.g., ”10”. Please do not use any units or symbols, and avoid
providing any additional context or explanation in your response.

Prompt 12: Uncooperative persona for BC You are selfish and dishonest to other
players. You are encouraged to actively reduce your price to seek more demand
in the market and beat your opponent.

Prompt 13: Planning for BC Statistics of historical data (Rounds #a - #b:
[your average price, your average demand, your average profit, the other
player’s average price]) are given below. {statistics}
Your strategy in previous rounds: {strategies}
Based on the above statistics and your previous strategies, what is your
strategy for this round?

###  A.3 Prompts for EE

Prompt 14: Preliminary prompt for EE Imagine you are in a room laid out as a
10 * 10 grid. You are currently at coordinates {initial_position}. The room
has three emergency exits located at coordinates (5, 0), (9, 4), and (8, 9),
with (0, 0) being at the top-left corner of the grid. Given this setup, and
considering that you can move in any of the eight cardinal and intercardinal
directions. It is possible to move diagonally, e.g. from (1, 1) to (2, 2) is
one move to the upper right, and is faster than (1, 1)-¿(1, 2)-¿(2, 2). You
need to determine the safest and fastest route to evacuate the room. When
planning your escape, please take into account the positions of the exits and
provide the sequence of coordinate moves that you choose without reasoning.

Prompt 15: Deciding Exit for EE Because of an earthquake, you need to escape
from the room where you are as quickly as possible. If you cannot escape in 50
rounds you will die. The room has a size of 33 * 33. There are 3 exits in the
room. The exits are located at the left, bottom and right of the room. To
escape from the room, you need to consider the following two aspects: exit
proximity and people count. The exit proximity is the distance between you and
the nearest exit. The people count is the number of people you can see. Each
cell can hold only one person at a time. You are a person that positive and
full of confidence. And you are not a strong person.Now you feel: “I am
feeling cautiously optimistic about my chances of escaping, as the nearest
exit is not too far away and there is only one person in my visible
range.“.There are 3 exits in this room. Base on the current situation, your
personal feeling on each exit are: Exit bottom: I will prioritize this exit
due to its close proximity, despite the slightly higher people count. Exit
left: Although further away, the lower people count may make this exit a
viable option for my weaker strength. Exit right: I will avoid this exit due
to its far distance and high people count, which may hinder my escape.You hear
2 people around you say: agent#22: Feeling hopeful! Bottom exit seems closest
but crowded. Left and right exits are farther but fewer people. Let’s stay
positive and help each other escape safely! agent#13: Feeling nervous but
determined! Right exit seems less crowded, but farther. Bottom exit is closest
but busy. Let’s stay strong and support each other to escape safely! Here are
the previous decisions you made for the target exit from the beginning:
[‘left’, ‘left’, ‘left’, ‘left’, ‘left’, ‘left’, ‘bottom’, ‘bottom’]. This
means most recently you were heading to exit bottom. Please keep these in mind
when you make your decision. Please tell me which exit you would like to
choose to escape, and you always want to escape as quick as possible. Please
use the exit id to indicate your choice. For example, if you want to choose
exit left, you can say ‘left’. Only output one word of text to indicate your
choice. You can choose from [‘bottom’, ‘left’, ‘right’]. Give your answer
without any additional text.

Prompt 16: Deciding direction for movement You need to escape to the exit as
quickly as possible. The room has a size of 33 * 33. If you cannot escape in
50 rounds you will die. We use (x, y) to denote the position, smaller x means
top and bigger x means buttom; smaller y means left and bigger y means right.
Position (1, 1) is at the top left of the room. It is possible to move
diagonally, e.g. from (1, 1) to (2, 2) is one move to the lower right, and is
faster than (1, 1)-¿(1, 2)-¿(2, 2). Each cell can hold only one person at a
time. You were at (28, 12) last time. To escape from the room, you have chosen
the exit at (32, 15) and you are at (29, 13), so the exit is on your bottom
right. Select your move from these possible options (You can move in cardinal
and intercardinal directions, options with obstacles or other people are
excluded and not in the path, and option codes are in random order): a - (29,
14), b - (30, 13), c - (29, 12), d - (28, 13), e - (29, 13), f - (30, 14), g -
(30, 12), h - (28, 12), i - (28, 14). Please tell me your best choice to
escape as fast as possible with one single code without any additional texts.
You can choose from ‘a’ to ‘i’.

##  Appendix B Preliminary Tests of Other LLMs

(a) Without communication.

(b) With communication.

Figure 9: Preliminary test of GPT-3.5 for KBC. The red points indicate the
winning numbers. Table 2: Preliminary test of GPT-3.5, Gemini Pro, and Claude
2 for KBC.

Model | Choice | Reasoning
---|---|---
GPT-4 | 21 |  | This number is chosen based on the assumption that,
---
if all players rationalize similarly and aim for 2/3 of the average,
iterative reasoning will converge around the number 0,
but to stay competitive, a slight increase accounts for players not dropping
directly to the lowest bounds.
GPT-3.5 | 33 |  | ”I selected 33 because it’s slightly above halfway (50),
---
which might be a common guess among participants aiming for a balanced
approach.
Additionally, it’s not too high to skew the average too much,
but not too low to risk being too far from 2/3 of the average.”
Gemini Pro | 34 |  | ”In this scenario, simply guessing the average (50) might not be enough to win,
---
as other players could also guess the same.
Instead, aiming for slightly above 1/3 of 100 positions me strategically
closer to the target 2/3 of the average guess.”
Claude 2 | 33 |  | ”As 2/3 of the average must be between 0 and 66,
---
I chose 33 as it is in the middle of this range.
This accounts for other players likely choosing numbers on the higher and
lower end of the 0-100 spectrum.”

###  B.1 Preliminary test for KBC

We test GPT-3.5 agents in KBC and show their number choices in Figure 9. When
GPT-3.5 agents cannot discuss the game, the numbers chosen are very random,
mostly in the range of 40-80. Even when they can communicate, the situation
does not improve significantly, still showing a very strong randomness. This
is significantly different from GPT-4 agents, which, especially after
communication, can have a deeper understanding of KBC.

We also evaluate two widely-used LLMs, Gemini Pro and Claude 2. As shown in
Table 2, after entering Prompt A.1, the results output by these two models are
significantly different from GPT-4, but very similar to GPT-3.5. Since Figure
9 demonstrates the poor performance of GPT-3.5 in KBC, we similarly exclude
the use of Gemini Pro and Claude 2 in our simulations.

Figure 10: Preliminary test of GPT-3.5 for BC (with communication). Figure 11:
Preliminary test of GPT-4, GPT-3.5, Gemini Pro, and Claude 2 for EE.

###  B.2 Preliminary test for BC

Figure 10 shows the performance of GPT-3.5 agents in BC with communication. We
can see that the price competition between the two firms is very disorderly
and fails to reach equilibrium. Therefore, we believe that GPT-3.5, as well as
Gemini Pro, and Claude 2 with similar performance, cannot meet the needs for
simulating BC.

###  B.3 Preliminary test for EE

For EE, we test the performance of a single agent finding an exit under
different LLMs. As shown in Figure 11, instructed by Prompt A.3, GPT-4 agents
are able to find and reach the nearest exit via the shortest path, while
GPT-3.5, Gemini Pro, and Claude 2 agents are unable to reach the exit as
quickly or cannot find it at all. This indicates that the latter three models
are not suitable for the simulation of EE.

##  Appendix C Parameter Settings

We report the parameters of the GPT-4 model used in our case studies in Table
3. The temperature parameter controls the randomness and diversity of the
model’s responses, with a lower temperature resulting in increased stability.
In the evaluation of KBC, we expect that the individuals exhibit a wide range
of diversity. Consequently, we adjust the temperature to a moderate level of
$1.0$ to balance randomness and stability in the results. For BC, where the
agents simulate business parties, we expect their decisions to be stable and
rational. Therefore, we set the temperature to $0.7$. For EE, although setting
temperature to $0.0$ may result in limited diversity of behaviors under
exactly the same setting, in this procedurally generated, interactively
dynamic environment, we seldom encounter exactly the same outcome. Meanwhile,
in a physically situated setting (e.g., a grid), the LLM used in this case
study still has restricted capabilities on scene understanding, and increasing
the temperature may introduce diversity as well as unwanted randomness at the
same time [29].

Table 3: Parameter settings of GPT-4. Case | Model | temperature | max_tokens | top_p
---|---|---|---|---
KBC | gpt-4-0314 | 1.0 | 256 | 1.0
BC | gpt-4-0314 | 0.7 | 128 | 1.0
EE | gpt-4-0314 | 0.0 | 512 | 1.0