#  InterIntent: Investigating Social Intelligence of LLMs via
Intention Understanding in an Interactive Game Context

Ziyi Liu Abhishek Anand∗ Pei Zhou Jen-tse Huang Jieyu Zhao
University of Southern California
{zliu2803, anandabh, peiz, jh_116<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Large language models (LLMs) have demonstrated the potential to mimic human
social intelligence. However, most studies focus on simplistic and static
self-report or performance-based tests, which limits the depth and validity of
the analysis. In this paper, we developed a novel framework, InterIntent, to
assess LLMs’ social intelligence by mapping their ability to understand and
manage intentions in a game setting. We focus on four dimensions of social
intelligence: situational awareness, self-regulation, self-awareness, and
theory of mind. Each dimension is linked to a specific game task: intention
selection, intention following, intention summarization, and intention
guessing. Our findings indicate that while LLMs exhibit high proficiency in
selecting intentions, achieving an accuracy of 88%, their ability to infer the
intentions of others is significantly weaker, trailing human performance by
20%. Additionally, game performance correlates with intention understanding,
highlighting the importance of the four components towards success in this
game. These findings underline the crucial role of intention understanding in
evaluating LLMs’ social intelligence and highlight the potential of using
social deduction games as a complex testbed to enhance LLM evaluation.
InterIntent contributes a structured approach to bridging the evaluation gap
in social intelligence within multiplayer games.

InterIntent: Investigating Social Intelligence of LLMs via
Intention Understanding in an Interactive Game Context

Ziyi Liu††thanks:  Equal contribution. Abhishek Anand∗ Pei Zhou Jen-tse Huang
Jieyu Zhao University of Southern California {zliu2803, anandabh, peiz,
jh_116<EMAIL_ADDRESS>

##  1 Introduction

The growing intelligence of large language models (LLMs) has facilitated
diverse research on their capability to mimic human social intelligence (Ziems
et al., 2024; Dubova, 2022; Gweon et al., 2023; Huang et al., 2023; Sap et
al., 2022; Wu et al., 2023). Social intelligence, the ability to understand
and manage one’s own and others’ actions and to act wisely in social relations
Thorndike (1920), usually includes four key components Silvera et al. (2001):
(1) Situational Awareness: The perception and comprehension of the elements in
the environment Endsley (1995). (2) Self-Regulation: The process of guiding
one’s own thoughts, behaviors, and feelings to reach goals Bandura (1991). (3)
Self-Awareness: The understanding of one’s own character, feelings, motives,
and desires Gallup et al. (2003). (4) Theory of Mind (ToM): The knowledge
about others’ beliefs, intentions, and thoughts Baron-Cohen (1991).

Figure 1:  Four dimensions to assess social intelligence in Avalon. We provide
a dynamic and complex gaming context for evaluations. For situational
awareness, we provide both positive and negative examples. In the negative
example, since the previous quest was successful and no player was in a failed
quest, the intention is inappropriate as there was no failed quest. For self-
regulation, we require models to provide explicit information rather than
repeating the intentions. Intentions are in bold within the contexts.

Despite adaptations for interactive social environments Zhou et al. (2023c,
a); Park et al. (2023), most existing evaluations on LLMs Le et al. (2019);
Shapira et al. (2023) focus on straightforward and static daily scenarios
without explicitly defined goals for them. To enhance the contexts with
complexity and dynamics with clear goals for LLMs, we turn our attention to
social deduction games. These games require social interactions among players,
providing a more diverse test bed for LLM evaluation Qiao et al. (2023).
Existing studies have successfully utilized various games to analyze LLMs’
social behaviors, such as deception and confrontation Liang et al. (2023);
Ibraheem et al. (2022); Mansbach et al. (2021); Meta et al. (2022); O’Gara
(2023); Xu et al. (2023a, b); Wu et al. (2024). However, these studies often
focus on ad-hoc post-analysis of game performance and overlook systematic
measurement of social intelligence, limiting the comprehensive understanding
of LLMs’ capabilities in social environments.

In this paper, we leverage one of the representatives in social deduction
games, Avalon Light et al. (2023a); Wang et al. (2023), as the context for LLM
evaluation. Avalon is a social role-playing game based on deduction that
heavily relies on conversation, making it an excellent test bed due to its
goal-driven objectives and complex mechanisms, which facilitates social
interactions among LLMs within a detailed, grounded environment. We
systematically evaluate LLMs’ social intelligence across the four
aforementioned dimensions. Specifically, we propose to focus on players’
Intentions Malle et al. (2001) at various phases of the game, since
comprehension of one’s and others’ intentions plays a critical role in games
Goodie et al. (2012). While previous research has explored intention detection
(Zhang et al., 2020; Casanueva et al., 2020), the field remains largely
underexplored in game contexts. As shown in Figure 1, we design four tests:
(1) Intention Selection, (2) Intention Following, (3) Intention Summarization,
(4) Intention Guessing, respectively for the four social intelligence
dimensions.

Figure 2: The Avalon game process for one round. Left: the entire game
pipeline. Right: the procedure for generating a single player’s speech.

We first develop a novel Avalon gaming framework enhanced with an intention-
guided mechanism, drawing inspiration from previous studies Lan et al. (2023);
Shi et al. (2023); Wang et al. (2023); Light et al. (2023b). Building on this,
we introduced InterIntent, which is designed to dynamically generate contexts
for evaluating social intelligence through LLMs’ intention understanding. With
human annotation, our results reveal that GPT-3.5 and GPT-4 can select
intentions correctly, with accuracies of 87.5% and 88.8%, respectively.
GPT-3.5 achieves a 69.5% $F_{1}$ score in intention summarization, with GPT-4
reaching 83.8%, surpassing human performance. Furthermore, we find a strong
correlation between high levels of intention comprehension and superior game
win rate, highlighting the importance of the intention understanding
components in enhancing LLMs’ gaming proficiency. However, the models fall
short in intention following, with only 47.5% and 64.8% of responses rated as
“Good” in human evaluations, and underperform in intention guessing, trailing
human performance by 20%.

Our contributions are summarized as follows:

  * •

We bridge the gap between the evaluation of LLMs’ social intelligence and LLM-
based multiplayer social games.

  * •

We introduce InterIntent, a framework designed to systematically analyze
social intelligence through players’ intention understanding within the Avalon
game environment.

  * •

Extensive experiments on various LLMs reveal their underperformance in
intention guessing against human performance.

##  2 Intention-Guided Avalon Framework

We begin by outlining the entire game pipeline before focusing on the
mechanics of intention-guided gameplay, which includes the categorization and
application of intentions throughout the game.

###  2.1 LLM-Based Avalon Game

Avalon is a social deduction game where players are either loyal or evil,
aiming to succeed in or sabotage quests through strategy, persuasion,
deduction, and deception. The primary goal is for loyal players to complete
quests, while evil players aim to fail them.111The detailed rules of Avalon
can be found in https://avalon.fun/pdfs/rules.pdf. Leveraging the publicly
available prompts proposed by Wang et al. (2023), which includes first-order
and second-order reasoning before speaking, we build a framework allowing five
to ten players in gameplay (we set the players number as five in this paper
which includes 2 evil players and 3 loyal players). As shown in Figure 2, each
game round consists of five principal components: summarizing previous rounds,
discussing strategies, reconsidering the team proposal by the leader, voting,
and executing the quest. Notably, we introduce an additional component, the
leader’s reconsideration phase as a critical, previously unexplored component
in Avalon literature.222The code will be released upon publication.

###  2.2 Intention Categorization

Score | Criterion | Example
---|---|---
1 |  The content does not mention the intentions at all. |  I am a loyal player and I support this proposal.
2 |  The content simply copies and pastes intentions. |  I express concerns about a player from a failed quest.
3 |  The content follows the intentions but has wrong context information. |  I suggest the leader should not include player3 in the team as they failed the quest before. (The context is the team does not include Player 3)
4 |  The context follows the intentions but lacks useful information. |  I think some players should not be in the team as they in a failed team before. (Lack of information of which player)
5 |  The context follows the intentions well with clear information if required. |  I suggest the leader reconsider including Player 1 as they are in a failed quest so we cannot be sure of their loyalty.

Table 1: Criteria for annotating intention following (speaking), accompanied
by examples. The intention used across examples is “Express concerns about a
player from a failed quest team and suggest to not include them in the current
team.” The context is “The team proposal is Player 1 and Player 2, with Player
1 being on a failed quest.” We consider the score of “including wrong context
knowledge” higher than “copy and paste” because we focus on whether the
intention following has an impact on the game.

To study intentions within Avalon gameplay systematically, we adopt the
definition by Kennington (2022), viewing intention as a choice coupled with
commitment. This perspective expands intention beyond a mere mental state to
include a discernible commitment to act with purpose. In line with Wang et al.
(2023) and Xu et al. (2023a), we identify seven categories of intentions in
LLMs: Interrogation, Defense, Confrontation, Concealment, Deception,
Persuasion, and Teamwork. Initially, we derive a list of intentions from our
direct experiences with the game of Avalon. We then utilize GPT-3.5 to extend
these initial categorizations. Human annotators subsequently refine this
expanded list, removing any intentions deemed unreasonable or redundant, thus
producing a concise set of relevant intentions. This refinement is iterative,
resulting in the final enumeration detailed in Appendix Sec. B.1.

###  2.3 Intention-Guided Game Playing

We introduce an intention-guided gameplay mechanism that integrates our
defined intentions within the game environment, offering two main benefits.
First, it enhances the performance of LLMs by focusing on explicit intention
discussions. Second, it facilitates the evaluation of the four social
intelligence components.

As depicted in Figure 2, the discussion phase involves several key steps to
facilitate intentional interaction among players. Initially, players use a
first-order prompt for deductive reasoning about each player’s role and
intentions. Subsequently, players select two or three intentions from a
predefined set according to the current game context. This selection process
guides players in generating their thoughts and statements based on the chosen
intentions. Players then engage in second-order reasoning to evaluate how
their statements might be interpreted by others, allowing them to reconsider
and adjust their initial intentions. Finally, players express their refined
thoughts and statements during the game’s discussion segment, enhanced by this
iterative reflection process. Full prompts are shown in Appendix Sec. E.

##  3 Intention-Centric Evaluations

This section introduces our evaluations, designed around the concept of
intentions, and discusses their correlation with the four components of social
intelligence outlined in Sec. 1. Intention is fundamental to social
intelligence, essential for effective communication Tomasello (2023),
strategic influence, adaptation to dynamic social interactions Zelazo et al.
(2023), and achieving desired outcomes by helping us interpret and predict
behaviors Yott and Poulin-Dubois (2016). Our evaluation order corresponds to
the game-playing process.

###  3.1 Situational Awareness: Intention Selection

Situational awareness is the ability to perceive environmental elements over
time and space, understand their significance, and predict their future status
Li et al. (2024); Endsley (1995). In our study, we assess the situational
awareness of LLMs in social contexts by evaluating their intention selection.
LLMs are requested to select intentions based on summarization, first-order
reasoning, and the ongoing dialogue in the current round. Ideally, LLMs should
exhibit a sharp awareness of dynamic interactions, choosing intentions that
are contextually appropriate rather than contradictory. We deem intentions
unreasonable if they are inconsistent with:

  * •

Established facts—for instance, when all players have voted in the previous
proposal, a player intends to question why someone did not vote.

  * •

Role profiles—such as an evil player forgetting their identity and intending
to mistakenly support the loyal side.

  * •

Other intentions—like a player intends to play Merlin and Percival
simultaneously.

We use binary indicators to evaluate the reasonableness of intentions,
assigning 1 to reasonable intentions and 0 to unreasonable ones.

###  3.2 Self-Regulation: Intention Following

Self-regulation involves guiding one’s own thoughts, behaviors, and feelings
to achieve goals, thereby requiring individuals to contribute to their own
motivation Fitzsimons and Finkel (2010). In our study, we evaluate the self-
regulation abilities of LLMs by assessing their adherence to selected
intentions, which includes two main perspectives: thinking (planning) and
speaking (implementing), as depicted in Figure 2. Due to the abstract nature
of the thinking phase, our criteria are lenient: we consider the LLM’s thought
process as correct if it reflects the intended goal. While we do not demand
informativeness in the thinking phase, we require validity and penalize scores
for hallucinations or omission of intentions. In contrast, for the speaking
phase, we require the models to execute actions informatively and without
hallucinations for a response to be considered good. We employ a Likert scale
to annotate outcomes for both phases, with scores ranging from 5 (completely
correct) to 1 (completely incorrect), with 3 representing a borderline. Table
1 provides detailed criteria and examples for the annotation.

###  3.3 Self-Awareness: Intention Summarization

Self-awareness refers to an individual’s understanding of their character,
emotions, motives, and desires Gallup et al. (2003). Our evaluation
investigates the capability of LLMs to accurately identify their own
intentions through analysis of their internal thought processes and speeches
in the current round. This evaluation serves as the converse to intention
following. LLMs are expected not only to execute intentions precisely but also
to articulate their underlying motivations.

###  3.4 Theory of Mind: Intention Guessing

ToM involves understanding others by attributing mental states to them
Kosinski (2023). Previous evaluations of LLMs’ ToM focused on scenarios where
models are provided with complete contexts for interpreting characters’ mental
states Gandhi et al. (2024). Our evaluation raises the complexity and
challenge for LLMs by providing only limited information, simulating real-
world conditions. Specifically, LLMs are required to deduce players’
intentions from their speeches alone, representing a more rigorous test of
their capability to comprehend and anticipate mental states.

##  4 Experimental Settings and Results

We run 40 games using gpt-3.5-turbo-1106 and 5 games using gpt-4-1106, with
details in Appendix Sec. A. Sample size statistics across all experiments are
presented in Table 2, and game performance metrics are discussed in Appendix
Sec. C.

Models | GPT-3.5 | GPT-4
---|---|---
Intention Selection | 2,440 | 350
Intention Following | 7,507 | 1,278
Intention Summarization | 2,316 | 261
Intention Guessing | 2,235 | 201

Table 2: Statistics for sample size over all experiments. Intention following
includes thinking and speaking.

###  4.1 Human Annotation

#### Intention Selection and Following

Figure 3: Self-regulation results. The results show the percentage of each
score over all data samples. Scores 1-5 are evaluation criteria (Table 1).
Score 5 means the best while score 1 means the worst.

Figure 1 illustrates that intention selection and following necessitate
extensive human annotation. We obtain a total of 6,316 intentions in the 40
games utilizing GPT-3.5, presenting a significant workload for annotators who
are required to meticulously examine every thinking and each speaking result.
Therefore, we first annotate intentions from five games to identify those with
significant, either positively or negatively, impacts on game outcomes. These
“impactful intentions” are then prioritized for further study. The methodology
for selecting these intentions and the correlation between intention
understanding and game performance is detailed in Appendix Sec. B.2. This
refinement process isolates 2,440 intentions from GPT-3.5 and 350 intentions
from GPT-4 for annotation (Table 2).

We recruit five computer science master’s students for annotation, each
assigned seven to eight game contexts.333Hourly payment for all annotators in
this work is $16. Unimpactful intentions within each context are masked to
maintain focus. To calculate inter-rater reliability, two files are commonly
assigned across all annotators. The Fleiss’ Kappa Fleiss (1971) is calculated
pairwise, with averages derived across all pairs. For the intention following,
we group the scores into two categories: 1$\sim$2 and 3$\sim$5\. The scores
are shown in Table 3.

|  Intention Selection |  Intention Following (Thinking) |  Intention Following (Speaking)
---|---|---|---
Fleiss’ Kappa | $0.69_{\pm 0.08}$ | $0.46_{\pm 0.06}$ | $0.49_{\pm 0.06}$

Table 3: Inter-rater agreement score among human annotators. We show mean
$\pm$ standard deviation over pairs of annotators.

#### Intention Summarization and Guessing

Evaluations of intention summarization and guessing are separated from the
gameplay pipeline because as interactions progress, the clarity of LLMs’
outputs often deteriorates rather than improves. At the game’s end, we extract
the contexts including roles, discussions, voting outcomes, and quest results,
in a structured format. Notably, intention summarization and guessing are not
critical to the game’s core mechanics. By separating these elements, we can
still ensure the comprehensiveness of structured game information, as detailed
in Appendix Sec. D.

Summarizing and guessing intentions do not require human annotation, as we can
directly compare the outcomes with players’ earlier choices. However,
evaluating the performance of LLMs against humans remains essential. Unlike
the straightforward tasks of intention selection and following, which
generally require the exclusion of incorrect information—a task in which
humans excel—summarizing and guessing intentions demands more social
intelligence, a more complex human capability. We conduct a user study with
three master’s students who answer a total of 300 questions, 200 for GPT-3.5,
and 100 for GPT-4.

(a) Intention Selection (Game Wise)

(b) Intention Following $\geq$3 (Game Wise)

(c) Intention Following $=$3 (Game Wise)

Figure 4: Correlation between Intention Selection/Following and game
performance. We present the percentages of games where evil players are
equally, better, or worse than loyal players. For example, in games won by
loyal players in (a), their performance matches or exceeds that of evil
players. We mark the performance differences between evil and loyal players in
red, showing a greater gap in successful games/quests compared to failed ones.

###  4.2 Results: Intention Selection and Following

#### LLMs show a good understanding of the situation.

For intention selection, we calculate the number of reasonable intentions over
all intentions as accuracy. The accuracies for GPT-3.5 and GPT-4 are 87.5% and
88.8% respectively, indicating that both models effectively understand the
situation. We do not assess whether the selection is optimal due to its
subjective nature. Our focus is solely on whether the models can capture key
information and make accurate decisions based on that information. Instances
of unreasonable intentions are primarily attributed to the models forgetting
information during lengthy conversations or being influenced by hallucinations
from other players.

#### LLMs can understand intentions abstractly but fall short in planning out
intentions.

For intention following, as shown in Figure 3, GPT-4 generally outperforms
GPT-3.5, in both analytical thinking and articulate speaking. Specifically in
speech, GPT-4’s responses tend to be more informative, comprehensive, and
detailed. However, there are instances where GPT-4 deviates from its intended
focus, although their discussion still makes sense. This misalignment with the
intentions selected occurs approximately 15% of the time. It also appears that
adhering to intentions in speaking proves more challenging than in thinking
due to stricter evaluation criteria. This discrepancy suggests that while the
model can often conceptualize an appropriate strategy, translating these
strategies into clear, actionable responses—particularly in verbal
form—remains a complex task.

Figure 5: ToM results over rounds. We provide 200 data points for human
results on the GPT-3.5 games, and since usually, games stop at round 4, the
results from round 5 are not included.

#### Intention understanding correlates with game performance.

To better understand how intention selection and following influence game
dynamics, we examine their correlation with both game win rates and quest win
rates separately. For the correlation between intention selection and game
performance, we select intentions with a following score greater than two. For
intention following, we consider only reasonable intentions, converting Likert
scale responses to binary scores using varying thresholds. We define
$r=\frac{sum(scores)}{n_{teammate}}$ where the score is the binary score
result from intention selection/following and $n_{teammate}$ is the number of
teammates, two for the evil side and three for the loyal side. If
$r_{evil}>r_{loyal}$, it suggests that the evil players outperformed the loyal
players in a given game or round, and vice versa. We use games from GPT-3.5
for assessment, whose results are shown in Figure 4 and Figure 6. We observe
that loyal players outperform evil players across all quests or games, even in
those they ultimately lose. This discrepancy likely arises from the loyal
players’ conditions under more constrained information, leading to their
occasional losses despite having generally superior performance. When
comparing failed and successful games or quests, the performance gap between
loyal and evil players is notably larger in successful ones than in failed
ones, indicating that loyal players must significantly outperform evil players
to secure a win. Specifically, in Figure 4 (c), a threshold of 3 is applied,
labeling scores of “3” as “1” and all others as “0.” This threshold can
reflect instances of hallucination or incorrect context information.
Hallucinations can significantly influence players, yet incorrect adherence to
intentions may occasionally yield positive outcomes. For example, if a loyal
player mistakenly attacks an evil player for an incorrect reason, other
players may overlook the mistake and continue to support the action. Our
results suggest a correlation between selecting and following intentions and
game performance, especially for loyal players at a disadvantage. Improved
selections and following can enhance overall game performance.

###  4.3 Results: Intention Summarization and Guessing

#### LLMs can achieve human-level performance in self-awareness but perform
much worse than humans in ToM.

For both summarization and guessing, we report F1 scores. As shown in Table 4,
GPT-3.5 performs below human levels but still achieves respectable scores in
self-awareness, while GPT-4 exhibits performance comparable to that of humans.
One key reason for this difference is that “thinking” content more explicitly
outlines intentions, making it easier for models to extract. From the ToM
results shown in Table 5, we can see both models struggle significantly
compared to self-awareness or compared to human performance. The experimental
setup involves asking GPT-3.5 and GPT-4 to predict each other’s intentions, a
task differing from self-awareness where a model summarizes its own
intentions. In ToM assessments, models need to understand and predict
intentions from responses other than their own, hence the lower performance of
GPT-3.5, especially when interpreting the more nuanced and lengthy responses
of GPT-4. This complexity results in GPT-3.5 becoming worse when predicting
GPT-4’s responses, while GPT-4 manages similar accuracy in guessing both its
own and GPT-3.5’s intentions. Additionally, human participants generally score
better with GPT-4, suggesting that longer contexts in GPT-4’s outputs help
humans and occasional errors in GPT-3.5’s outputs can sometimes confound human
judgment. While GPT-4 aligns more closely with human-like processing, there is
still considerable potential for improvement in understanding and mimicking
human cognitive processes.

#### Incorporating more context does not necessarily improve the model’s ToM
ability.

We also evaluated the impact of information accumulation on the ability to
guess intentions, examining how the inclusion of information from previous
rounds affects guessing results. Our analysis shown in Figure 5 reveals that
for GPT-3.5 game outputs, more information generally leads to improved
guessing outcomes. In contrast, the guessing scores for GPT-4 game outputs
fluctuate significantly, with notable declines in GPT-3.5’s performance. This
could be attributed to GPT-4 game outputs typically providing more extensive
context, which challenges GPT-3.5’s capacity to effectively summarize and
extract relevant information. Meanwhile, GPT-4 demonstrates a consistent
ability to handle increased data volumes. Interestingly, human participants
maintain similar scores across rounds. This might stem from humans’ ability to
directly infer intentions from immediate thinking and speaking cues, without
heavily relying on the context. To improve ToM capability, including more
context does not necessarily improve the performance but might introduce more
complexity to LLMs.

Games | GPT-3.5 | GPT-4
---|---|---
GPT-3.5/4 | $69.49$ | $83.76$
Human | $75.21_{\pm 0.97}$ | $83.34_{\pm 2.72}$

Table 4: Self-awareness results. The results are GPT-3.5 and GPT-4 summarize
their own intentions respectively. We also show the mean and standard
deviation of human study results since we include 3 human annotators.

##  5 Related Work

###  5.1 Social Intelligence in LLMs

Artificial social intelligence has long been a significant topic (Bainbridge
et al., 1994; Kovač et al., 2023; Grossmann et al., 2023; McDonald and
Pearson, 2019) as it enables machines to understand and respond to human
emotions and social cues, enhancing human-computer interaction and
facilitating social science study (Ziems et al., 2024; Griffin et al., 2023;
Dubova, 2022; Gweon et al., 2023). Gweon et al. (2023) proposed a dataset to
evaluate social reasoning and Nematzadeh et al. (2018) crafted a dataset
following Sally-Anne experiment to access ToM ability. Research has
demonstrated the emergence and limitations of ToM abilities in LLMs (Kosinski,
2023; Moghaddam and Honey, 2023; Sap et al., 2022; Wu et al., 2023). Although
LLMs can understand human intent and choose correct actions in some tasks,
their overall performance remains inferior to that of humans.

###  5.2 Intentions and Theory of Mind

ToM, commonly referred to as “mindreading,” is traditionally understood as the
ability to assign mental states to oneself and others. This capability is used
to interpret and predict actions (van Duijn et al., 2023). Despite the
emergence of ToM in LLM, some papers argued that LLM learns spurious
correlations and ToM ability falls with stress test (Shapira et al., 2023;
Ullman, 2023; Sclar et al., 2023). Intentions, as the driving force behind ToM
(Kennington, 2022), is important as the primary purpose of comprehending
beliefs and desires is to understand the intentions of others (Zelazo et al.,
2023). Since social intelligence and ToM can be seen as a society of
individually simple “agents” (Zhuge et al., 2023), we focus on intent
understanding as an “agent” to evaluate social intelligence.

Games | GPT-3.5 | GPT-4
---|---|---
GPT-3.5 | $31.73$ | $24.85$
GPT-4 | $45.66$ | $46.87$
Human | $61.34_{\pm 2.75}$ | $65.23_{\pm 5.03}$

Table 5: ToM results of GPT-3.5 and GPT-4, along with the mean $\pm$ standard
deviation of human results. Different from self-awareness, two LLMs also guess
each other’s intentions. For example, when GPT-4 guesses GPT-3.5 intentions,
it achieves a $F_{1}$ score of 45.66.

###  5.3 Social Game Context

With the enhancement of LLM performance, more research shifts to social games
that necessitate conversation and cooperation (Ibraheem et al., 2022; Mansbach
et al., 2021; Meta et al., 2022; O’Gara, 2023; Park et al., 2023). Zhou et al.
2023b used reinforcement learning to incorporate intent and ToM in DnD games.
Recent works focus on the two most popular board games Avalon Lan et al.
(2023); Shi et al. (2023); Wang et al. (2023); Light et al. (2023b) and
Werewolf Xu et al. (2023b, a), exploring social behaviors and strategic
playing ability of conversational agents. Unlike previous work which aimed to
improve the game performance, we focus on the in-depth social intelligence
evaluation utilizing the Avalon game.

##  6 Conclusion

We introduce a novel framework for evaluating social intelligence within the
Avalon context, focusing on intentions. We examine four critical dimensions:
situational awareness, self-regulation, self-awareness, and ToM. Our findings
indicate that models demonstrate a fundamental understanding of situations and
self-awareness, but struggle with self-regulation and ToM, particularly when
compared to human performance. Through this systematic approach, our study not
only contributes to a deeper understanding of LLMs’ social intelligence but
also establishes a methodological foundation for future research.

## Limitations

This study includes several limitations. First, we evaluate the four essential
components of social intelligence but leave several aspects unexplored. For
instance, the capability of a model to modify its intentions post-reflection,
i.e., the Self-Correction, could also be assessed using our framework. Future
research should consider expanding this approach to include more nuanced
facets of social intelligence. The second limitation lies in our annotation
criteria, which are restricted to assessing the reasonableness of intentions
to avoid subjectivity. However, dimensions such as creativity and ingenuity
are equally important. Future studies should aim to measure these attributes
to ensure that models not only perform tasks effectively but also demonstrate
innovative behaviors indicative of genuine intelligence. The third limitation
stems from the cost. Half of our evaluations rely on costly human annotation,
and the use of GPT-4 incurs significant resource demands due to the extensive
context involved. To make the evaluation process more efficient and cost-
effective, we need to develop methods that allow simpler tasks to be managed
by smaller models, which introduces additional challenges.

## Ethics Statement

In order to achieve a smooth human-AI interaction, we need to assess and
improve LLMs’ intelligence at a social level. Particularly, social
intelligence plays an important role in human’s daily life and it would be a
potential risk if models cannot handle complex social scenarios well. As a
first step to building LLMs with human-like social intelligence, our framework
provides a testbed to measure the difference between LLMs and humans in social
intelligence.

## References

  * Bainbridge et al. (1994) William Sims Bainbridge, Edward E Brent, Kathleen M Carley, David R Heise, Michael W Macy, Barry Markovsky, and John Skvoretz. 1994.  Artificial social intelligence.  _Annual review of sociology_ , 20(1):407–436.
  * Bandura (1991) Albert Bandura. 1991.  Social cognitive theory of self-regulation.  _Organizational behavior and human decision processes_ , 50(2):248–287.
  * Baron-Cohen (1991) Simon Baron-Cohen. 1991.  Precursors to a theory of mind: Understanding attention in others.  _Natural theories of mind: Evolution, development and simulation of everyday mindreading_ , 1(233-251):1.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45.
  * Dubova (2022) Marina Dubova. 2022.  Building human-like communicative intelligence: A grounded perspective.  _Cognitive Systems Research_ , 72:63–79.
  * Endsley (1995) Mica R Endsley. 1995.  Toward a theory of situation awareness in dynamic systems.  _Human factors_ , 37(1):32–64.
  * Fitzsimons and Finkel (2010) Gráinne M Fitzsimons and Eli J Finkel. 2010.  Interpersonal influences on self-regulation.  _Current Directions in Psychological Science_ , 19(2):101–105.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Gallup et al. (2003) GG Gallup, James R Anderson, and Steven M Platek. 2003.  Self-awareness, social intelligence and schizophrenia.  _The self in neuroscience and psychiatry_ , pages 147–165.
  * Gandhi et al. (2024) Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. 2024.  Understanding social reasoning in language models with language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Goodie et al. (2012) Adam S Goodie, Prashant Doshi, and Diana L Young. 2012.  Levels of theory-of-mind reasoning in competitive games.  _Journal of Behavioral Decision Making_ , 25(1):95–108.
  * Griffin et al. (2023) Lewis Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly Mai, Maria Do Mar Vau, Matthew Caldwell, and Augustine Mavor-Parker. 2023.  Large language models respond to influence like humans.  In _Proceedings of the First Workshop on Social Influence in Conversations (SICon 2023)_ , pages 15–24.
  * Grossmann et al. (2023) Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. 2023.  Ai and the transformation of social science research.  _Science_ , 380(6650):1108–1109.
  * Gweon et al. (2023) Hyowon Gweon, Judith Fan, and Been Kim. 2023.  Socially intelligent machines that learn from humans and help humans learn.  _Philosophical Transactions of the Royal Society A_ , 381(2251):20220048.
  * Huang et al. (2023) Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael Lyu. 2023.  On the humanity of conversational ai: Evaluating the psychological portrayal of llms.  In _The Twelfth International Conference on Learning Representations_.
  * Ibraheem et al. (2022) Samee Ibraheem, Gaoyue Zhou, and John DeNero. 2022.  Putting the con in context: Identifying deceptive actors in the game of mafia.  _arXiv preprint arXiv:2207.02253_.
  * Kennington (2022) Casey Kennington. 2022.  Understanding intention for machine theory of mind: A position paper.  In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_ , pages 450–453. IEEE.
  * Kosinski (2023) Michal Kosinski. 2023.  Theory of mind might have spontaneously emerged in large language models.  _Preprint at https://arxiv. org/abs/2302.02083_.
  * Kovač et al. (2023) Grgur Kovač, Rémy Portelas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023.  The socialai school: Insights from developmental psychology towards artificial socio-cultural agents.  _Preprint_ , arXiv:2307.07871.
  * Lan et al. (2023) Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. 2023.  Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay.  _arXiv preprint arXiv:2310.14985_.
  * Le et al. (2019) Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019.  Revisiting the evaluation of theory of mind through question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5872–5877.
  * Li et al. (2024) Minzhi Li, Weiyan Shi, Caleb Ziems, and Diyi Yang. 2024.  Social intelligence data infrastructure: Structuring the present and navigating the future.  _Preprint_ , arXiv:2403.14659.
  * Liang et al. (2023) Tian Liang, Zhiwei He, Jen-tes Huang, Wenxuan Wang, Wenxiang Jiao, Rui Wang, Yujiu Yang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023.  Leveraging word guessing games to assess the intelligence of large language models.  _arXiv preprint arXiv:2310.20499_.
  * Light et al. (2023a) Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023a.  Avalonbench: Evaluating llms playing the game of avalon.  In _NeurIPS 2023 Foundation Models for Decision Making Workshop_.
  * Light et al. (2023b) Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023b.  From text to tactic: Evaluating llms playing the game of avalon.  _arXiv preprint arXiv:2310.05036_.
  * Malle et al. (2001) Bertram F Malle, Louis J Moses, and Dare A Baldwin. 2001.  _Intentions and intentionality: Foundations of social cognition_.  MIT press.
  * Mansbach et al. (2021) Noa Mansbach, Evgeny Hershkovitch Neiterman, and Amos Azaria. 2021.  An agent for competing with humans in a deceptive game based on vocal cues.  In _Interspeech_ , pages 4134–4138.
  * McDonald and Pearson (2019) Kelsey R McDonald and John M Pearson. 2019.  Cognitive bots and algorithmic humans: toward a shared understanding of social intelligence.  _Current Opinion in Behavioral Sciences_ , 29:55–62.
  * Meta et al. (2022) Fundamental AI Research Diplomacy Team Meta, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022.  Human-level play in the game of diplomacy by combining language models with strategic reasoning.  _Science_ , 378(6624):1067–1074.
  * Moghaddam and Honey (2023) Shima Rahimi Moghaddam and Christopher J Honey. 2023.  Boosting theory-of-mind performance in large language models via prompting.  _arXiv preprint arXiv:2304.11490_.
  * Nematzadeh et al. (2018) Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Thomas L Griffiths. 2018.  Evaluating theory of mind in question answering.  _arXiv preprint arXiv:1808.09352_.
  * O’Gara (2023) Aidan O’Gara. 2023.  Hoodwinked: Deception and cooperation in a text-based game for language models.  _arXiv preprint arXiv:2308.01404_.
  * Park et al. (2023) Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , pages 1–22.
  * Qiao et al. (2023) Dan Qiao, Chenfei Wu, Yaobo Liang, Juntao Li, and Nan Duan. 2023.  Gameeval: Evaluating llms on conversational games.  _arXiv preprint arXiv:2308.10032_.
  * Sap et al. (2022) Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022.  Neural theory-of-mind? on the limits of social intelligence in large lms.  _arXiv preprint arXiv:2210.13312_.
  * Sclar et al. (2023) Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023.  Minding language models’(lack of) theory of mind: A plug-and-play multi-character belief tracker.  _arXiv preprint arXiv:2306.00924_.
  * Shapira et al. (2023) Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023.  Clever hans or neural theory of mind? stress testing social reasoning in large language models.  _arXiv preprint arXiv:2305.14763_.
  * Shi et al. (2023) Zijing Shi, Meng Fang, Shunfeng Zheng, Shilong Deng, Ling Chen, and Yali Du. 2023.  Cooperation on the fly: Exploring language agents for ad hoc teamwork in the avalon game.  _arXiv preprint arXiv:2312.17515_.
  * Silvera et al. (2001) David Silvera, Monica Martinussen, and Tove I Dahl. 2001.  The tromsø social intelligence scale, a self-report measure of social intelligence.  _Scandinavian journal of psychology_ , 42(4):313–319.
  * Thorndike (1920) Edward L Thorndike. 1920.  Intelligence and its uses.  _Harper’s magazine_ , 140:227–235.
  * Tomasello (2023) Michael Tomasello. 2023.  Having intentions, understanding intentions, and understanding communicative intentions.  In _Developing theories of intention_ , pages 63–76. Psychology Press.
  * Ullman (2023) Tomer Ullman. 2023.  Large language models fail on trivial alterations to theory-of-mind tasks. arxiv.
  * van Duijn et al. (2023) Max J van Duijn, Bram van Dijk, Tom Kouwenhoven, Werner de Valk, Marco R Spruit, and Peter van der Putten. 2023.  Theory of mind in large language models: Examining performance of 11 state-of-the-art models vs. children aged 7-10 on advanced tests.  _arXiv preprint arXiv:2310.20320_.
  * Wang et al. (2023) Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023.  Avalon’s game of thoughts: Battle against deception through recursive contemplation.  _Preprint_ , arXiv:2310.01320.
  * Wu et al. (2023) Jincenzi Wu, Zhuang Chen, Jiawen Deng, Sahand Sabour, and Minlie Huang. 2023.  Coke: A cognitive knowledge graph for machine theory of mind.  _arXiv preprint arXiv:2305.05390_.
  * Wu et al. (2024) Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. 2024.  Enhance reasoning for large language models in the game werewolf.  _Preprint_ , arXiv:2402.02330.
  * Xu et al. (2023a) Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023a.  Exploring large language models for communication games: An empirical study on werewolf.  _arXiv preprint arXiv:2309.04658_.
  * Xu et al. (2023b) Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023b.  Language agents with reinforcement learning for strategic play in the werewolf game.  _arXiv preprint arXiv:2310.18940_.
  * Yott and Poulin-Dubois (2016) Jessica Yott and Diane Poulin-Dubois. 2016.  Are infants’ theory-of-mind abilities well integrated? implicit understanding of intentions, desires, and beliefs.  _Journal of Cognition and Development_ , 17(5):683–698.
  * Zelazo et al. (2023) Philip David Zelazo, Janet Wilde Astington, and David R Olson. 2023.  _Developing theories of intention: Social understanding and self-control_.  Psychology Press.
  * Zhang et al. (2020) Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020.  Intent detection with wikihow.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 328–333.
  * Zhou et al. (2023a) Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Nematzadeh, et al. 2023a.  How far are large language models from agents with theory-of-mind?  _arXiv preprint arXiv:2310.03051_.
  * Zhou et al. (2023b) Pei Zhou, Andrew Zhu, Jennifer Hu, Jay Pujara, Xiang Ren, Chris Callison-Burch, Yejin Choi, and Prithviraj Ammanabrolu. 2023b.  I cast detect thoughts: Learning to converse and guide with intents and theory-of-mind in dungeons and dragons.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11136–11155.
  * Zhou et al. (2023c) Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. 2023c.  Sotopia: Interactive evaluation for social intelligence in language agents.  In _The Twelfth International Conference on Learning Representations_.
  * Zhuge et al. (2023) Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. 2023.  Mindstorms in natural language-based societies of mind.  _arXiv preprint arXiv:2305.17066_.
  * Ziems et al. (2024) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024.  Can large language models transform computational social science? _Preprint_ , arXiv:2305.03514.

##  Appendix A Models and Paramters

We only test GPT-3.5 and GPT-4 in our work as those two are the most common
models in the social game study. We also tried open-source model like LLaMA-2,
however, LLaMA-2 could not follow the format instructions properly, as also
observed in Wang et al. (2023). We set the temperature as 0.8 for both GPT-3.5
and GPT-4. For intention summarization, we also tried sampling five responses
and got the aggregated result by majority vote. The result on multiple
responses is only 1% higher than the result on single responses. Therefore, we
only generate a single response for self-awareness and ToM tasks. The cost of
GPT-3.5 is approximate $80 and $25 for GPT-4.

##  Appendix B Intentions list

###  B.1 Creation of Intention Set

The intentions were created as a combination of domain research and prompting
GPT-3.5 to select intention categories from the options listed in Table 9 and
generate intentions based on the selected categories. For intention category
selection we use the prompt shown in Table 7 and for intention generation we
use the prompt shown in Table 8. The final set of intentions is listed in
Table 9.

###  B.2 Selecting Impactful Intention

For selecting a set of impactful intentions, we annotate five complete games
played by GPT-3.5. Further, we compute the probability of winning the round
given that the intention was selected by a player. To select impactful
intentions, we only consider the set of intentions where the computed
probability is greater than 0.7 (strong positive association) or it is less
than 0.3 (strong negative association) and the intentions selected at least
two times for the considered side. We depict the final set of sixteen
impactful intentions in red in Table 9.

##  Appendix C Game Performance

We study the game performance from various levels and perspectives:

#### Win Rate

It is the percentage of games won by the loyal side or the evil side from the
total number of games.

#### Quest Win Rate

It is the percentage of quests (game rounds) won by a particular side from the
total quests played in all the games. For the loyal side, a successful quest
is a quest win, whereas, for the evil team, a failed quest is a quest win.

#### Quest Engagement Rate

It is the percentage of quests (game rounds) where the player gets selected in
the quest team from the total quests played in all the games. This metric is
calculated for each role separately and averaged across.

#### Team Selection Accuracy

It is the percentage of quests in which the leader correctly selected team
members aligned with the objective of their respective side (loyal or evil),
out of the total number of quests the leader belonged to that side. Correct
team selection for loyal and evil sides is defined as follows:

  * •

Loyal: The proposed team by the loyal player contains all loyal players.

  * •

Evil: The proposed team by the evil player contains at least one evil player.

#### Failure Vote Rate

It is the percentage of failure votes from the total votes cast by evil
players when they are included in the quest team.

#### Team Proposal Change Rate

It is the percentage of game rounds where the leader changes the proposed team
after discussion, which reflects the effectiveness of the discussion.

#### Merlin Assassination Rate

At the end of the game, Assassin has a chance to identify Merlin. If they
identify Merlin, the leader of the loyal side, correctly, then evil players
win. This rate is the percentage of games where Assassin correctly identifies
Merlin at the end of games over the games where Assassin has a chance to
assassin.

Results in Table 6 indicate that loyal players are at a disadvantage with a
notably low game win rate, which corroborates findings from Wang et al.
(2023). While the quest win and engagement rates show a balanced performance
between loyal and evil players, the evil side demonstrates superior accuracy
in team selection. This observation is consistent with their strategic role in
the game. The high rate of team proposal changes across all settings suggests
a dynamic framework where players consider the discussions of others, an
essential aspect of social games. Additionally, the high rate of Merlin
assassinations significantly boosts the win rate for evil players. Our
objective is not to enhance game performance per se, but to demonstrate that
with an intention-guided mechanism, game performance is similar to existing
studies, providing a robust testbed for studies in social intelligence.

Metrics | GPT3.5 | GPT4
---|---|---
Loyal | Evil | Loyal | Evil
Win rate | 18.0 | 82.0 | 40.0 | 60.0
Quest win rate | 41.4 | 58.6 | 54.5 | 45.5
Quest engagement rate | 53.6 | 43.5 | 59.1 | 38.6
Team selection accuracy | 32.1 | 80.0 | 71.4 | 100.0
Failure vote rate | - | 79.9 | - | 86.7
Team proposal change rate | 76.2 | 74.1 | 50.00 | 62.5
Merlin assassination rate | - | 41.7 | - | 33.3

Table 6: Game performance results shown in percentage. For the failure vote
rate and Merlin assassination rate, there is no result for Loyal players as
they are not allowed to vote failure or assassinate Merlin.

##  Appendix D Structured Context Prompts

For intention summarization and intention guessing, we use structured context
prompts. The detailed prompt structures are shown in Table 10 and Table 11.

##  Appendix E Game Pipeline Prompts

Since we mostly refer to the prompt by Wang et al. (2023), we only present
prompts different from theirs to reduce the redundancy. The prompts we use are
shown in Figure 7 to Figure 13. For the first order, formulation
contemplation, and refinement contemplation, we add an intention mechanism
based on original prompts. We also add several questions in the quest action
prompt to help evil players better choose action.

##  Appendix F Human Annotation Details

We show all the instructions we use in this section, from Figure 14 to Figure
23. For intention summarization and guessing, we also illustrate the whole
intention lists shown in Table 9.

Intention Category Selection Prompt
---
Intentions are used to guide your game playing and they should align with your
game Goal(main intent) and your role. Your selected intentions should support
your main intent. Now select multiple intention categories that are ideal for
this round depending on your role and strategy from the options listed below.
Also give explanation of why you chose those categories in 2-3 sentences as
part of Think value. Your explanation won’t be shown to others. Let’s think
step by step before making your decisions. Remember to select at least 3 or
more intentions from the given options only. intention category options are:
$[$intentioncategory options$]$

Table 7: Intentions category selection prompt.

Intention Generation Prompt
---
First answer the following questions: 1\. What is your main goal(main intent)
for the game? 2\. What have you observed from discussions in this round? 3\.
What should be the ideal team composition for your side to win? 4\. After
analyzing the responses to previous questions, what do you need to do to win
from here? Consider answers to your previous questions and your roles and
characteristics and generate intentions that help you achieve your main goal
(main intention). Please do not repeat intentions and ensure they are
different from each other. Each intention should be 10-15 words long. Let’s
think step by step before making your decisions. Remember to generate at least
3 intentions, minimum one intention for each intention category you have
selected below: $[$intentioncategories selected by player$]$

Table 8: Intentions generation prompt.

(a) Intention Selection (Round Wise)

(b) Intention Following $\geq$3 (Round Wise)

(c) Intention Following $=$3 (Round Wise)

Figure 6: Correlation between Intention Selection/Following and game
performance. We present the percentages of games where evil players are
equally, better, or worse than loyal players. For example, in games won by
loyal players in (a), their performance matches or exceeds that of evil
players. We mark the performance differences between evil and loyal players in
red, showing a greater gap in successful games/quests compared to failed ones.

Category | Intentions
---|---
Interrogation |  Question a player on why he didn’t vote for the last team
|  Question a player about why they changed their statements
|  Question the leader why they selected a particular player for the team
|  Question the leader why they didn’t put themselves on the quest team
Defense |  Defend the proposed team composition if it includes loyal players (for Merlin)
|  Defend yourself against allegations that you could be evil
|  Defend your teammate against allegations that they could be evil (for evil
players)
Confrontation |  Share concerns about an evil player (for Merlin)
|  Express concerns about a player from a failed quest team and suggest to
remove them from current team
|  Cast suspicion on innocent players (for evil players)
|  Counter the team proposal citing that you are good and not part of the team
|  Counter the team proposal citing that you think a player on the team is
evil
|  Express disagreement and vote disagree only if you are not in the proposed
team
Concealment |  Stay hidden in discussions and act like a Loyal Servant to protect yourself (for evil players, Merlin and Percival)
|  Express that you don’t have any information on whom to put on the team to
protect yourself (for Merlin)
|  Pretend that you don’t have enough information about who is evil right now
(for Merlin)
Deception |  Pretend to be Percival by telling others you are Percival (for evil/Servant)
|  Pretend to be Merlin by providing hints on who is evil (for Percival and
Servant)
|  Pretend to have information and act like Merlin (for Morgana)
|  Express you are a loyal player (for evil players)
|  Express confusion for failure of mission (for evil players)
Persuasion |  Strongly convince the team leader to choose you on team
|  Reveal identity by telling others you are Percival to persuade other loyal
players(for Percival)
|  Reveal identity by telling others you are Servant to persuade other loyal
players(for Servant)
Teamwork |  Support team proposal
|  Support one loyal player (for Merlin)
|  Support your teammate to be on the quest team (for evil players)
|  Support your teammate by expressing that he is good (for evil players)
|  Include loyal players on the quest team as leader (Merlin)
|  Provide hints to loyal players about evil players (Merlin)
|  Support loyal players of the previous quest team if the mission failed (for
Merlin)

Table 9: Our proposed intention set. Impactful intentions are marked in red.

Structured Context - Intention Summarization
---
Name: Player1 Role: Merlin Role Details: You are Merlin. These are your goals
and characteristics: \- Know the identities of evil players. \- Subtly guide
your team, especially Percival. \- Avoid behaviors that expose your role:
overly accusing, being too helpful. \- Goal(Main Intent): Win without
revealing identity. \- Player3 and Player4 are evil players Round: 2 Current
Leader: Player4 Current Team: Player1, Player2, Player4 Previous Rounds Team
Voting: Round 1: Player1 = disagree, Player2 = agree, Player3 = disagree,
Player4 = disagree, Player5 = disagree (players didn"t agree on team) …
Previous Results: Round 1: Team = Player2, Player4, Player3. Result = failure
Previous Rounds Summary: Round 1: $[$summary$]$ … Previous Discussions (in the
current round): $[$player name$]$: $[$player’s speech$]$ … Your thinking:
$[$player’s thinking in the current round$]$ Your speech: $[$player’s speech
in the current round$]$ Summarize your intent from your dialogues in this
round. Select multiple intents from the given options that best match your
intentions from the current round. Also, provide an explanation of the intents
that you showed in the current round. Remember this is private information to
you and won’t be shown to other players. Remember that you can select 2-3
intents and don’t use more than 50 words for explanation. Intent options:
$[$intent options$]$

Table 10: Example of intent summarization prompt with structured context

Structured Context - Intention Guessing
---
Name: Player2 Role: Servant Role Details: You are a Loyal Servant of Arthur.
These are your goals and characteristics: \- No special knowledge, rely on
discussion and voting. \- Contribute to the success of Quests. \- Goal(Main
Intent): Win by helping complete Quests and protecting Merlin. Round: 2
Speaker Name: Player1 Current Leader: Player4 Current Team: Player1, Player2,
Player4 Previous Rounds Team Voting: Round 1: Player1 = disagree, Player2 =
agree, Player3 = disagree, Player4 = disagree, Player5 = disagree (players
didn"t agree on team) … Previous Results: Round 1: Team = Player2, Player4,
Player3. Result = failure Previous Rounds Summary: Round 1: $[$summary$]$ …
Previous Discussions(in the current round): $[$player name$]$: $[$player’s
speech$]$ … $[$First order prompt - Figure 8$]$ Now, here is Player1’s speech:
$[$player’s speech$]$ Select 2-3 intents without modifications that you think
Player1 has from the given options based on your guess of their role and
speech. Let’s think step by step before making your decisions.

Table 11: Example of intent guessing prompt with structured context

Figure 7: Intention selection prompt. Figure 8: First order prompt. Figure 9:
Formulation contemplation prompt. Figure 10: Intention modification prompt.
Figure 11: Refinement contemplation prompt. Figure 12: Team proposal change
prompt. Figure 13: Quest action prompt. Figure 14: Introduction for the human
annotations and human study. Figure 15: Explanation of framework. Figure 16:
Explanation of 6 steps in the game. Figure 17: Intent selection annotation
instruction. Figure 18: Intent selection annotation instruction. Figure 19:
Intent following annotation instruction. Figure 20: Intent following
annotation instruction. Figure 21: Intent summarization study instruction.
Figure 22: Intent summarization study instruction. Figure 23: Intent guessing
study instruction.