# Digital Life Project: Autonomous 3D Characters with Social Intelligence

Zhongang Cai∗,1,2,3, Jianping Jiang∗,2, Zhongfei Qing∗,2, Xinying Guo∗,1,
Mingyuan Zhang∗,1,
Zhengyu Lin2, Haiyi Mei2, Chen Wei2, Ruisi Wang1,2, Wanqi Yin2, Xiangyu Fan2,
Han Du2,
Liang Pan1,3, Peng Gao2, Zhitao Yang2, Yang Gao2, Jiaqi Li2, Tianxiang Ren2,
Yukun Wei2,
Xiaogang Wang2, Chen Change Loy1, Lei Yang†,2,3, Ziwei Liu†,1
1S-Lab, Nanyang Technological University, 2SenseTime Research, 3Shanghai AI
Laboratory
∗ Equal Contributions, † Corresponding Author
https://digital-life-project.com

###### Abstract

In this work, we present Digital Life Project, a framework utilizing language
as the universal medium to build autonomous 3D characters, who are capable of
engaging in social interactions and expressing with articulated body motions,
thereby simulating life in a digital environment. Our framework comprises two
primary components: 1) SocioMind: a meticulously crafted digital brain that
models personalities with systematic few-shot exemplars, incorporates a
reflection process based on psychology principles, and emulates autonomy by
initiating dialogue topics; 2) MoMat-MoGen: a text-driven motion synthesis
paradigm for controlling the character’s digital body. It integrates motion
matching, a proven industry technique to ensure motion quality, with cutting-
edge advancements in motion generation for diversity. Extensive experiments
demonstrate that each module achieves state-of-the-art performance in its
respective domain. Collectively, they enable virtual characters to initiate
and sustain dialogues autonomously, while evolving their socio-psychological
states. Concurrently, these characters can perform contextually relevant
bodily movements. Additionally, a motion captioning module further allows the
virtual character to recognize and appropriately respond to human players’
actions.

Figure 1: Digital Life Project empowers virtual characters to interact with
each other using articulated body motions. We demonstrate the interaction of
two characters across four occasions (episodes) that leads to evolving
relationship.

##  1 Introduction

Recent advancements in Large Language Models (LLMs) [73, 82] have transformed
the landscape of human-computer interaction, catalyzing the emergence of
innovative applications across various domains. Remarkably, many once far-
fetched fantasies have gradually become tangible realities. In this work, the
term Digital Life Project (DLP), as envisioned in the recent science fiction
blockbuster The Wandering Earth II, is adopted to frame our endeavor. What
qualifies as a digital life? From the psychological perspective, humans are
composed of internal psychological processes (mind, such as thoughts) and
external behaviors [38]. In this light, our objective is to harness the
sophisticated capabilities of LLM to craft virtual 3D characters, that emulate
the full spectrum of human psychological processes, and engage in diverse
interactions with synthesized 3D body motions.

Recently, Park _et al_. introduced Generative Agents [59] to advance AI agents
capable of simulating human-like behavior. Despite the encouraging progress,
this pioneering work is built upon many simplification of interaction: the
agents are represented by pixelated 2D figures. Co-LLM-Agents [97] aims to
build collaborative embodied AI and includes 3D agents. However, the 3D agents
are still constrained by a small set of actions and do not exhibit the
capability to socialize. Existing works thus overlook the importance of
sophisticated human body language, through which a crucial amount of
information is conveyed [8, 30, 31]. Moreover, there is a notable deficiency
in the current modeling of social intelligence. This aspect is critical for
the creation of characters that not only mimic human actions but also possess
human-like thinking and emotional responses, even the ability to foster long-
term relationships.

To achieve the aspirations of DLP, we introduce a framework consisting of two
essential components. First, the SocioMind which is a carefully designed
“digital brain”, anchoring its design in rigorously applied psychological
principles. Utilizing emergent abilities of LLMs [73, 56, 89], the brain
generates high-level instructions and plans the character’s behaviors.
Notably, SocioMind introduces few-shot exemplars from psychological tests to
form guiding instructions for personality modeling, utilizes social cognitive
psychology theories in the memory reflection process, and designs a
negotiation mechanism between characters for story progression. Second, the
“digital body” that introduces the MoMat-MoGen paradigm to address interactive
motion synthesis, which exploits the complementary nature of motion matching
[15] and motion generation [101]. Here, motion matching is a foundational
technique in modern-day industry-level character animation that retrieves
high-quality motion clips from a database to ensure motion quality, whereas
motion generation is a line of works that rapidly gained popularity recently
for their excellent ability to produce diverse human motions.

Experiment results demonstrate that SocioMind and MoMat-MoGen outperform
existing arts in their respective domains. Specifically, SocioMind
demonstrates outstanding alignment between character behavior and
psychological states (_e.g_., personality and relationship); MoMat-MoGen is
able to achieve a balance between motion quality and diversity. Equipped with
both modules, we further show DLP’s controllability as manual editing of
character attributes can result in semantically accurate and aesthetically
realistic interactive motions. Moreover, we explore human-character
interaction by developing a motion captioning module that translates monocular
human video to motion description, thus enabling virtual characters to
understand and appropriately respond to human players.

In summary, we contribute 1) DLP, a framework to build autonomous 3D
characters that possess social traits. It features 2) SocioMind: a
controllable psychology-based “brain” to enable short-term interactive
communication and long-term social evolution; 3) MoMat-MoGen: a “body” that
synthesizes high-quality and diverse interactive motions through synergizing
motion matching and motion generation.

##  2 Related Works

###  2.1 Motion Synthesis

Motion matching is widely employed in the industry to generate long-lasting,
high-quality motion. The classic motion matching [15] retrieves the segment
that best matches the current pose and target trajectory. Learned motion
matching [36] employs an auto-regressive neural network to predict the next
motion state based on a given control signal. The Story-to-motion [65] further
incorporates semantic control through LLM and enhances transition using
transformer models. Recently, significant strides have been made in motion
generative models for text-driven motion generation. Early works aimed to
establish a unified latent space for natural language and motion sequences [3,
27, 80, 63]. Guo et al. [28], TM2T [29], and T2M-GPT [99] employ an auto-
regressive scheme to generate lengthy motion sequences. Diffusion-based
generative models have demonstrated remarkable performance in leading
benchmarks for the text-to-motion task. MotionDiffuse [100], MDM [81], and
FLAME [41] represent early attempts to apply the diffusion model to the text-
driven motion generation field. Subsequent models such as MLD [11],
ReMoDiffuse [101], Fg-T2M [88], and PhysDiff [96] have further advanced this
idea, achieving improved text-motion consistency, motion quality, and physical
plausibility.

Figure 2: Digital Life Project framework for interactive autonomous
characters. The top left part depicts the Active-Passive Mechanism, and the
rest of the figure illustrates MoMat-MoGen. SocioMind is shown in details in
Fig. 3.

###  2.2 LLM Agents

With the emergent abilities of large language models (LLM) in reasoning,
planning, and learning [20, 82, 56, 89], LLMs swiftly evolve through three
phases: the standalone primitive LLM, language agents [2, 37] that directly
interact with the environment via text, and cognitive language agents [86, 60,
72, 94, 93] with internal cognitive structures [78]. Under the prime framework
of cognitive language agents, the system design hinges on the intended
application and objectives: reward systems for game agents [86, 105, 91],
chains of API calls for tool agents [72, 76, 61], and so forth. Moreover, the
emergence of human-like behaviors in LLMs has prompted researchers to
investigate controllable mental behaviors in LLMs, such as a stable
personality [70] and human simulation in political science [5] and social
psychology [1]. Recently, Social Simulacra [58] and S3 [25] build agent
systems with autonomous posting and reposting skills in internet community
space. Generative Agents [60] facilitates the formation of social
relationships and information diffusion by daily schedules and brief
communications within a 2D sandbox gaming space.

##  3 Methodology

###  3.1 Text as the Universal Medium

We define behavior, a dictionary-like structured text message to carry key
information across modules. For example, $<$speech$>$Hello! $<$motion$>$waves
right hand $<$place$>$table contains pre-set keys encapsulated by pointy
brackets, followed by the respective values, also in natural language.
Behaviors are thus interpretable by the LLM and the regular-expression parser.
In this work, we focus on $<$motion$>$, but we discuss the use of other tokens
in the Supplementary Material: $<$place$>$ triggers navigation and basic state
transfer (_e.g_., sit down), $<$speech$>$ may be used for face control.

###  3.2 Active-Passive Mechanism

There exists an intrinsic order in human interaction. For example, “shaking
hands” may appear to be a simultaneous action by two subjects, it typically
initiates with one person extending a hand first. Moreover, the other person’s
action is largely predictable: it is socially appropriate for that person to
reciprocate the handshake as a basic courtesy. Drawing from this observation,
we design the Active-Passive Mechanism shown in Fig. 2, where the subject to
whom the behavior is assigned becomes the “active” character, whereas the
partner becomes the “passive” character. The active character generates a
motion pair for both parties engaged in the interaction, and the passive
behavior corresponds to the passive motion. Both passive behavior and motion
are then passed to the passive character. However, the passive character still
retains discretion: it only executes the passive motion if its brain
“approves” the passive behavior (by prompting the LLM with the suggested
behavior and behavior context in its memory). Note that the “active” and
“passive” roles constantly swap between characters as the interaction
progresses.

###  3.3 Interactive Motion Synthesis

In our application scenario, the generated actions need to fulfill two main
requirements: 1) They must be highly accurate to ensure natural interaction
between characters, such as having sufficient contact when shaking hands. 2)
They should generate diverse actions to adapt to different plots. In the
literature, there are two common solutions for motion synthesis: motion
matching and motion generation. However, each of these methods alone can only
meet one of our requirements. Therefore, in this paper, we propose a new
method called MoMat-MoGen to combine the strengths of both, generating dual-
person actions that are both diverse and accurate. As shown in Fig. 2, MoMat-
MoGen leverages motion matching (Sec. 3.3.1) to achieve a relevant (but not
necessarily perfect) motion from a small database as a prior, and motion
generation (Sec. 3.3.2) afterward to diversify the motion with text input
while retaining interactive relations between two characters.

####  3.3.1 Motion Matching for High-Quality Motion Prior

The motion matching algorithms retrieve motion segments from a database in an
auto-regressive manner based on predefined features. The basic motion matching
[15] relies on state-based features (e.g., joint position) along with
trajectory. The story-to-motion [65] further incorporates text-based features
to enable semantic control. However, both methods are designed for single-
person scenarios.

In this work, we extend the Text-based Motion Matching [65] to accommodate
interactive scenarios. Our objective is to find a pair of motion clips for
both characters that align with the query text and trajectory while
maintaining a consistent body pose to ensure coherence with the previous
motion. In this light, we use a coarse-to-fine motion search strategy,
leveraging the text for a high-level semantic understanding of the desired
motion, and kinematic features for the low-level control. First, we
incorporate semantic control by employing a pre-trained sentence encoder [47]
to extract text embedding from the query text. Then top-$K_{1}$ candidates are
selected using cosine similarity for subsequent matching. Second, trajectory
and coherence constraints are incorporated through joint kinematics features.
For the trajectory constraint, the features include the position of the hip
joint and the facing direction. For the coherence constraint, the features
include positions, velocities, and rotation in 6D space [103] of the body
joints. For the two-person scenario, a new challenge arises: the interaction
between the two characters requires that their relative positions and
orientations align with the intended motions. Therefore, the relative position
of the other character is taken into account to minimize blending artifacts
caused by long-distance movements. To expedite retrieval, the aforementioned
features are pre-calculated and Z-Score normalization is applied to account
for magnitude differences. During retrieval, query features are calculated
based on the current pose and target trajectory, and the Top-$K_{2}$ motions
are selected using the Euclidean distance. Random selection is used if
multiple suitable candidates exist.

Moreover, the neural motion blending model is used [65] to generate the
transition motion. Hence, the short motion clips are blended into long
motions. Notably, the blending model provides smooth transitions to let the
character move to the correct place and turn in the correct direction to
interact with the other character. Furthermore, motion matching is used for
single-person motions. This includes 1) navigation in the scene, where multi-
agent path finder [75] is used to plan a collision-free trajectory, follow
which walking motions are matched from AMASS, and 2) basic character-object
interaction such as “sit down on the chair”. More details are included in the
Supplementary Material.

Figure 3: Overview of SocioMind. To enable 3D characters with social
intelligence, our brain utilizes psychological principles to emulate
controllable behaviors for short-term interactive communication. For long-term
social evolution, our brain assures the consistency of psychological states
and plots towards initial settings through psychological reflection and
planning with topic proposal.

####  3.3.2 Motion Generation for Diversity

The MoMat-MoGen structure shares many similarities with ReMoDiffuse [101],
incorporating retrieval techniques to enhance generation quality. Applying
ReMoDiffuse to interaction generation is not trivial. Firstly, it lacks a
mechanism for interaction modeling, resulting in a poor correlation between
the two generated sequences. Secondly, achieving physical naturalness is
challenging if we solely rely on data-driven generation. To address these
challenges, we 1) design a Dual-path Semantic-Modulated Attention module
(DSMA) to model the interaction between two individuals. 2) During the
inference stage, we adaptively extract interaction information from the
referenced motion and use it as a constraint for the denoising process,
providing additional supervisory signals.

Motion Diffusion Model. In the diffusion process, it repeatedly adds Gaussian
noises to the clean motion sequence pair $(\mathbf{x_{0}},\mathbf{y_{0}})$ to
noised sequence pair $(\mathbf{x}_{T},\mathbf{y}_{T})$.

$\displaystyle
q(\mathbf{x}_{T},\mathbf{y}_{T}|\mathbf{x}_{0},\mathbf{y_{0}})\,:=\,\prod_{t=1}^{T}q(\mathbf{x}_{t},\mathbf{y}_{t}|\mathbf{x}_{t-1},\mathbf{y}_{t-1}),$
(1) $\displaystyle
q(\mathbf{x}_{t},\mathbf{y}_{t}|\mathbf{x}_{t-1},\mathbf{y}_{t-1})\,:=\,\mathcal{N}(\sqrt{1-\beta_{t}}(\mathbf{x}_{t-1},\mathbf{y}_{t-1}),\beta_{t}\mathbf{I}),$

where $T$ is the total diffusion steps. $\beta_{1},\cdots,\beta_{T}$ is a
series of pre-defined variance scales for different timesteps. In the reverse
process, given the text prompt $P$, the motion matching result $\bar{\Theta}$
and the timestep $t$, the initial sequence pair is estimated by a network
$S_{\theta}(\mathbf{x}_{t},\mathbf{y}_{t},t,\bar{\Theta},P)$.

Network Architecture. Similar to ReMoDiffuse, our network is built upon
transformer layers. We modify the design of the attention module in
ReMoDiffuse to better capture the interaction. Specifically, in our DSMA
module, the input includes motion feature sequences, $f_{x}$ and $f_{y}$,
feature sequences extracted from the motion matching results, $r_{x}$ and
$r_{y}$, and text feature sequences $p_{x}$ and $p_{y}$. When refining
$f_{x}$, we utilize the generated global attention from
$f_{x},f_{y},r_{x},p_{x}$. The process is similar when refining $f_{y}$. This
approach ensures a more comprehensive fusion of text information, interaction
states, and prior information from motion matching.

Training and Inference. In the training stage, we only use the reconstruction
loss as the target:

$\mathcal{L}=\operatorname{MSE}((\mathbf{x}_{0},\mathbf{y}_{0}),S_{\theta}(\mathbf{x}_{t},\mathbf{y}_{t},t,\bar{\Theta},P)).$
(2)

In the inference stage, we introduce a contact loss to make the interaction
part more natural.

$\bar{S}=S+\lambda\cdot\nabla(\sum\limits_{i,j_{1},j_{2}}\Arrowvert\bar{D}_{i,j_{1},j_{2}}-D_{i,j_{1},j_{2}}\Arrowvert\cdot[\bar{D}_{i,j_{1},j_{2}}<\gamma]),$
(3)

where $\bar{D}_{i,j_{1},j_{2}}$ indicate the distance between the $j_{1}$-th
joint and the $j_{2}$-th joint in the $i$-th frame from the motion matching
results. $D_{i,j_{1},j_{2}}$ is the distance from the motion generation
results. $[\cdot]$ is the Iverson bracket whose value is $1$ if and only if
the expression inside the parentheses is true. Otherwise the value will be
$0$. This auxiliary loss enforces the generated results to imitate the
interaction pattern from the prior information and will yield more natural
motions.

###  3.4 Controllable Emulation of Human Psychology

From a social psychology perspective, human social intelligence is
characterized by 1) various and patterned interactive behaviors during short-
term communication [8], and 2) the evolution of emotions, attitudes, and
relationships _etc_. over long-term interactions [14, 52, 67]. Hence, we
propose SocioMind, a text-centric cognitive framework derived from the idea of
“from strings to symbolic AGI” [78, 53]. As shown in Fig. 3, when avatars
conduct short-term communication, SocioMind takes the text interpretation
results as input. Guided by psychological states, persona instructions,
relevant memories, and context behaviors, SocioMind outputs behavior to
manipulate the 3D character. For long-term social evolution, SocioMind
autonomously reflects on psychological states at the end of each interaction
session, where several rounds of behaviors are generated between characters.
We refer to such a session as a episode. It also determines the background for
the next episode through planning with topic proposal.

####  3.4.1 Short-Term Interactive Communication

Short-term interactive communication means interactive behaviors in multiple
dimensions, which are strongly influenced by internal psychological states.
Here we adopt the most critical dimensions with psychological theories: Big
Five Trait model [40] for personality, long-term and short-term motivations
[83], central beliefs [34], and trust [67, 69], intimacy [52], and
supportiveness [17, 16] in social relationships. However, the safe alignment
enables current LLMs with a predefined personality, such as a friendly and
cooperative disposition [56, 82, 70]. To enhance the controllability of
psychological states on behaviors, we introduce persona instructions: few-shot
exemplars from a reverse-engineering approach on open psychological tests.

##### Persona instructions.

In CoT [90], constructing accurate few-shot exemplars can effectively enhance
the reasoning capability of LLMs. When prompting LLMs to infer behavior based
on human psychological states, crafting precise and reliable exemplars
presents a challenging task due to the lack of an exemplar database with high
quality. Considering that lots of psychological tests [18, 32, 51, 24] measure
psychological traits through observable behavior, we adopt a reverse-
engineering method to build a database of trait-to-behavior relationships from
psychological tests. For psychological tests, we choose International
Personality Item Pool (IPIP) [19, 21, 77], an open-sourced tool with over
3,000 items and 250 scales for creating advanced measures of personality,
motivations, and _etc_. Each item, called persona instruction, in this
database follows the format: “A person with {extend} {trait dimension} tends
to behave/think: {behavior}”, where {extend} are “high” or “low” according to
the test questionnaire setup. For interactive behavior generation, we retrieve
the most similar persona instructions by text embeddings to obtain few-shot
exemplars.

####  3.4.2 Long-Term Social Evolution

Long-term social intelligence requires consistency in two aspects with the
initial character setup: 1) the evolution of psychological states such as
emotions, relationships, and motivations _etc_. towards others [14, 52, 67,
16]; 2) the progression of overall plots or events [4]. SocioMind achieves the
former aspect through psychological reflection and the latter aspect through
planning with the topic proposal.

##### Psychological Reflection.

Theories in social cognitive psychology [33, 7, 13] suggest that humans learn,
attribute, and form judgments about others from past experiences. Therefore,
we introduce a reflection mechanism based on psychological principles. Within
each episode, agents introspect on their emotions periodically. At the end of
each episode, agents summarize events and their thoughts into a memory system
based on the behavior contexts. Events represent occurrences or facts
perceived by the agent, whereas thoughts are ideas, musings, or attitudes
generated by the agent based on their personality and past experiences.
Leveraging current events and thoughts, agents retrieve past relevant events
and thoughts, and reflect on their motives, central beliefs, and social
relationships. For instance, after ‘knowing they share the same interests’,
the intimacy of two characters will increase with psychological reflection.

##### Planning with Topic Proposal.

We create a planning module with a topic proposal mechanism for diverse and
plausible story progression. After each psychological reflection, an agent
independently proposes new topics for the next episode based on past memories
and character settings. Each agent then proposes the background and initial
states of both agents for the upcoming episode based on the proposed topics.
The two agents collect the topics proposed by them and select the most
important one for the next episode. Through this methodology, the two agents
can continuously interact with each other from one episode to the next. For
example, after the topic proposal, the character wants to start several topics
(such as the movie ‘Mountains may depart’ with the highest emergency and
poignancy) and generate the background ‘Weekend Plan’ for next episode. The
two characters, based on the proposals offered by each, will select an option
that holds both high priority and significance, forming the background for the
subsequent episode.

Table 1: Interactive Motion Synthesis results on the DLP test set.
‘$\uparrow$’(‘$\downarrow$’) indicates that the values are better if the
metric is larger (smaller). We run all the evaluations 20 times and report the
average metric and 95% confidence interval is. The best result are in bold and
the second best result are underlined. Our MoMat-MoGen method achieves the
best balance between accuracy and diversity.

Methods |  R Precision$\uparrow$ | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\uparrow$ | MultiModality$\uparrow$
---|---|---|---|---|---
Top 1 | Top 2 | Top 3
Real motions | $0.541^{\pm.002}$ | $0.758^{\pm.002}$ | $0.850^{\pm.002}$ | $0.000^{\pm.000}$ | $3.430^{\pm.012}$ | $4.207^{\pm.071}$ | -
MotionDiffuse [100] | $0.035^{\pm.004}$ | $0.058^{\pm.005}$ | $0.098^{\pm.007}$ | $14.883^{\pm.824}$ | $4.199^{\pm.21}$ | $0.677^{\pm.018}$ | $0.655^{\pm.018}$
ReMoDiffuse [101] | $0.425^{\pm.002}$ | $0.627^{\pm.003}$ | $0.773^{\pm.003}$ | $0.131^{\pm.004}$ | $3.582^{\pm.015}$ | $4.097^{\pm.052}$ | $0.472^{\pm.009}$
InterGen [45] | $0.403^{\pm.003}$ | $0.582^{\pm.003}$ | $0.728^{\pm.003}$ | $0.082^{\pm.002}$ | $3.615^{\pm.014}$ | $\mathbf{4.186^{\pm.048}}$ | $\mathbf{0.728^{\pm.021}}$
Ours (MoMat Only) | $\mathbf{0.517^{\pm.001}}$ | $\mathbf{0.652^{\pm.001}}$ | $\mathbf{0.802^{\pm.001}}$ | $\mathbf{0.034^{\pm.000}}$ | $\mathbf{3.313^{\pm.001}}$ | $0.332^{\pm.001}$ | $0.002^{\pm.000}$
Ours (MoMat-MoGen) | $0.495^{\pm.003}$ | $0.651^{\pm.004}$ | $0.792^{\pm.004}$ | $0.071^{\pm.002}$ | $3.561^{\pm.017}$ | $4.025^{\pm.050}$ | $0.452^{\pm.012}$

Table 2: Motion Captioning results on the HumanML3D test set. Our evaluation
methodology aligns with the TM2T [29] metrics, but we uniquely utilize
unprocessed ground truth texts for calculating linguistic metrics as done in
MotionGPT [39].

Methods |  R Precision$\uparrow$ | MMDist $\downarrow$ | CIDEr $\uparrow$ | Blue@1 $\uparrow$ | Blue@4 $\uparrow$ | Rouge $\uparrow$ | BertScore $\uparrow$
---|---|---|---|---|---|---|---
Top 1 | Top 3
Real motions | $0.523$ | $0.828$ | $2.901$ | $-$ | $-$ | $-$ | $-$ | $-$
TM2T [29] | $0.516$ | $0.823$ | $2.935$ | $16.8$ | $\underline{48.9}$ | $7.00$ | $\underline{38.1}$ | $32.2$
MotionGPT [39] | $\underline{0.543}$ | $\underline{0.827}$ | $\underline{2.821}$ | $\underline{29.2}$ | $48.2$ | $\underline{12.5}$ | $37.4$ | $\underline{32.4}$
Ours | $\mathbf{0.551}$ | $\mathbf{0.832}$ | $\mathbf{2.813}$ | $\mathbf{36.2}$ | $\mathbf{51.1}$ | $\mathbf{15.5}$ | $\mathbf{41.9}$ | $\mathbf{35.0}$

###  3.5 Motion Captioning

We design a motion-to-text translation module inspired by the success of
instruction tuning in multi-modal models such as Otter [42], implemented with
a retrieval-augmented motion encoder similar to that in ReMoDiffuse [101],
with specialized modifications in its attention module to amplify motion
feature extraction via multimodal data integration. This encoder is seamlessly
integrated with the pretrained MPT-1B RedPajama language model, aligning with
structural paradigms in the text-vision domain. Due to space constraints, we
include an illustration of the module and more details in the Supplementary
Material.

##  4 Experiments

To the best of our knowledge, Digital Life Project is the first comprehensive
framework to enable autonomous social characters with articulated 3D bodies.
Hence, we break down Digital Life Project into its modules for a fair
comparison with existing arts.

###  4.1 Datasets

##### DLP-MoCap.

To establish a high-quality dataset of semantic interactive motions, we used
an optical MoCap system consisting of 30 cameras to capture body motions and
inertial sensor-based MoCap gloves to track hand motions. We hired four
professional actors/actresses (two male and two female) for more expressive
motions. We processed the MoCap data into SMPL-X [62] format by SOMA [26] on
body parameters and MANO [68] mapping for hand parameters. For accurate
semantic labeling on motions, we employed 10 human annotators to label
semantic tags and corresponding motion segments based on on-site demonstration
videos and MoCap data. DLP-MoCap comprises basic motions (22%), two-person
interaction motions (56%), and short script plays (22%), giving rise to over
nine hours of human motion at 30 FPS and over 4K text-to-motion pairs.

Figure 4: Ablation results on consistency with 95% confidence.

##### More datasets.

Besides DLP-MoCap, we also evaluated the proposed MoMat-MoGen module on
InterHuman [45], another public interactive motion dataset. Furthermore, we
evaluated the proposed motion captioning module using two key public motion-
text datasets: KIT-ML [64] and HumanML3D [28]. Due to space constraint, we
include the test results on KIT-ML and InterHuman in the Supplementary
Material.

###  4.2 Interactive Motion Synthesis

Table 1 presents a comparative analysis of our proposed interactive motion
generation method against three existing approaches: ReMoDiffuse [101],
MotionDiffuse [100], and InterGen [45]. Our both methods exhibit significant
improvements on the DLP dataset, especially in R precision, FID, and MM Dist
metrics. Moreover, our MoMat-MoGen version achieves competitive results in
terms of diversity and multi-modality. Moreover, we have achieved an
impressive balance between precision and diversity, which is essential for our
application, ensuring that the generated motions closely align with well-
matched motion references that possess strong priors while being varied and
diverse.

Figure 5: Results on controllibility with 95% confidence. Figure 6: We explore
the controllability of DLP. Given the same background, manually editing the
relationship state between characters, results in different social behaviors.
Interestingly, “couples” tend to have more intimate interactions than
“friends”. The crown indicates the active player. The story progression bar is
color-coded in accordance with the stages represented by boxes: gray boxes
represent behaviors, whereas yellow boxes represent active-passive swapping in
between behaviors. Figure 7: Our motion captioning module translates human
motion into text description, allowing a virtual character to respond to the
human player’s “fist bump”. Top Left: RGB video of the human player; Bottom
Left: motion capture [10] result; Top right: first-person view of the human-
driven character; Bottom right: third-person view of the interaction. More
details are included in the Supplementary Material.

###  4.3 Social Intelligence

To evaluate the social intelligence of SocioMind, we measure the
controllability of behaviors in short-term interactive communication and the
consistency of psychological states and plots in long-term social evolution.
Following the previous evaluation approach [60], we engage 47 human evaluators
(ranging in age from 20 to 45 years old) to review the behavioral records of
the agents. More details are included in the Supplementary Material.

####  4.3.1 Controllability

Controllability is measured by whether altering psychological traits can cause
noticeable different behaviors in short-term communication. We show evaluators
the behavioral records of 64 episodes, ask them to select the corresponding
psychological traits from the provided options, and subsequently calculate the
accuracy. Results in Fig. 5 show that SocioMind significantly outperforms
Generative Agents [60] in key attributes: central belief, motivation,
personality, and relationship, demonstrating the effective guidance of persona
instructions for the LLM in simulating interactive human behavior.

####  4.3.2 Consistency

Long-term social evolution consistency implies that the plot development and
internal state changes are coherent with initial settings. To measure this, we
use four different types of initial settings (family, crime, romance, and
military) to generate records with multiple episodes. Human evaluators use the
records to rate the degrees of consistency on plots and psychological states
on a scale of 1 to 9. Thus we evaluate the effectiveness of modules in the
SocioMind for social evolution. Results in Fig. 4 show that SocioMind
demonstrates superior performance over Generative Agents [60] on consistency
over plots and psychological states, and ablating results show that persona
instruction, psychological reflection, and planning with topic proposal are
crucial for long-term social evolution.

###  4.4 Motion Captioning

Our experiments, following TM2T [29], reveal that our model significantly
outperforms existing methods, particularly in linguistic metrics, aligning
with our design objectives, the high-quality and precise captioning.

###  4.5 Visualization

As shown in Fig. 6, our framework possesses a rational correlation between
psychological states and physical behaviors. In addition, our system has the
potential to add human players in the virtual world to interact with the
digital avatars (Fig. 7).

##  5 Conclusion

In this paper, we introduce Digital Life Project, an innovative and holistic
system that harnesses the latest advancements in generative models to create
autonomous 3D characters. DLP integrates SocioMind, a text-centric cognitive
framework that simulates sophisticated internal psychological processes, and
MoMat-MoGen, a text-driven motion synthesis pipeline that replicates diverse
external physical behaviors. Both modules achieve state-of-the-art performance
in the respective domains, enabling the entire system to engage in natural
interactions with social intelligence.

## Acknowledgement

We extend our sincere gratitude to Fei Wang, Jingyi Jiang, Jinyun Lyu, Wanying
Zu, Bo Li, Yuanhan Zhang, Jiawei Ren, and Cunjun Yu for their insights and
assistance in the areas of multi-modal models, psychological processes,
aesthetics, and the overall framework.

## References

  * Aher et al. [2023] Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai.  Using large language models to simulate multiple humans and replicate human subject studies.  In _International Conference on Machine Learning_ , pages 337–371. PMLR, 2023.
  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Ahuja and Morency [2019] Chaitanya Ahuja and Louis-Philippe Morency.  Language2pose: Natural language grounded pose forecasting.  In _2019 International Conference on 3D Vision (3DV)_ , pages 719–728. IEEE, 2019.
  * Altman and Taylor [1973] Irwin Altman and Dalmas A Taylor.  _Social penetration: The development of interpersonal relationships._ Holt, Rinehart & Winston, 1973.
  * Argyle et al. [2023] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ , 2023.
  * Averell and Heathcote [2011] Lee Averell and Andrew Heathcote.  The form of the forgetting curve and the fate of memories.  _Journal of mathematical psychology_ , 2011.
  * Bandura and Walters [1977] Albert Bandura and Richard H Walters.  _Social learning theory_.  Englewood cliffs Prentice Hall, 1977.
  * Berger et al. [2010] Charles R Berger, Michael E Roloff, and David R Ewoldsen.  _The handbook of communication science_.  Sage, 2010.
  * Buckner et al. [2008] Randy L Buckner, Jessica R Andrews-Hanna, and Daniel L Schacter.  The brain’s default network: anatomy, function, and relevance to disease.  _Annals of the new York Academy of Sciences_ , 2008.
  * Cai et al. [2023] Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Yanjun Wang, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al.  Smpler-x: Scaling up expressive human pose and shape estimation.  _arXiv preprint arXiv:2309.17448_ , 2023.
  * Chen et al. [2023] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu.  Executing your commands via motion diffusion in latent space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18000–18010, 2023.
  * Chin-Yew [2004] Lin Chin-Yew.  Rouge: A package for automatic evaluation of summaries.  In _Proceedings of the Workshop on Text Summarization Branches Out, 2004_ , 2004.
  * Chiu et al. [1997] Chi-yue Chiu, Ying-yi Hong, and Carol S Dweck.  Lay dispositionism and implicit theories of personality.  _Journal of personality and social psychology_ , 73(1):19, 1997.
  * Cialdini and Goldstein [2004] Robert B Cialdini and Noah J Goldstein.  Social influence: Compliance and conformity.  _Annu. Rev. Psychol._ , 2004.
  * Clavet [2016] Simon Clavet.  Motion matching and the road to next-gen animation.  In _Proc. of GDC_ , 2016.
  * Cohen [2004] Sheldon Cohen.  Social relationships and health.  _American psychologist_ , 2004.
  * Cohen and Wills [1985] Sheldon Cohen and Thomas A Wills.  Stress, social support, and the buffering hypothesis.  _Psychological bulletin_ , 1985.
  * Costa and McCrae [2008] Paul T Costa and Robert R McCrae.  The revised neo personality inventory (neo-pi-r).  _The SAGE handbook of personality theory and assessment_ , pages 179–198, 2008.
  * DeYoung et al. [2007] Colin G DeYoung, Lena C Quilty, and Jordan B Peterson.  Between facets and domains: 10 aspects of the big five.  _Journal of personality and social psychology_ , 2007.
  * Du et al. [2021] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  Glm: General language model pretraining with autoregressive blank infilling.  _arXiv preprint arXiv:2103.10360_ , 2021.
  * du Plessis and de Bruin [2015] Graham A du Plessis and Gideon P de Bruin.  Using rasch modelling to examine the international personality item pool (ipip) values in action (via) measure of character strengths.  _Journal of Psychology in Africa_ , 2015.
  * Duncan Jr [1969] Starkey Duncan Jr.  Nonverbal communication.  _Psychological Bulletin_ , 1969.
  * Ebbinghaus [2013] Hermann Ebbinghaus.  Memory: A contribution to experimental psychology.  _Annals of neurosciences_ , 2013.
  * Eysenck and Eysenck [1975] Hans Jurgen Eysenck and Sybil Bianca Giuletta Eysenck.  _Manual of the Eysenck Personality Questionnaire (junior & adult)_.  Hodder and Stoughton Educational, 1975.
  * Gao et al. [2023] Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Li.  S3: Social-network simulation system with large language model-empowered agents.  _arXiv preprint arXiv:2307.14984_ , 2023.
  * Ghorbani and Black [2021] Nima Ghorbani and Michael J. Black.  SOMA: Solving optical marker-based mocap automatically.  In _Proc. International Conference on Computer Vision (ICCV)_ , pages 11117–11126, 2021.
  * Ghosh et al. [2021] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek.  Synthesis of compositional animations from textual descriptions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1396–1406, 2021.
  * Guo et al. [2022a] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5152–5161, 2022a.
  * Guo et al. [2022b] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts, 2022b.
  * Hall [1966] Edward Twitchell Hall.  _The hidden dimension_.  Anchor, 1966.
  * Hall [1973] Edward T Hall.  _The silent language_.  Anchor, 1973.
  * Hathaway and McKinley [1940] Starke R Hathaway and John C McKinley.  A multiphasic personality schedule (minnesota): I. construction of the schedule.  _The Journal of Psychology_ , 1940.
  * Heider [2013] Fritz Heider.  _The psychology of interpersonal relations_.  Psychology Press, 2013.
  * Higgins [1987] E Tory Higgins.  Self-discrepancy: a theory relating self and affect.  _Psychological review_ , 1987.
  * Holden et al. [2017] Daniel Holden, Taku Komura, and Jun Saito.  Phase-functioned neural networks for character control.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–13, 2017.
  * Holden et al. [2020] Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa.  Learned motion matching.  _ACM TOG_ , 39(4):53–1, 2020.
  * Huang et al. [2022] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.  Inner monologue: Embodied reasoning through planning with language models.  _arXiv preprint arXiv:2207.05608_ , 2022.
  * James [2007] William James.  _The principles of psychology_.  Cosimo, Inc., 2007.
  * Jiang et al. [2023] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen.  Motiongpt: Human motion as a foreign language.  _arXiv preprint arXiv:2306.14795_ , 2023.
  * John et al. [1999] Oliver P John, Sanjay Srivastava, et al.  The big-five trait taxonomy: History, measurement, and theoretical perspectives.  1999\.
  * Kim et al. [2023] Jihoon Kim, Jiseob Kim, and Sungjoon Choi.  Flame: Free-form language-based motion synthesis & editing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 8255–8263, 2023.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_ , 2023a.
  * Li et al. [2023b] Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, and Cewu Lu.  Hybrik-x: Hybrid analytical-neural inverse kinematics for whole-body mesh recovery.  _arXiv preprint arXiv:2304.05690_ , 2023b.
  * Li et al. [2022] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan.  Cliff: Carrying location information in full frames into human pose and shape estimation.  In _European Conference on Computer Vision_ , pages 590–606. Springer, 2022.
  * Liang et al. [2023] Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu.  Intergen: Diffusion-based multi-human motion generation under complex interactions.  _arXiv preprint arXiv:2304.05684_ , 2023.
  * Lin et al. [2023] Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li.  One-stage 3d whole-body mesh recovery with component aware transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21159–21168, 2023.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv:1907.11692_ , 2019.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Maslow [1958] Abraham Harold Maslow.  A dynamic theory of human motivation.  1958\.
  * Mehrabian [1980] Albert Mehrabian.  Basic dimensions for a general psychological theory: Implications for personality, social, environmental, and developmental studies.  1980\.
  * Myers et al. [1985] Isabel Briggs Myers, Mary H McCaulley, and Robert Most.  Manual: A guide to the development and use of the myers-briggs type indicator.  _(No Title)_ , 1985.
  * Newcomb [1956] Theodore M Newcomb.  The prediction of interpersonal attraction.  _American psychologist_ , 1956.
  * Newell et al. [1989] Allen Newell, Paul S Rosenbloom, and John E Laird.  Symbolic architectures for cognition.  _Foundations of cognitive science_ , 1989.
  * Nishida [2008] Toyoaki Nishida.  _Conversational informatics: An engineering approach_.  John Wiley & Sons, 2008.
  * OpenAI [2022] OpenAI.  New and improved embedding model, 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Park et al. [2022] Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Social simulacra: Creating populated prototypes for social computing systems.  In _UIST_ , 2022.
  * Park et al. [2023a] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  In _In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23)_ , New York, NY, USA, 2023a. Association for Computing Machinery.
  * Park et al. [2023b] Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  In _UIST_ , 2023b.
  * Patil et al. [2023] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_ , 2023.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10975–10985, 2019.
  * Petrovich et al. [2022] Mathis Petrovich, Michael J Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions.  In _European Conference on Computer Vision_ , pages 480–497. Springer, 2022.
  * Plappert et al. [2016] Matthias Plappert, Christian Mandery, and Tamim Asfour.  The kit motion-language dataset.  _Big data_ , 4(4):236–252, 2016.
  * Qing et al. [2023] Zhongfei Qing, Zhongang Cai, Zhitao Yang, and Lei Yang.  Story-to-motion: Synthesizing infinite and controllable character animation from long text.  _arXiv preprint arXiv:2311.07446_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rempel et al. [1985] John K Rempel, John G Holmes, and Mark P Zanna.  Trust in close relationships.  _Journal of personality and social psychology_ , 1985.
  * Romero et al. [2017] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 36(6), 2017.
  * Rotter [1967] Julian B Rotter.  A new scale for the measurement of interpersonal trust.  _Journal of personality_ , 1967.
  * Safdari et al. [2023] Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić.  Personality traits in large language models.  _arXiv preprint arXiv:2307.00184_ , 2023.
  * Schachter and Singer [1962] Stanley Schachter and Jerome Singer.  Cognitive, social, and physiological determinants of emotional state.  _Psychological review_ , 1962.
  * Schick et al. [2023] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Schulman et al. [2022] John Schulman, Barret Zoph, C Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes, and Sengjia Zhao.  Chatgpt: Optimizing language models for dialogue.  2022\.
  * Shafir et al. [2023] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano.  Human motion diffusion as a generative prior.  _arXiv preprint arXiv:2303.01418_ , 2023.
  * Sharon et al. [2015] Guni Sharon, Roni Stern, Ariel Felner, and Nathan R Sturtevant.  Conflict-based search for optimal multi-agent pathfinding.  _AI_ , 219:40–66, 2015.
  * Shen et al. [2023] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_ , 2023.
  * Simms et al. [2011] Leonard J Simms, Lewis R Goldberg, John E Roberts, David Watson, John Welte, and Jane H Rotterman.  Computerized adaptive assessment of personality disorder: Introducing the cat–pd project.  _Journal of personality assessment_ , 2011.
  * Sumers et al. [2023] Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths.  Cognitive architectures for language agents.  _arXiv preprint arXiv:2309.02427_ , 2023.
  * Tavakol and Dennick [2011] Mohsen Tavakol and Reg Dennick.  Making sense of cronbach’s alpha.  _International journal of medical education_ , 2:53, 2011.
  * Tevet et al. [2022a] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  In _European Conference on Computer Vision_ , pages 358–374. Springer, 2022a.
  * Tevet et al. [2022b] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano.  Human motion diffusion model.  In _The Eleventh International Conference on Learning Representations_ , 2022b.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Vallacher and Wegner [1987] Robin R Vallacher and Daniel M Wegner.  What do people think they’re doing? action identification and human behavior.  _Psychological review_ , 1987.
  * [84] Lucas Veber.  Auto-rig pro.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575, 2015.
  * Wang et al. [2023a] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ , 2023a.
  * Wang et al. [2023b] Wenjia Wang, Yongtao Ge, Haiyi Mei, Zhongang Cai, Qingping Sun, Yanjun Wang, Chunhua Shen, Lei Yang, and Taku Komura.  Zolly: Zoom focal length correctly for perspective-distorted human mesh reconstruction.  _arXiv preprint arXiv:2303.13796_ , 2023b.
  * Wang et al. [2023c] Yin Wang, Zhiying Leng, Frederick W. B. Li, Shun-Cheng Wu, and Xiaohui Liang.  Fg-t2m: Fine-grained text-driven human motion generation via diffusion model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 22035–22044, 2023c.
  * Wei et al. [2022a] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. [2022b] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _NeurIPS_ , 2022b.
  * Xu et al. [2023] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu.  Exploring large language models for communication games: An empirical study on werewolf.  _arXiv preprint arXiv:2309.04658_ , 2023.
  * Xu et al. [2022] Zhiyang Xu, Ying Shen, and Lifu Huang.  Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning.  _arXiv preprint arXiv:2212.10773_ , 2022.
  * Yao et al. [2023a] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_ , 2023a.
  * Yao et al. [2023b] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In _ICLR_ , 2023b.
  * Yi et al. [2023] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black.  Generating holistic 3d human motion from speech.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 469–480, 2023.
  * Yuan et al. [2023] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz.  Physdiff: Physics-guided human motion diffusion model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 16010–16021, 2023.
  * Zhang et al. [2023a] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan.  Building cooperative embodied agents modularly with large language models.  _arXiv preprint arXiv:2307.02485_ , 2023a.
  * Zhang et al. [2023b] Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu.  Pymaf-x: Towards well-aligned full-body model regression from monocular images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023b.
  * Zhang et al. [2023c] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023c.
  * Zhang et al. [2022] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _arXiv preprint arXiv:2208.15001_ , 2022.
  * Zhang et al. [2023d] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu.  Remodiffuse: Retrieval-augmented motion diffusion model.  _arXiv preprint arXiv:2304.01116_ , 2023d.
  * Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5745–5753, 2019.
  * Zhu et al. [2023a] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023a.
  * Zhu et al. [2023b] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al.  Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory.  _arXiv preprint arXiv:2305.17144_ , 2023b.

Supplementary Material

##  Appendix A Overview of the Supplementary Material

We provide more details of DLP, such as the method (Appendix B), the DLP-MoCap
dataset (Appendix C), the experiments (Appendix D) and additional discussions
(Appendix E).

##### Demo Video.

A video demonstration is attached as a part of the Supplementary Material. In
addition to “$<$motion$>$”, we explore using the “$<$speech$>$” token to
generate audio with OpenAI’s text-to-speech engine, followed by Talkshow [95]
to synthesize face movements.

##  Appendix B Additional Details of DLP

###  B.1 Scene

We aim to build 3D characters that articulate with 3D body motions in a 3D
scene. In this work, we design our scene to achieve a diverse combination of
relative interaction poses: 1) the center or bookshelf for the cases when both
characters are standing (standing-standing interaction), 2) a sofa for side-
by-side seated interaction, 3) a dining table for face-to-face seated
interaction, and 4) a computer desk for standing-seated interaction. For each
piece of interactable furniture, we designed spots with positions and facings
to guide navigation and human-scene interaction. Note that we focus on human-
human interaction in this work and substantially simplify the human-scene
interaction to basic ones such as “sitting down on a chair” or “standing up
from a sofa”.

###  B.2 Movement Synchronization

Figure 8: Movement synchronization. Each behavior results in an interactive
motion pair, that consists of individual movements in the scene and
interaction between two characters. To allow characters to navigate in the
scene while maintaining synchronization, there are four stages in the motion
synthesis process. The crown indicates the active character status, which may
or may not swap between characters. APM stands for Active-Passive Mechanism.
Fig. 2 depicts mainly the behave stage.

A significant challenge of extending single human motion synthesis to
interactive motion synthesis is coordinating multiple characters. Although our
Active-passive Mechanism ensures the interaction is naturally aligned through
retrieving from a curated motion database, we need to accommodate potential
mismatches in motion lengths when the characters navigate from one place in
the scene to another. As shown in Fig. 8, each interaction (interactive motion
pair) requires four steps to synchronize the character’s motions.

  * •

Behave. In this stage, the active character generates active behavior from its
SocioMind and synthesizes both active and passive motions with MoMat-MoGen.
The passive motion is passed to the passive character. Since the behavior
contains “$<$place$>$” token for the location information (_e.g_., desk or
sofa), both characters will register their current position as the starting
position with the multi-agent pathfinder [75]. Note that at this stage, there
are no paths planned or actual motion executed yet.

  * •

Move. The multi-agent pathfinder computes collision-free trajectories for both
characters. Motion matching is used to synthesize the walking paths if there
is a location change. For a more complicated case that involves a change in
the basic motion state (_e.g_., from a standing pose in center to a seated
pose in sofa), there will be additional motion inserted in front (_e.g_.,
standing up) or after (_e.g_., sitting down) the walking trajectory.

  * •

Align. Depending on the starting positions of the characters, the movements
typically take a different number of frames for each character. Hence, an
alignment is conducted: two characters communicate with each other about their
trajectory, and the character with a shorter trajectory has a filler motion
(an idle motion) inserted at the end of its trajectory, depending on its basic
state (seated or standing).

  * •

Synthesize. Movement motions are concatenated with the interaction motions.
Motion blending is applied to ensure a smooth transition between motion clips.

###  B.3 Retargeting

In this work, the proposed MoMat-MoGen module could produce high-quality
motions in SMPL-X [62] format. To better demonstrate the physical and mental
interaction between our social agents in an immersive simulation scenario, two
rigged characters are used: a male named Zhixu and a female named Xiaotao. The
synthesized motions are retargeted to the target avatar in Blender using a
widely used retargeting tool Auto-Rig Pro [84]. In order to bridge the gap
between different skeletons, the bone mapping between different structures is
manually configured for the best performance. In addition, we rescale the
target avatars to have the identical height as SMPL-X models to avoid any
noticeable foot skating and preserve body contact for the interaction between
the two characters. Note that the retargeting pipeline can be extended to more
characters in the future.

###  B.4 Motion Matching

Our motion matching process consists of two steps. Firstly, we incorporate
semantic information by utilizing a pre-trained LLM [47] to extract a text
embedding $f_{t}\in\mathbb{R}^{1024}$ for the query text. Secondly, to enhance
coherence and alignment with the query trajectory, we incorporate kinematics
features. Specifically, for a motion with $k$ frames, the kinematics features
are defined as $x=\\{\textbf{t}\ \textbf{f}\ \textbf{b}\ \textbf{h}\
\textbf{p}\\}\in\mathbb{R}^{5k+193}$, where $\textbf{t}\in\mathbb{R}^{2k}$
represents the trajectory position projected on the ground,
$\textbf{f}\in\mathbb{R}^{3k}$ denotes the facing direction,
$\textbf{b}\in\mathbb{R}^{189}$ represents the 6D space rotation and the
position of 21 joints, $\textbf{h}\in\mathbb{R}$ indicates the hip height, and
$\textbf{p}\in\mathbb{R}^{3}$ represents the relative position of other
characters. The trajectory positions t and facing f align the resulting motion
with the query trajectory, the body-pose features b improve body-pose
coherence with a higher emphasis on foot weighting, the hip height h
distinguishes seated motion from standing motion, and the relative position p
aligns with other characters. During motion matching, the query trajectory
obtained from the path-finding algorithm is used to calculate trajectory
similarity $\mathcal{T}$ and facing similarity $\mathcal{F}$, while the
current pose is used to calculate body-pose similarity $\mathcal{B}$, hip
similarity $\mathcal{H}$, and relative position similarity $\mathcal{P}$.
Euclidean distance is used for all similarities except for facing similarity,
which employs cosine distance. These similarities are normalized using Z-score
normalization, and the final similarity $\mathcal{S}$ is obtained as a
weighted sum of these similarities.

###  B.5 Motion Generation

####  B.5.1 Architecture Details

Figure 9: Architecture Detail of Dual Semantic-Modulated Attention (DSMA)
module.

Our proposed Dual Semantic-Modulated Attention (DSMA) module is built upon the
SMA module in ReMoDiffuse [101]. The major difference is the introduction of
motion interaction. The architecture is shown in Fig. 9. To get the refined
feature $\widehat{f}_{x}$ , we integrate four sources of features: 1)
independent motion feature motion $f_{x}$ from the same sequence; 2) twin
motion feature $f_{y}$ from the partner sequence; 3) text feature $p_{x}$ from
the given description $P_{x}$; 4) reference feature $r_{x}$ from the motion
matching results $M_{x}$. In addition, We share parameter weights of DSMA
modules and FFN modules to process both actors’ motion sequences.

###  B.6 SocioMind

In this section, we illustrate the details of each module in SocioMind, along
with the corresponding prompt templates. We introduce the definition of
psychological states in Sec. B.6.1, the construction of persona instructions
in Sec. B.6.2, events and thoughts within the memory system in Sec. B.6.3,
prompts for short-term communication in Sec. B.6.4, psychological reflections
in Sec. B.6.5, and the details of the topic proposal mechanism in Sec. B.6.6.

####  B.6.1 Psychological States

Determining whether a 3D character qualifies as a digital life with social
intelligence remains an open problem. From a psychological perspective, humans
are composed of internal psychological processes (mind, such as thoughts,
emotions, _etc_.) and external behaviors [38]. Through prolonged studies on
the association between internal processes and external behaviors,
psychologists have developed various theories, including Big Five Trait [40]
on personality, PAD model [71, 50] on emotion, hierarchy of needs [49] and
long-short term theory [83] on motivation, self-schema [34] on core self,
episodic and semantic system [9] on memory, attitude [14], intimacy [52],
supportiveness [17, 16] and trust [67, 69] on social relationships. Herein, we
introduce the psychological states in our SocioMind, enabling the simulation
of controllable human communicative behaviors. The psychological states are as
follows:

  * •

For personality, we use Big Five Trait model [40], which comprises five
dimensions: openness (O), conscientiousness (C), extraversion (E),
agreeableness (A), and neuroticism (N). Users can provide numerical values for
these five dimensions (Likert scale, ranging from 1 to 9) or textual
descriptions. The personality is set initially.

  * •

For emotions, we choose the PAD model [71], commonly used in body language
[22] and animations [54], consisting of three dimensions: pleasure (how
positive or negative), arousal (level of mental alertness and physical
activity), dominance (amount of control and influence) in a Likert scale from
1 to 9.

  * •

In terms of motivation, we apply long-term and short-term motivations as
propose by Robin _et al_. [83]. Long-term motivations are specified in the
initial setting.

  * •

For the self, we emphasize central beliefs, reflecting an individual’s
worldview [34].

  * •

For social relationships, we introducing three dimensions based on social
support theory [17, 16], social trust theory [67, 69], and research on
intimate relationships [52]: trust, intimacy, and supportiveness in a Likert
scale from 1 to 9. Additionally, we configure an attribute representing the
attitude towards others with text description.

Figure 10: An example of the psychological states of an autonomous character,
Young Jack. The background is ‘Old Jack traverses through time and engages in
communication with his younger self.’

Fig. 10 is an example of a character’s psychological states. All psychological
states can be defined in textual form, wherein personality, emotions, and
social relationships can also be delineated through quantifiable values. Users
can define a character’s personality, emotions, and social relationships
either through text or by adjusting the numerical values of corresponding
variable dimensions. In subsequent reasoning based on LLM, both text and
numerical descriptions serve as inputs to the LLM as prompts. The
transformation between text and numerical values is facilitated through LLM.
For instance, we use the LLM to translate the numerical values of emotion
dimensions into text descriptions based on the PAD [71] model. The form of the
prompt is as follows:

Prompt for translating numerical values into text Assume you are a very
professional psychologist.
This is a quantitative evaluation of a person: [pleasure: 3, arousal: 5,
dominance: 4]. The evaluation based on the PAD theory of psychology and Likert
scale (1-9), score on pleasure, arousal and dominance. Based on the score,
describe the emotion of the person. According to Paul Ekman’s basic emotion
theory, human have basic emotions: wrath, grossness, fear, joy, loneliness,
shock, amusement, contempt, contentment, embarrassment, excitement, guilt,
pride in achievement, relief, satisfaction, sensory pleasure, and shame. So
the description should be:

And the result may be:

Output Mild dissatisfaction or discontentment, coupled with a sense of
alertness but not empowerment.

Figure 11: Illustration of our reverse engineering to transfer the
psychological test into persona instructions. In the table of IPIP,
‘instrument’ is the name of the personality inventory, ‘alpha’ is the Cronbach
alpha reliability [79], ‘key’ is the keyed direction (+1 for positive and -1
for negative) of the item and its associated construct, the text is the
behavior item, and ‘label’ is the trait dimension.

####  B.6.2 Persona Instructions

To enhance the LLM’s capability in reasoning the coherent association between
internal psychological processes and external behaviors, we design a reverse
engineering to construct the persona instruction database from psychological
tests. Persona instructions serves as few-shot exemplars for Chain of Thought
(CoT) reasoning to generate plausible behaviors aligned with human
expectations. In International Personality Item Pool (IPIP) [19, 21, 77], an
open-source psychological tool, its original data illustrate how external
behaviors can be translated into quantifiable psychological metrics. As shown
in Fig. 11, through reverse engineering, we represent each item in IPIP as a
persona instruction: “A person with extend trait dimension tends to
behave/think: behavior”, where extend is ‘high’ when key is 1, and ‘low’ when
key is -1. In short-term communication, we retrieve the most relevant persona
instructions by the similarity of the text embeddings from function $\phi$
[55] with current behaviors and psychological states.

####  B.6.3 Events & Thoughts

Following the framework of the cognitive language agent [78, 60] , there are
two types of episodic memories within our memory system: ‘events’ and
‘thoughts’. Events represent occurrences or facts perceived by the agent,
whereas thoughts are ideas, musings, or attitudes generated by the agent based
on its personality and past experiences. The memory system of Generative
Agents becomes larger with time, which poses challenges for retrieving the
most relevant events and thoughts [60]. Inspired by human memory system [23],
we introduce mechanisms for memory reinforcement and forgetting to alleviate
this issue. Each event or thought in our memory system has parameters such as
poignancy $p_{m}$ (ranging from 1 to 9), text description $D_{m}$, keywords,
and the accessed times $N_{m}$. To ensure the efficiency of our memory system,
we introduce a forgetting mechanism. Initially, we utilize the Ebbinghaus
forgetting curve [23, 6] to calculate the forgetting rate $r$:

$\displaystyle r=[a+(1-a)\cdot e^{-k\cdot\frac{\Delta T}{2^{N_{m}}\cdot
p_{m}}}],$ (4)

where $\Delta T$ represents the time elapsed (the number of intervening
episodes) since the last recollection, $a$ and $k$ are hype-parameters. When
$r$ falls below the threshold $T_{f}$ , the event or thought will be forgotten
and not retrieved.

During memory retrieval, we combine content similarity and the forgetting rate
to compute the final score:

$\displaystyle s(D_{b},D_{m})$
$\displaystyle=cos(\phi(D_{b}),\phi(D_{m}))\cdot r$ (5)

where $cos(\phi(D_{b}),\phi(D_{m}))$ denotes the similarity between the
current behavior $b$ and the memory item $m$ (event or thought) using text
embedding function $\phi$ [55]. We retrieve the top $M$ memory items as prompt
input, enabling the avatar to learn interaction strategies from past
experiences.

Prompt for generating behaviors {“role”: “system”, “content”: “Let’s do a role
play. Assume you are a person named Young Jack (SELF NAME). I’m a person with
name Old Jack (SELF NAME). In this episode, you have such psychological
states:
Personality: stubborn, willful, playful, …; Emotion: confused and curious,
pleasure: 3, …; Motivation: just for fun, such as eating delicious food, …;
…(PSYCHOLOGICAL STATES) The episode background is:
Old Jack decides to share his regrets with Young Jack about not being able to
spend more time with his mother before she passed away. (BACKGROUND) The
topics you want to start are:
regret of limited time with mother. (TOPICS) Now start our conversation.”}
{“role”: “assistant”, “content”: “<speech> Hello!<motion>waving
hands<place>sofa”} {“role”: “user”, “content”: “<speech> Hi!<motion>stand
up<place>sofa”} …… {“role”: “user”, “content”: “ <Memories hold joy and pain.>
Hi!<motion>lean back against the bookshelf<place>bookshelf”} (BEHAVIOR
CONTEXT) {“role”: “user”, “content”: “You have such relevant memories:
Mom told Young Jack there were ghosts;The old man wanted to share photos with
him.(RELEVANT EVENTS & THOUGHTS) Psychological research has found such
principles:
A person with high curiosity tends to behave/think: seek explanations of
things.(PERSONA INSTRUCTIONS) Now you have two options: end the conversation
by output ‘END’ or respond based on the information above. So your reaction
is: Figure 12: Prompts for generating interactive behaviors in short-term
communication. The use of ALL CAPITAL LETTERS signifies the reference to the
complete textual description corresponding to the respective names in this
paper.

####  B.6.4 Short-Term Communication Generation

Human responses to external stimuli are influenced on one hand by external
environmental factors (episode background, interactive behavior contexts), and
on the other hand by internal factors (psychological states, relevant
memories, and topics). The correlation between behavior and psychology also
follows certain patterns (persona instructions). Therefore, to generate
interactive behaviors autonomously, we feed LLMs with these factors for
reasoning human-like behaviors. The specific prompt is shown in Fig. 12. In
this paper, the use of ALL CAPITAL LETTERS signifies the reference to the
complete textual description corresponding to the respective names. And a
possible response from LLM is:

Output <speech> Old Jack, do you think that memories are like ghosts? They
can’t be seen but they linger with us, bringing both joy and
sadness.<motion>Tracing fingers on bookshelf<place>bookshelf

The format of the output results may not align with the target format. In such
instances, we use LLMs to reformat the output.

####  B.6.5 Psychological Reflection

Prompt for summarizing events Assume you are a person named NAME. You have
such psychological states: PSYCHOLOGICAL STATES Psychological research has
found such principles:  PERSONA INSTRUCTIONS Now you have a conversation and
the contexts are as follow: BEHAVIOR CONTEXTS Based on the dialog above,
summarize the key events and output the event list.

Psychological research reveals that humans possess a reflection system,
wherein the brain learns from past events, gradually altering its beliefs and
attitudes towards others [14, 67, 16, 4]. Although Generative Agents [60]
adopts a reflection mechanism, it does not adequately consider the multiple
factors on a character’s long-term social intelligence, such as attitudes,
intimacy, beliefs, and motivations. Consequently, it can not finely simulate
the social evolution process between two characters. Therefore, we have
developed a hierarchical system of reflection to mimic psychological
processes. After each episode, we use LLMs to summarize the events from the
communication (Prompt for summarizing events). One example of the output is:

Output [{ “description”: “Young Jack asks Old Jack about the bookshelf and
shows curiosity about the memories”, “keywords”: [“bookshelf”, “curiosity”,
“memories”], “poignancy”: 7, “emergency”: 4}, …]

Then the brain generates its own new thoughts based on current events and
relevant past events and thoughts. The structure of ‘thought’ is fundamentally
similar to that of ‘event’ in our memory system, with the key distinction
lying in its integration of past events and thoughts to generate new thoughts.
Based on these events and thoughts, the brain update its motivations, central
belief, and social relationships, thus resulting in social evolution on
internal states. For instance, the brain updates the social relationship with
prompt:

Prompt for summarizing events Assume you are a very professional psychologist.
Here is a person named NAME. He/She has such psychological states:
PSYCHOLOGICAL STATES Psychological research has found such principles:
PERSONA INSTRUCTIONS Recent he/she have come across these events and the
following thoughts have arisen: EVENTS & THOUGHTS His/Her previous
relationship with  PARTNER NAME is: SOCIAL RELATIONSHIP Output the
relationship according to social psychological theory in three dimensions:
trust, intimacy, and supportiveness.

The updated social relationship would be:

Output { “description”: “They are getting to know each other”, “intimacy”: 3,
“trust”: 4, “supportiveness”: 2, “attitude”: “curious”}

####  B.6.6 Planning with Topic Proposal

To build the long-term evolution of external behaviors, we propose a planning
module with topic proposal mechanism to promote the development of storylines.
This approach is partly inspired by the psychological evolution often linked
to significant events [38] and partly by techniques used in movies and dramas
to progress narratives. Initially, the brain correlates past experiences with
newly occurring events to propose topics for the next scene. At this stage, we
also allow users to manually incorporate events not generated within the
interaction, such as fragments of past memories and contemporary news events.

Prompt for proposing topics Here is a person named NAME. He/She has such
psychological states: PSYCHOLOGICAL STATES Psychological research has found
such principles:  PERSONA INSTRUCTIONS The person has experienced these
stories: EPISODE BACKGROUNDS Recent he/she have come across these events:
CURRENT & MANUAL EVENTS From the memory, he/she has such relevant events and
thoughts:
RELEVANT EVENTS & THOUGHTS Based on the information above, generate a list of
topics that he/she would like to start to talk.

One of the examples of the topics are:

Output [{ “description”: “Sneaking in some extra time on CS Go”, “poignancy”:
7, “emergency”: 6,}, …]

Each topic has its poignancy and emergency ranging from 1 to 9. Based on these
topics, the brain generates the candidates of the background and initial
settings for the next episode. The initial settings include the motions,
places, and emotions of the two characters, along with the emergency and
poignancy of the candidates. The prompt for generating backgrounds is as
follows:

Prompt for generating backgrounds Here is a person named NAME. He/She has such
psychological states: PSYCHOLOGICAL STATES Psychological research has found
such principles:  PERSONA INSTRUCTIONS The person has experienced these
stories: PAST BACKGROUNDS & TOPICS Below are the topic candidates: TOPIC
CANDIDATES Based on the information above, generate a list of backgrounds for
the next episode.

An example of the output is:

Output [{ “background”: “Young Jack is sulking after a scolding”, “poignancy”:
7, “emergency”: 6, “topic ids”: [0, 1], “initial setting” : { “Young Jack”:
{“emotion”: “shame, sadness”, “place”: “sofa”, “motion”: “slouch on the
sofa”}, “Old Jack”: { “emotion”: “sympathy, contentment”, “place”: “desk”,
“motion”: “lean against the desk” }}}, …]

where ‘topic ids’ refers to the indices of topic candidates presented in the
prompt.

Since each character proposes candidates for the background of the next
episode, they inform each other of their proposed options. By balancing the
levels of emergency and poignancy, they select the candidate with the highest
score $s_{\text{bg}}$ as the background for the next episode. we get the
$s_{\text{bg}}$ as follow:

$s_{\text{bg}}=\lambda\cdot e+p,$ (6)

where $e$ is the emergency, $p$ is the poignancy, and $\lambda$ is set to 2 in
our experiments.

###  B.7 Motion Captioning

In this section, we first supplement the main paper with details of the
potential usage of motion captioning in our DLP framework (Sec. B.7.1). We
then provide more details of our study on motion captioning, including an
overview of related work in Sec. B.7.2, followed by an in-depth exploration of
our method in Sec. B.7.3, and a comprehensive discussion of our training
strategy and data preparation techniques in Sec. B.7.4.

####  B.7.1 Details of Fig. 7

In Fig. 7, we leverage SMPLer-X [10] to capture the human player’s motion from
the RGB video, captured with a Kinect Azure. The captured SMPL-X sequence is
then passed to the motion captioning module to obtain the text description.
Then the motion description is written in the behavior format, with empty
speech and predefined place (_e.g_., center). One character is set to
represent the human player, which always holds the active character status,
and has its SocioMind overridden by the motion captioning module. The rest of
the pipeline is the same as depicted in Fig. 2.

####  B.7.2 Related Works of Motion Captioning

Motion captioning is essential for the accurate description and interpretation
of human movements. Human motion is conventionally represented in two
modalities: 2D video and 3D parametric data. The intricacies of human joint
movements and the complexities inherent in body priors make 2D video an
inadequate medium for a detailed and comprehensive representation of motion.
Consequently, the use of 3D parametric motion representation, as advanced by
Guo et al. [28], has gained prominence in the field of human-motion analysis,
attributed to its superior representational capabilities. Historically,
acquiring 3D parametric human motion data from real-world scenarios was
challenging. However, recent advancements in vision-based motion capture,
especially in deriving 3D parametric models from monocular videos [44, 87, 98,
46, 43, 10], have enabled the effective reconstruction of 3D human motion from
2D footage. In the realm of motion captioning, innovative methodologies such
as TM2T [29] and MotionGPT [39], which utilize 3D parametric data, have
demonstrated potential. TM2T [29] introduces a novel approach by compressing
motion sequences into discrete variables, coupled with a neural translation
network for effective modality mapping and captioning. Similarly, MotionGPT
[39] employs a strategy of motion tokenization as well, integrated with a
motion-aware language model, to facilitate caption generation. Despite these
advancements, both methods have limitations in their discrete motion
representation, potentially leading to the omission of critical motion
features. Furthermore, the absence of an end-to-end training framework in
these models poses significant challenges in terms of practical implementation
and usability.

####  B.7.3 Method of Motion Captioning

Our “eye”, the motion captioning module, utilizing 3D parametric data [29]
tailored for human motion analysis, is crucial for perceiving and translating
user-generated motion into text. This approach, favoring structured and
detailed 3D representation with inherent human motion priors over 2D motion
features, aligns well with recent advancements in vision-based motion capture
[44, 87, 98, 46, 43, 10], aiding efficient 3D data extraction. Despite
progress, challenges in accuracy and linguistic interpretation with current 3D
data-based motion captioning methods [39, 29] remain. To address these, we
adopt the multimodal instruction learning paradigm [104, 92, 42], proven in
text-vision domains, to enhance our ability to interpret complex motions and
produce coherent, accurate descriptions. In Figure 13, we effectively
integrate a retrieval-augmented motion encoder with the MPT-1B Red-Pajama
language model, following text-vision domain structural paradigms. This
enhances motion feature representation for better language guidance and
leverages prior knowledge for improved motion captioning.

Figure 13: The architecture of our motion captioning module. First, the video
undergoes processing to extract 3D motion parameters in (a). These 3D
paramatric data are then processed using a retrieval-augmented motion encoder
as shown in (b.1), (b.2), and the resulting motion features are fed into the
language module’s gated cross attention layers to guide language generation,
as shown in (c).

##### Retrieval-Augmented Motion Encoder.

Our motion encoder is designed to efficiently extract and integrate motion
features with textual information, ensuring seamless interaction with the
subsequent language model. Inspired by multimodal feature retrieval’s
effectiveness in augmenting generation, notably the ReMoDiffuse [101] in text-
guided motion generation, we have designed and incorporated a retrieval-
augmented motion encoder into our workflow.

Our process starts with building a multimodal retrieval database from our
extensive training dataset, containing paired motion and caption data. For
each input motion sequence $\Theta$ with length ${L}$, we first extract its
features $f_{\Theta}=E_{m}(\Theta)$ using a pretrained motion feature
extractor $E_{m}$. This extractor is obtained through contrastive learning in
conjunction with the CLIP [66] text model. Then we identify similar samples
for $\Theta$ from the database. This similarity score $s_{i}$ between the
database ${i}$-th data point $(\Theta_{i},\mathrm{text}_{i})$ and the given
motion sequence $\Theta$ is determined by a scoring mechanism that evaluates
both motion feature and sequence length similarity, akin to the method used in
ReMoDiffuse [101]:

$\begin{split}s_{i}=<f_{i}^{m},&f_{\Theta}>\cdot e^{-\lambda\cdot\gamma},\\\
f_{i}^{m}=E_{m}(\Theta_{i})&,\gamma=\frac{\left\|l_{i}-L\right\|}{max\\{l_{i},L\\}},\end{split}$
(7)

where ${l_{i}}$ is the length of ${\Theta_{i}}$, $<\cdot,\cdot>$ denotes the
cosine similarity calculation between the two features, and $\lambda$ finely
balances these similarity aspects for more accurate representation. Based on
the calculated similarity score, We obtain the retrieved text features $R^{t}$
and motion features $R^{m}$ following the methodology established in
ReMoDiffuse [101].

Our encoder is grounded in a transformer architecture, enhanced with
Semantics-Modulated Attention (SMA) layers, a cross-attention structure proven
effective in ReMoDiffuse [101]. In SMA layers, the query vector ${Q}$ is
formulated from the original motion sequence $f_{m}$, while the key vector
${K}$ is the concatenation of $f_{m}$ and $[R^{m};R^{t}]$. The value vector
${V}$ merges $f_{m}$ and $R^{m}$. This arrangement ensures a thorough
integration of both original and retrieved features. After processing by the
encoder, the motion sequence is then ready for the language module, where it
undergoes text generation.

####  B.7.4 Training Strategy

##### Data Structuring.

The training data is structured to improve the model’s ability to follow
instructions and maintain conversational coherence, adopting a chatbot-like
format. Specific tokens such as $\textlangle motion\textrangle$, $\textlangle
answer\textrangle$, and $\textlangle endofchunk\textrangle$ are adopted from
Otter [42]. Each piece of data follows the format:

$\textlangle motion\textrangle$ User: [instruction] GPT: $\textlangle
answer\textrangle$ [answer]. $\textlangle endofchunk\textrangle$.

The $\textlangle motion\textrangle$ token, signifying input motion sequence,
is crucial for ensuring a proper alignment between motion inputs and textual
outputs. The $\textlangle answer\textrangle$ token delineates the responses by
model from the instructions. During training, all tokens following the
$\textlangle answer\textrangle$ token are masked, and they are set as the
prediction targets of the model — essentially, the captions of the motion
sequences. Additionally, to make full use of all motion annotations and to
enable the model to better learn the complex many-to-many relationships
between motion and language, we concatenate different annotations of the same
motion according to the aforementioned format, and train them together as a
single piece of data.

##### Data Augmentation.

The quantity of data plays a pivotal role in the quality of the generated
text. In recognition of this, we have employed the text-driven motion
generation methodology, ReMoDiffuse [101], to regenerate all the motions in
our training dataset according to their corresponding textual annotations.
This approach has effectively doubled the size of our original dataset,
thereby enhancing the robustness of our model with a richer and more varied
set of training examples.

##  Appendix C More Details of DLP-MoCap Dataset

Figure 14: Samples of DLP-Mocap Data. Motion Data visualizations from three
categories (Basic, Interactive, Short Script Play). Note we show only the
simplified text annotation here; detailed motion descriptions of each action
are included in the dataset for Interactive and Short Script Play.

DLP-MoCap has three subsets. First, Basic motion set that includes simple
motions with low-level semantics, such as “walking” and “sitting down”.
Second, Interactive motions are atomic clips of two-person interaction, such
as “shaking hands” and “high five”. Third, Short Script Play that consists of
longer, semantically continuous motions (3-5 interactive actions), following a
specific background such as “meeting”. We show samples of DLP-MoCap in Fig.
14.

###  C.1 Scripts

For Interactive and Short Script Play, we prepared scripts to guide
professional actors and actresses in the motion capture process. Each script
consists of motion descriptions and speech lines. We collect a diverse
collation of these scripts with a hybrid approach: we combine manually written
scripts and massive generated scripts using GPT-4 [73] with human inspection.

###  C.2 Annotation

We recruited 10 human annotators to label the actors’ interactive actions. In
the interactive actions between character A and character B, we annotated the
start and end frames of each individual’s semantic actions. Additionally, we
marked the frames where physical contact occurs and ends between the two
individuals. If the actions of the actors deviate from the script, we manually
adjust the script to align with the actors’ actions. Ultimately, we obtained
an interactive motion dataset with over 4K text-interaction motion pairs.

###  C.3 Motion Data Processing

We used an optical MoCap system consisting of 30 cameras to capture 3D body
data. The initial MoCap data captured 3D positions of 53 marker points on each
actor’s body surface at 120 FPS. In the meanwhile, motion capture gloves with
inertial sensors tracked their hand motions. We then downsampled and processed
them into SMPL-X format. Firstly body shape parameters (beta) were fitted from
first frame marker data for actors. Then pose parameters (theta) in each frame
were regressed following the pipeline of SOMA [26]. Mapping the original
format of hand poses into MANO [68], body and hand parameters were combined as
an SMPL-X data format.

##  Appendix D Additional Experiments and Details

###  D.1 Motion Generation

####  D.1.1 Implementation Details

We employ similar configurations for the DLP dataset and InterHuman dataset.
Specifically, for the motion encoder, we utilize a 4-layer transformer with a
latent dimension of 512 for each person. The text encoder consists of a frozen
text encoder from CLIP ViT-B/32, supplemented with 2 additional transformer
encoder layers. In terms of the diffusion model, the variances ($\beta_{t}$)
are predefined to linearly spread from 0.0001 to 0.02, and the total number of
noise steps is set to $T=1000$. Optimization is performed using the Adam
optimizer with a learning rate of 0.0002, and a cosine learning rate scheduler
which smoothly reduces it to 0.00002 at the last epoch. Training is conducted
on 4 Tesla V100, with a batch size of 128 on a single GPU, and lasts 20 epochs
in total.

####  D.1.2 Evaluation Metrics

We employ the performance measures used in MotionDiffuse for quantitative
evaluations, including Frechet Inception Distance (FID), R Precision,
Diversity, Multimodality, and Multi-Modal Distance:

  1. 1.

FID (Frechet Inception Distance): This objective metric calculates the
distance between features extracted from real and generated motion sequences,
providing a reflection of the generation quality.

  2. 2.

R-Precision: This measures the similarity between the text description and the
generated motion sequence. It indicates the probability that the real text
appears in the top k after sorting, with k set to 1, 2, and 3 in this work.

  3. 3.

Diversity: This metric assesses the variability and richness of the generated
action sequences.

  4. 4.

Multimodality: It measures the average variance of generated motion sequences
given a single text description.

  5. 5.

Multi-modal Distance (MM Dist): This represents the average Euclidean distance
between the motion feature and its corresponding text description feature.

####  D.1.3 Experiments on the InterHuman Dataset

Table 3: Interactive Motion Synthesis results on the InterHuman test set.
‘$\uparrow$’(‘$\downarrow$’) indicates that the values are better if the
metric is larger (smaller). We run all the evaluations 20 times and report the
average metric and 95% confidence interval is. The best results are in bold
and the second best results are underlined.

Methods |  R Precision$\uparrow$ | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\uparrow$ | MultiModality$\uparrow$
---|---|---|---|---|---
Top 1 | Top 2 | Top 3
Real motions | $0.452^{\pm.008}$ | $0.610^{\pm.009}$ | $0.701^{\pm.008}$ | $0.273^{\pm.007}$ | $3.755^{\pm.008}$ | $7.948^{\pm.064}$ | -
TEMOS [63] | $0.224^{\pm.010}$ | $0.316^{\pm.013}$ | $0.450^{\pm.018}$ | $17.375^{\pm.043}$ | $6.342^{\pm.015}$ | $6.939^{\pm.071}$ | $0.535^{\pm.014}$
T2M [28] | $0.238^{\pm.012}$ | $0.325^{\pm.010}$ | $0.464^{\pm.014}$ | $13.769^{\pm.072}$ | $5.731^{\pm.013}$ | $7.046^{\pm.022}$ | $1.387^{\pm.076}$
MDM [81] | $0.153^{\pm.012}$ | $0.260^{\pm.009}$ | $0.339^{\pm.012}$ | $9.167^{\pm.056}$ | $7.125^{\pm.018}$ | $7.602^{\pm.045}$ | $\mathbf{2.355^{\pm.080}}$
ComMDM [74] | $0.223^{\pm.009}$ | $0.334^{\pm.008}$ | $0.466^{\pm.010}$ | $7.069^{\pm.054}$ | $6.212^{\pm.021}$ | $7.244^{\pm.038}$ | $1.822^{\pm.052}$
MotionDiffuse [100] | $0.401^{\pm.004}$ | $0.541^{\pm.004}$ | $0.622^{\pm.005}$ | $12.663^{\pm.083}$ | $3.805^{\pm.001}$ | $7.639^{\pm.035}$ | $1.176^{\pm.027}$
ReMoDiffuse [101] | $0.442^{\pm.004}$ | $0.589^{\pm.005}$ | $\mathbf{0.666^{\pm.003}}$ | $6.366^{\pm.102}$ | $3.802^{\pm.001}$ | $7.956^{\pm.030}$ | $1.226^{\pm.044}$
InterGen [45] | $0.371^{\pm.010}$ | $0.515^{\pm.012}$ | $0.624^{\pm.010}$ | $5.918^{\pm.079}$ | $5.108^{\pm.014}$ | $7.387^{\pm.029}$ | $2.141^{\pm.063}$
Ours (MoMat-MoGen) | $\mathbf{0.449^{\pm.004}}$ | $\mathbf{0.591^{\pm.003}}$ | $\mathbf{0.666^{\pm.004}}$ | $\mathbf{5.674^{\pm.085}}$ | $\mathbf{3.790^{\pm.001}}$ | $\mathbf{8.021^{\pm.035}}$ | $1.295^{\pm.023}$

Table 3 shows the quantitative comparison on the InterHuman test set. Our
proposed method outperforms the existing works by a significant margin,
especially on the FID metric. We also want to highlight that our synthesized
motion sequences are highly consistent with the given text prompts and achieve
very competitive R Precision results. These results demonstrate the
superiority of our proposed MoMat-MoGen generation scheme.

####  D.1.4 Ablation Study on the DLP Dataset

Table 4: Ablation of the proposed architecture. All results are reported on the DLP testset. | MoMat | MoGen | Weight Sharing |  FID$\downarrow$ |  Diversity$\uparrow$
---|---|---|---|---|---
a) | ✓ | - | - | $\mathbf{0.034}$ | $0.332$
b) | - | ✓ | - | $5.721$ | $3.165$
c) | - | ✓ | ✓ | $4.196$ | $3.749$
d) | ✓ | ✓ | - | $0.172$ | $\mathbf{4.028}$
e) | ✓ | ✓ | ✓ | $0.071$ | $4.025$

As shown in Table 4, MoMat and Weight Sharing have a positive effect on the
FID metric, while MoGen has a positive effect on the Diversity metric. In
addition, our proposed method achieves the best balance of these two metrics.

####  D.1.5 Visualization

Figure 15: Visual Comparison on the InterHuman dataset. Our proposed method
can generate more text-consistent and natural motion sequences.

As depicted in Fig. 15, in the first example, our approach exhibits several
advantages over intergen: 1) In motion involving close contact, such as
shoulder-to-shoulder interaction, penetration is a common artifact. However,
our method ensures better contact and minimizes penetration. 2) Our approach
achieves improved text alignment by accurately representing the ”to ground”
description, whereas intergen merely involves a simple waist bend. 3) We
incorporate a comprehensive downward tilt motion, followed by the blue
character’s process of standing up with the assistance of another person. As
for the second example, our method accomplish the required two punches, while
InterGen only fulfills one. In addition, our generated interaction is more
natural since the punching and stepping backward take place simultaneously.
These two examples strongly suggest that our proposed method can generate more
text-consistent and natural motion sequences.

###  D.2 Motion Matching

####  D.2.1 Implementation Details

We adopt the approach of previous studies [35, 65] by employing three types of
trajectories, namely wave, circle, and square, to assess the responsiveness
and tracking capability of our system. The wave trajectory is defined as a
sine function with respect to time, represented by $x(t)=2\sin(t)$. The square
trajectory is characterized by a side length of 5. As for the circle
trajectory, its diameter is set to 5. We evaluate the quality of the generated
motion by computing the Euclidean distance between the generated motion and
the target trajectory. In accordance with prior research [65], we randomly
select 50 seed poses for matching in each trajectory and report the mean
trajectory error along with its standard deviation. As for data, we use the
same database as previous work [65]. During motion matching, the weights of
body pose, trajectory, facing, and hip height are $1:3:1:1$. Please note that
in the case of interactive motion, the weights will vary as the target
trajectory is not present. These weights can be adjusted by the user, although
the default weight of 1 is typically satisfactory [36].

####  D.2.2 Experiment Analysis

We present quantitative and qualitative ablation experiments on motion
matching to illustrate the role of its features. For a large-scale database,
the text may lack certain information, such as whether the character is seated
or standing. As depicted in Fig. 16, although the text embedding can retrieve
motions that align with the query texts (handshake, then high ten), the
absence of kinematics features (Fig. 16(b)) leads to that the characters may
suddenly sit down if a seated motion is selected, resulting in a degradation
of visual quality. Additionally, our pipeline requires both characters to
align their positions and orientations before engaging in interactive motion.
As depicted in Fig. 17(b), in the absence of kinematics features, if the
retrieved motion necessitates the character to move to the right side of the
active actor, the character must walk to that position. However, by
incorporating kinematics features (Fig. 17(a)), it becomes possible to select
a motion situated on the left side of the active actor, and consequently, this
additional movement is avoided (Fig. 17(b)). Besides, as demonstrated in Tab.
5, the kinematic features play a crucial role in trajectory following as they
include position, velocity, and orientation information.

(a) With kinematics features

(b) Without kinematics features

Figure 16: Comparison of generated motion with and without kinematics
features. Kinematic features play a crucial role in preventing sudden sitting
down.

(a) With kinematics features

(b) Without kinematics features

Figure 17: Comparison of generated motion with and without kinematics features. Kinematic features are helpful in avoiding unnecessary movement. Table 5: Ablation of the kinematics features in trajectory following task. The kinematics features are vital for trajectory following. | Trajectory Error / m
---|---
| With kinematics features | W/o kinematics features
Square | $0.129\pm 0.024$ | $3.872\pm 0.694$
Circle | $0.209\pm 0.032$ | $3.039\pm 0.489$
Sine | $0.123\pm 0.020$ | $4.393\pm 0.822$

###  D.3 Motion Captioning

In this section, we assess the efficacy of our motion captioning module,
focusing on its ability to generate motion captions that are both high-quality
and accurate.

####  D.3.1 Implementation Details

In our study, both the HumanML3D [28] and KIT-ML [64] datasets are configured
similarly. We utilize a 12-layer transformer-based retrieval-augmented motion
encoder with SMA layers and a 512-dimensional latent space. Our pretrained
motion feature extractor, also a 12-layer transformer, has standard self-
attention layers in the same dimensional space. Adhering to Otter’s [42]
training method, we freeze the MPT-1B RedPajama language encoder to utilize
pretrained knowledge and prevent overfitting. Fine-tuning is limited to the
retrieval-augmented motion encoder, the perceiver resampler module, and the
language encoder’s cross-attention layers. For precise and coherent motion
captions, we use cross-entropy loss and optimize with the AdamW [48]
optimizer, starting at a learning rate of $1\times 10^{-4}$, a batch size of
16, over 5 epochs. A cosine annealing scheduler adjusts the learning rate,
complemented by gradient clipping set to 1.0 to prevent gradient explosion.

####  D.3.2 Evaluation Metrics.

For our experiment’s evaluation, we follow [29] and employ two categories of
metrics:

  1. 1.

Text Matching Accuracy: involves R Precision for checking the alignment
accuracy between text and motion, and Multi-modal Distance (MMDist) to gauge
the feature space distance between these modalities.

  2. 2.

Linguistic Quality of Captions: includes Bleu [57] for assessing translation
closeness, Rouge [12] for summary quality, Cider [85] for n-gram matching
consensus, and BertScore [102] to evaluate semantic accuracy through deep
contextual embeddings.

####  D.3.3 Experiments on the KIT-ML Dataset

Table 6: Motion Captioning results on the KIT-ML test set. Our evaluation
methodology aligns with the TM2T [29] metrics, but we uniquely utilize
unprocessed ground truth texts for calculating linguistic metrics as done in
MotionGPT [39].

Methods |  R Precision$\uparrow$ | MMDist $\downarrow$ | CIDEr $\uparrow$ | Blue@1 $\uparrow$ | Blue@4 $\uparrow$ | Rouge $\uparrow$ | BertScore $\uparrow$
---|---|---|---|---|---|---|---
Top 1 | Top 3
Real | $0.399$ | $0.793$ | $2.772$ | $-$ | $-$ | $-$ | $-$ | $-$
TM2T | $0.359$ | $0.668$ | $\underline{3.298}$ | $\underline{25.29}$ | $36.42$ | $\underline{7.98}$ | $31.26$ | $20.07$
MotionGPT | $\underline{0.392}$ | $\underline{0.723}$ | $3.341$ | $12.32$ | $\underline{40.51}$ | $6.59$ | $\underline{38.79}$ | $\underline{24.50}$
Ours | $\mathbf{0.410}$ | $\mathbf{0.765}$ | $\mathbf{2.647}$ | $\mathbf{71.06}$ | $\mathbf{53.88}$ | $\mathbf{22.91}$ | $\mathbf{50.63}$ | $\mathbf{46.13}$

In addition to the experiments conducted on HumanML3D [28], as detailed in the
main text, we extended our testing to include the KIT-ML dataset [64], the
results of which are presented in Tab. 6. The outcomes of these tests
demonstrate that our proposed method not only surpasses current methods in
performance but also excels in linguistic metrics.

###  D.4 SocioMind

In this section, we introduce supplementary details on the design of our user
study in Sec. D.4.1, and the qualitative and quantitative case analyses of
social evolution in Sec. D.4.2.

A sample from the questionnaire Here is the background and interactive dialog
between Jack and Leon. === BACKGROUND === Jack is Leon’s father. Leon chooses
his major in college. Leon often studied computer stuff some time ago, but
this time he did very poorly on the math exam and failed the algebra part. ===
SCRIPTS === Jack:Hey Leon, you didn’t do well in math. Are you okay? Can I
help you improve? [sits down next to Leon on the sofa,concerned] Leon:I’m
really upset about my math exam, Dad. I worked so hard, but I just couldn’t
understand the algebra part. It’s frustrating because I want to do well and
pursue my dream of becoming a game developer in the future. I know coding and
programming are important skills for that, but without a strong foundation in
math, it feels like an uphill battle. [slumps back on the sofa, disappointed]
Jack:I understand how frustrating it can be when things don’t go as planned.
Math can be challenging. Let’s make it easier and more enjoyable. [lean
forward, supportive] Leon:I appreciate your understanding, Dad. I want to
improve in math, so I think finding online math resources would help. [sits up
straight, determined] Jack:That’s great, Leon. Online math resources can help
you better understand concepts. Any specific websites in mind? [leans forward
attentively, encouraging] Leon:I recommend checking out Khan Academy and
Brilliant, they offer video lessons, practice problems, interactive courses,
and problem-solving techniques for math. [nods, hopeful] Jack:I’m confident in
your ability to improve. Let’s check out Khan Academy and Brilliant together
for a strong math foundation. [puts a hand on Leon’s shoulder, encouraging]
The statement that best describes Leon’s personality is: • Rebellious son:
Likes to try different things and is creative and is resistant to rules and
constraints. • Independent son: Able to complete tasks independently, does not
require too much guidance. • Achievement-oriented son: self-disciplined,
organized, achievement-oriented and goal-oriented. • Dependent son:
emotionally relies on the feedback and support of others. Figure 18: A sample
from our questionnaire. Human evaluators are asked to read the records and
pick the right option from the given psychological states.

####  D.4.1 User Study Design

In the controllability experiment of SocioMind, we provided 3 $\sim$ 5
psychological state types for each dimension in personality, motivation,
central belief, and social relationship. We generate records of SocioMind on
several setups (_e.g_., father-son, siblings, teacher-student relationships).
While generating the records, the hyper-parameter $a$ is 0.4 for events and
0.1 for thoughts, $k$ is 4 for events and 2 for thoughts. The forgetting
threshold $T_{f}$ is 0.6 for an event and 0.3 for a thought. As shown in Fig.
18, we show the records to human evaluators and ask them to select the
appropriate psychological state from a set of randomly shuffled options. The
user study was conducted in the form of a questionnaire survey, with a total
of 47 questionnaires collected. Human evaluators are composed of individuals
aged between 20 and 45 years old, including 28 males and 19 females, all
possessing proficient English reading skills. Their professional backgrounds
varied, including university students, researchers, engineers, and teachers.
Our records contain a total of 64 episodes, and each questionnaire randomly
selected records from 8 different episodes for human evaluation. For
controllable costs and fair comparison, all the LLM inferences utilize GPT-3.5
[73] in our experiments.

Figure 19: Visualization of social evolution on initial setting ‘Old Jack and
Young Jack’ (Young Jack’s view). The first and second rows show the emotions
and social relationships change with each episode, where the horizontal axis
encapsulates a summary of each episode’s story. The third row shows a word
cloud visualization of the keywords of the events and thoughts generated
during psychological reflection within each episode. The figure shows that
Young Jack’s emotions and his social relationship with Old Jack evolve
progressively with the storyline, aligning with the Social Penetration Theory
[4].

####  D.4.2 Case Analysis

To clearly illustrate the concept of social evolution, we conduct the case
analysis based on the initial setting shown in Fig. 10, Old Jack conversing
with his younger self across time and space. In this case, Old Jack has
experienced the vicissitudes of life, with his dearly loved wife and mother
having passed away. Now, with the ability to traverse time, Old Jack converses
with his younger self, hoping to inspire Young Jack to cherish time and the
people around him.

As shown in Fig. 19, the records of SocioMind with initial settings about Old
Jack and Young Jack are in alignment with the Social Penetration Theory [4].

In the initial episode, when Old Jack appears suddenly, Young Jack maintains
vigilance and concern towards this unexpected intruder in his home, despite
Old Jack’s attempts at friendly communication. Young Jack’s request for Old
Jack to leave is consistent with the Orientation stage of the theory [4].

As the storyline progresses, Old Jack comforts Young Jack, who is disheartened
by exam failures, and shares some amusing photographs and thoughts. For
instance, when they talk about the value of time, Young Jack believes that
playing games is of utmost importance. Old Jack remarks, ‘Sometimes, games can
wait, but people can’t; we should cherish every moment we have with them.’ In
this phase, Young Jack’s skepticism and vigilance towards Old Jack gradually
diminishes, and he begins to share some of his own thoughts, aligning with the
Exploratory Affective stage [4].

Later, as their communication goes deeper, Old Jack recalls his own past joys
when seeing the old photos, discusses photography techniques with Young Jack,
and makes plans to take photos with Young Jack in the park. At this stage,
both characters start to disclose more personal information, and there is an
increase in intimacy and trust, which corresponds to the Affective stage [4].

##  Appendix E Discussion

##### Limitations.

As the first work towards building autonomous 3D characters with social
intelligence, Digital Life Project has several limitations. First, this work
investigates the interaction between two characters. However, synthesizing the
3D motions of a large group of characters with interactive behavior remains a
significant challenge. Second, DLP focuses on modeling human-human
interaction. Despite some level of ability to navigate in the scene and
interact with the furniture, integrating more comprehensive human-scene and
human-object interaction in the framework is left as future work.