#  MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics
Education

Murong Yueα, Wijdane Mifdalα, Yixuan Zhangβ, Jennifer Suhγ, Ziyu Yaoα
αDepartment of Computer Science, George Mason University
βDepartment of Computer Science, College of William&Mary
γMathematics Education, School of Education, George Mason University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Mathematical modeling (MM) is considered a fundamental skill for students in
STEM disciplines. Practicing the MM skill is often the most effective when
students can engage in group discussion and collaborative problem-solving.
However, due to unevenly distributed teachers and educational resources needed
to monitor such group activities, students do not always receive equal
opportunities for this practice. Excitingly, large language models (LLMs) have
recently demonstrated strong capability in both modeling mathematical problems
(Wei et al., 2022) and simulating characters with different traits and
properties (Park et al., 2023). Drawing inspiration from the advancement of
LLMs, in this work, we present MathVC, the very first LLM-powered virtual
classroom containing multiple LLM-simulated student characters, with whom a
human student can practice their MM skill. To encourage each LLM character’s
behaviors to be aligned with their specified math-relevant properties (termed
_“characteristics alignment”_) and the overall conversational procedure to be
close to an authentic student MM discussion (termed _“conversational
procedural alignment”_), we proposed three innovations: integrating _MM domain
knowledge_ into the simulation, defining _symbolic schema_ as the ground for
character simulation, and designing a _meta planner_ at the platform level to
drive the conversational procedure. Through experiments and ablation studies,
we confirmed the effectiveness of our simulation approach and showed the
promise for MathVC to benefit real-life students in the future.111Source code
and examples will be available at our project website soon.

##  1 Introduction

Mathematical modeling (MM), the process of creating a mathematical
representation of a real-world problem, is a critical skill for students
pursuing STEM fields. Practicing this skill often requires students to engage
in effective discussions of relevant concepts and collaboratively solve
mathematics problems with their classmates (Vygotsky & Cole, 1978; Garfunkel &
Montgomery, 2019). However, orchestrating such discussions and collaborative
learning activities can be very challenging for teachers (Bush et al., 2020).
As a consequence, the Opportunity Myth report (TNTP, 2018) has highlighted
that students from marginalized communities where only limited teachers and
educational resources are available, are less likely to have opportunities to
participate in effective practices of MM skills.

The rapidly developing large language models (LLMs) have shown huge potential
to reshape the future of education (including Mathematics) (Denny et al.,
2024). This potential stems from two recent advancements uniquely for LLMs.
Firstly, LLMs present strong math reasoning capabilities in understanding and
solving mathematics problems (Wei et al., 2022; Lu et al., 2023; Ahn et al.,
2024). In particular, on the GSM8k dataset (Cobbe et al., 2021) representing
middle-school mathematics level, the state-of-the-art GPT-4 model has achieved
close to perfect performance (OpenAI et al., 2024). Secondly, by generating
human-like thoughts and actions in explicit texts, autonomous LLM-based
generative agents have showcased the capacity to simulate human activities
(Park et al., 2023; Wang et al., 2023c; Zhou et al., 2023). Leveraging this
capability, studies have explored simulating human behaviors in different
scenarios, such as games (Wang et al., 2023a), social activities (Li et al.,
2023b; Cheng & Chin, 2024), software engineering (Hong et al., 2023; Qian et
al., 2023).

Figure 1: An example demonstrating the application of MathVC, where students
are presented a math problem (optionally with accompanied data) and engage in
effective discussions on math modeling. Alice, Bob, and Charlie333We use the
three names as “placeholders” following the conversion in science and
engineering literature (Wikipedia, n.d.); however, we envision the system to
be used by various communities. are LLM-simulated student characters. A human
student can choose to respond or skip the current turn.

Observing the pressing need to enhance MM education for resource-limited
communities and getting inspired by the striking progress made by LLMs, in
this work, we propose MathVC, a Mathematics Virtual Classroom (Figure 1)
featuring multiple LLM-simulated student characters, with whom a human student
can discuss and collaboratively solve MM problems, which thus allows them to
practice their MM skills without needing the teachers monitoring it. In
particular, the platform focused on _middle-school_ MM problems, a difficulty
level that has been well handled by GPT-4. To the best of our knowledge, our
research is the _first_ work exploring the potential of leveraging LLMs and
LLM-powered generative agents for mathematics education. We envision that such
a platform can eventually be deployed as part of the middle-school curriculum,
enabling effective MM learning to go beyond the space constraint in
traditional classroom studies while enhancing students’ MM capability.

We identified two unique challenges in the development of MathVC. The first
challenge is characteristic alignment, i.e., aligning an LLM’s character
simulation to the authentic characteristics of real human students. Middle-
school students in real life could struggle with rapid and accurate modeling.
They may initially produce incomplete or erroneous mathematical models and
refine them iteratively as the discussion progresses. This process of
gradually establishing mutual understanding with teammates, interleaved by
continual self-reflection, is the key for students to improve their MM skills.
However, in our preliminary experiments using GPT-4, a vanilla character
simulation via prompt engineering the profile instruction to LLM cannot yield
such an authentic and fine-grained student simulation. Instead, GPT-4 often
directly gives a close-to-perfect solution, even when prompted to simulate
students with poor math skills. Simulating how a student’s problem
understanding and math modeling evolve during the collaboration, is even more
difficult. The second challenge is conversational procedural alignment, i.e.,
aligning the overall conversational procedure to an authentic MM discussion
among middle-school students. While the first challenge concerns individual
characters’ simulation, this second challenge considers inter-character
interaction. In practice, a collaborative problem-solving process typically
consists of multiple stages, including problem understanding, task division,
solution planning, plan execution, etc. (PISA, 2018). However, when we
directly exposed the group of LLM characters to interact with each other,
these LLM characters often bypassed the earlier stages and instead, initiated
the conversation by immediately describing a modeling solution. For both of
the two challenges, we conjecture that the vanilla simulation fails because
LLMs were pre-trained in a question-answering style (e.g., they tend to
directly give answers to the input mathematics questions), while lacking
sufficient _instructiblity_ to follow the prompt instruction and simulate the
deliberation of a middle-school student. Despite existing exploration of LLM
character simulations, we have not seen any prior work carefully exploring
these two alignment challenges and the solutions to them.

To tackle these challenges, we propose to integrate relevant theories of MM
and collaborative problem-solving (CPS) (Standards, 2010; PISA, 2018) into the
development of MathVC. Specifically, to enhance characteristics alignment, we
propose using a symbolic _character schema_ to represent the _dynamic_ problem
understanding process of a character. The character schema centers around
“variables” that are essential for solving the mathematics problem, which are
considered fundamental elements in MM theory (Standards, 2010). By considering
it as a “ground”, we can simulate characters with more intricate behaviors,
including how they make a mistake initially and then gradually fix it during
the collaboration with other students. To enhance conversational procedural
alignment, we further propose a _meta planner_ at the platform level, which
monitors the overall conversation and plans its procedure following the stages
defined in the CPS theory (PISA, 2018).

Through experiments and ablation studies, we confirmed the effectiveness of
our simulation. Specifically, we compared MathVC with vanilla character
simulation as well as its variants when it is augmented with math theories in
prompt formulation or is equipped with our proposed character schema and meta
planner. The results showed that MathVC can achieve the best performance for
simulating both individual characters and character interaction. Overall, our
generated multi-character dialogues can better reflect the predefined student
characteristics and are more akin to conversations among real middle-school
students in collaborative MM tasks. We also conducted a series of ablation
studies for two key components in MathVC (i.e., meta planner and the character
schema); the results validated the necessity of our proposed prompt designs.
In summary, our results indicate that MathVC has huge potential for being
deployed to serve middle-school students in real life, and our proposed
approaches can be generalized to resolve similar alignment problems in other
applications.

##  2 MathVC: An LLM-Powered Mathematics Virtual Classroom

###  2.1 Overview

Our goal is to develop a “virtual classroom” platform with multiple LLM-
simulated student characters discussing and collaboratively solving MM tasks.
We propose to design MathVC with a modular architecture as shown in Figure 3.
Specifically, the system consists of two parts, i.e., meta planner for
organizing the overall conversation and facilitating a smooth multi-stage
student discussion, and character simulation for creating and updating
individual student characters. All modules in MathVC are implemented by
prompting an LLM (e.g., GPT-4) with specialized scripts, mostly in zero-shot
(except for Dialogue Act Generator in one-shot). A complete workflow of the
system is presented in Appendix A.

###  2.2 Symbolic Representations for Task and Character Modeling

A key innovation of MathVC lies in the use of _symbolic schemas_ describing
how characters form and gradually update their understanding of the
mathematics problem. Specifically, we design two schema representations. Task
Schema is a template describing elements that are necessary for solving a
given MM task. As shown in Figure 2 (left), a task schema specifies how the
original mathematics problem can be decomposed into multiple sub-tasks, and
for each sub-task, what variables should be defined to resolve it. In other
words, a task schema shows the ground-truth understanding and modeling of a
mathematics problem. We define the schema centering around “variables” because
they are considered the fundamental elements in MM (Standards, 2010). In
practice, this schema can be automatically generated by prompting an LLM with
the mathematics question and the ground-truth step-by-step solution following
the Chain-of-Thought (Wei et al., 2022, CoT) formulation. Note that while a
task schema could be challenging and time-consuming to manually annotate for
teachers using this platform, the step-by-step solution is easy to compose or
can be scraped from the answer notes of textbook exercises or directly
generated by GPT-4.

Built on top of the task schema, we further devise Character Schema, a
symbolic representation showing a student character’s understanding and
modeling plan on the given mathematics problem. As shown in Figure 2 (right),
a character schema is a task schema injected with errors that could be made by
real human students. We consider a wide range of errors including
misunderstanding on a sub-task or even omitting the sub-task, as well as any
variable mistakes resulting from miscounting, incorrect modeling, or
calculation errors. In mathematics education, materials such as the lecture
notes in the Mathematics Assessment Project (MAP) (MAP, n.d.), often summarize
common mistakes in each math problem, which can be directly used for
simulating how students had incorrect solutions in the beginning. In MathVC,
the character schema is synthesized by prompting an LLM to _edit_ a task
schema based on the student’s characteristics as well as a set of possible
mistakes. Throughout the conversation with other students, a character will
gradually update its character schema to reflect its dynamic thought process,
as we will introduce in Section 2.4.

Figure 2: Example task and character schema. The LLM edits the correct value
16 into 15 in the task schema to simulate the initial mistake made by the
simulated student Charlie.

###  2.3 Meta Planning

The main goal of the meta-planning module is to perform meta-level control of
the entire conversation to encourage conversational procedural alignment. It
consists of three components: (1) Task Schema Generator: As Section 2.2
mentioned, the initial step of MathVC is to generate the task schema, which
can then be edited to form character schemas. We instruct the LLM to produce
the task schema with the specified format shown in Figure 2. A rule-based text
checker will be applied to verify if the generated schema follows the format;
if not, we regenerate a new one. (2) Collaboration Stage Monitor: Prior work
(PISA, 2018) has observed that the collaborative problem-solving (CPS) process
among students often consists of a sequence of stages, such as first
establishing a shared task understanding and then establishing a team
organization, as we summarize in Figure 3 and explained in Appendix B. To
ensure that our simulated discussions closely resemble real CPS discussions,
every dialogue will start from Stage 1, with the collaboration stage monitor
dynamically determining whether the current stage is completed and whether the
discussion should proceed to the next stage. This decision is made based on
the dialogue history as well as a definition of every stage. When all stages
are completed, we end the session. We note that simulating this multi-stage
process is also important for human students to engage in effective MM
practices, as it enables more _scaffolded_ human and character interaction.
(3) Dialogue Speaker Control: To facilitate smoother conversation
interactions, we have developed a dialogue speaker control module that
predicts the next speaker based on the dialogue context. For instance, if the
last response is “Hey Bob, you made a mistake…”, the next speaker is likely
Bob. The LLM captures this intuition by learning from the dialogue history,
and it is prompted to output a single character name as the next speaker.

Figure 3: Architecture of MathVC for simulating student characters and multi-
character conversations in collaborative MM problem-solving.

###  2.4 Character Simulation

Each character is simulated with four specified traits: name, gender, career
(e.g., grade level in our application), and MM skill level (i.e., good or
bad444In this work, we define a student’s MM skill in a coarse way. Future
work could enhance it to be more fine-grained based on the practices in
mathematics education.). We included the first two traits mainly for
characters to address each other (via names or pronouns) and the last two for
configuring each character’s MM background.

Our simulation functions with four components (Figure 3, bottom). The first
two components are responsible for generating and maintaining the character
schema. (1) Character Schema Generator: As we introduced in Section 2.2, every
simulated student character has its own character schema to reflect their
thought. In our simulation, a character schema generator decides the specific
mistake(s) a student character will make based on their traits and then edits
the task schema into their initial character schema (Figure 2). To facilitate
this editing, as shown in Appendix C, we represent the task schema as a Python
dictionary and express every edit as a line of Python code that specifies the
replacement of a variable or a sub-task description. (2) Character Schema
Modifier: During real-life student discussions, a human student’s thought for
a given question often evolves as the conversation progresses. We simulate
these dynamics by continuously modifying the character schema based on the
dialogue history. We prompt the LLM to decide whether the current schema needs
to be modified. When it does, the modification will be achieved by generating
a Python code that edits the previous character schema, similar to the initial
character schema generation.

Generating character responses is not a trivial task in our application. A
major challenge we observed is their _lack of diversity_. For example, when we
directly prompted LLMs to generate responses simulating how students discuss
their modeling plans (i.e., Stage 3), all responses were uniformly assertive
statements of a plan, without other forms of communication that are also
commonly seen in authentic student discussions (e.g., question-answering). To
address the issue, we propose a two-step response generation with the
following two components. (3) Dialogue Act Generator: Before the response
generation, we first prompt the LLM character to pick a “dialogue act”
indicating the intent of their response. Dialogue acts are widely used in
traditional dialogue systems (Stolcke et al., 2000; Li et al., 2016; Zhao et
al., 2017). In our application, we define a set of dialogue acts for each
conversational stage, such as “present a task understanding” and “ask for a
clarifying question about a task understanding” for Stage 1, as shown in
Figure 3; the complete definitions can be found in Appendix B. A dialogue act
generator is then tasked to _generate_ a dialogue act (e.g., “ask Bob a
clarifying question about…”) based on the character schema, the dialogue
history, and the set of dialogue acts for the current stage. We found that the
use of dialogue acts could dramatically increase the response diversity when
an LLM is explicitly prompted to reason about the proper response (e.g.,
asking a clarifying question when the character has a different modeling plan
than their teammates, or generating a narrative answer when a character is
asked for clarification). (4) Character Response Generator: Once deciding the
dialogue act, a character response can be generated conditionally. However,
even when the character schema is provided in the prompt, we still observe
inconsistency between the character schema and the generated response (e.g.,
characters referring to inconsistent variable values than those in their
schemas). To make the response follow the character schema better, instead of
directly generating, we prompt an LLM to first retrieve relevant variables
from the character schema, and then generate a response conditionally on the
variables and their values. In practice, we found this prompt design to be
very effective. Additionally, in the prompt, we instruct the LLM to mimic the
language style of middle-school students (e.g., keeping the response short and
colloquial). We present the prompts for the dialogue act generator and the
response generator in Appendix I.

##  3 Experiment

###  3.1 Experimental Setup

Dataset and Experimental Setting  We conducted experiments using middle-school
mathematics problems from the MAP project (MAP, n.d.) and GSM8k (Cobbe et al.,
2021). Questions from the MAP include both answers and common student
mistakes, while those from the GSM8k only have answers; therefore, we manually
annotate mistakes for them.

The most ideal evaluation of MathVC is to include real middle-school students
in the loop, but this is very challenging and unsafe to implement (see our
Ethics Statement). As our purpose of this work is mainly to showcase the
feasibility of such a virtual classroom, during evaluation, we exclude real-
student interaction and only focus on whether the multiple (3 in our
experiments) simulated LLM characters can reasonably collaborate on solving MM
tasks. The evaluation is then conducted by sampling multiple simulated
dialogues and manually assessing their quality. Due to the cost of manual
evaluation, we sample a small set of questions (5 as the dev set and 10 as the
test set). For each question on the test set, we consider two student
character configurations with varying MM skill combinations, and sample one
multi-character dialogue in each for every system in comparison. We used
GPT-4-turbo as the backbone LLM. We include more experimental details in
Appendix D.

Baseline Systems  We compare MathVC with four baselines. (1) Vanilla
Simulation follows the prior work (Park et al., 2023) in prompt engineering
the profile instruction to an LLM. Specifically, we directly present the LLM
the math question, the student characteristics, and the dialogue history, and
then instruct the LLM to generate responses as a middle-school student. (2)
Domain-Specified Simulation further augments the vanilla simulation with
domain-specific knowledge in its prompt, including the stages and possible
acts in each stage, common student mistakes, the language style of a middle-
school student, and so on. To demonstrate the effectiveness of our character
schema and the meta planner, we include two more variants on top of domain-
specified simulation, i.e., (2.1) w/ only character schema, where we include
our proposed character schema in the prompt to the LLM and allow for continual
schema modification, and (2.2) w/ only meta planner, where we augment the
domain-specified simulation with our meta planner. The planned stage is added
to the prompt for character response generation.

Evaluation Metrics The human evaluation was conducted by two student
annotators. The annotators were instructed to rate each sampled dialogue on
the degrees of characteristic alignment and conversational procedural
alignment it reflected (1: worst; 4: best). We report an average score. More
details and the annotation guideline are presented in Appendix H.

System | Characteristic Alignment | Conversational Procedural Alignment
---|---|---
Vanilla Simulation | 2.85 | 2.88
Domain-Specified Simulation | 3.30 | 3.15
w/ only character schema | 3.50 | 3.20
w/ only meta planner | 3.00 | 3.10
MathVC | 3.73 | 3.53

Table 1: Human evaluation results comparing various simulations (1: worst; 4:
best).

###  3.2 Experimental Results

The human evaluation results are shown in the Table 1. We make the following
observations:

Incorporating domain knowledge improves the simulation quality. All systems
built with domain knowledge outperform the vanilla simulation. In our
inspection, we observed that the vanilla simulation cannot simulate fine-
grained student behaviors; for example, it rarely exhibits modeling mistakes
even when simulating students with bad MM skills and often starts the
discussion unnaturally with solution statements. By introducing knowledge of
common mistakes and stage information to its prompt, these issues are
mitigated in domain-specified simulation, leading to improved alignment on
both characteristics and conversational procedures.

Including character schema enhances characteristic alignment, but including
meta planner enhances alignment only when used with character schema.
Incorporating solely character schema could improve the characteristic
alignment and result in comparable conversational procedures. Surprisingly, we
observe that simply adding the meta planner may even degrade the dialogue
quality. We found that when adding the natural-language stage information
(e.g., “execute actions for problem-solving”) explicitly to the prompt of
domain-specified simulation, the LLM was biased to respond with a direct
solution (e.g., presenting the execution results). This not only broke the
characteristics but also resulted in unnaturally shorter conversations. The
observation, on the other hand, confirms the necessity of the dialogue act
generator of MathVC, which avoids this issue by inferring the likely dialogue
acts in each stage and feeding the dialogue acts rather than the explicit
stage information for response generation.

Figure 4: Case study for different simulation approaches. We manually
annotated the simulated stages in blue and indicated responses reflecting
Alice’s bad math skill in red. In the vanilla simulation, Alice merely
verbally indicates that she struggles with math but still gives a correct
calculation “48”, whereas in MathVC, she makes a genuine mathematical modeling
mistake.

MathVC offers the most aligned simulation by harmoniously integrating
character schema with meta planner. Our system obtained the highest scores on
both alignment aspects, demonstrating the effectiveness of integrating the
symbolic schema representations with meta-planning under our framework. We
qualitatively compare MathVC with other approaches in Figure 4. In the vanilla
simulation, the discussion bypasses many discussion stages (e.g., establishing
the team organization), and Alice, who is bad at mathematics, does not make
any mistakes. The domain-specified simulation made some improvements, such as
simulating responses that are shorter and can simulate how students establish
shared task understanding before planning on problem-solving, although it is
still unable to simulate a fully extended, multi-stage conversation (e.g.,
team organization is still missing). In MathVC, the conversation successfully
goes over all stages. It also simulates how Alice makes a calculation mistake
initially, which is then corrected by Charlie. Like a real student, Alice then
adjusts her modeling plan based on Charlie’s explanation. This case study
demonstrates the superiority of our system in both the characteristics
alignment and the conversational procedural alignment.

###  3.3 Further Analysis and Discussion

Ablation Studies Evaluating Key Prompt Designs in MathVC We further conducted
ablation studies to validate the key prompt designs in MathVC, hoping to share
insights with future research that will simulate characters at the same
complexity level. Our investigation reveals that (1) editing is superior to
regenerating in the character schema modifier; (2) generating acts is
preferable to solely selecting acts in the dialogue act generator; and (3)
two-step generation is more effective than the direct generation in the
response generator. The results and analysis are in Appendix E.

Error Analysis  Through analyzing the sampled dialogues, we identify two key
issues of MathVC that can be addressed in the future. First, we observed that
the LLM-simulated students sometimes cannot elaborate on their deliberation
process. Rather, they tend to directly offer an opinion, which makes the
conversation less engaging. Second, _instructibility_ remains a major
challenge in the character simulation. For example, even when a character has
been simulated to take charge of a certain sub-task during the team
organization, the LLM may still generate a response mistakenly representing
the character’s actions in solving a different sub-task. We conjecture that
this discrepancy is sourced from the LLM’s attention to dominantly the local
context (e.g., the current sub-task focus), while instructions indicated in
the relatively distance context are overlooked (e.g., Alice’s sub-task
assignment at the beginning of the discussion). Future work could address
these issues by designing an appropriate schema to encourage responses with
details and dynamically store and retrieve information in both long- and
short-term character memory.

Preliminary Results with Humans in the Loop  While we are not able to recruit
middle-school students in our current research, we conducted a preliminary
study showing the potential for real humans to engage in collaborative MM
tasks with LLM characters. An example dialogue conducted by one author is
shown in Appendix F. We found that the interaction between the human and the
simulated student characters is natural and effective. It showcases the
promise for future deployment with real middle-school students.

Simulation with Varying Character Names, Genders, and Careers  Our main
evaluation has focused on the simulation of students at different MM skill
levels, given that it is the most essential demand for MathVC. To further
understand the impact of other traits, we present sampled dialogues with
different character names (other than the “placeholders” Alice, Bob, and
Charlie), genders (e.g., when all characters are females), and careers (e.g.,
when characters come from different grade levels or countries). Our ablation
analysis as described in Appendix G shows that these characteristics have only
a minor impact on the response, and we did not observe any ethical concerns
(e.g., biases) in our examination. We include further discussions in the
Ethics Statement.

Limitations and Future Work  Like most existing LLM agents, MathVC faces the
issues of token costs and latency, when it has to query an LLM multiple times
for each character’s response generation. In our early study, we experimented
with the relatively cheaper and faster GPT-3.5-turbo but encountered
unsatisfactory results. On the other hand, the pipeline design of MathVC’s
architecture could lead to error accumulation, i.e., when one module does not
yield satisfying outcomes, the following ones suffer the same. However,
overall MathVC still demonstrated a huge potential to be deployed to serve
real students and be used as a tool for mathematics education. In the future,
one important extension is to allow for “personalization” of MathVC, such as
simulating student characters based on the specific human student’s need
(e.g., setting up each character’s dialogue act frequency depending on what
kinds of dialogue interaction can better scaffold the student’s problem
modeling and solving). Finally, we clarify that the close-to-4 score of MathVC
in Table 1 by no means implies a perfect performance with our system, and we
believe that more effort is still required for the system to benefit student
communities in practice. More systematic evaluation of both the system and the
human student (e.g., whether they exhibit enhanced MM skills after interacting
with MathVC) should be conducted in collaboration with experts from
Mathematics Education.

##  4 Related Work

LLM Character Research  There is a growing number of work employing LLMs to
construct autonomous agents for playing specific roles (Wang et al., 2023b;
Park et al., 2023), such as programmers (Qian et al., 2023; Hong et al.,
2023). Some studies have found the characteristic alignment of LLMs is
sometimes unsatisfactory (Fischer, 2023; Wang et al., 2024). Approaches to
improving the characteristic alignment are mainly in two categories: fine-
tuning LLMs (Shao et al., 2023; Zhou et al., 2023; Liu et al., 2023; Yu et
al., 2024), which requires high-quality data, and carefully designing specific
prompts and architectures (Yu et al., 2022; Li et al., 2023a; Tang et al.,
2024; Hua et al., 2024). Given the challenge of gathering realistic data from
middle-school students due to privacy concerns, we opted for the second line
of approach. In particular, we propose a novel idea of leveraging symbolic
representations to enhance characteristics alignment, and we explore an
innovative application of simulating less knowledgeable populations (e.g.,
students with bad math skills) for mathematics education.

LLMs in Mathematics Education  Leveraging the mathematical reasoning ability
of LLMs, numerous studies have explored how to enhance mathematics education
with LLMs. McNichols et al. (2024) utilize LLMs to generate multiple-choice
mathematical questions. Levonian et al. (2023) investigate how to select math
problems to assist math tutoring. Bulusu et al. (2023) integrate LLMs and a
math graphing tool to plot math figures by language. Macina et al. (2023)
propose a framework for human teachers to interact with LLMs to collect data
for math tutoring and the resulting data can be used to fine-tune smaller
tutoring models. Most relevant to us are the works of Markel et al. (2023) and
Lee et al. (2023), which construct platforms with LLM-simulated students for
teaching assistants and teachers training. However, existing work has only
focused on the vanilla simulation with humans-agents interaction, while we
look into an innovative yet challenging setting of simulating _multiple_
students in a collaborative problem-solving scenario with both human-agent and
agent-agent interaction and carefully design the system to emulate authentic
students. Our work will thus offer unique insights into the applications of
LLMs to complicated multi-agent settings both in mathematics education and
beyond.

##  5 Conclusion

We present MathVC, a novel LLM-powered platform simulating middle-school
students collaborating in mathematics problem-solving. To this end, we
proposed the use of symbolic schema and meta planner to encourage simulation
alignment. Our experimental results demonstrated the effectiveness of our
simulation and the promise for MathVC to be deployed to enhance mathematics
education in the future.

##  6 Ethics Statement

Our research is motivated by the pressing need to alleviate the dependence on
teachers to orchestrate collaborative student learning for mathematics
modeling (MM), a critical skill in STEM fields. To the best of our knowledge,
MathVC is the very first LLM-powered multi-character virtual classroom
designed for this purpose. We envision that such a platform can eventually be
deployed as effective take-home exercises for students practicing their MM
skills. This is particularly important for marginalized communities where only
limited teachers and educational resources are available, and our system has
the potential to become a tool to promote educational equity. In addition, we
also identify a few other benefits of how MathVC could assist students in
learning. For example, MathVC can reduce the pressure and anxiety of students
for taking peer discussions, thus increasing their opportunities to
participate. This is especially beneficial for students who may be reluctant
to participate due to shyness, low self-esteem, language barriers, or poor
academic performance. Our system thus promotes inclusion, allowing a greater
variety of students to participate, thereby increasing diversity and breadth.
Finally, we note that teachers have the flexibility to configure the student
characters according to each human student’s background and performance, which
can thus enable personalized learning experiences. In the future, one could
also extend our system to automatically optimize the character configuration
based on the student’s past performance.

Despite all the promises discussed above, however, we advocate careful
discussion and comprehensive evaluation of any AI system’s safety before
applying them to serve real humans. Due to this reason, in our project aiming
for the first prototype system of a mathematics virtual classroom, we opted
_not_ to involve real middle-school students in the loop, before we can
confirm the system’s safety. On the other hand, performing human subject
studies with middle-school students requires collecting informed consent from
their parents or guardians and obtaining permission from the school, which
typically needs months of preparation time or longer. Therefore, our main
evaluation was only conducted by examining the dialogues simulated among
solely virtual characters.

As we discussed in Section 3.3, one extension of our system could be to enable
richer characteristics simulation, including but not limited to gender, age,
name, culture, language, and academic background. The significant advantage of
such features is to allow for more engaging character simulation because, for
example, students may enjoy the interaction with classmates sharing similar
backgrounds with them. However, the introduction of these features may trigger
biases in the LLM, such as leading to an unfair simulation of students from
different backgrounds. When human students are exposed to such unfairly
simulated characters, stereotypes may be exaggerated. In our analysis
(Appendix G), we have not observed any such unethical simulation. However,
future research following our work should remain cautious, ensuring that the
simulated students objectively and fairly reflect the diversity presented in
the real world.

##  7 Reproducibility Statement

We have documented the major implementation details, both in the main text and
the appendix sections, including the input and output for all the modules and
the prompts for key components (Appendix I). Our source code will be publicly
available in the future. Our experimental setup is described in detail in the
appendix. We will publicize our sampled data and LLM-generated dialogues with
annotators’ scores and comments.

##  8 Acknowledgment

We were grateful to receive support from Microsoft’s Accelerating Foundation
Models Research program (https://www.microsoft.com/en-
us/research/collaboration/accelerating-foundation-models-research/) for
conducting this research. We also appreciate computing resources from the
Office of Research Computing (https://orc.gmu.edu) and funding support from
the Department of Computer Science at GMU. W. Mifdal was sponsored by the
Office of Student Creative Activities and Research (OSCAR) at GMU. We thank
Dr. Anthony Eamonn Kelly (GMU, Educational Psychology) and Dr. Melissa A.
Gallagher (University of Houston, Department of Curriculum & Instruction) for
early discussions about the idea, and students in the GMU NLP group
(https://nlp.cs.gmu.edu/) for their thoughtful comments.

## References

  * Ahn et al. (2024) Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin.  Large language models for mathematical reasoning: Progresses and challenges, 2024.
  * Bulusu et al. (2023) Arya Bulusu, Brandon Man, Ashish Jagmohan, Aditya Vempaty, and Jennifer Mari-Wyka.  An automated graphing system for mathematical pedagogy.  In _NeurIPS’23 Workshop on Generative AI for Education (GAIED)_ , 2023.
  * Bush et al. (2020) Sarah B Bush, George J Roy, and Christa Jackson.  Catalyzing change in middle school mathematics: Initiating critical conversations.  2020\.
  * Cheng & Chin (2024) Junyan Cheng and Peter Chin.  Sociodojo: Building lifelong analytical agents with real-world text and time series.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=s9z0HzWJJp.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Denny et al. (2024) Paul Denny, Sumit Gulwani, Neil T. Heffernan, Tanja Käser, Steven Moore, Anna N. Rafferty, and Adish Singla.  Generative ai for education (gaied): Advances, opportunities, and challenges, 2024.
  * Fischer (2023) Kevin A. Fischer.  Reflective linguistic programming (rlp): A stepping stone in socially-aware agi (socialagi), 2023.
  * Garfunkel & Montgomery (2019) Sol Garfunkel and Michelle Montgomery.  _GAIMME—Guidelines for Assessment & Instruction in Mathematical Modeling Education_.  SIAM, 2019.
  * Hong et al. (2023) Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber.  Metagpt: Meta programming for a multi-agent collaborative framework, 2023.
  * Hua et al. (2024) Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang.  War and peace (waragent): Large language model-based multi-agent simulation of world wars, 2024.
  * Lee et al. (2023) Unggi Lee, Sanghyeok Lee, Junbo Koh, Yeil Jeong, Haewon Jung, Gyuri Byun, Jewoong Moon, Jieun Lim, and † HyeoncheolKim.  Generative agent for teacher training: Designing educational problem-solving simulations with large language model-based agents for pre-service teachers.  2023\.  URL https://api.semanticscholar.org/CorpusID:266874743.
  * Levonian et al. (2023) Zachary Levonian, Chenglu Li, Wangda Zhu, Anoushka Gade, Owen Henkel, Millie-Ellen Postle, and Wanli Xing.  Retrieval-augmented generation to improve math question-answering: Trade-offs between groundedness and human preference, 2023.
  * Li et al. (2023a) Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie.  Large language models understand and can be enhanced by emotional stimuli, 2023a.
  * Li et al. (2016) Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen.  A user simulator for task-completion dialogues.  _arXiv preprint arXiv:1612.05688_ , 2016.
  * Li et al. (2023b) Yuan Li, Yixuan Zhang, and Lichao Sun.  Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents, 2023b.
  * Liu et al. (2023) Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang, and Soroush Vosoughi.  Training socially aligned language models in simulated human society, 2023.
  * Lu et al. (2023) Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang.  A survey of deep learning for mathematical reasoning.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 14605–14631, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.817.  URL https://aclanthology.org/2023.acl-long.817.
  * Macina et al. (2023) Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan.  MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 5602–5621, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.372.  URL https://aclanthology.org/2023.findings-emnlp.372.
  * MAP (n.d.) MAP.  Mathematics Assessment Project, n.d.  Accessed March 2024. https://www.map.mathshell.org/.
  * Markel et al. (2023) Julia M. Markel, Steven G. Opferman, James A. Landay, and Chris Piech.  Gpteach: Interactive ta training with gpt-based students.  _Proceedings of the Tenth ACM Conference on Learning @ Scale_ , 2023.  URL https://api.semanticscholar.org/CorpusID:259949836.
  * McNichols et al. (2024) Hunter McNichols, Wanyong Feng, Jaewook Lee, Alexander Scarlatos, Digory Smith, Simon Woodhead, and Andrew Lan.  Automated distractor and feedback generation for math multiple-choice questions via in-context learning, 2024.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2024.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  In _In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23)_ , UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * PISA (2018) OECD PISA.  assessment and analytical framework. 2019, 2018.
  * Qian et al. (2023) Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Communicative agents for software development, 2023.
  * Shao et al. (2023) Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu.  Character-llm: A trainable agent for role-playing, 2023.
  * Standards (2010) Common Core State Standards.  Common core state standards for mathematics, national governors association center for best practices and the council of chief state school officers.  _Washington, DC_ , 2010.
  * Stolcke et al. (2000) Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer.  Dialogue act modeling for automatic tagging and recognition of conversational speech.  _Computational linguistics_ , 26(3):339–373, 2000.
  * Tang et al. (2024) Yihong Tang, Jiao Ou, Che Liu, Fuzheng Zhang, Di Zhang, and Kun Gai.  Enhancing role-playing systems through aggressive queries: Evaluation and improvement, 2024.
  * TNTP (2018) TNTP.  _The Opportunity Myth: What Students Can Show Us About How School Is Letting Them Down—And How to Fix It_.  2018\.
  * Vygotsky & Cole (1978) Lev Semenovich Vygotsky and Michael Cole.  _Mind in society: Development of higher psychological processes_.  Harvard university press, 1978.
  * Wang et al. (2024) Angelina Wang, Jamie Morgenstern, and John P. Dickerson.  Large language models cannot replace human participants because they cannot portray identity groups, 2024.
  * Wang et al. (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv: Arxiv-2305.16291_ , 2023a.
  * Wang et al. (2023b) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_ , 2023b.
  * Wang et al. (2023c) Zhilin Wang, Yu Ying Chiu, and Yu Cheung Chiu.  Humanoid agents: Platform for simulating human-like generative agents.  In Yansong Feng and Els Lefever (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 167–176, Singapore, December 2023c. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-demo.15.  URL https://aclanthology.org/2023.emnlp-demo.15.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * Wikipedia (n.d.) Wikipedia.  Alice and bob, n.d.  Accessed March 2024. https://en.wikipedia.org/wiki/Alice_and_Bob.
  * Xu et al. (2023) Binfeng Xu, Xukun Liu, Hua Shen, Zeyu Han, Yuhan Li, Murong Yue, Zhiyuan Peng, Yuchen Liu, Ziyu Yao, and Dongkuan Xu.  Gentopia.AI: A collaborative platform for tool-augmented LLMs.  In Yansong Feng and Els Lefever (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 237–245, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-demo.20.  URL https://aclanthology.org/2023.emnlp-demo.20.
  * Yu et al. (2022) Jifan Yu, Xiaohan Zhang, Yifan Xu, Xuanyu Lei, Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and Jie Tang.  Xdai: A tuning-free framework for exploiting pre-trained language models in knowledge grounded dialogue generation.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , KDD ’22, pp. 4422–4432, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450393850.  doi: 10.1145/3534678.3539135.  URL https://doi.org/10.1145/3534678.3539135.
  * Yu et al. (2024) Xiaoyan Yu, Tongxu Luo, Yifan Wei, Fangyu Lei, Yiming Huang, Hao Peng, and Liehuang Zhu.  Neeko: Leveraging dynamic lora for efficient multi-character role-playing agent, 2024.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi.  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 654–664, 2017.
  * Zhou et al. (2023) Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al.  Characterglm: Customizing chinese conversational ai characters with large language models.  _arXiv preprint arXiv:2311.16832_ , 2023.

##  Appendix A Workflow Algorithm of MathVC

The workflow algorithm of MathVC is shown in Algorithm 1.

Algorithm 1 MathVC Workflow

1:Characteristics $C$, Math Question $Q$, Answer $A$, Common Mistakes $M$,
Dialogue Act Candidates $DAC$, End Stage $end$, $Max\ Conv.\ Turn$

2:Dialogue $D$

3:Initialize stage $\mathrm{s}\leftarrow 0$, $D\leftarrow[]$

4:$i\leftarrow 0$

5:while $i\leq Max\ Conv.\ Turn$ do

6: if $i=0$ then

7: $S_{t}\leftarrow\textsc{Task\ Schema\ Generator}(Q,A)$ $\triangleright$
generate task schema

8: $S_{c}\leftarrow\textsc{Character\ Schema\ Generator}(S_{t},M,C)$
$\triangleright$ generate character schema

9: $DA\leftarrow\textsc{Dialogue\ Act\
Generator}(S_{c},\mathrm{s},DAC[\mathrm{s}],D)$ $\triangleright$ generate
dialogue act

10: $R\leftarrow\textsc{response\ generator}(S_{c},DA)$ $\triangleright$
generate response

11: else

12: $Next\ Speaker\leftarrow\textsc{Next\ Speaker\ Predictor}(D)$
$\triangleright$ get next speaker

13: $\mathrm{s}\leftarrow\textsc{Dialogue\ Stage\ Monitor}(D,\mathrm{s})$
$\triangleright$ stage update

14: if $\mathrm{s}\ is\ end$ then

15: break

16: end if

17: $S_{c}\leftarrow\textsc{Character\ Schema\ Modifier}(S_{c},D)$
$\triangleright$ modity character schema

18: $DA\leftarrow\textsc{Dialogue\ Act\
Generator}(S_{c},\mathrm{s},DAC[\mathrm{s}],D)$

19: $R\leftarrow\textsc{Response\ Generator}(S_{c},DA,D)$

20: end if

21: $D\leftarrow D+[R]$

22: if $Human\\_Response$ then

23: $D\leftarrow D+[Human\\_Response]$

24: end if

25: $i\leftarrow i+1$

26:end while

##  Appendix B Mathematical Modeling Discussion Stages and Dialogue Acts

Stage | Dialogue Act Candidates
---|---
Establish and maintain shared understanding |  | Present a task understanding
---
Ask a clarifying question about task understanding
Answer a clarifying question about task understanding
Second a task understanding
Ask for agreement on a task understanding
Establish team organization |  | Initiate the workload division
---
Volunteer to serve a role
Second role designation plan
Ask for agreement on the role designation
Plan actions for problem-solving |  | Prompt a teammate to join the discussion
---
State an action plan
Ask a clarifying question about an action plan
Answer a clarifying question about an action plan
Second an action plan
Ask for agreement on an action plan
Execute actions for problem-solving |  | Execute an action plan and state the execution result
---
Ask a clarifying question about an execution result
Answer a clarifying question about an execution result
Second an execution result
Ask for agreement on an execution result
Validating the answer |  | Reflection on the implication of modeling outcomes
---
Summarize the results

Table 2: Stages and Dialogue Acts

We list the MM discussion stages and dialogue acts in the Table 2. “Plan(ning)
actions” in the table refers to planning on the modeling of certain variables.
“Execut(ing) actions” refers to executing the planned variable calculation and
obtaining the result.

##  Appendix C Example and Prompt for Character Schema Generation

Here is an example of character schema generation:

Question: Martha wants to set up a soup stall at a Farmer’s Market. She hopes
to sell 500 mugs of soup, each with a white or brown bread roll. She will sell
a mug of soup with a bread roll for $1.25. Martha knows that She can buy the
soup in 2.5-liter bottles. Each bottle of soup costs $5 and provides ten
servings. Bread rolls are sold in packs of 10. Each pack costs $2. The mugs
will not cost her anything, as she can borrow them from a friend. Martha
surveys 40 people to find out what flavor of soup they would be most likely to
buy and what kind of bread roll they would prefer. The survey responses show
that Among the 40 people, 6 voted for soup of carrot and coriander, 16 for
tomato, 10 for leek and potato, and 8 for chicken and vegetable. Among the
same 40 people, 30 voted for white bread rolls, and 10 voted for brown bread
rolls. What exactly should Martha buy so that she can make the most profit and
not have lots of soup and rolls left over at the end?

Characteristics:

  * •

Name: Alice

  * •

Gender: Female

  * •

Career: 6-grade student in the US

  * •

MM skill level: Bad

Common mistakes:

  1. 1.

The student suggests that Martha should buy 12.5 bottles of leek potato soup,
and forget to round up.

  2. 2.

The student assumes that Martha will buy equal quantities of each kind of
soup.

  3. 3.

The student’s solution contains incorrect calculations, such as calculating
40% of 500 is 125.25.

Task Schema in Python (generated by GPT-4)

⬇

1{

2 "task 1": {

3 "description": "Calculate the percentage of each soup flavor preference
based on the survey.",

4 "variables": {

5 "total_surveyed": 40,

6 "carrot_coriander_votes": 6,

7 "tomato_votes": 16,

8 "leek_potato_votes": 10,

9 "chicken_vegetable_votes": 8,

10 "carrot_coriander_percentage": 15, # 6/40 * 100

11 "tomato_percentage": 40, # 16/40 * 100

12 "leek_potato_percentage": 25, # 10/40 * 100

13 "chicken_vegetable_percentage": 20 # 8/40 * 100

14 }

15 },

16 "task 2": {

17 "description": "Calculate the percentage of each bread roll preference
based on the survey.",

18 "variables": {

19 "white_bread_votes": 30,

20 "brown_bread_votes": 10,

21 "white_bread_percentage": 75, # 30/40 * 100

22 "brown_bread_percentage": 25 # 10/40 * 100

23 }

24 },

25 "task 3": {

26 "description": "Calculate the number of servings needed for each soup
flavor to cater to 500 customers.",

27 "variables": {

28 "total_customers": 500,

29 "servings_carrot_coriander": 75, # 500 * 15%

30 "servings_tomato": 200, # 500 * 40%

31 "servings_leek_potato": 125, # 500 * 25%

32 "servings_chicken_vegetable": 100 # 500 * 20%

33 }

34 },

35 "task 4": {

36 "description": "Calculate the number of bottles needed for each soup
flavor.",

37 "variables": {

38 "servings_per_bottle": 10,

39 "bottles_carrot_coriander": 8, # ceil(75 / 10)

40 "bottles_tomato": 20, # 200 / 10

41 "bottles_leek_potato": 13, # ceil(125 / 10)

42 "bottles_chicken_vegetable": 10 # 100 / 10

43 }

44 },

45 "task 5": {

46 "description": "Calculate the number of packs of bread rolls needed for
each type.",

47 "variables": {

48 "rolls_per_pack": 10,

49 "packs_white_bread": 38, # ceil(375 / 10)

50 "packs_brown_bread": 13 # ceil(125 / 10)

51 }

52 },

53 "task 6": {

54 "description": "Calculate the total cost of soup and bread rolls.",

55 "variables": {

56 "cost_per_bottle_soup": 5,

57 "cost_per_pack_bread": 2,

58 "total_cost_soup": 255, # (8 + 20 + 13 + 10) * 5

59 "total_cost_bread": 102, # (38 + 13) * 2

60 "total_cost": 357 # 255 + 102

61 }

62 },

63 "task 7": {

64 "description": "Calculate the total revenue from selling 500 mugs of soup
with bread rolls.",

65 "variables": {

66 "price_per_mug_with_roll": 1.25,

67 "total_revenue": 625 # 500 * 1.25

68 }

69 },

70 "task 8": {

71 "description": "Calculate the profit Martha will make.",

72 "variables": {

73 "total_revenue": 625,

74 "total_cost": 357,

75 "profit": 268 # 625 - 357

76 }

77 }

78}

Listing 1: An example task schema represented in Python

Example of Variables to Edit for Character Schema Modification (generated by
GPT-4):

⬇

1{

2 "bottles_leek_potato": 12.5,

3 "total_cost_soup": 252.5,

4 "total_cost": 354.5,

5 "profit": 270.5

6}

Listing 2: An example output of the Character Schema Modifier which indicates
variables to edit in Python

We will then apply these variable edits to the task schema to initialize the
character schema.

Prompt for Character Schema Modifier:

⬇

1Question: Martha wants to set up a soup stall...

2task schema: {

3 "task 1": {

4 ...

5}

6The error student will make:

71. The student suggests that Martha should buy 12.5 bottles of leek potato soup, and forget to round up.

82. The student assumes that Martha will buy equal quantities of each kind of soup.

93. The student’s solution contains incorrect calculations, such as calculating 0.4 of 500 is 125.25.

10Character:

11Name: Alice

12Gender: Female

13Career: 6 Grade Student in the US

14Skill at Math: Bad

15Will Alice make a mistake in solving math questions based on his/her
characteristics?

16If the answer is no, reply no.

17If the answer is yes, please select one or multiple mistakes from above.
Based on the error a student would make, variables need to be edited in the
thoughts schema. Generate a Python dict without any comment(with "#") to
indicate which variables should be edited, the format is:

18‘‘‘python

19{var1_name: incorrect_var1_value, var2_name: incorrect_var2_value,...}

20‘‘‘

Listing 3: Prompt for character schema modification

##  Appendix D Details of Experimental Setup

The GPT-4 we used was GPT-4-turbo version “11-06-preview”. The temperature is
set to 0.7, top_p to 1, repetition_penalty to 1 and max_tokens to 1024. The
student character implementation is based on a lightweight open-source
platform Gentopia (Xu et al., 2023), on top of which we further developed the
proposed meta planner to form MathVC. In our simulation, we have two different
setups of simulated characters:

  * •

Characteristics Setup I:

    * –

Name: Alice; Gender: Female; Career: 6 Grade student in the US; MM skill
level: Bad

    * –

Name: Bob; Gender: Male; Career: 6 Grade student in the US; MM skill level:
Good

    * –

Name: Charlie; Gender: Male; Career: 6 Grade student in the US; MM skill
level: Good

  * •

Characteristics Setup II:

    * –

Name: Alice; Gender: Female; Career: 6 Grade student in the US; MM skill
level: Bad

    * –

Name: Bob; Gender: Male; Career: 6 Grade student in the US; MM skill level:
Bad

    * –

Name: Charlie; Gender: Male; Career: 6 Grade student in the US; MM skill
level: Good

We manually add some common mistakes to the questions from the GSM8k. One
example is as follows:

Question: Jon runs a triathlon. It takes him 40 minutes for the swim, an hour
and 20 minutes for the bike ride, and 50 minutes for the run. Compared to Jon,
James finishes the swim 10% faster but takes 5 minutes longer on the bike. If
Jon won by 10 minutes, how long did it take James to do the run?

Answer: Jon did the bike ride in 60+20=80 minutes. So his total time was
40+80+50=170 minutes. James finished the swim 40*.1=4 minutes faster. So he
finished the swim in 40-4=36 minutes. He finished the bike ride in 80+5=85
minutes. James’ total time was 170+10=180. So it took him 180-85-36=59 minutes
to do the run.

Common Mistakes:

  1. 1.

The student thinks the time to take the bike for Jon is 20 min and calculates
the time for Jon to finish the game into 40+20+50=110 minutes.

  2. 2.

The student calculates the time it took James to complete the swim correctly
but then forgets to convert Jon’s bike time to minutes before adding the 5
minutes that James took longer.

  3. 3.

The student adds the 10 minutes that Jon won by to the total time it took
James to complete the swim and bike ride instead of subtracting it from Jon’s
total time to find James’ run time.

##  Appendix E Ablation Studies Evaluating Key Prompt Designs in MathVC

In this section, we investigate the effects of selecting different prompt
strategies in each module. Through the prompting study, we would like to
carefully design the prompt for each component to allow them to reach their
full potential. In addition, we aim to provide some insights from a prompt
engineering perspective for future simulations of other characters. When we
explore and select the best prompt, we manually check the generation of the
components to determine whether they are satisfactory. The analysis was
performed manually on the dev set. Our results are presented in Table 3.

Character schema update: regeneration or editing  While we prompt the LLM to
generate a sequence of edits that should be applied to the present character
schema, an alternative approach could be to directly generate the updated
schema. As shown in Table 3(a), we found that editing schema yields a higher
satisfaction rate than regenerating it. Although regeneration can generate
schema flexibly, it is prone to add some comments, such as “Note: this
variable is incorrect”, which is the information that the simulated student
should not know. Besides, regeneration takes a longer time when it generates a
schema from scratch.

Method | Satisfaction
---|---
Generation | 0.73
Editing (ours) | 1.00

(a) Character Schema Update

Method | Satisfaction
---|---
Selection | 0.59
Generation (ours) | 0.85

(b) Action Generation

Method | Satisfaction
---|---
Direct Generation | 0.92
Two-step Generation | 1.00

(c) Response Generation

Table 3: Ablation studies validating the effectiveness (satisfaction rate) of
our prompt design.

Dialogue act generation: selection or generation  Alternative to our approach
of prompting the LLM to generate a plausible dialogue act and describe a more
detailed plan, one could prompt the LLM to directly select an act from the
list. Our results in Table 3(b) showed that our approach often yields more
contextually reasonable acts. We conjecture that the list of dialogue acts is
too generic for the LLM to select while allowing it to articulate its next
action facilitates its planning.

Response generation: direct or two-step generation As described in Section
2.4, we utilize a two-step generation in response generation. A simple
approach is to directly generate the response without variable retrieval from
the character schema. However, our results in Table 3(c) suggest a
disadvantage of this direct approach.

##  Appendix F Human Interaction

In the main experiment (Section 3.2), we only show the dialogues between
simulated student characters. In this section, we would like to show the
interaction between human and simulated characters. We are not able to recruit
middle-school students. Therefore, one of our authors tries to mimic a middle-
school student to conduct a preliminary study.

We show an example below. In this example, we use Daniel as the placeholder
for human. At every turn, Daniel could choose to respond or skip.

In this example, we have observed that the dialogue flows smoothly, and the
interactions between human and simulated characters are natural. For example,
when a student asks a question, the simulated character could respond
appropriately to address the query; if a student provides an incorrect answer,
the simulated character could correct it; and if a student would like to talk
with a specific character, that character responds accordingly. This is
largely attributed to the powerful text understanding and interaction
capabilities of the LLM, showcasing the promise for future deployment.

⬇

1We use Daniel as the placeholder name of human

2Now let’s start a group discussion within the group of Alice, Bob, Charlie,
and Daniel

3Daniel: Hi good to see you guys! Let’s work together!

4Alice: Hey everyone, I think we should split the tasks based on our
strengths. I’m happy to take on the research part. What roles do you guys feel
comfortable with? Let’s make sure we’re all okay with our responsibilities

5Daniel: I will take responsibility for validating the answers!

6Bob: I’m good with organizing and keeping track of our progress, so I can
handle the project management side of things. Does that work for everyone?

7Daniel: How about you Charlie?

8(When Daniel asks the reply from Charlie, the next speaker is correctly
predicted as Charlie.)

9Charlie: I’m comfortable with handling the writing and compilation of our
work. Does that fit well with everyone else’s roles?

10Daniel: It works for me!

11Charlie: Let’s start the problem solving by calculating the soup flavor
preferences from the survey data we’ve collected.

12Daniel: How to get this?

13(When Daniel asks the question, the simulated character will answer it.)

14Charlie: So, we just take the number of votes each soup got, like 6 votes
for Carrot Coriander, and divide it by the total number of people surveyed,
which is 40, then multiply by 100 to get the percentage, which for Carrot
Coriander is 15%!

15Daniel: Oh, it makes sense to me! So the tomato is 16/40, which is 40% and
leek is 10/40=25%.

16(When Daniel gives correct solution, the simulated character will encourage
him.)

17Charlie: Exactly, Daniel! And for the Chicken Vegetable, it’s 8 out of 40,
which gives us 20%. Now let’s figure out the bread roll preferences; 30 people
chose White Bread, so that’s 75%, and 10 for Brown Bread, which is 25% of our
survey.

18Alice: Great job on the percentages, guys! Now let’s figure out how many
servings we need for each soup to serve 500 people; we’ll use the percentages
we just calculated to get the right amounts for each flavor!

19Daniel: So the number of people to buy carrot soup is 500*0.4=20?

20(When Daniel makes a mistake, the simulated character will correct it with
details.)

21Charlie: Oh, Daniel, you got a little mixed up! For the Carrot Coriander
soup, we need to use its percentage, which is 15%, not 40%. So we multiply 500
customers by 0.15, and that gives us 75 servings for Carrot Coriander. The
Tomato soup needs 200 servings because it’s 40%, Leek Potato needs 125
servings for 25%, and Chicken Vegetable needs 100 servings for 20%.

22Daniel: Oh! I understand it now! What’s your thought, Bob?

23Bob: Yep, Charlie, you’ve got it right! We need 75 servings of Carrot
Coriander, 200 for Tomato, 125 for Leek Potato, and 100 for Chicken Vegetable.

24Daniel: What’s your thought, Alice?

25(When Daniel asks Alice, she replies him.)

26Alice: I think we need 8 bottles for Carrot Coriander, 20 for Tomato, 13 for
Leek Potato, and 10 for Chicken Vegetable. Now, let’s figure out how many
packs of bread rolls we need, and then we can work out the total cost for all
the soup and bread!"

27Bob: Actually, for the Leek and Potato soup, we need exactly 12.5 bottles
because 125 servings divided by 10 servings per bottle equals 12.5.

28Daniel: But I think we need to round it up to 13!

29(When Daniel corrects other’s mistake, the simulated character will convey
agreement.)

30Alice: Yep, rounding up to 13 bottles for the Leek Potato soup makes sense,
and we’ll need 38 packs for White Bread and 13 for Brown Bread. After that, we
can add up the costs to see how much all the soup and bread will set us back!"

31Bob: Alright, we’re set with 13 bottles for the Leek Potato soup! Let’s add
up the costs now; each bottle is $5 and each pack of rolls is $2, so if we do
the math, we’ll see how much we’ll spend on all the soup and bread rolls
combined!

32Daniel: I like the current progress! Now let’s find the final answer!

33Alice: We’ve got 13 bottles for the Leek Potato soup since we always round
up, and we need 38 packs for White Bread and 13 for Brown Bread. If we
multiply the number of bottles and packs by their costs, we’ll spend $252.5 on
soup and $102 on bread rolls, making our total $354.5!

34Daniel: I don’t agree with that. Could anyone help to check the answer?

35(When Daniel express disagreement, other simulated characters will actively
discuss this.)

36Charlie: Hey guys, I think there’s a little mix-up; the total cost for the
soup is actually $255 and for the bread rolls, it’s $102, so our total cost
should be $357. And yep, we’re using 13 bottles for the Leek Potato soup.
Let’s double-check these numbers to make sure everything adds up right!

37Daniel: I agree with that!

38Alice: You’re right, the soup costs $255 and the bread rolls are $102, so
our total is $357, and we definitely need 13 bottles for the Leek Potato soup.
Let’s double-check to make sure we’ve got it all correct!

39Bob: Yep, the soup definitely costs us $255 and with the bread rolls at
$102, our grand total is $357, and we’re getting 13 bottles for the Leek
Potato soup to cover all the servings we need!

Listing 4: An example dialogue with human interaction

##  Appendix G Characteristics Analysis

To analyze the impact of traits other than MM skill levels, we use different
characteristics with different names such as Alice/Bob/Charlie compared with
Cecelia/Lucas/Leo, as well as varying gender setups, such as all male or
female. Additionally, we explore different career setups, such as varying
grades (ranging from 6 to 8) or countries (e.g., US or Brazil). These
variations are tested on our dev set.

Instead of running the simulation under each characteristic configuration end
to end, we conducted a series of controlled ablation studies. That is, for
each component in MathVC, we first randomly sampled a few LLM queries from the
log of our main experiments (e.g., a query for response generation is shown in
the listing below). We then prompted the LLM with two different characteristic
configurations and checked the difference between the generation results. For
instance, we initially set the character names as Alice, Bob, and Charlie and
let the LLM generate a response. In our ablation study, we replaced the names
with Cecelia, Lucas, and Leo without any other changes in the prompt
(including the conversational context) and prompted the LLM to generate
another response. Finally, we compare the differences between these two
generations. To mitigate the effect of randomness, we set the sampling
temperature to 0. The advantage of this analysis compared with end-to-end
dialogue generation lies in that it offers us a more controlled and direct
comparison between two characteristic configurations, with all other factors
being exactly the same.

On the dev set, we randomly selected about 30 prompts containing
characteristics from our log file, covering different modules and discussion
stages. We only replaced one variable each time, such as only replacing names
or genders. Then we manually examined the differences. An example is shown
below.

⬇

1Input with names Alice, Bob and Charlie:

2Charlie’s thoughts schema:

3{

4 "task 1": {

5 ...

6}

7

8Conversation:

9Now let’s start a group discussion!

10Bob: " Hey everyone, so we’ve got several tasks ahead of us, including
calculating preferences for soup flavors and bread rolls from a survey,
determining the number of servings and bottles needed for each soup flavor,
figuring out the packs of bread rolls required, and then calculating the total
costs, revenue, and profit for Martha’s event. Let’s tackle these one by one!"

11Alice: " Okay, so we’re starting with calculating the soup and bread roll
preferences from the survey, then we’ll work out how much of each we need, and
finally, we’ll figure out the costs, revenue, and profit for the event,
right?"

12Charlie: " I can start by calculating the soup and bread roll preferences
from the survey. Does anyone want to tackle the servings and bottles needed,
or should we decide that together? Also, great job outlining the tasks, Bob!"

13Alice: " I can handle the calculations for the servings and bottles needed
for each soup flavor if that’s okay with everyone?"

14Bob: " Thanks, Charlie, for taking on the survey preferences, and Alice,
your help with the servings and bottles is perfect. I can calculate the total
costs, revenue, and profit unless someone else wants to. Does this sound good
to everyone?"

15Alice: " Let’s start the problem solving by figuring out how many people
preferred each soup flavor by calculating the percentage based on our survey
results."

16Generate a reply from Charlie based on the action and the variables from
Charlie thought schema. Never mention the "task" word in the reply. Remember
you’re a grade school student, please reply in one sentence with the tone of a
grade school student. You believe that your thought schema is completely
correct, and all the variables in reply must be perfectly aligned to your own
thought schema.

17The action to take is " Charlie will begin by discussing task 1 and task 2,
using the variables from his thought schema to calculate the percentages of
each soup flavor and bread roll preference based on the survey results.
Charlie will then present these calculations to the group.". Before generating
the reply, ground the tasks and variables mentioned in the action from your
thought schema. Then generate the response. The output format is Related tasks
index:... Grounding values in those tasks:... Charlie:...

18Output with names Alice, Bob and Charlie:

19Related tasks index: 1, 2.

20Grounding values in those tasks: For the soup flavors, we have carrot
coriander at 15%, tomato at 40%, leek potato at 25%, and chicken vegetable at
20%. For the bread rolls, white bread is preferred by 75% and brown bread by
25%.

21Charlie: "Based on our survey, 15% like carrot coriander soup, 40% prefer
tomato, 25% go for leek potato, and 20% choose chicken vegetable. For bread
rolls, 75% enjoy white bread and 25% prefer brown bread."

22

23Input with names Cecelia, Lucas and Leo:

24Leo’s thoughts schema:

25{

26 "task 1": {

27 ...

28}

29

30Conversation:

31Now let’s start a group discussion!

32Lucas: " Hey everyone, so we’ve got several tasks ahead of us, including
calculating preferences for soup flavors and bread rolls from a survey,
determining the number of servings and bottles needed for each soup flavor,
figuring out the packs of bread rolls required, and then calculating the total
costs, revenue, and profit for Martha’s event. Let’s tackle these one by one!"

33Cecelia: " Okay, so we’re starting with calculating the soup and bread roll
preferences from the survey, then we’ll work out how much of each we need, and
finally, we’ll figure out the costs, revenue, and profit for the event,
right?"

34Leo: " I can start by calculating the soup and bread roll preferences from
the survey. Does anyone want to tackle the servings and bottles needed, or
should we decide that together? Also, great job outlining the tasks, Lucas!"

35Cecelia: " I can handle the calculations for the servings and bottles needed
for each soup flavor, if that’s okay with everyone?"

36Lucas: " Thanks, Leo, for taking on the survey preferences, and Cecelia,
your help with the servings and bottles is perfect. I can calculate the total
costs, revenue, and profit unless someone else wants to. Does this sound good
to everyone?"

37Cecelia: " Let’s start the problem solving by figuring out how many people
preferred each soup flavor by calculating the percentage based on our survey
results."

38Generate a reply from Leo based on the action and the variables from Leo
thought schema. Never mention the "task" word in the reply. Remember you’re a
grade school student, please reply in one sentence with the tone of a grade
school student. You believe that your thought schema is completely correct,
and all the variables in reply must be perfectly aligned to your own thought
schema.

39The action to take is " Leo will begin by discussing task 1 and task 2,
using the variables from his thought schema to calculate the percentages of
each soup flavor and bread roll preference based on the survey results. Leo
will then present these calculations to the group.". Before generating the
reply, ground the tasks and variables mentioned in the action from your
thought schema. Then generate the response. The output format is Related tasks
index:... Grounding values in those tasks:... Leo:...

40Output with names Cecelia, Lucas and Leo:

41Related tasks index: 1, 2

42Grounding values in those tasks:

43- For soup preferences, 15% like carrot coriander, 40% prefer tomato, 25% go
for leek potato, and 20% choose chicken vegetable.

44- For bread roll preferences, 75% prefer white bread and 25% prefer brown
bread.

45Leo: "Based on our survey, 15% of people like carrot coriander soup, 40%
prefer tomato, 25% enjoy leek potato, and 20% choose chicken vegetable. For
bread rolls, 75% of folks prefer white bread, and 25% go for brown bread."

Listing 5: An example of characteristics analysis with varying character names

From the example, although we changed the names, the responses are quite
similar (except for negligible language variations). Overall, we observe that
these characteristic variations other than the math skills only have a very
minor impact on the LLM’s response. Most of the generations with different
variations remain the same and none of the generations have been greatly
changed. In particular, we did not observe any unfair simulation, such as
simulating students with certain names or from a certain gender population
with excessively good or bad behaviors. The reason could be that through
reinforcement learning with human feedback (Ouyang et al., 2022), GPT-4 has
been fine-tuned to avoid potential biases of these variations. Additionally,
our prompts focus on discussing mathematical problems, without emphasizing
backgrounds other than the MM skill levels of students. Under this well-
defined scope, these background variations are not modeled as decisive factors
in our current simulation.

In the future, we consider it as a potentially impactful topic to
systematically investigate approaches to properly simulate different cultural
or linguistic backgrounds of students. This could be an important step toward
providing personalized learning environments to individual human students
because, intuitively, students can benefit from interacting with teammates who
share a similar background with them. Systematic evaluation of the potential
biases should then be carefully designed and conducted.

##  Appendix H Human Evaluation Guideline

The annotators were instructed to evaluate the characteristics alignment and
conversational procedural alignment of each sampled multi-character dialogue.
For the former, the annotators evaluate whether the multi-turn responses from
each character are well aligned to its configured characteristics and present
coherently changing problem understanding and solving. For the latter, the
annotators evaluate whether the entire conversation follows a procedure close
to a real collaborative mathematics problem-solving conversation among middle-
school students, particularly showing the transition across different stages.
For each metric, the annotators assigned a score ranging from 1 (worst) to 4
(best) accompanied by fine-grained comments for our post-analysis. The
annotation guideline is shown below.

⬇

1

2First, thank you for your help! This document is the guideline for the
evaluation. Every case contains a math question, its answer, each student’s
character traits, and multiple dialogues. The dialogues are shuffled to avoid
tricks. Please read the question, answer and character traits first. Then
evaluate the following aspects for each dialogue:

3Evaluate individual character responses

4Character Alignment: Do the multi-turn responses from a character coherently
show the change in problem understanding and modeling? Does a student with the
given character have the possibility to do math modeling like this?

5Scores:

61: not coherent

72: somewhat coherent

83: mostly coherent

94: perfectly coherent

10

11Evaluate inter-character conversations

12Conversational Procedure: Does the entire conversation show proper progress?
In our simulation, we define a successful conversation to go over stages:
understanding sharing, team organization, problem-solving, validation. Is
there likely that the given discussion is real from the middle school student?

13Scores:

141: not organized, not realistic

152: somewhat organized and realistic

163: mostly fine

174: perfect and realistic

18Lastly, please add your thoughts in the comment, especially for those
dialogues you give a low score.

Listing 6: Our annotation guideline for system evaluation

##  Appendix I Prompts for Dialogue Act Generation and Response Generation

⬇

1During conversations, people generate reactions based on their own thought
schemas and the content of the conversation and then respond with
corresponding conversational behaviors. Please determine the conversational
actions that the next speaker would take based on the following conversation
content and thought schema.

2Charlie thoughts schema:

3{{

4 "task 1": {{

5 "description": "Calculate the percentage of each soup flavor preference
based on the survey.",

6 "variables": {{

7 "total_surveyed": 40,

8 "carrot_coriander_votes": 6,

9 "tomato_votes": 16,

10 "leek_potato_votes": 10,

11 "chicken_vegetable_votes": 8,

12 "carrot_coriander_percentage": 15, # (6/40) * 100

13 "tomato_percentage": 40, # (16/40) * 100

14 "leek_potato_percentage": 25, # (10/40) * 100

15 "chicken_vegetable_percentage": 20 # (8/40) * 100

16 }}

17 }},

18 "task 2": {{

19 "description": "Calculate the percentage of each bread roll preference
based on the survey.",

20 "variables": {{

21 "white_bread_votes": 30,

22 "brown_bread_votes": 10,

23 "white_bread_percentage": 75, # (30/40) * 100

24 "brown_bread_percentage": 25 # (10/40) * 100

25 }}

26 }},

27 "task 3": {{

28 "description": "Calculate the number of servings needed for each soup
flavor to cater to 500 customers.",

29 "variables": {{

30 "total_customers": 500,

31 "servings_carrot_coriander": 75, # 500 * 15%

32 "servings_tomato": 200, # 500 * 40%

33 "servings_leek_potato": 125, # 500 * 25%

34 "servings_chicken_vegetable": 100 # 500 * 20%

35 }}

36 }},

37 "task 4": {{

38 "description": "Calculate the number of bottles needed for each soup
flavor.",

39 "variables": {{

40 "servings_per_bottle": 10,

41 "bottles_carrot_coriander": 8, # ceil(75 / 10)

42 "bottles_tomato": 20, # 200 / 10

43 "bottles_leek_potato": 13, # ceil(125 / 10)

44 "bottles_chicken_vegetable": 10 # 100 / 10

45 }}

46 }},

47 "task 5": {{

48 "description": "Calculate the number of bread roll packs needed for each
type to cater to 500 customers.",

49 "variables": {{

50 "rolls_per_pack": 10,

51 "packs_white_bread": 38, # ceil(375 / 10)

52 "packs_brown_bread": 13 # ceil(125 / 10)

53 }}

54 }},

55 "task 6": {{

56 "description": "Calculate the total cost for soup and bread rolls.",

57 "variables": {{

58 "cost_per_bottle_soup": 5,

59 "cost_per_pack_bread": 2,

60 "total_cost_soup": 255, # (8 + 20 + 13 + 10) * 5

61 "total_cost_bread": 102, # (38 + 13) * 2

62 "total_cost": 357 # 255 + 102

63 }}

64 }},

65 "task 7": {{

66 "description": "Calculate the total revenue from selling 500 mugs of soup
with bread rolls.",

67 "variables": {{

68 "price_per_mug_with_roll": 1.25,

69 "total_revenue": 625 # 500 * 1.25

70 }}

71 }},

72 "task 8": {{

73 "description": "Calculate the profit Martha will make.",

74 "variables": {{

75 "profit": 268 # 625 - 357

76 }}

77 }}

78}}

79Actions to take:

80(1)Plan actions for problem-solving:

81-Prompt a teammate to join the discussion

82-State an action plan

83-Ask a clarifying question about an action plan

84-Answer a clarifying question about an action plan

85-Second an action plan

86-Ask for agreement on an action plan

87(2) Execute actions for problem-solving:

88-Execute an action plan and state the execution result

89-Ask a clarifying question about an execution result

90-Answer a clarifying question about an execution result

91-Second an execution result

92-Ask for agreement on an execution result

93Conversation:

94Now let’s start a group discussion!

95Alice: " Hey team, I think we should divide the tasks. I can analyze the
survey data to determine the most popular soup and bread roll choices. What
roles would you all prefer, and does everyone agree with this plan?"

96Bob: " Sounds good, Alice! I can handle the cost analysis and figure out how
much we’ll need to spend. Does anyone want to tackle the logistics of
purchasing and setting up the stall? Let’s make sure we’re all on the same
page with these roles."

97Charlie: " I can take on the role of organizing the purchasing and setup of
the stall. Does that work for everyone, and is there anyone else who wants to
join in on a specific task? Let’s start the problem-solving by calculating how
much of each soup flavor we need based on the survey preferences to ensure we
cater to everyone’s taste without excess waste."

98Alice: "Bob, before we proceed, could you help me verify if we’re
calculating the servings for each soup flavor correctly? We need to ensure our
numbers are accurate to avoid over or underestimating the amount needed."

99Bob: "Alice, based on the survey, tomato is the most popular one. So we
should provide 500 servings of tomatoes."

100

101The next speaker is Charlie. Now please decide what action Charlie will
take, including discussing which tasks, discussing which variables, etc.
Remember you are Charlie, and your actions should follow your thought schema.
Please carefully examine the variables mentioned by others in the
conversation. For any variables in the dialogue that are inconsistent with
your thought schema values, you should try to request others to examine them
carefully. Please remember, "Action" only indicates how Charlie will take
action in the next response, such as asking questions or continuing to discuss
the next task, and should not include what Charlie might say. The output
format is:

102variables and values in the current task:...

103Charlie thought about these variables:...

104Explain:...

105Action with variables and task from schema:...

106####

107variables and values in the current task: "tomatos_servings": 500

108Charlie thought about these variables: "tomatos_servings": 200

109Explain: Everyone is discussing task 3. Bob believes the value of tomato
servings should be 500. According to the thought schema, Charlie believes the
value of tomato servings is 200. Therefore, Charlie’s action will be to refute
Bob’s point of view regarding tomato and explain why he thinks it’s 200.

110Action with variables and task from schema: Charlie will continue to
discuss task 3 and will explain why he thinks the tomato servings is 200.

111####

112...(more examples)

113####

114{name} thoughts schema:

115{character_schema}

116Conversation:

117{conversation_input}

118Action to take:

119{action}

120The next speaker is {name}. Now please decide what action {name} will take,
including discussing which tasks, discussing which variables, etc. Remember
you are {name}, and your actions should follow your thought schema. Please
carefully examine the variables mentioned by others in the conversation. For
any variables in the dialogue that are inconsistent with your thought schema
values, you should try to request others to examine them carefully. Please
remember, "Action" only indicates how {name} will take action in the next
response, such as asking questions or continuing to discuss the next task, and
should not include what {name} might say. The output format is:

121variables and values in the current task:...

122{name} thought about these variables:...

123Explain:...

124Action with variables and tasks from schema:...

Listing 7: Prompt for dialogue act generation

⬇

1{name}’s thoughts schema:

2{character_schema}

3

4Conversation:

5{conversation_input}

6Generate a reply from {name} based on the action and the variables from
{name} thought schema. Never mention the "task" word in the reply. Remember
you’re a middle school student, please reply in one sentence with the tone of
a middle school student. You believe that your thought schema is completely
correct, and all the variables in reply must be perfectly aligned to your own
thought schema.

7The action to take is "{action}". Before generating the reply, ground the
tasks and variables mentioned in the action from your thought schema. Then
generate the response. The output format is:

8Related tasks index:...

9Grounding values from {name} thoughts schema in those tasks:...

10Now, as a middle school student, {name} would say:...

Listing 8: Prompt for response generation