# CLASS meet SPOCK: An Education Tutoring Chatbot
based on Learning Science principles

Shashank Sonkar, Lucy Liu, Debshila Basu Mallick, Richard G. Baraniuk
Rice University
Houston
{ss164, nl35, debshila<EMAIL_ADDRESS>

###### Abstract

We present a design framework called Conversational Learning with Analytical
Step-by-Step Strategies (CLASS) for developing high-performance Intelligent
Tutoring Systems (ITS). The CLASS framework aims to empower ITS with with two
critical capabilities: _imparting tutor-like step-by-step guidance_ and
_enabling tutor-like conversations in natural language_ to effectively engage
learners. To empower ITS with the aforementioned capabilities, the CLASS
framework employs two carefully curated synthetic datasets. The first
scaffolding dataset encompasses a variety of elements, including problems,
their corresponding subproblems, hints, incorrect solutions, and tailored
feedback. This dataset provides ITS with essential problem-solving strategies
necessary for guiding students through each step of the conversation. The
second conversational dataset contains simulated student-tutor conversations
that involve the application of problem-solving strategies learned from the
first dataset. In the second dataset, the tutoring system adheres to a pre-
defined response template, which helps to maintain consistency and structure
in ITS’s responses during its interactions. This structured methodology
facilitates seamless integration of user feedback and yields valuable insights
into ITS’s internal decision-making process, allowing for continuous
refinement and improvement of the system. We also present a proof-of-concept
ITS, referred to as SPOCK, trained using the CLASS framework with a focus on
college level introductory biology content. A carefully constructed protocol
was developed for SPOCK’s preliminary evaluation, examining aspects such as
the factual accuracy and relevance of its responses. Experts in the field of
biology offered favorable remarks, particularly highlighting SPOCK’s
capability to break down questions into manageable subproblems and provide
step-by-step guidance to students.

##  1 Introduction

The development of Transformer-based models, specifically Large Language
Models (LLMs), such as ChatGPT and other GPT-based models [1, 2], has
significantly impacted the field of education, resulting in innovative tools
and methods that enhance the learning experience [3]. These advancements have
led to various applications for both students and instructors, including
personalized learning, support for students with disabilities, assessment
generation, and lesson planning. Among these advancements, integrating
artificial intelligence (AI) in the form of Intelligent Tutoring Systems (ITS)
has gained considerable attention for its potential to provide personalized,
accessible, and interactive learning support. However, designing and
implementing effective ITS remains a challenge, particularly in addressing
students’ diverse learning needs and promoting a deeper understanding of
complex concepts. This highlights the need to develop innovative frameworks
that leverage the power of LLMs, pushing the potential of ITS to benefit
students across various grade levels.

In this study, we present a novel design framework called Conversational
Learning with Analytical Step-by-Step Strategies (CLASS) that utilizes LLMs
for building ITS. CLASS aims to provide learners with thorough scaffolded
step-by-step support, addressing their queries and rectifying misconceptions.
As a proof of concept to demonstrate the practicality and effectiveness of
CLASS framework, we develop a Biology ITS referred to as SPOCK. Biology is
chosen as a representative example due to the subject’s inherent complexity
and the need for students to grasp intricate concepts.

Figure 1: A demonstration of CLASS framework and SPOCK’s training process. The
framework adopts a two-pronged approach for creating ITS, utilizing two
synthetic datasets with distinct objectives. The first scaffolding dataset
aims to equip SPOCK with step-by-step problem-solving skills. This dataset
consists of problems, corresponding subproblems, hints, incorrect student
responses and corresponding feedback. The second conversational dataset has an
objective of helping SPOCK apply these problem-solving skills effectively in
real-time conversations with students. This dataset contains simulated mock
interactions between students and a hypothetical AI tutorbot. Both datasets
are created using GPT-4 and a brief description of the specially designed
prompt instructions and outputs are displayed in the figure. CLASS framework
also uses an indexed search over related educational contents to reduce
hallucination and maintain factual consistency during conversations. In the
top part, we also present an interaction example between students and SPOCK.

The core objective of the CLASS framework is to equip SPOCK with two critical
capabilities: 1) providing tutor-like step-by-step guidance that fosters a
deeper understanding and significantly augments students’ learning experience
and 2) engaging in tutor-like conversations using natural language suitable
for students, ensuring conversational fluidity and adaptability. To achieve
these capabilities, CLASS implements an innovative, dual-faceted approach that
utilizes two distinct training datasets. These datasets are specifically
curated to instill the desired capabilities in SPOCK, thereby leveraging the
datasets as a solution for fulfilling CLASS’s core objective. CLASS’s approach
of utilizing the two training datasets, each to addressing a specific aspect
of SPOCK’s functionality is illustrated in Figure 1.

The first dataset, referred to as the _“scaffolding dataset”_ lays the
foundation for equipping SPOCK with essential skills necessary to effectively
guide students through the learning process. The scaffolding dataset,
generated using GPT-4 [1], covers a wide range of components, such as problems
along with their related subproblems, hints to provide guidance, solutions
that are incorrect, and customized feedback for the incorrect solutions. By
endowing SPOCK with a holistic understanding of problem-solving strategies and
methods for addressing misconceptions, the scaffolding dataset ensures the
system’s effective performance during interactions with students.

Building on the foundation established by the scaffolding dataset, the second
_“conversational dataset”_ focuses on simulated conversational student-tutor
interactions. The conversational dataset, also generated by GPT-4, consists of
simulated student-tutor conversations that involve the application of problem-
solving strategies acquired from the first dataset within structured,
question-focused exchanges. Training on this conversational dataset enhances
SPOCK’s ability to effectively apply the skills learned in the scaffolding
dataset during real-time interactions with students, providing guidance in
decomposing the main problem into smaller tasks and assisting learners in
solving these tasks sequentially through hints and direction.

Within the interactions contained in the second dataset, a pre-defined
response template is employed to ensure consistency and coherence in SPOCK’s
responses. This template addresses various situations, such as handling
incorrect, correct, and ambiguous responses, as well as guiding students
through subproblems and effectively addressing their inquiries. By adhering to
this template, SPOCK benefits from a structured approach that enables the
seamless incorporation of user feedback, promoting system enhancement and
fine-tuning. Additionally, this method offers valuable insights into the
internal decision-making mechanisms of SPOCK, contributing to its continuous
improvement and refinement.

Our contributions can be summarized as follows:

  1. 1.

We introduce a novel Conversational Learning with Analytical Step-by-Step
Strategies (CLASS) framework for constructing Intelligent Tutoring Systems
(ITS), focusing on two primary goals: facilitating tutor-like, step-by-step
guidance and promoting engaging interactions with students. The CLASS
framework incorporates two unique synthetic datasets generated using GPT-4
with specially crafted prompts, working together to enable the achievement of
our goals. The first, the scaffolding dataset, cultivates essential skills
such as providing hints, feedback, and segmenting problems into smaller tasks.
The second, the conversational dataset, facilitates the effective application
of these acquired skills, enabling ITS to engage in cohesive and interactive
dialogues with learners.

  2. 2.

Under the CLASS framework, we present SPOCK, a proof-of-concept ITS system
designed for a college-level introductory biology course, which showcases
CLASS’s capabilities. SPOCK is trained on the scaffolding and conversational
datasets, generated using GPT-4, allowing it to deliver tailored support
through engaging, interactive conversations.

  3. 3.

We develop a thoughtfully designed response template for SPOCK to ensure
consistency and clarity in its responses. This template enables users to offer
feedback on SPOCK’s decisions while providing valuable insights into SPOCK’s
internal decision-making process.

  4. 4.

We use an application of indexed search over educational course materials to
ensure factual correctness and relevancy of SPOCK’s responses, by fetching
relevant passages for problems and subproblems.

  5. 5.

We also develop a comprehensive evaluation protocol and conduct preliminary
human evaluations of SPOCK with biology domain experts, providing valuable
insights into the effectiveness of SPOCK.

##  2 Background

The concept of using ITS in education has a rich history, with various
approaches and techniques being explored over the years. In this section, we
first provide an overview of ITS, then emphasize the influence of LLMs in the
ITS designing. Additionally, we highlight the fundamental principles of
learning science that have motivated our dual-faceted approach.

###  2.1 Intelligent Tutoring Systems

ITS have gained popularity due to their ability to provide students with cost-
effective and personalized learning experience, particularly in online classes
where individualized support from educators is challenging [4]. ITS can
typically be divided into four categories. [5] The most common type of ITS
incorporates natural language dialogues as a means to address students’
knowledge gaps and identify their misconceptions, which allows the tutoring
system to intervene and correct the students. An example is AutoTutor [6] that
uses natural language dialogues to facilitate students learning in STEM field,
such as physics.

A different type of ITS uses the constraint-based modeling scaffolding
approach [7], which involves predefined constraints consisting of a set of
problems and ideal solutions written by human experts. These constraints serve
as a basis for the ITS to respond to student inquiries. The adoption of
constraint-based modeling approach offers advantages in tutoring subjects that
involve interactive, activity-based learning, such as database modeling,
taught by the tutoring system KERMIT [8].

The other two major categories of ITS directly models on the students: model
tracing and Bayesian network modeling. Model tracing formulates a set of
production rules that capture and represent students’ problem-solving
processes by monitoring their knowledge states. Bayesian network modeling
expands model tracing by utilizing Bayesian networks to predict students’
future performances based on their previous responses.

Our proposed framework CLASS incorporates both the first and second type of
ITS by first using the scaffolding approach of breaking down a set of problems
into subproblems and then guiding students through the problem using natural
language conversations. We also utilizes LLMs that possess a robust
understanding of natural languages to construct conversational systems.
Additionally, instead of relying on labor-intensive manual approaches to
develop scaffolding constraints, we rely on LLMs, which are already endowed
with great reasoning abilities, to autonomously derive the scaffolding
strategy.

###  2.2 Large Language Models

The development of LLMs has further revolutionized the field of ITS . These
LLMs have demonstrated remarkable abilities to generate human-like text and
understand complex language patterns, making them well-suited for creating ITS
that can engage with students in a more natural and interactive manner. Recent
advances enabled the training of LLMs on a massive scale, such as GPT-4 [1]
from OpenAI with 1 trillion parameters or PaLM [9] from Google with 540
billion parameters. However, smaller language models, such as LLaMA [10], have
yet still proven to achieve competitive performance. They have advantages of
being highly customizable, safer to deploy and less costly. Currently to our
knowledge, the practice of training a custom language model for ITS has remain
under-explored. To our knowledge, most large language model based ITS are
simply using large model APIs and prompting, which restrict scalability and
can result in a paywall.

In order to take the benefits of training custom language models, we use
Vicuna-13b [11], an open-source transformer-based language model with 13
billion parameters, to develop SPOCK . Vicuna-13b is created by fine-tuning a
foundation model LLaMA using ShareGPT dataset which contains approximately 70K
user-shared conversations. An essential aspect of utilizing Vicuna model is
the instruction-based training process [12], which allows the model to learn
from explicit instructions provided during the fine-tuning stages. This
instruction-based training enables SPOCK to better comprehend user intentions
and then generate appropriate responses accordingly.

###  2.3 Learning Science Principles

The development of our two-pronged approach for creating SPOCK is grounded in
learning science principles, which emphasize the importance of breaking down
complex problems into smaller, more manageable subproblems to facilitate
students’ learning, often referred to as problem decomposition in
computational thinking [13, 14]. Scaffolding, driven by the socio-
constructivist model of learning [15], is the feedback support that a student
receives when interacting with a better-informed person or, in this case,
SPOCK , thereby facilitating learning [16, 17]. SPOCK provides students with
supports that are tailored to their current levels of understanding and are
constantly monitored and adapted [18]. Optimal learning outcomes are achieved
when the complexity of the task aligns appropriately with the learner’s
current abilities.

By focusing on creating subproblems in the first dataset, our design framework
aligns with these learning theories, enabling SPOCK to guide students through
the problem-solving process in a step-by-step manner. This approach not only
facilitates students in overcoming their learning obstacles but also
encourages them to engage with progressively challenging problems, ultimately
promoting a deeper understanding of the subject matter.

##  3 Proposed CLASS framework

⬇

Prompt:

Generate a hard, challenging problem which can be broken down into subproblems
for the following section on Photosynthesis whose learning objective is:
‘Describe the main structures involved in photosynthesis’. For each
subproblem, generate a hint, one incorrect student response to the subproblem,
and corresponding feedback to the student. Put all the output in the following
JSON structure:

{{

"Problem": "..",

"SubProblems": [

"Question": "..",

"Answer": "..",

"Hint": "..",

"Incorrect Response": "..",

"Feedback": ".."

],

"Facts": [

"..",

".."

],

"Solution": ".."

}}

⬇

Example:

"Problem": "Describe the main structures involved in photosynthesis.",

"SubProblems": [

{

"Question": "What is the primary structure responsible for capturing sunlight
in photosynthesis?",

"Answer": "Chloroplasts",

"Hint": "It is a specialized organelle found in plant cells.",

"Incorrect Response": "Mitochondria",

"Feedback": "Good effort, but mitochondria are responsible for cellular
respiration, not photosynthesis. The correct answer is chloroplasts, which
contain pigments that capture sunlight."

},..]

Table 1: Scaffolding dataset generation prompt example and the resulting
content, featuring a problem, its subproblems, hints, an incorrect response,
and feedback.

The Conversational Learning with Analytical Step-by-Step Strategies (CLASS)
design framework for SPOCK focuses on developing a model that offers 1) tutor-
like step-by-step assistance and 2) engaging, natural language interactions
that mimic the conversational experience with human tutors. To achieve these
objectives, the CLASS framework emphasizes the use and curation of synthetic
datasets as an essential means to develop the desired capabilities
successfully. In this section, we delve into the organization of the datasets,
and outline the methodology for training SPOCK using these datasets.

###  3.1 Scaffolding Dataset

The first scaffolding dataset encompasses challenging problems specifically
designed for the subject of Biology, along with their corresponding
subproblems, essential facts for solving the main problem, hints, incorrect
student responses, and relevant feedback for each subproblem. This
comprehensive set of elements, especially with the focus on breaking down
problems into manageable subproblems, aims to equip SPOCK with crucial skills
to provide step-by-step assistance to learners, promoting deeper understanding
and engagement with the subject matter.

To construct the dataset, we employ a carefully crafted prompt for generating
content, which plays a pivotal role in directing generative LLMs such as GPT-4
to produce contextually relevant and focused output. An example of the prompt
and the generated content can be found in Table 1. The given prompt guides the
LLM in crafting a complex, challenging problem corresponding to a specific
section and learning objective Additionally, it provides the necessary facts
to answer the main problem while breaking it down into subproblems. The prompt
also ensures that the final solution to the main problem is included, and for
each subproblem, it generates a hint, an incorrect student response, and the
corresponding feedback

As the first dataset equips the Tutorbot with essential problem-solving
skills, such as providing hints and segmenting problems into smaller tasks, it
paves the way for the application of these capabilities through the second
dataset. In the following section, we will explore the creation of the second
dataset, which focuses on engaging and cohesive student-Tutorbot.

###  3.2 Conversational Dataset

After being trained on the scaffolding dataset, SPOCK acquires essential
techniques such as generating hints, providing feedback, and breaking problems
into smaller tasks. However, to effectively teach SPOCK how to apply these
skills seamlessly within real-time conversations, a second and even more
crucial dataset is needed. The second conversational dataset, also generated
by GPT-4, contains simulated conversations between a student and an AI-powered
Tutorbot, designed to assist students with their questions. The focus of this
dataset is to help SPOCK effectively apply the acquired skills from the
scaffolding dataset in real-time interactions through a question-driven
approach. The prompt, generated by GPT-4, consists of the following essential
parts:

  1. 1.

Problem: This part contains a problem or a question that the student needs
help with.

  2. 2.

Student-Tutorbot Interaction: The interaction begins with the student
requesting assistance for the given problem or question.

  3. 3.

Thoughts of Tutorbot: This part provides insights into the Tutorbot’s thought
process while handling different student responses. The thought process
focuses on various situations, such as: a) Handling incorrect responses b)
Handling correct responses c) Handling partially correct responses d) Handling
ambiguous or unclear responses e) Redirecting off-topic responses f)
Responding to student inquiries g) Guiding through subproblems h) Continuing
the conversation if none of the above apply

  4. 4.

Decision by Tutorbot: Based on the Tutorbot’s thought process, a decision is
made on how to address the student’s response. This decision includes choosing
the appropriate actions from the thought process (e.g., a, b, c, etc.) to
ensure a coherent and effective interaction with the student.

  5. 5.

Subproblem: This part describes the specific subproblem being solved within
the main problem. It helps the Tutorbot to break down the task into smaller,
manageable components.

  6. 6.

Tutorbot Response: Based on the decisions made, the Tutorbot provides a
response to the student. These responses include helpful hints

A part of a mock interaction generated is provided as example here:

⬇

{

"Problem": "Explain how the steps of digestion will change for a patient who
has recently undergone a subtotal gastrectomy."

"Student": "There will be less gastric juice produced.",

"Thoughts of Tutorbot": "The student’s response is partially correct but
missed the impact on the churning process.",

"Decision by Tutorbot": "c",

"Subproblem": "Changes in digestion",

"Tutorbot": "You’re correct that there will be less gastric juice produced,
but there’s another important aspect of digestion that will be affected. Think
about the physical process in the stomach. Can you identify it?"

}

Please refer to Appendix A.2 and B.2 respectively for the whole prompt and a
full mock conversation example. To sum it up, by training SPOCK on this second
dataset, SPOCK becomes adept at applying the foundational skills acquired from
the scaffolding dataset during conversations.

###  3.3 Tutorbot’s Response Template in Mock Conversations facilitates Model
Refinement and Explainability

A pivotal aspect of CLASS framework rests in the implementation of a fixed
response template for SPOCK in simulated chat interactions of the
conversational dataset. Centered around SPOCK’s thought process and decision-
making, this template ensures consistent and coherent engagement with
students. It enables SPOCK to systematically address a wide range of student
responses and inquiries, effectively guiding learners through the problem-
solving process. The output of SPOCK’s response comprises the following
components:

  * •

Thoughts of Tutorbot (labeled as ‘a’ to ‘h’):

    * a)

Handling Incorrect Responses

    * b)

Handling Correct Responses

    * c)

Handling Partially Correct Responses

    * d)

Handling Ambiguous or Unclear Responses

    * e)

Redirecting Off-topic Responses

    * f)

Responding to Student Inquiries

    * g)

Guiding Through Subproblems

    * h)

None of the above apply, Continue the Conversation

  * •

Decision:

    * –

Choose all that apply from the above thought process (labeled as ‘a’, ‘b’,
‘c’, etc.)

  * •

Subproblem:

    * –

Subproblem field describes the Subproblem being solved

Using this response template enables better explainability of SPOCK’s
decision-making. It offers insights into the rationale behind the model’s
choices, including the assessment of student responses and the resulting
decisions the Tutorbot performs. By leveraging the template’s decision field,
which encompasses both the evaluation of student responses and the subproblem,
we can create a loss function that quantifies potential errors and
inaccuracies in the SPOCK’s responses. This iterative refinement approach
ensures that SPOCK remains informed by real-world student interactions and
steadily enhances its problem-solving and conversational capabilities. It
enables the model to evolve continually, becoming more accurate and effective
in providing step-by-step guidance.

In conclusion, it is essential to emphasize that the central aim of our CLASS
design framework is to equip SPOCK with capabilities that facilitate tutor-
like step-by-step guidance and allow it to engage in tutor-like conversations
using natural language appropriate for students. The synergistic relationship
between the scaffolding and conversational datasets serves as an instrumental
method to achieve these goals in the development process. The scaffolding
dataset provides a solid foundation for SPOCK by imparting vital problem-
solving strategies, hints, and feedback mechanisms for various subproblems.
Upon this basis, the conversational dataset, primarily through its fixed
response template, effectively applies the acquired skills in crafting a more
captivating and personalized learning experience for students.

###  3.4 Indexed Search Over Educational Resources: A Robust Approach to
Maximize SPOCK’s Response Relevance and Minimize Hallucinations

To further enhance the accuracy and relevancy of the SPOCK’s responses, we
employ an indexed search over educational contents, such as textbooks, as an
additional verification method and a means to reduce hallucinations

By indexing the paragraphs within the educational contents, we can retrieve
relevant passages that correspond to the main problems and subproblems
encountered during the simulated conversations. This approach not only ensures
that SPOCK’s responses are factually correct but also aids in providing
contextually relevant and helpful hints to students when they are stuck.

The indexing process begins with preprocessing of text-based educational
resources, such as books, which includes tokenizing and cleaning the text and
extracting relevant paragraphs and sections. Next, these resources are indexed
to create an efficient search structure, allowing for fast retrieval of
relevant passages based on the input query, such as a problem or subproblem
field derived from the response template. The integration of the indexed
search mechanism with SPOCK’s response template empowers the Tutorbot to fetch
relevant content when generating hints or providing feedback, ensuring that
its responses are both factually accurate and contextually suitable. This
approach adds an additional layer of validation to SPOCK’s responses,
contributing to an enriched learning experience for students.

##  4 Implementation of SPOCK

In this section, we provide the implementation details of SPOCK using proposed
CLASS framework as a proof-of-concept. The implementation process involves
several interconnected steps that ensure the development of an effective and
engaging AI-driven educational tool. We began by selecting a powerful 13
billion parameter Vicuna model [11], which is based on the LLaMA model [10]
and trained on the ShareGPT dataset. This model was chosen for its ability to
understand complex language patterns and generate coherent, contextually
relevant responses.

To provide SPOCK with domain-specific knowledge, we fine-tuned the model using
the Causal Language Model (CLM) loss on the OpenStax Biology 2e textbook. This
step allowed the model to gain a deeper understanding of biology concepts,
ensuring more accurate and relevant guidance for students.

Following the fine-tuning, we focused on creating the two datasets that form
the backbone of the CLASS framework. We generated the scaffolding dataset by
prompting GPT-4 [1, 2] to create challenging Bloom Level 5 [19] problems,
based on 648 learning objectives covering 207 sections across 47 chapters of
the OpenStax Biology 2e textbook [20]. This dataset, containing problems along
with their corresponding subproblems, hints, incorrect solutions, and
feedback, is designed to equip SPOCK with the necessary skills for engaging in
collaborative problem-solving conversations with students.

Next, we created the conversational dataset by prompting GPT-4 to generate
mock conversations between a student and an AI-Tutorbot using the problems
from the scaffolding dataset. The GPT-4 model utilized the subproblems to
guide the student with step-by-step assistance, simulating a realistic
learning environment.

Once the two datasets were prepared, we trained the Vicuna model using both
datasets, equipping SPOCK with problem-solving strategies, hints, feedback
mechanisms, and conversation management skills. This comprehensive training
process ensures that SPOCK can effectively guide students through a wide range
of biology problems. Finally, during the inference process, SPOCK leverages
the knowledge and skills acquired during the training phase to provide
students with step-by-step guidance, hints, and feedback based on the
question-centric design.

The cost of training SPOCK can be broken down into two primary components.
First, the creation of both datasets involves prompting GPT-4, which costs
approximately $50 each. Second, we train the model using the CLM loss on the
biology textbook and fine-tune it utilizing both datasets with the help of the
DeepSpeed [21] and FastChat [22] libraries. This process is executed on 8
NVIDIA RTX A6000 GPUs and runs for a day.

In summary, the implementation of SPOCK is a carefully orchestrated process
that combines model selection, domain-specific fine-tuning, dataset creation,
model training, and inference, resulting in a powerful and responsive AI-
driven educational tool for biology learning. Having detailed the
implementation, we now turn our attention to the evaluation of SPOCK’s
effectiveness in the following section.

##  5 Evaluation

In this section, we will first provide an evaluation protocol for our tutoring
system, SPOCK . We conducted a preliminary evaluation of the ITS with one
educator at a tribal college and three graduate-level biology students as
biology subject matter experts (SMEs). Each evaluation session involved the
SMEs selecting three specific biology section, followed by their interaction
with the ITS while they engaged in "thinking out aloud". The SMEs focused on
four main dimensions at the end of their interaction: i) factual correctness:
the information returned by SPOCK is true and accurate. ii) relevance: the
information returned by SPOCK is relevant to the context that is being
studied. iii) completeness of the responses: the information returned is
comprehensive of the concept being studied. iv) perceived motivational impact
of SPOCK on students, including whether students would find the interaction
engaging and motivating. In this case specifically, this perception is from
the perspective of introductory biology college students.

###  5.1 Evaluation Protocol

In this section, we present the specific asceps we evaluated in the four main
dimensions – factual correctness, relevancy, completeness and motivation.
These criteria will help us determine not only the accuracy of the information
provided by SPOCK, but also its ability to engage with students and guide them
effectively through the problem-solving process. The insights gained from this
evaluation process will help us identify areas for improvement and inform the
continuous refinement of SPOCK , ultimately resulting in a more effective and
engaging AI-driven ITS .

##### Factual Correctness

The factual correctness of the SPOCK’s responses is crucial in ensuring that
students receive accurate information while solving problems. To evaluate this
aspect, we will examine the following

  * •

F1: Are the decisions (see Section 3.3) made by SPOCK accurate?

  * •

F2: Are generated hints accurate and factually valid?

  * •

F3: Are generated final answers accurate and factually valid?

##### Relevancy

The relevancy of SPOCK’s responses is essential in providing helpful guidance
to students when they encounter difficulties. To assess the relevancy, we will
consider the following:

  * •

R1: Are generated subproblems (see Section 3.3) relevant to the question being
asked?

  * •

R2: Are generated hints relevant or helpful when a student is stuck (provided
the hints are factually correct)?

  * •

R3: Is this line of dialogue similar to what instructors generally use for
explaining a concept?

##### Completeness

The completeness of the SPOCK’s responses ensures that all aspects of a
question are addressed before proceeding to the next problem. To evaluate the
completeness, we will examine the following:

  * •

C1: Are all aspects of an answer completed before the next question is asked?

  * •

C2: Are there good guardrails for handling off-topic conversations?

##### Motivation

The motivation level of the SPOCK evaluates on whether it successfully
captures and maintains students’ interest and attention throughout the
learning process. To evaluate this aspect, we will examine the following:

  * •

M1: Are these conversations engaging for students?

  * •

M2: Will these conversations not lead to frustration for students?

Factual Correctness | Relevance | Completeness | Motivation
---|---|---|---
F1 | F2 | F3 | R1 | R2 | R3 | C1 | C2 | M1 | M2
4.50 | 4.83 | 4.33 | 4.33 | 4.33 | 4.00 | 3.83 | 4.83 | 4.00 | 4.67
Table 2:  We provide the average rating of three biology domain experts based
on our evaluation protocol. The protocol examines the factual accuracy,
relevance, completeness, and motivational impact of SPOCK in response to
students’ questions (see section 5.1 for more details). The ratings are based
on a scale of 5 (1 – Poor, 2 – Fair, 3 – Good, 4 – Very Good, 5 – Excellent).
We attained ratings above a scale of 4 for the majority of our evaluation
aspects, showcasing a strong and satisfactory level of performance in each
area.

###  5.2 Preliminary Evaluation Results

We conducted the first phase of evaluation following the evaluation protocol
with four SMEs who possess extensive knowledge and expertise in biology. To
ensure a comprehensive evaluation, every domain expert is asked to behave like
a student learning biology and provide incorrect answer, correct answer,
irrelevant answers and also ask for hints during the interaction. At the end
of the evaluation, we give them the above questions and get a rating on a
scale of 5 (1 – Poor, 2 – Fair, 3 – Good, 4 – Very Good, 5 – Excellent) with
their comments. Average ratings by the biology domain experts are reported in
Table 2. We also include some interactions between the evaluators and SPOCK in
Appendix B.3.

To elaborate on the results obtained from the evaluation process, the domain
experts expressed positive feedback on the strategy of SPOCK where it breaks
down a question into subproblems and gives step-by-step hints and responses to
guide the students through the question. Additionally, they enjoyed the
encouraging nature of SPOCK, which motivated students to persevere and engage
with challenging biology concepts. They believe that positive reinforcement
and supportive feedback from SPOCK could foster a conducive learning
environment, boosting students’ confidence and enthusiasm in their studies.
Also, all domain experts agree that SPOCK will be a useful learning aid for
self-learning and they would prefer the interactive learning experience over
reading books or simply browsing for answers. Potential use cases of SPOCK
include but not limited to previewing for classes, consolidating unanswered or
confused topics after class and preparing for quizzes or exams.

##  6 Future Work

We have identified various opportunities to enhance the proposed CLASS
framework. A key priority for future updates is to address the challenge of
maintaining factual accuracy in the information generated by ITS. LLMs are
prone to occasional inaccuracies and hallucinations, and these limitations are
also inherited by any ITS built upon LLMs. To mitigate these issues, we have
implemented indexed search over the educational content which significantly
reduced concerns regarding factual accuracy. However, we acknowledge that
additional creative guardrails are needed to further improve the accuracy of
the returned information in future iterations of CLASS powered ITS.
Additionally, we only conducted a preliminary evaluation so far, which only
include interactions between ITS and biology domain expert. In the next stage,
we will modify ITS to address the concerns of biology-learning students,
followed by use ITS in a small classroom to perform more comprehensive and
practical evaluations, and ultimately test the tool in the real world. The key
to investigating the impact of using ITS will be to see how students perform
on a content assessment with and without its use. After a small scale
evaluation, we will evaluate the impact of ITS on a large scale research
infrastructure.

##  7 Conclusions

The Conversational Learning with Analytical Step-by-Step Strategies (CLASS)
framework revolutionizes ITS training with LLMs, equipping models with tutor-
like step-by-step guidance and interactive conversational capabilities. SPOCK,
our biology ITS proof-of-concept, showcases the effectiveness of these
capabilities. The CLASS framework utilizes two distinct training datasets and
automated feedback for continual improvement of SPOCK. The scaffolding dataset
imparts problem-solving strategies, while the conversational dataset enhances
interactive skills with simulated student interactions. Our work contributes
to the AI in education literature by laying the foundation for future ITS
designs across various disciplines. We aim to address current limitations by
conducting additional evaluation studies that encompass feedback from not only
subject matter experts but also a diverse sample of students for a more
comprehensive understanding of the ITS ’s impact. Furthermore, we plan to
expand the scope of our research by exploring different subjects and improving
the CLASS framework based on user feedback and experiences.

## References

  * [1] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [2] OpenAI.  GPT-4 technical report, 2023.
  * [3] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al.  Chatgpt for good? on opportunities and challenges of large language models for education.  Learning and Individual Differences, 103:102274, 2023.
  * [4] Rainer Winkler and Matthias Söllner.  Unleashing the potential of chatbots in education: A state-of-the-art analysis.  In Academy of management annual meeting (AOM), 2018.
  * [5] Shi Feng, Alejandra J Magana, and Dominic Kao.  A systematic review of literature on the effectiveness of intelligent tutoring systems in stem.  In 2021 IEEE Frontiers in Education Conference (FIE), pages 1–9. IEEE, 2021.
  * [6] Arthur C Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M Louwerse.  Autotutor: A tutor with dialogue in natural language.  Behavior Research Methods, Instruments, & Computers, 36:180–192, 2004.
  * [7] Antonija Mitrovic, Stellan Ohlsson, and Devon K Barrow.  The effect of positive feedback in a constraint-based intelligent tutoring system.  Computers & Education, 60(1):264–272, 2013.
  * [8] Pramuditha Suraweera and Antonija Mitrovic.  Kermit: A constraint-based tutor for database modeling.  In Intelligent Tutoring Systems: 6th International Conference, ITS 2002 Biarritz, France and San Sebastian, Spain, June 2–7, 2002 Proceedings 6, pages 377–387. Springer, 2002.
  * [9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [10] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [11] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [12] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [13] Jeannette M Wing.  Computational thinking.  Communications of the ACM, 49(3):33–35, 2006.
  * [14] Valerie J Shute, Chen Sun, and Jodi Asbell-Clarke.  Demystifying computational thinking.  Educational research review, 22:142–158, 2017.
  * [15] Lev Semenovich Vygotsky and Michael Cole.  Mind in society: Development of higher psychological processes.  Harvard university press, 1978.
  * [16] Nicole D Martin, Catherine Dornfeld Tissenbaum, Dana Gnesdilow, and Sadhana Puntambekar.  Fading distributed scaffolds: The importance of complementarity between teacher and material scaffolds.  Instructional Science, 47:69–98, 2019.
  * [17] Sadhana Puntambekar and Janet L Kolodner.  Toward implementing distributed scaffolding: Helping students learn science from design.  Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 42(2):185–217, 2005\.
  * [18] C Addison Stone.  The metaphor of scaffolding: Its utility for the field of learning disabilities.  Journal of learning disabilities, 31(4):344–364, 1998.
  * [19] Jack Conklin.  A taxonomy for learning, teaching, and assessing: A revision of bloom’s taxonomy of educational objectives complete edition, 2005.
  * [20] Mary Ann Clark, Jung Choi, and Matthew Douglas.  Biology 2e.  OpenStax, 2021.
  * [21] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
  * [22] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Fastchat: An open platform for training, serving, and evaluating large languages - release repo for vicuna and fastchat-t5.  https://github.com/lm-sys/FastChat, 2023.  Accessed: 2023-04-04.

##  Appendix A Prompts

###  A.1 Prompt for the first dataset

⬇

Generate a hard, challenging problem which can be broken down into subproblems
for the following section on {section_name} whose learning objective is:
{section_learning_objs}.

For the generated main problem for this learning objective, also output the
following:

1) Facts necessary to answer it,

2) Subproblems that the main problem can be broken down into, and

3) The final answer.

For each subproblem, generate a hint, one incorrect student response to the
subproblem, and corresponding feedback to the student. Put all the output in
the following JSON structure:

{{

"Problem": "..",

"SubProblems": [

"Question": "..",

"Answer": "..",

"Hint": "..",

"Incorrect Response": "..",

"Feedback": ".."

],

"Facts": [

"..",

".."

],

"Solution": ".."

}}

###  A.2 Prompt for the second dataset

⬇

Your goal is to create a mock conversation between Student and a Tutorbot, an
AI-powered chatbot designed to help Student’s with a question:

Question: {problem}

"Student": "Help me with Q. {problem}",

"Thoughts of Tutorbot": "..."

"Decision by Tutorbot": "..."

"Subproblem": "..."

"Tutorbot": "No problem! Let’s break the problem into sub-problems down. Let’s
begin with the first subproblem... First subproblem is ...",

Function of Thoughts of Tutorbot:

a) Handling Incorrect Responses:

1) Promptly notify the student about the mistake or ambiguous reply.

2) Provide constructive feedback to pinpoint the errors.

3) Offer helpful hints to guide the student towards the correct solution.

4) Step in to provide a solution if the student is unable to answer even after
multiple attempts.

b) Handling Correct Responses:

1) Meticulously examine if all components of the current question have been
addressed.

2) Ensure no essential elements are overlooked or omitted.

c) Handling Partially Correct Responses:

1) Acknowledge the accurate parts.

2) Highlight the mistakes or missing details.

3) Assist the student in rectifying and refining their answer.

d) Handling Ambiguous or Unclear or Short Responses:

1) Actively seek clarification through relevant follow-up questions.

2) Request the student to provide more specific information.

e) Redirecting Off-topic Responses:

1) Skillfully redirect the student’s attention to the subject matter.

2) Provide guidance on how to approach the question appropriately.

f) Responding to Student Inquiries:

1) Prioritize addressing the inquiry.

2) Offer relevant support and guidance to meet the student’s specific needs.

g) Guiding Through Subproblems:

1) Present subproblems sequentially.

2) Validate the completion and understanding of each subproblem before moving
to the next.

h) None of the above apply. Continue the Conversation.

Function of Decision by Tutorbot:

Choose all that apply from the above
"a1,a2,a3,b1,b2,c1,c2,c3,d1,d2,e1,e2,f1,f2,g1,g2,h" thought process.

Function of Subproblem:

Subproblem field describes the Subproblem being solved.

Now, let’s begin. Your goal is to create a mock conversation between Student
and a Tutorbot, an AI-powered chatbot designed to help Student’s with a
question.

Please create a mock conversation now. Tutorbot helps the student by breaking
down the main problem into subproblems, and the help student to solve each
sub-problem sequentially. Tutorbot only provide hints.

Remember, in this mock conversation, simulate many incorrect responses from
the student.

Use the following json format:

Put all the output in the following JSON structure

[{{

"Student": "..",

"Decision": ".."

"Subproblem": ".."

"Tutorbot": "..",

}},

Repeat above N times.

]

Remember, in this mock conversation, simulate many incorrect responses from
the student.

###  A.3 Prompt for the second dataset (New Version)

⬇

Your goal is to create a mock conversation between Student and a Tutorbot, an
AI-powered chatbot designed to help Student’s with a question:

Question: {problem}

"Student": "Q. {problem}",

"Thoughts of Tutorbot": ".."

"Evaluation of Student Response": ".."

"Action Based on Evaluation": ".."

"Subproblem State": ".."

"Subproblem": ".."

"Tutorbot": "Let’s break the problem into subproblems and tackle the
subproblems one by one. Let’s begin with the first subproblem...",

The function of Thoughts of Tutorbot is to decide the evaluation and also the
subproblem state:

a) Evaluating Incorrect Responses

b) Evaluating Correct Responses

c) Evaluating Partially Correct Responses

d) Evaluating Ambiguous or Unclear or Short Responses

e) Redirecting Off-topic Responses

f) Responding to Student Inquiries

g) N/A

Tutorbot Actions Based on the Evaluation:

If "a" is the evaluation, then:

1) Promptly notify the student about the mistake, Provide constructive
feedback to pinpoint the errors, Offer helpful hints

2) Step in to provide a solution if the student is unable to answer even after
multiple attempts.

If "b" is the evaluation, then:

3) Confirm the correct answer. Check for completeness for the answer to the
subproblem. If solution is incomplete, notify the student to complete the
solution.

If "c" is the evaluation, then:

4) Acknowledge the accurate parts, Promptly notify the student about the
mistake, Provide constructive feedback to pinpoint the errors, Offer helpful
hints

5) Step in to provide a solution if the student is unable to answer even after
multiple attempts.

If "d" is the evaluation, then:

6) Actively seek clarification through relevant follow-up questions. Request
the student to provide more specific information.

If "e" is the evaluation, then:

7) Skillfully redirect the student’s attention to the subject matter. Provide
guidance on how to approach the question appropriately.

If "f" is the evaluation, then:

8) If student asks for a hint, provide a hint for the current subproblem.

9) If student asks for a solution, give student the solution, marked current
subproblem finished, and move to the next subproblem.

10) If student asks to move to previous subproblem, marked current subproblem
finished, and move to the previous subproblem.

11) If none apply, prioritize addressing the inquiry. Offer relevant support
and guidance to meet the student’s specific needs.

If "g" is the evaluation, then:

12) N/A

Function of Subproblem State is to guide through subproblems:

w) N/A

x) One of the subproblems is currently being solved

y) Subproblem finished, moving to next subproblem that is not finished

z) Subproblem finished, no next subproblem, problem finished

Now, let’s begin. Your goal is to create a mock conversation between Student
and a Tutorbot, an AI-powered chatbot designed to help Student’s with a
question.

Please create a mock conversation now. Tutorbot helps the student by breaking
down the main problem into subproblems, and the help student to solve each
sub-problem sequentially. Tutorbot only provide hints.

Remember, in this mock conversation, simulate many incorrect responses from
the student.

Use the following json format:

Put all the output in the following JSON structure

[{{

"Student": "..",

"Thoughts of Tutorbot": ".."

"Evaluation of Student Response": "a,b,c,d,e,f,g"

"Action Based on Evaluation": "1,2,3,4,5,6,7,8,9,10,11,12"

"Subproblem State": "w,x,y,z"

"Subproblem": ".."

"Tutorbot": "..",

}},

Repeat above N times.

]

Remember, in this mock conversation, simulate many incorrect responses from
the student.

###  A.4 Inference Prompt

⬇

Instructions to Act as a Tutorbot:

You are a Tutorbot, an AI-powered chatbot designed to help Student’s with a
question.

For each response from the student, first think about which category your
response falls on, and then use these thoughts to frame you reply

"Thoughts of Tutorbot": "..."

"Decision by Tutorbot": "..."

"Subproblem": "..."

"Tutorbot": "No problem! Let’s break the problem into sub-problems down. Let’s
begin with the first subproblem... First subproblem is ...",

a) Handling Incorrect Responses:

1) Promptly notify the student about the mistake or ambiguous reply.

2) Provide constructive feedback to pinpoint the errors.

3) Offer helpful hints to guide the student towards the correct solution.

4) Step in to provide a solution if the student is unable to answer even after
multiple attempts.

b) Handling Correct Responses:

1) Meticulously examine if all components of the current question have been
addressed.

2) Ensure no essential elements are overlooked or omitted.

c) Handling Partially Correct Responses:

1) Acknowledge the accurate parts.

2) Highlight the mistakes or missing details.

3) Assist the student in rectifying and refining their answer.

d) Handling Ambiguous or Unclear or Short Responses:

1) Actively seek clarification through relevant follow-up questions.

2) Request the student to provide more specific information.

e) Redirecting Off-topic Responses:

1) Skillfully redirect the student’s attention to the subject matter.

2) Provide guidance on how to approach the question appropriately.

f) Responding to Student Inquiries:

1) Prioritize addressing the inquiry.

2) Offer relevant support and guidance to meet the student’s specific needs.

g) Guiding Through Subproblems:

1) Present subproblems sequentially.

2) Validate the completion and understanding of each subproblem before moving
to the next.

h) None of the above apply. Continue the Conversation.

Function of Decision by Tutorbot:

Choose all that apply from the above
"a1,a2,a3,b1,b2,c1,c2,c3,d1,d2,e1,e2,f1,f2,g1,g2,h" thought process.

Function of Subproblem:

Subproblem field describes the Subproblem being solved.

Helpful Information for Tutorbot:

{retrieved bio passages}

End of Helpful Information for Tutorbot.

Now, let’s begin. Your goal as a Tutorbot is to help the student with a
question.

Remember Tutorbot helps the student by breaking down the main problem into
subproblems, and the help student to solve each sub-problem sequentially.
Tutorbot only provide hints.

Use the following json format for your reply:

Put all the output in the following JSON structure

{{

"Decision": ".."

"Subproblem": ".."

"Tutorbot": "..",

}}

Also, make sure that all your responses/ statements to the student are
factually correct and TRUE.

##  Appendix B Examples

###  B.1 Example for the first dataset

⬇

{

""Problem"": ""Analyze the ecological consequences of a hypothetical scenario
where all fungi were to suddenly disappear from different ecosystems. Discuss
the effects on nutrient cycling, soil formation, and symbiotic
relationships."",

""SubProblems"": [

{

""Question"": ""What is the role of fungi in nutrient cycling?"",

""Answer"": ""Fungi play a crucial role in nutrient cycling by decomposing
organic matter and releasing essential nutrients back into the soil, which can
be then used by plants and other organisms."",

""Hint"": ""Think about the decomposition process and how fungi break down
organic matter."",

""Incorrect Response"": ""Fungi consume the nutrients, making them unavailable
to other organisms."",

""Feedback"": ""That’s incorrect. Fungi decompose organic matter and release
nutrients back into the soil, making them available for other organisms.""

},

{

""Question"": ""How do fungi contribute to soil formation?"",

""Answer"": ""Fungi help in soil formation by breaking down rocks and other
inorganic materials and turning them into soil. They also help in binding soil
particles together, contributing to soil stability."",

""Hint"": ""Consider the processes of weathering and binding of soil
particles."",

""Incorrect Response"": ""Fungi merely consume nutrients from the soil and do
not contribute to its formation."",

""Feedback"": ""That’s not accurate. Fungi actually help in soil formation by
breaking down rocks and other materials, as well as binding soil particles
together.""

},

{

""Question"": ""What are the symbiotic relationships between fungi and other
organisms?"",

""Answer"": ""Fungi form various symbiotic relationships, such as mycorrhizal
associations with plant roots, which help plants absorb nutrients and water
from the soil; and lichens, where fungi partner with photosynthetic organisms
like algae or cyanobacteria."",

""Hint"": ""Think about the mutualistic relationships where both fungi and
another organism receive benefits."",

""Incorrect Response"": ""Fungi are parasitic on other organisms and harm
them."",

""Feedback"": ""While some fungi are indeed parasitic, there are many more
fungi that form beneficial symbiotic relationships, such as mycorrhizal
associations with plants and lichen partnerships with algae or
cyanobacteria.""

}

],

""Facts"": [

""Fungi play a crucial role in nutrient cycling by decomposing organic matter
and releasing essential nutrients back into the soil."",

""Fungi help in soil formation by breaking down rocks and other inorganic
materials and turning them into soil. They also help in binding soil particles
together."",

""Fungi form various symbiotic relationships with other organisms, such as
mycorrhizal associations with plant roots and lichen partnerships with
photosynthetic organisms.""

],

""Solution"": ""If all fungi were to suddenly disappear from ecosystems,
nutrient cycling would be severely disrupted, leading to a decline in soil
fertility and ultimately plant growth. Soil formation processes would be
affected, resulting in less stable soil structures and increased erosion.
Furthermore, the loss of symbiotic relationships between fungi and other
organisms, such as mycorrhizal associations and lichen partnerships, would
lead to reduced nutrient uptake capabilities for plants and the possible
extinction of some photosynthetic organisms. Altogether, the disappearance of
fungi would have devastating consequences on the overall functioning of
ecosystems.""

}

###  B.2 Example for the second dataset

⬇

[{

""Student"": ""Q. Imagine a team of scientists uncovers a new bacterial
species and a new single-celled eukaryotic species. They are tasked with
differentiating their DNA structure and organization. Describe the key
differences and similarities in the DNA structure of these species."",

""Thoughts of Tutorbot"": ""Introduce the first subproblem."",

""Evaluation of Student Response"": ""g"",

""Action Based on Evaluation"": ""12"",

""Subproblem State"": ""x"",

""Subproblem"": ""Identify the differences in DNA packaging between the
bacterial and eukaryotic species."",

""Tutorbot"": ""Let’s break the problem into subproblems and tackle them one
by one. The first subproblem is: Identify the differences in DNA packaging
between the bacterial and eukaryotic species."",

},

{

""Student"": ""I think in both species, DNA is wrapped around histones to form
nucleosomes."",

""Thoughts of Tutorbot"": ""Student is partially correct. Only eukaryotes
package DNA with histones."",

""Evaluation of Student Response"": ""c"",

""Action Based on Evaluation"": ""4"",

""Subproblem State"": ""x"",

""Subproblem"": ""Identify the differences in DNA packaging between the
bacterial and eukaryotic species."",

""Tutorbot"": ""You’re partially correct. Only eukaryotic species have their
DNA wrapped around histones to form nucleosomes. Can you identify how
bacterial species package their DNA?"",

},

{

""Student"": ""Bacterial species package their DNA with histone-like
proteins."",

""Thoughts of Tutorbot"": ""Student is correct."",

""Evaluation of Student Response"": ""b"",

""Action Based on Evaluation"": ""3"",

""Subproblem State"": ""y"",

""Subproblem"": ""Discuss the differences in DNA organization and chromosomes
between bacterial and eukaryotic species."",

""Tutorbot"": ""That’s correct! Bacteria package their DNA with histone-like
proteins. Now for the next subproblem: Discuss the differences in DNA
organization and chromosomes between bacterial and eukaryotic species."",

},

{

""Student"": ""Eukaryotic species have circular DNA, while bacterial species
have linear DNA."",

""Thoughts of Tutorbot"": ""Student is incorrect. The organization of DNA is
the opposite of what the student mentioned."",

""Evaluation of Student Response"": ""a"",

""Action Based on Evaluation"": ""1"",

""Subproblem State"": ""x"",

""Subproblem"": ""Discuss the differences in DNA organization and chromosomes
between bacterial and eukaryotic species."",

""Tutorbot"": ""Actually, that’s not quite right. You have the organization of
DNA mixed up. Let me give you a hint: Bacterial species typically have one
type of chromosome, while eukaryotic species have multiple. Can you identify
the correct organization now?"",

},

{

""Student"": ""Bacterial species have circular DNA, while eukaryotic species
have linear DNA."",

""Thoughts of Tutorbot"": ""Student is correct."",

""Evaluation of Student Response"": ""b"",

""Action Based on Evaluation"": ""3"",

""Subproblem State"": ""y"",

""Subproblem"": ""Discuss the similarities between bacterial and eukaryotic
species DNA structure."",

""Tutorbot"": ""That’s correct! Bacterial species have circular DNA, while
eukaryotic species have linear DNA. Now let’s move to the last subproblem:
Discuss the similarities between bacterial and eukaryotic species’ DNA
structure."",

},

{

""Student"": ""Both have a double helix structure."",

""Thoughts of Tutorbot"": ""Student is correct."",

""Evaluation of Student Response"": ""b"",

""Action Based on Evaluation"": ""3"",

""Subproblem State"": ""z"",

""Subproblem"": ""Discuss the similarities between bacterial and eukaryotic
species DNA structure."",

""Tutorbot"": ""That’s correct! Both bacterial and eukaryotic species’ DNA are
composed of a double helix structure. Great job! You have successfully
described the key differences and similarities in the DNA structure of these
species."",

}]

###  B.3 Examples for the interaction with Tutorbot

####  B.3.1 Mitochondria

⬇

$\blacktriangleright$ Student

Help with Q: What is mitochondria?

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "g1,g2",

"Subproblem": "What are the key characteristics of mitochondria?",

"Tutorbot": "No problem! Let’s break the question down into subproblems.
First, let’s discuss the key characteristics of mitochondria."

}

$\blacktriangleright$ Student

hint

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "f2",

"Subproblem": "What are the key characteristics of mitochondria?",

"Tutorbot": "Some key characteristics of mitochondria include its double
membrane, oval shape, presence of ribosomes and DNA, and its role in cellular
respiration."

}

$\blacktriangleright$ Student

Yes. What are other things you could teach me about mitochondria?

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "h",

"Subproblem": "",

"Tutorbot": "Some other things you could know about mitochondria include its
discovery, its size and position in the cell, and its connection with the
process of cell division."

}

$\blacktriangleright$ Student

okay break it into questions for me

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "g1,g2",

"Subproblem": "Discovery of mitochondria",

"Tutorbot": "Sure! Let’s start with the discovery of mitochondria. Can you
tell me who first identified mitochondria?"

}

$\blacktriangleright$ Student

Darwin

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "a1,a2,a3",

"Subproblem": "Discovery of mitochondria",

"Tutorbot": "Not quite. Charles Darwin was a scientist, but it was another
person who discovered mitochondria. Do you remember who it was?"

}

$\blacktriangleright$ Student

i don’t know

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "a4",

"Subproblem": "Discovery of mitochondria",

"Tutorbot": "That’s okay. The person who discovered mitochondria was a
scientist named Antony van Leeuwenhoek. He was a Dutch tradesman who lived in
the 1600s. Moving on, let’s discuss the size and position of mitochondria in
the cell."

}

$\blacktriangleright$ Student

The size is bacteria size and in the cytoplasm

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "a1,a2,a3",

"Subproblem": "Size and position of mitochondria",

"Tutorbot": "That’s not quite accurate. Mitochondria are larger than bacteria,
and they are not located in the cytoplasm. Can you try again to describe their
size and position?"

}

$\blacktriangleright$ Student

mitochondria is in the cytoplasm

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "a1,a2,a3",

"Subproblem": "Size and position of mitochondria",

"Tutorbot": "Not quite. Mitochondria may be located within the cytoplasm, but
they have their own distinct membrane. Can you tell me about the membrane and
its role?"

}

$\blacktriangleright$ Student

it is to generate energy

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "a1,a2,a3",

"Subproblem": "Mitochondrial membrane",

"Tutorbot": "That’s true, the mitochondrial membrane plays a role in
generating energy, but there’s more to it. Can you provide more detail about
the structure and function of the mitochondrial membrane?"

}

$\blacktriangleright$ Student

hint

$\blacktriangleright$ TutorBot

{

"Decision by Tutorbot": "f2",

"Subproblem": "Mitochondrial membrane",

"Tutorbot": "Some key aspects of the mitochondrial membrane include its
double-layered structure, the cristae, and its role in the electron transport
chain."

}

####  B.3.2 Can animal photosynthesis?

⬇

$\blacktriangleright$ Student

Help me with Q: Can animals photosynthesize?

$\blacktriangleright$ TutorBot

{

"Decision": "a1,a2",

"Subproblem": "Define photosynthesis",

"Tutorbot": "No problem! Let’s break the problem into subproblems. Let’s begin
with the first subproblem: Define photosynthesis."

}

$\blacktriangleright$ Student

Absorb CO2 and provide water?

$\blacktriangleright$ TutorBot

{

"Decision": "c1,c2",

"Subproblem": "Define photosynthesis",

"Tutorbot": "That’s partly correct! Photosynthesis is the process of using
sunlight, carbon dioxide, and water to produce glucose and oxygen. Can you try
defining it again?"

}

$\blacktriangleright$ Student

I get it

$\blacktriangleright$ TutorBot

{

"Decision": "h",

"Subproblem": "Define photosynthesis",

"Tutorbot": "Great! Now that you understand what photosynthesis is, let’s move
on to the next subproblem: Can animals, like humans, photosynthesize?"

}

$\blacktriangleright$ Student

I hope so

$\blacktriangleright$ TutorBot

{

"Decision": "a1,a2",

"Subproblem": "Can animals photosynthesize?",

"Tutorbot": "Well, not quite. Unlike plants, animals do not have the necessary
structures, like chloroplasts, to perform photosynthesis. What do you think,
can animals photosynthesize?"

}

$\blacktriangleright$ Student

Seems like animal can’t, but is there any exceptions?

$\blacktriangleright$ TutorBot

{

"Decision": "f1,f2",

"Subproblem": "Can animals photosynthesize?",

"Tutorbot": "There are some exceptions, like some marine invertebrates, that
have specialized cells that can perform photosynthesis. However, most animals
do not have the ability to photosynthesize like plants do.

}


######  Corollary 2.45.

The change-of-parahoric map between central leaves is finite.

######  Proof:

We have just seen it to be quasi-finite.

Corollary 2.13, combined with the properness of the change-of-parahoric itself
and therefore of the change-of-parahoric map restricted to Newton strata
(similar argument as in footnote 21), implies the properness of the change-of-
parahoric map between central leaves. □

######  Remark 2.46.

By [Kim19, Cor. 5.3.1], leaves are equidimensional smooth and the dimension of
a leaf depends only on the Newton stratum it is in; in particular, if we
consider a change-of-parahoric map between leaves,

$\pi_{K,K^{\prime}}\colon\Upsilon^{-1}_{K}(y)\to\Upsilon_{K^{\prime}}^{-1}(y^{\prime}),$

then $\dim\Upsilon^{-1}_{K}(y)=\dim\Upsilon_{K^{\prime}}^{-1}(y^{\prime})$.

######  Corollary 2.47.

The change-of-parahoric map between central leaves is finite locally free.

######  Proof:

Combine Corollary 2.45 with the preceding remark and [GW10, Cor.
14.128]222222Note that “$y\in Y$” may be replaced by “$y\in f(X)$” in the
statement of [GW10, Cor. 14.128].. □

######  Corollary 2.48.

The change-of-parahoric morphism between central leaves is the composition of
a flat universal homeomorphism of finite type and a finite Ã©tale morphism.

######  Proof:

This follows from what has been established about the morphism by using
[Mes72, Lemma 4.8]. □

## References

  * [1] P. Hamacher and W. Kim  “$l$-adic étale cohomology of Shimura varieties of Hodge type with non-trivial coefficients”  In _Math. Ann._ 375.3-4 Springer, Berlin/Heidelberg, 2019, pp. 973–1044
  * [2] M. Kisin and G. Pappas  “Integral models of Shimura varieties with parahoric level structure”  In _Publ. Math., Inst. Hautes Étud. Sci._ 128 Springer, Berlin/Heidelberg; Institut des Hautes Études Scientifiques, Bures-sur-Yvette, 2018, pp. 121–218
  * [Ahs11] Tobias Ahsendorf  “$\mathcal{O}$-displays and $\pi$-divisible formal $\mathcal{O}$-modules”, 2011  URL: http://nbn-resolving.de/urn:nbn:de:hbz:361-24713520
  * [AT08] Alexander Arhangel’skii and Mikhail Tkachenko  “Topological groups and related structures”  Hackensack, NJ: World Scientific; Paris: Atlantis Press, 2008
  * [BBM82] Pierre Berthelot, Lawrence Breen and William Messing  “Théorie de Dieudonné cristalline. II” 930, Lecture Notes in Mathematics  Springer-Verlag, Berlin, 1982, pp. x+261  DOI: 10.1007/BFb0093025
  * [BG18] Alessandra Bertapelle and Cristian D. González-Avilés  “On the perfection of schemes”  In _Expo. Math._ 36.2 Elsevier, Munich, 2018, pp. 197–220
  * [BM90] Pierre Berthelot and William Messing  “Théorie de Dieudonné cristalline. III. Théorèmes d’équivalence et de pleine fidélité”  In _The Grothendieck Festschrift, Vol. I_ 86, Progr. Math.  Birkhäuser Boston, Boston, MA, 1990, pp. 173–247
  * [Bor69] Armand Borel  “Introduction aux groupes arithmétiques”, Publications de l’Institut de Mathématique de l’Université de Strasbourg, XV. Actualités Scientifiques et Industrielles, No. 1341  Hermann, Paris, 1969
  * [BT84] François Bruhat and Jacques Tits  “Groupes réductifs sur un corps local. II. Schémas en groupes. Existence d’une donnée radicielle valuée.”  In _Publ. Math., Inst. Hautes Étud. Sci._ 60 Springer, Berlin/Heidelberg; Institut des Hautes Études Scientifiques, Bures-sur-Yvette, 1984, pp. 1–194
  * [BT84a] François Bruhat and Jacques Tits  “Schémas en groupes et immeubles des groupes classiques sur un corps local”  In _Bull. Soc. Math. Fr._ 112 Société Mathématique de France (SMF), Paris, 1984, pp. 259–301  DOI: 10.24033/bsmf.2006
  * [CS17] Ana Caraiani and Peter Scholze  “On the generic part of the cohomology of compact unitary Shimura varieties.”  In _Ann. Math. (2)_ 186.3 Princeton University, Mathematics Department, Princeton, NJ; Mathematical Sciences Publishers (MSP), Berkeley, CA, 2017, pp. 649–766  DOI: 10.4007/annals.2017.186.3.1
  * [Dan36] D. Dantzig  “Zur topologischen Algebra. III: Brouwersche und Cantorsche Gruppen”  In _Compos. Math._ 3 Cambridge University Press, Cambridge; London Mathematical Society, London, 1936, pp. 408–426
  * [Del11] Pierre Deligne  “Letter to Kisin”, http://people.math.binghamton.edu/adrian/Letter_Deligne.pdf, 2011
  * [Del71] Pierre Deligne  “Travaux de Shimura”  In _Séminaire Bourbaki, 23ème année (1970/71), Exp. No. 389_ 244, Lecture Notes in Math.  Springer, Berlin, 1971, pp. 123–165
  * [Del82] Pierre Deligne  “Hodge cycles on abelian varieties. (Notes by J. S. Milne).”, Hodge cycles, motives, and Shimura varieties, Lect. Notes Math. 900, 9-100 (1982)., 1982
  * [EGA2] A. Grothendieck  “Éléments de géométrie algébrique. II. Étude globale élémentaire de quelques classes de morphismes”  In _Inst. Hautes Études Sci. Publ. Math._ , 1961  URL: http://www.numdam.org/item?id=PMIHES_1961__8__222_0
  * [EGA4] Alexander Grothendieck  “Éléments de géométrie algébrique IV: Étude locale des schémas et des morphismes de schémas”  Publications mathématiques de l’I.H.É.S., 1964–1967
  * [Gro74] Alexandre Grothendieck  “Groupes de Barsotti-Tate et cristaux de Dieudonné” Séminaire de Mathématiques Supérieures, No. 45 (Été, 1970)  Les Presses de l’Université de Montréal, Montreal, Que., 1974
  * [GW10] Ulrich Görtz and Torsten Wedhorn  “Algebraic geometry I” Schemes with examples and exercises, Advanced Lectures in Mathematics  Vieweg + Teubner, Wiesbaden, 2010  DOI: 10.1007/978-3-8348-9722-0
  * [Hai05] Thomas J. Haines  “Introduction to Shimura varieties with bad reduction of parahoric type”  In _Harmonic analysis, the trace formula, and Shimura varieties. Proceedings of the Clay Mathematics Institute 2003 summer school, Toronto, Canada, June 2–27, 2003_ Providence, RI: American Mathematical Society (AMS), 2005, pp. 583–642
  * [Ham17] Paul Hamacher  “The almost product structure of Newton strata in the deformation space of a Barsotti-Tate group with crystalline Tate tensors”  In _Math. Z._ 287.3-4, 2017, pp. 1255–1277  DOI: 10.1007/s00209-017-1867-2
  * [Haz78] Michiel Hazewinkel  “Formal groups and applications”, Pure and Applied Mathematics, 78. New York-San Francisco-London: Academic Press. XXII, 573 p., 1978
  * [Hes20] Jens Hesse  “Central leaves and EKOR strata on Shimura varieties with parahoric reduction”, 2020  URL: http://nbn-resolving.de/urn:nbn:de:tuda-tuprints-115430
  * [Hes20a] Jens Hesse  “EKOR strata on Shimura varieties with parahoric reduction” Preprint, 2020  arXiv:2003.04738 [math.AG]
  * [HK17] P. Hamacher and W. Kim  “$l$-adic étale cohomology of Shimura varieties of Hodge type with non-trivial coefficients” Preprint, 2017  arXiv:1711.07123v1 [math.NT]
  * [HR17] X. He and M. Rapoport  “Stratifications in the reduction of Shimura varieties.”  In _Manuscr. Math._ 152.3-4 Springer, Berlin/Heidelberg, 2017, pp. 317–343  DOI: 10.1007/s00229-016-0863-x
  * [Jon95] A.. Jong  “Crystalline Dieudonné module theory via formal and rigid geometry”  In _Inst. Hautes Études Sci. Publ. Math._ , 1995, pp. 5–96  URL: http://www.numdam.org/item?id=PMIHES_1995__82__5_0
  * [Kim19] Wansu Kim  “On central leaves of Hodge-type Shimura varieties with parahoric level structure”  In _Math. Z._ 291.1-2 Springer, Berlin/Heidelberg, 2019, pp. 329–363
  * [Kis10] Mark Kisin  “Integral models for Shimura varieties of abelian type”  In _J. Amer. Math. Soc._ 23.4, 2010, pp. 967–1012  DOI: 10.1090/S0894-0347-10-00667-3
  * [Kis17] Mark Kisin  “Mod $p$ points on Shimura varieties of abelian type”  In _J. Am. Math. Soc._ 30.3 American Mathematical Society (AMS), Providence, RI, 2017, pp. 819–914
  * [Kot92] Robert E. Kottwitz  “Points on some Shimura varieties over finite fields”  In _J. Amer. Math. Soc._ 5.2, 1992, pp. 373–444  DOI: 10.2307/2152772
  * [Kot97] Robert E. Kottwitz  “Isocrystals with additional structure. II”  In _Compositio Math._ 109.3, 1997, pp. 255–339  DOI: 10.1023/A:1000102604688
  * [KP15] M. Kisin and G. Pappas  “Integral models of Shimura varieties with parahoric level structure” Preprint, 2015  arXiv:1512.01149v2 [math.AG]
  * [Lan00] Erasmus Landvogt  “Some functorial properties of the Bruhat-Tits building”  In _J. Reine Angew. Math._ 518 De Gruyter, Berlin, 2000, pp. 213–241  DOI: 10.1515/crll.2000.006
  * [Lan96] Erasmus Landvogt  “A compactification of the Bruhat-Tits building” 1619, Lecture Notes in Mathematics  Springer-Verlag, Berlin, 1996, pp. viii+152  DOI: 10.1007/BFb0094594
  * [Lau14] Eike Lau  “Relations between Dieudonné displays and crystalline Dieudonné theory”  In _Algebra Number Theory_ 8.9, 2014, pp. 2201–2262  DOI: 10.2140/ant.2014.8.2201
  * [Man04] Elena Mantovan  “On certain unitary group Shimura varieties” Variétés de Shimura, espaces de Rapoport-Zink et correspondances de Langlands locales  In _Astérisque_ , 2004, pp. 201–331
  * [Man05] Elena Mantovan  “On the cohomology of certain PEL-type Shimura varieties”  In _Duke Math. J._ 129.3, 2005, pp. 573–610  DOI: 10.1215/S0012-7094-05-12935-0
  * [Mes72] William Messing  “The crystals associated to Barsotti-Tate groups: with applications to abelian schemes”, Lecture Notes in Mathematics, Vol. 264  Springer-Verlag, Berlin-New York, 1972
  * [Mil05] J.. Milne  “Introduction to Shimura varieties.”  In _Harmonic analysis, the trace formula, and Shimura varieties. Proceedings of the Clay Mathematics Institute 2003 summer school, Toronto, Canada, June 2–27, 2003_ Providence, RI: American Mathematical Society (AMS), 2005, pp. 265–378
  * [Oor04] Frans Oort  “Foliations in moduli spaces of abelian varieties”  In _J. Amer. Math. Soc._ 17.2, 2004, pp. 267–296  DOI: 10.1090/S0894-0347-04-00449-7
  * [Oor09] Frans Oort  “Foliations in moduli spaces of abelian varieties and dimension of leaves”  In _Algebra, arithmetic, and geometry: in honor of Yu. I. Manin. Vol. II_ 270, Progr. Math.  Birkhäuser Boston, Inc., Boston, MA, 2009, pp. 465–501  DOI: 10.1007/978-0-8176-4747-6_15
  * [OZ02] Frans Oort and Thomas Zink  “Families of $p$-divisible groups with constant Newton polygon”  In _Doc. Math._ 7, 2002, pp. 183–201
  * [Pin90] Richard Pink  “Arithmetical compactification of mixed Shimura varieties” 209, Bonner Mathematische Schriften [Bonn Mathematical Publications]  Universität Bonn, Mathematisches Institut, 1990
  * [PRS13] Georgios Pappas, Michael Rapoport and Brian Smithling  “Local models of Shimura varieties, I. Geometry and combinatorics”  In _Handbook of moduli. Vol. III_ 26, Adv. Lect. Math. (ALM)  Int. Press, Somerville, MA, 2013, pp. 135–217
  * [PZ13] Georgios Pappas and Xinwen Zhu  “Local models of Shimura varieties and a conjecture of Kottwitz”  In _Invent. Math._ 194.1 Springer, Berlin/Heidelberg, 2013, pp. 147–254  DOI: 10.1007/s00222-012-0442-z
  * [Rap05] Michael Rapoport  “A guide to the reduction modulo $p$ of Shimura varieties.”  In _Formes automorphes (I). Actes du Semestre du Centre Émile Borel, Paris, France, 17 février au 11 juillet 2000_ Paris: Société Mathématique de France, 2005, pp. 271–318
  * [RV14] Michael Rapoport and Eva Viehmann  “Towards a theory of local Shimura varieties”  In _Münster J. Math._ 7.1 Universität Münster, Mathematical Institutes, Münster, 2014, pp. 273–326
  * [RZ96] M. Rapoport and Th. Zink  “Period spaces for $p$-divisible groups” 141, Annals of Mathematics Studies  Princeton University Press, Princeton, NJ, 1996  DOI: 10.1515/9781400882601
  * [SGA3] “Schémas en groupes”, Séminaire de Géométrie Algébrique du Bois Marie 1962/64 (SGA 3). Dirigé par M. Demazure et A. Grothendieck. Lecture Notes in Mathematics, Vols. 151–153  Springer-Verlag, Berlin-New York, 1970
  * [Stacks] The Stacks Project Authors  “Stacks Project”, http://stacks.math.columbia.edu, 2020
  * [SW13] Peter Scholze and Jared Weinstein “Moduli of $p$-divisible groups” In _Camb. J. Math._ 1.2 International Press of Boston, Somerville, MA, 2013, pp. 145–237 DOI: 10.4310/CJM.2013.v1.n2.a1
  * [Vas08] Adrian Vasiu “Level $m$ stratifications of versal deformations of $p$-divisible groups” In _J. Algebraic Geom._ 17.4, 2008, pp. 599–641 DOI: 10.1090/S1056-3911-08-00495-5
  * [Wor13] D. Wortmann “The $\mu$-ordinary locus for Shimura varieties of Hodge type” Preprint, 2013 arXiv:1310.6444v1 [math.AG]
  * [Zha15] C. Zhang “Stratifications and foliations for good reductions of Shimura varieties of Hodge type” Preprint, 2015 arXiv:1512.08102v1 [math.AG]
  * [Zho18] R. Zhou “Mod-$p$ isogeny classes on Shimura varieties with parahoric level structure” Preprint, 2018 arXiv:1707.09685v2 [math.NT]
  * [Zhu17] Xinwen Zhu “Affine Grassmannians and the geometric Satake in mixed characteristic.” In _Ann. Math. (2)_ 185.2 Princeton University, Mathematics Department, Princeton, NJ; Mathematical Sciences Publishers (MSP), Berkeley, CA, 2017, pp. 403–492
  * [Zin01] Thomas Zink “A Dieudonné theory for $p$-divisible groups.” In _Class field theory – its centenary and prospect. Proceedings of the 7th MSJ International Research Institute of the Mathematical Society of Japan, Tokyo, Japan, June 3–12, 1998_ Tokyo: Mathematical Society of Japan, 2001, pp. 139–160
  * [Zin01a] Thomas Zink “On the slope filtration” In _Duke Math. J._ 109.1, 2001, pp. 79–95 DOI: 10.1215/S0012-7094-01-10913-7
  * [Zin02] Thomas Zink “The display of a formal $p$-divisible group.” In _Cohomologies $p$-adiques et applications arithmétiques (I)_ Paris: Société Mathématique de France, 2002, pp. 127–248