# Leveraging Large Language Model and Story-Based Gamification in Intelligent
Tutoring System to Scaffold Introductory Programming Courses: A Design-Based
Research Study

Chen Cao<EMAIL_ADDRESS>0000-0003-4368-0336 University of
SheffieldUnited KingdomS10 2TN

(2023)

###### Abstract.

Programming skills are rapidly becoming essential for many educational paths
and career opportunities. Yet, for many international students, the
traditional approach to teaching introductory programming courses can be a
significant challenge due to the complexities of the language, the lack of
prior programming knowledge, and the language and cultural barriers. This
study explores how large language models and gamification can scaffold coding
learning and increase Chinese students’ sense of belonging in introductory
programming courses. In this project, a gamification intelligent tutoring
system was developed to adapt to Chinese international students’ learning
needs and provides scaffolding to support their success in introductory
computer programming courses.

My research includes three studies: a formative study, a user study of an
initial prototype, and a computer simulation study with a user study in
progress. Both qualitative and quantitative data were collected through
surveys, observations, focus group discussions and computer simulation. The
preliminary findings suggest that GPT-3-enhanced gamification has great
potential in scaffolding introductory programming learning by providing
adaptive and personalised feedback, increasing students’ sense of belonging,
and reducing their anxiety about learning programming.

††conference: IUI’23: 28th International Conference on Intelligent User
Interfaces; March 27 – 31, 2023; Sydney, Australia††booktitle: IUI’23: 28th
International Conference on Intelligent User Interfaces, March 27 – 31, 2023,
Sydney, Australia††journalyear: 2023††copyright: rightsretained††conference:
28th International Conference on Intelligent User Interfaces; March 27–31,
2023; Sydney, NSW, Australia††booktitle: 28th International Conference on
Intelligent User Interfaces (IUI ’23 Companion), March 27–31, 2023, Sydney,
NSW, Australia††doi: 10.1145/3581754.3584111††isbn: 979-8-4007-0107-8/23/03

##  1\. Problem Statement

Computing and technology are increasingly ubiquitous and have become a
necessary part of many educational paths, professional opportunities, and
industries (Pedro et al., 2019). Consequently, the importance of programming
knowledge and coding skills has grown significantly in recent years.
Introductory programming courses, often referred to as computer science 1
(CS1) courses are designed to introduce students to the fundamentals of
programming and coding (Becker and Quille, 2019). However, these courses can
be challenging for many international students, who may lack prior programming
expertise and are confronted with a language and cultural barriers(Khanal and
Gaulee, 2019).

Recent years have also seen an increased number of Chinese international
students enrolling in universities in the UK(de Wit, 2020). Despite the clear
benefits of international student enrollment, Chinese international students
can often feel isolated in their new environment and struggle to integrate
into university life(Chen and Zhou, 2019). In the context of programming
education, Chinese international students often face hardship due to a lack of
prior exposure to coding concepts and language (Alaofi and Russell, 2022). As
a result, they are more likely to experience anxiety and a feeling of low
belonging. However, previous research showed bias and stereotypes when
describing Chinese students’ learning behaviours in global higher education
(Heng, 2018). Chinese international students are often perceived as passive
and reluctant learners when adapting to British educational systems (Zhu and
O’Sullivan, 2022). This stereotype, however, fails to take into account the
cultural and educational factors driving these behaviours.

At the same time, gamification has seen a surge in popularity in the
educational sector. It has been used as a tool to engage students and
encourage learning in a range of contexts (Welbers et al., 2019). This
presents an opportunity to investigate the potential of gamification to
scaffold coding learning and increase the sense of belonging among Chinese
international students in introductory programming courses. Additionally, the
use of educational technologies, such as intelligent tutoring systems in
educational settings has been growing steadily in recent years, as an
increasing number of educators recognize the potential of this approach to
engage and motivate learners (Szymkowiak et al., 2021). An intelligent
tutoring system is particularly attractive due to its ability to personalise
the learning experience, offering tailored activities to meet the needs of
diverse learners (Shemshack et al., 2021). In addition, an intelligent
tutoring system can provide feedback in a timely and effective manner,
offering learners the opportunity to improve their skills in a supportive and
engaging environment (AlShaikh and Hewahi, 2021).

The research background of this thesis is rooted in the idea that AI-enhanced
gamification can be used to scaffold learning and increase belonging among
Chinese international students in introductory programming courses. AI-
enhanced gamification combines the use of gamification techniques with AI-
driven intelligent tutoring systems to create a personalised learning
experience. This approach encourages learners to engage in the learning
process and offers them the opportunity to practise and improve their coding
skills in a supportive and engaging environment.

This research is motivated by the need to bridge the gaps in programming
knowledge between Chinese international students and their domestic peers. It
is also driven by the urgent need to create an inclusive learning environment
that is accessible and welcoming to all students. The research findings could
have a considerable impact on the teaching and learning of programming for
Chinese international students. It has the potential to provide insight into
how AI-enhanced gamification can be used to make the learning process easier
and more effective for these students, as well as to foster a sense of
belonging. The findings could inform instructors, administrators, and
policymakers of the most effective strategies for teaching and learning to
program for this population. This study can also be expanded to support other
underrepresented student groups (such as female STEM students and other
international students from different countries and cohorts) who also need a
sense of belonging to their peers, faculty, and subject-related careers.

##  2\. Research aim and questions

The main focus of this research project is to explore the potential of AI-
enhanced gamification to scaffold coding learning and increase the sense of
belonging among Chinese international students in introductory programming
courses. Specifically, the focus will be on designing, evaluating, and
refining the use of AI-enhanced gamification to improve learning outcomes and
increase motivation among Chinese international students in introductory
programming courses.

The research project is framed within the context of design-based research,
which emphasises the importance of designing, implementing, and evaluating a
learning environment, with the goal of optimising the learning experience. The
research project is guided by two research questions:

  * •

RQ1: what are the challenges that Chinese international students are facing
regarding developing a sense of belonging and code learning in introductory
programming courses?

  * •

RQ2: how can AI and gamification be used to scaffold coding learning and
increase the sense of belonging among Chinese international students in
introductory programming courses?

##  3\. Related work

This research is situated within the larger context of educational technology
and game-based learning. The use of technology in education has become
increasingly popular, with a growing number of educators recognising the
potential of this approach to engage and motivate learners (Szymkowiak et al.,
2021). This study specifically focuses on the application of AI-enhanced
gamification to scaffold coding learning among Chinese international students.

There has been a considerable amount of research examining the challenges and
barriers associated with teaching introductory programming courses, e.g.
(Alam, 2022). These studies have highlighted the need for more effective
pedagogical approaches to engage learners in CS1 courses. Several researchers
have proposed the use of game-based learning techniques such as serious games
and simulations to increase engagement and motivation among students
(Papadakis and Kalogiannakis, 2019) . Gamification combines elements of gaming
(e.g., points, rewards, leaderboards) with traditional educational activities
to create an engaging learning experience.

The use of intelligent tutoring systems (ITS) in programming education has
also been studied extensively. Research has shown that ITS can improve student
performance and reduce cognitive load in programming courses. For example,
Grenander et al. (Grenander et al., 2021) proposed an AI-enhanced educational
system that was designed to provide personalised feedback based on individual
learners’ needs and evaluated its effectiveness using deep discourse analysis.
Similarly, Eguchi (Eguchi et al., 2021) investigated the use of AI-enhanced
games to support STEM education for children with visual impairments. They
found that the AI-enhanced game had a positive impact on engagement and
motivation among participants. Furthermore, ITS can provide personalised
instruction and targeted remediation, which can be particularly beneficial for
those who lack prior experience in programming. In particular, GPT-3
(Generative Pre-trained Transformer 3), an advanced language model developed
by OpenAI, has been used to improve the performance of ITSs by providing
natural language understanding capabilities(Tack and Piech, 2022).

This research project is also informed by recent studies on belonging in
higher education. It has been argued that belonging is an important factor
when it comes to understanding and supporting the learning experiences of
international students (Cureton and Gravestock, 2019). Studies have also shown
that international students often face difficulties in developing a sense of
belonging due to language and cultural barriers (Cena et al., 2021).

##  4\. Research method

This research project applies design-based research (DBR) as its
methodological approach. DBR is an iterative process that emphasises the
importance of designing, implementing, and evaluating educational technology
to optimise the learning experience. This approach has been widely used in
educational settings to investigate the effectiveness of new technologies and
approaches in engaging and motivating learners. The use of DBR provides a
unique opportunity to combine theoretical insights with practical
implementation to create meaningful interventions that are responsive to the
needs of diverse learners.

The research is divided into three phases: 1) a survey to identify Chinese
students’ needs and the challenges they met regarding a sense of belonging and
learning experience in introductory programming courses; 2) a design probe
with a story-based gamification prototype to increase Chinese students’ sense
of belonging and improve their learning experience in CS1 courses with user
evaluation; 3) the development and systematic evaluations of an intelligent
tutoring system leveraging large language model and gamification features for
CS1 courses.

The study used a mixed-methods approach, incorporating both quantitative and
qualitative data. The quantitative data were collected through a survey
measuring students’ sense of belonging, academic performance, and academic
emotions, and a computer simulation study evaluating the performances of the
ITS. The qualitative data were collected through participatory observations
and focus group interviews to explore students’ experiences and perceptions in
more depth. The data were analyzed using descriptive statistics, thematic
analysis, and computational analysis.

##  5\. Preliminary results

###  5.1. Study 1: Formative study

The initial survey was conducted as a formative study in the first semester of
the 21/22 academic year. Based on the questionnaire with 57 Chinese
international students and in-depth interviews with nine of them, the study
found a number of unique challenges faced by the participants in the CS1
courses. Chinese students generally found it difficult to cope with the
demands of the programming course. A number of factors were identified as
contributing to this difficulty, including the challenges to adapt to new
teaching methods emphasizing independent thinking, critical thinking, and
innovative learning; less interaction with teachers and classmates;
disconnection between the knowledge and real-life cases and not receiving
enough academic support with timely help and real-time feedback. As a result,
they have low engagement in classes, low retention and negative academic
emotions. One of the key factors that have been identified as contributing to
these unsmooth academic transition experiences is a lack of intrinsic
motivation, especially a lack of sense of belonging.

###  5.2. Study 2: A prototype of story-based ITS

The second study designed, deployed and evaluated a prototype of a story-based
gamification design on a learning management system Blackboard. Based on the
initial findings from the survey, the prototype adopted a set of gamification
features, including animated trailers, story-telling, role-play, teamwork,
points, leaderboards and feedback. The study was conducted over a period of
two weeks with a total of 34 Chinese students enrolled in the introductory
programming course INF6032 Big Data Analytics, which is one of the core
modules of the MSc in Data Science programme at the University of Sheffield.
The practical session in week 6 was set in the context of airline companies’
survival during the pandemic. In week 7, the story was about solving a
criminal case. The narrative storytelling of each session with real-life cases
was designed to make students feel more connected to the programme.

The findings from questionnaires (n=32), focus groups (n=32) and participatory
observation revealed that Chinese students generally had positive perceptions
of the effect of story-based gamification design on their sense of belonging
in the introductory programming course. The majority of students reported
feeling more motivated to learn, more engaged in the course, and more
connected to their classmates as a result of the story-based gamification
design. In addition, the findings suggested that story-based gamification
design had a positive effect on Chinese students’ sense of belonging in the
introductory programming course. Most students reported feeling a greater
sense of belonging in the course after the story-based gamification design was
implemented. It was also found that Chinese students have some unique cultural
needs which should be considered in the story-based gamification design. These
findings suggest that story-based gamification design can be an effective
means of improving Chinese students’ sense of belonging in the introductory
programming course.

###  5.3. Study 3: A model of the story-based ITS empowered by GPT-3

The reflection upon the findings from the previous study encouraged the
development of the current model of the story-based intelligent tutoring
system (ITS). The ITS empowered by GPT-3 was designed to better fit the needs
of individual learners, and increase their sense of belonging to the class,
institution and career-related subject, in order to further foster inclusion
in higher education.

The design of the user interface is inspired by the English TV series Sherlock
with abundant modern English elements, which is ideal to engage international
students coming to the UK for higher education. The user interface of the ITS
was built on a popular open-source template111Bootstrap 4 UI kit,
https://demos.creative-tim.com/now-ui-kit-react/#/index?ref=nukr-github-readme
in React, a mainstream JavaScript library for developing front-ends. The
overall design principles of the web app are inspired by the famous detective,
Sherlock Holmes. The colour scheme, typography and navigation are all based on
the idea of the Mind Palace, with a focus on the dark blue and white colours
to convey a feeling of sophistication and intelligence.

The development of the learning system consists of four main components: the
instructional content, the gamification mechanism design, the user interface,
and the generative language model. The learning content consists of the
introduction of programming knowledge, demo explanations and exercises to
provide the learning materials to the users. The game engine was designed to
provide a fun and engaging learning experience. It consisted of a set of
gamification elements, including alternative reality, points, badges,
personalized feedback and encouragements, levels and challenges, an avatar, a
progress bar and an exploratory word cloud. The user interface is responsible
for displaying the learning content and gamification mechanisms to the users
and accepting user input. The generative language model is used for providing
AI capabilities such as Q&A and explaining code to the system.

The system is designed to be a web-based application. It consists of three
main components, which are the front end, the back end and the GPT-3 platform.
The front end used React library to develop the user interface and enable
interactions. The back end is responsible for data storage, processing and
retrieval. In this system, Firebase is applied to connect with the interface
and synchronise data. All the states (user behaviours and inputs) in the front
end were centralised with Redux and synchronised to Firebase, where all the
user data, such as the user’s avatar, learning progress and chat history with
the AI conversational agents were stored for further analysis.

GPT-3 was used in this system to train the AI model for the intelligent
agents. The agents were designed to interact with the users, and provide
guidance and support throughout the learning process. In addition, the agents
were also responsible for marking the user’s progress and giving feedback.
There were four chatbots providing support to students with different
functions, including the intelligent tutoring bot Sherlock, the peer and
critical thinking bot Watson, the career engagement bot Inspector Lestrade and
the emotion support bot Mrs Hudson. When the student asked a question,
Sherlock answered it first. Then Watson, Inspector Lestrade and Mrs Hudson
generated follow-up questions to continue the conversation, in order to foster
inquiry-based learning and improve students’ computational thinking and
understanding of job market needs.

In the interface of marking and feedback, students can submit the codes for
quizzes and get real-time and tailored feedback about their answers. The model
first extracts the code snippet from the user input and then generates an
explanation for the code snippet by calling the GPT-3 API with the code
snippet and a prompt as input. The prompt is a natural language question that
can be used to generate an explanation for the code snippet. The code-
explaining module uses a set of predefined prompts to generate explanations
for the code snippets. The prompts are selected to cover a wide range of
programming concepts.

A computer simulation study and a pilot user study were conducted to evaluate
the ITS. By using prompt programming, a dataset containing 360 rounds of
simulated questions that students may ask in the class and answers were
simulated based on the learning objectives and instructors’ reflections. The
findings from semantic similarity analysis, topic modelling and sentiment
analysis indicated that GPT-3 empowered agents performed well in providing
feedback and having insightful conversations, but the quality of answers
depend on the form and level of the questions. The ITS was also piloted with
different stakeholders, including potential users, instructors, practitioners
and researchers to assess the usability, acceptability and effectiveness of
the system. Findings from the pilot study indicated that the system performs
well in terms of usability and acceptability. Most participants reported
feeling positive about their experience with the intelligent tutoring system,
with many saying they were impressed and felt supported by the AI agents. They
also commented positively on the gamification elements in the system, such as
the intriguing storyline of Sherlock.

##  6\. Implications and future work

The current studies suggested that GPT-3 as a large language model trained on
a vast amount of text data is particularly well-suited to answering questions
at the remembering and understanding levels of Bloom’s taxonomy. These levels
involve recalling and comprehending information, which is a strength of large
language models like GPT-3. IT also performed well in answering questions at
the higher levels of the taxonomy, such as analysis, synthesis, and evaluation
if the questions are well-formed. Generally, GPT-3’s ability to answer
questions in the educational scenario of CS1 courses is promising but limited
by its lack of domain-specific knowledge and expertise.

Despite the promising results of this study, there are still a number of areas
for further research and development. Firstly, more detailed studies should be
conducted to investigate the long-term impact of AI-enhanced gamification on
students’ learning outcomes and sense of belonging in introductory programming
courses. Secondly, contextual factors such as culture, language and prior
knowledge should be taken into account when designing AI-enhanced gamification
systems to better support Chinese international students. Thirdly, more
sophisticated machine learning models can be explored to improve the
performance of AI agents. Finally, more user studies need to be conducted to
explore how AI-enhanced gamification can be used to foster collaboration among
team members, reduce anxiety and facilitate learning transfer.

In order to fully realize the potential of the GPT-3 enhanced ITS to increase
student sense of belonging in higher education, more research must be
conducted in real educational scenarios. More user studies are needed to
assess how GPT-3 affects student academic performance, motivation, and sense
of belonging. In addition, researchers must continue to refine GPT-3’s
capabilities to improve the ITS’s ability to pick up on subtle social cues and
to develop a better understanding of student emotions and motivations.

## References

  * (1)
  * Alam (2022) Ashraf Alam. 2022\.  Platform Utilising Blockchain Technology for eLearning and Online Education for Open Sharing of Academic Proficiency and Progress Records.  In _Smart Data Intelligence_. Springer, 307–320.
  * Alaofi and Russell (2022) Suad Alaofi and Seán Russell. 2022.  The Influence of Foreign Language Classroom Anxiety on Academic Performance in English-based CS1 Courses.  _ACM International Conference Proceeding Series_.  https://doi.org/10.1145/3555009.3555020
  * AlShaikh and Hewahi (2021) Fatema AlShaikh and Nabil Hewahi. 2021.  Ai and machine learning techniques in the development of Intelligent Tutoring System: A review. In _2021 International Conference on innovation and Intelligence for informatics, computing, and technologies (3ICT)_. IEEE, 403–410.
  * Becker and Quille (2019) Brett A Becker and Keith Quille. 2019.  50 years of cs1 at sigcse: A review of the evolution of introductory programming education research. In _Proceedings of the 50th acm technical symposium on computer science education_. 338–344.
  * Cena et al. (2021) Elida Cena, Stephanie Burns, and Paul Wilson. 2021\.  Sense of Belonging, Intercultural and Academic Experiences among International Students at a University in Northern Ireland.  _Journal of International Students_ (2021).
  * Chen and Zhou (2019) Jia Chen and George Zhou. 2019.  Chinese international students’ sense of belonging in North American postsecondary institutions: A critical literature review.  _Brock Education Journal_ 28, 2 (2019), 48–63.
  * Cureton and Gravestock (2019) Debra Cureton and Phil Gravestock. 2019.  ‘We Belong’: differential sense of belonging and its meaning for different ethnicity groups in higher education.  _Compass: Journal of Learning and Teaching_ (2019).
  * de Wit (2020) Hans de Wit. 2020\.  Internationalization of Higher Education: The Need for a More Ethical and Qualitative Approach.  _Journal of International Students_ 10 (2020).
  * Eguchi et al. (2021) Amy Eguchi, Hiroyuki Okada, and Yumiko Muto. 2021\.  Contextualizing AI education for K-12 students to enhance their learning of AI literacy through culturally responsive approaches.  _KI-Künstliche Intelligenz_ 35, 2 (2021), 153–161.
  * Grenander et al. (2021) Matt Grenander, Robert Belfer, Ekaterina Kochmar, Iulian V Serban, François St-Hilaire, and Jackie C K Cheung. 2021.  Deep Discourse Analysis for Generating Personalized Feedback in Intelligent Tutor Systems.  https://github.com/UKPLab/sentence-transformers
  * Heng (2018) Tang T. Heng. 2018\.  Different is not deficient: contradicting stereotypes of Chinese international students in US higher education.  _Studies in Higher Education_ 43 (1 2018), 22–36.  Issue 1. https://doi.org/10.1080/03075079.2016.1152466
  * Khanal and Gaulee (2019) Jeevan Khanal and Uttam Gaulee. 2019.  Challenges of international students from pre-departure to post-study: A literature review.  _Journal of International Students_ 9, 2 (2019), 560–581.
  * Papadakis and Kalogiannakis (2019) Stamatios Papadakis and Michail Kalogiannakis. 2019.  Evaluating the effectiveness of a game-based learning approach in modifying students’ behavioural outcomes and competence, in an introductory programming course. A case study in Greece.  _International Journal of Teaching and Case Studies_ 10, 3 (2019), 235–250.
  * Pedro et al. (2019) Francesc Pedro, Miguel Subosa, Axel Rivas, and Paula Valverde. 2019.  Artificial intelligence in education: Challenges and opportunities for sustainable development.  (2019).
  * Shemshack et al. (2021) Atikah Shemshack, Jonathan Michael Spector, et al. 2021\.  A comprehensive analysis of personalized learning components.  _Journal of Computers in Education_ 8, 4 (2021), 485–503.
  * Szymkowiak et al. (2021) Andrzej Szymkowiak, Boban Melović, Marina Dabić, Kishokanth Jeganathan, and Gagandeep Singh Kundi. 2021\.  Information technology and Gen Z: The role of teachers, the internet, and technology in the education of young people.  _Technology in Society_ 65 (2021), 101565.
  * Tack and Piech (2022) Anaïs Tack and Chris Piech. 2022.  The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues.  _ArXiv_ abs/2205.07540 (2022).
  * Welbers et al. (2019) Kasper Welbers, Elly A Konijn, Christian Burgers, Anna Bij De Vaate, Allison Eden, and Britta C Brugman. 2019.  Gamification as a tool for engaging student learning: A field experiment with a gamified app.  _E-Learning and Digital Media_ 16, 2 (2019), 92–109.
  * Zhu and O’Sullivan (2022) Haiping Zhu and Helen O’Sullivan. 2022.  Shhhh! Chinese students are studying quietly in the UK.  _Innovations in Education and Teaching International_ 59, 3 (2022), 275–284.