# Educational data augmentation in physics education research using ChatGPT

Fabian Kieser1 Peter Wulff1 Jochen Kuhn2 Stefan Küchemann2 mailto:
<EMAIL_ADDRESS>1 Physics and Physics Education Research, Heidelberg
University of Education, Im Neuenheimer Feld 561, 69120 Heidelberg, Germany  2
Chair of Physics Education, Faculty of Physics, Ludwig-Maximilians-Universität
München (LMU Munich), Geschwister-Scholl-Platz 1, 80539 Munich, Germany

###### Abstract

Generative AI technologies such as large language models show novel potentials
to enhance educational research. For example, generative large language models
were shown to be capable to solve quantitative reasoning tasks in physics and
concept tests such as the Force Concept Inventory (FCI). Given the importance
of such concept inventories for physics education research, and the challenges
in developing them such as field testing with representative populations, this
study seeks to examine to what extent a generative large language model could
be utilized to generate a synthetic data set for the FCI that exhibits
content-related variability in responses. We use the recently introduced
ChatGPT based on the GPT 4 generative large language model and investigate to
what extent ChatGPT could solve the FCI accurately (RQ1) and could be prompted
to solve the FCI as-if it were a student belonging to a different cohort
(RQ2). Furthermore, we study, to what extent ChatGPT could be prompted to
solve the FCI as-if it were a student having a different force- and mechanics-
related preconception (RQ3). In alignment with other research, we found that
ChatGPT could accurately solve the FCI. We furthermore found that prompting
ChatGPT to respond to the inventory as-if it belonged to a different cohort
yielded no variance in responses, however, responding as-if it had a certain
preconception introduced much variance in responses that approximate real
human responses on the FCI in some regards.

Large Language Models, ChatGPT, Concept inventories, FCI, Data augmentation

##  I Introduction

Conducting empirical educational research is at the center of learning and
instruction sciences, for instance, for evaluating the effectiveness of
instructional materials, proving theoretical frameworks, or designing test
instruments. The design of effective learning materials or test instruments
often requires a large amount of human effort to perform the studies and
several iterations within the empirical validation process for the
optimization of the materials. Additionally, conducting empirical studies
requires financial investments, such as providing incentives for
participation.

Furthermore, in the past decade, learning systems based on artificial
intelligence (AI), such as AI-based intelligent tutoring systems or automated
feedback systems, have received growing attention, and several efforts have
targeted the implementation of AI tools in learning environments. For a good
performance, such AI tools require training and fine-tuning with a large
amount of empirical data. Although there are a growing number of pre-trained
AI-based models that provide good results for cases similar to the training
data, it is often necessary to fine-tune them for desired performance in
specific applications.

Recent advances of large language models (LLM) could have several
opportunities for education. Kasneci and colleagues (2023) suggest the number
of opportunities for learning and teaching, such as a support in writing
tasks, development of critical thinking, individual learning opportunities,
semi-automated grading of students’ works and lesson planning [1]. For
educational research, LLM may also help with summarizing and categorization
tasks. For instance, van de Shoot and colleagues (2021) developed a tool for
automated title and abstract screening for systematic literature reviews [2],
which was shown to save a significant amount of time. Particularly, the
release of ChatGPT (based on GPT 3.5) has stimulated several research efforts
to explore the opportunities for education. For instance, ChatGPT can be used
summarizing texts, zero-shot text classification tasks and programming tasks
[3, 4].

It was also found that ChatGPT is able to solve a variety of problems with
zero-shot learning, it performs reasonably well on university exams in
medicine and can support in solving higher-order problems in pathology [5].
These results offer the opportunity to test developed assessments before
administering it to students and subsequently optimizing them based on the
answers of ChatGPT. This is an important benefit, as it can save time and
effort in the assessment development process and yield insights into
psychometric properties of the instruments. Additionally, in science education
research, concept tests, such as concept inventories, play an important role
to diagnose students’ conceptual understanding and inform teachers about
prevailing students’ difficulties.

In physics education research, the most used concept inventory (CI) is the
Force Concept Inventory (FCI) developed by Hestenes and colleagues [6]. As a
common concept test, it consists of several multiple-choice questions that
target specific preconceptions related to Newtonian mechanics, in particular
the force concept. Therefore, the answer alternatives are specifically
designed to reflect these preconceptions. To use AI-based methods such as LLM
to test features of concept tests, e.g., if they capture students’
difficulties, it is necessary that LLM are aware of these difficulties and
know how students would answer as-if they experience one or more of these
difficulties. Furthermore, if a LLM would be able to answer concept questions
from the perspective of a student who experiences difficulties, the created
synthetic data could be used to train and fine-tune smaller AI tools for
education that support learners or better inform teachers. Therefore, it would
be a great benefit if LLM are able to create a synthetic data set that closely
resembles a realistic student data set, as it would save a significant amount
of time and effort in the development of assessments and concept tests.

For this purpose, we explore in this work the opportunity of creating
synthetic empirical data with LLM. First, we provide an overview of relevant
previous works and then specify research questions that target certain aspects
of the suitability to generate a synthetic data set in physics education
research.

##  II Theoretical Background

###  II.1 Concept inventories in science subjects

Physics in particular and science subjects more generally are characterized by
a structured knowledge base that is centered around key concepts [7, 8].
Hence, the conceptual understanding of key concepts in scientific disciplines
such as evolution or force is important, and key to scientific literacy. To
measure high-level conceptual understanding and evaluate pedagogical
practices, science education researchers devised so-called concept inventories
[9, 10]. Concept inventories have been developed and widely employed to assess
the understanding of key concepts such as biological evolution [11], climate
change [12], or Newtonian force [13]. These concept inventories typically
involve multiple-choice questions, where designing distractors that capture
common preconceptions is of crucial importance. Concept inventories have
helped science education researchers to empirically show to what extent a
basic level of conceptual understanding is reached in school and university
education. Oftentimes, it is found that a full understanding of key concepts
such as in Newtonian mechanics the force concept is typically reached only by
few students even after (traditional) instruction. Moreover, concept
inventories helped researchers to glean evidence supporting the effectiveness
of interactive engagement instructional formats, which yielded higher learning
gains [13]. The concept inventories thus helped to substantially improve
instruction, e.g., of mechanics in introductory physics, across universities
[14].

###  II.2 Development of CIs

Designing concept inventories was described by Adams and Wieman as a process
consisting of consecutive phases [15], namely 1) delineation of purpose and
scope of the construct to be measured, 2) development and evaluation of test
specifications, 3) field testing, evaluation, and selection of items, and 4)
assembly and evaluation of the test. Phase 1 includes information about expert
thinking on the construct and requirements for the concept inventory to be
implemented, such as no required training and ease of use. In Phase 2,
decisions on psychometric properties are considered, as well as the target
population for which the concept inventory is designed. Phases 3 and 4
comprise the bulk of the work needed to design the concept inventory. Six
iterative steps are distinguished that include a) establishing topics of
interest, b) interviews and observations to identify student thinking, c)
create open-ended survey to probe student thinking, d) create forced-answer
test, e) carry out validation interviews, and f) administer to classes and run
statistical tests. Establishing the topics of interests typically requires a
resource-intense Delphi Process [10]. Often, 15 to 20 experts need to be
recruited to discuss and refine topics or constructs in concept inventories.
[10, 15]. Developing a concept inventory is thus an extensive and time-
consuming task that involves substantial resources, where AI technologies
might be useful in almost any phase and step. Once qualitative, open-ended
analyses are completed, suitable forced-choice (e.g., multiple-choice) items
have to be field-tested where students’ responses are sampled in order to
determine the underlying structure of the concept inventory in terms of
psychometric properties [14, 15]. In the field tests class sizes of a “few
hundred or more [students] are desirable” [15, p. 1303]. In line with
generalizability theory, administering parallel tests of tests to two
equivalent populations are desired [15]. However, these requirements are
difficult to meet, and place restrictions on small institutions that might not
be able to develop concept inventories in the first place. It might also be
difficult to sample representative student populations, due to various reasons
such as restricted access, ecological considerations, or conceptual problems
such as defining the target population. Concept inventories would also have to
be field tested to different age groups to determine potential differences
[14].

Given this resource intensive process of field testing concept inventories it
would be desirable for the PER community to enhance existing strategies to
field test CI. A recently evolved strategy might be the use of generative AI
technologies such as LLM that were found to excel at various language
generation tasks that even involved knowledge application. LLMs have been
found to be capable of generating synthetic data and solving problems without
being explicitly instructed to do so; therefore, they are valuable candidates
for outsourcing some of the field testing in concept inventory development—and
test instrument development more generally.

###  II.3 Large Language Models (LLM)

AI technologies have enabled tremendous progress in fields such as computer
vision and natural language processing (NLP). A first breakthrough moment for
machine learning (inductive problem solving by computers) was probably when
deep neural networks could be utilized to accurately classify pictures. Even
more so, since its inception [16], and especially since 2017 so-called LLM
have had a profound impact on solving all sorts of language-related problems
as diverse as question answering, translation, or summarizing text [17, 18,
19]. There is a vast variety of LLM architectures that accomplish specific
tasks. For example, sequence-to-sequence models were used to translate an
input to an output, or encoder architectures were used to classify an input
into predefined categories. Most often, some form of deep neural networks
underlie the architecture of LLM. A general feature for these LLM was that
they are sensitive to number of parameters of the artificial neural network,
train set size, or amount of compute time, where increases in all of these
features generally improve the performance of the LLM [20].

Many LLM are trained in two steps. The first step consists semi-supervised
learning in which a masked language model objective is pursued. In masked
language modeling context words are masked out and have to be predicted by the
algorithm, given a related sequence of words [21].111More precisely, tokens
are used instead of words, which are more general sequences of letters that
can be added to form words. However, they have the advantage that also unseen
words in the training data can be processed. Early LLM were trained based on
large text corpora such as the Common Crawl (i.e., dump of the Internet,
60%222Percentages are taken from https://en.wikipedia.org/wiki/GPT-3#GPT-3.5
(last access 26 May 2023) for training GPT 3.), WebText2 (22%), or Wikipedia
(3%) with the training objective to predict next words in a sequence (a sort
of Cloze test, [18]), and classify if a subsequent sentence actually follows a
given sentence [18]. To the surprise of many experts, such a simple training
objective enabled large enough models to gain capabilities to perform entirely
novel tasks that were not seen in the training process [24]. Some resemblance
of masked language modeling to human language processing and cognition, namely
predictive coding where future words are sought to be predicted given the
current words, was recognized [25]. The second step consists of a supervised
learning task in which the pre-trained model (also called foundational model,
[26]) is trained or fine-tuned on a specific task on a labeled data set. In
this part, the LLM predicts classes based on textual information.

Among the first LLM that utilized a transformer architecture and solved a wide
range of unseen tasks was the Generative Pretrained Transformer 3 (GPT 3) by
OpenAI. This model was tested on a wide range of tasks such as common sense
reasoning. It was found that it achieved substantial accuracy [24]. A novel
paradigm emerged, namely prompting LLM, i.e., providing it context with the
input that it uses to generate a response. One prompting strategy was to
provide the LLM examples of similar task solutions that could improve accuracy
[24]. Some researchers contended that LLM with their tremendous size (GPT 3
has 175B parameters) overfit the training data and are essentially “stochastic
parrots” [27] that regurgitate the training data. However, analyses of data
contamination (i.e., evaluation examples being present in the training data)
revealed that the LLM could solve tasks that were not contained in the
training data set [24].

While word analogy problems and other fairly simple problem have been solved
by LLM rather early, quantitative reasoning as in disciplines such as biology,
chemistry, or physics was a more difficult task to solve. AI researchers
introduced more advanced training tasks that enabled LLM to also solve
quantitative reasoning problems as well to some extent [28]. In particular,
chain-of-thought prompting was found to be particularly effective [29]. For
once, researchers used science-related content to specifically further fine-
tune (i.e., adapt the weights) the LLM in a learning paradigm called transfer
learning. It was later found that generative LLM such as GPT required task-
specific fine-tuning. However, AI researchers showed that scaling up LLM also
enabled them to solve tasks they have not been specifically fine-tuned for,
so-called few-shot or even zero-shot learning [24]. These LLM achieved
translation, question-answering, and Cloze tasks with only seeing few or zero
examples, and even accomplished reasoning-related capabilities [24, 30]. With
increases in model size (GPT 3 had 175B parameters) LLM became increasingly
capable to perform unseen tasks, such as two- or three-digit arithmetic.
Interestingly, some capabilities only emerge with sufficient model size
(number of parameters), compute steps during training (e.g., training epochs),
and training data size [31]. Nowadays, LLM are as flexible as to pass exams in
medicine, law, or business without specific fine-tuning [32]. Also for concept
inventories in physics, GPT- based LLM such as ChatGPT were found to achieve
almost 100% accuracy [33, 34]. Given these findings, it is a next step to find
out whether and how to utilize LLM-based tools such as ChatGPT for pilot-
testing purposes for CIs.

###  II.4 Data augmentation techniques

The widespread use artificial neural networks is mainly attributed to their
exceptional performance in certain classification and pattern recognition
tasks. To achieve such high performance, the training of neural networks
requires a large amount of data. However, in some cases, it is difficult to
collect a large amount of data, such as empirical data in education or time
series data [35]. This is critical, because a smaller number of training data
usually leads to a reduced generalization ability and a lower overall
performance of the model. One solution to increase the generalization ability
of trained models by reducing the chance of overfitting is the generation of
synthetic data, the so-called data augmentation [36].

In the field of image recognition, data augmentation is a common practice in
which, for example, the original image is cropped, mirrored, or its colors are
changed to enhance the number of images in the training set [37]. In time
series data, a recent literature review provides an overview of magnitude,
time domain, and frequency domain transformation techniques [35]. In an
additional empirical evaluation of various data augmentation methods for time
series data, the authors report both enhanced and decreased generalization
accuracy. For example, a permutation method in which the observations and time
of the observations is changed, leads to detrimental effects, as it breaks the
time dependence of the series.

In our case, the educational data to be enhanced are the solutions on a
standardized concept inventory, the FCI. The analysis of data augmentation of
educational data is important for test instrument development and validation
as well as for adaptive learning systems that provide a personalized response
to a learner’s input or behavior. In concept tests, the items are related to
the concepts, but there is no inherent sequence dependence. Often the items
are even administered in a random order to avoid potential effects of the item
sequence. Therefore, typical data augmentation techniques in the range of time
domain transformations are not suitable for educational data. In addition,
students’ responses to concept test items are not three dimensional as images
(x-coordinate, y-coordinate, color). Therefore, common techniques of image
augmentation are not a promising augmentation method for educational data.

One technique that was used across different fields of data augmentation is
jittering, in which a small offset is given to the original data. However,
educational data such as students’ responses to concept test items are
discrete and not continuous, and small variations in the responses or the
item, in which a certain response was given, would lead to randomly incorrect
responses. Consequently, this technique would have a similar effect as
permutation techniques and it is not likely to perform well.

Therefore, it is important not to translate these existing methods for
educational data augmentation but to be sensitive to the inherent structure of
the data and study options that consider the structure of concepts in the test
items. In contrast to the data types mentioned above, educational data often
exhibits a hierarchical structure, e.g., a student is in a specific class with
a certain teacher and the class is part of a specific school. To account for
this, it is important that a suitable augmentation technique for educational
data would consider different levels of prior knowledge. As mentioned earlier,
large language models have made a tremendous advance in the past years and
they have shown to be able to respond reasonably well on physics concept test
items. Therefore, they are a promising method to augment educational data, but
it is an open question to what extent and how systematically a large language
model can reflect different levels of prior knowledge and students’
difficulties.

To assess the suitability for generating a valuable synthetic dataset in the
field of education and to account for varying levels of prior knowledge, this
study explores the following research questions:

  1. RQ1:

What is the conceptual understanding of the force concept of ChatGPT?

  2. RQ2:

To what extent is ChatGPT able to simulate conceptual understanding of
different cohorts?

  3. RQ3:

Is ChatGPT able to simulate specific preconceptions of students?

##  III Methods

###  III.1 Participants

We compared the responses to the FCI given by ChatGPT with real students in
the first semester at the RPTU Kaiserslautern-Landau, Germany. The engineering
students attended a lecture of experimental physics covering the topic of
mechanics, among others. The FCI was administered at the beginning of the
semester, so at the moment of answering the test items, the students had only
covered these concepts in high school but not yet during the lecture. In
total, there were 121 engineering students (12 female, 105 male, and 4 made no
statement) with an average age of 20.6 years.

###  III.2 The Force Concept Inventory

To examine to what extent ChatGPT can be used fruitfully in educational
contexts for tasks such as solving standardized test instruments and generate
data, we evaluated performance in the domain of mechanics as it is coherently
structured with regards to knowledge. Physics education researchers devised
standardized test instruments that are widely employed across universities
throughout the world. Among the most well-known instruments are concept
inventories that probe learners’ deep understanding of a particular concept.
Physics educators established that learners come with intuitive conceptions of
certain physics-related concepts, also referred to as common-sense knowledge
[38]. These might not coincide with more normative ideas about physics, such
as Newtonian physics, and are typically context-dependent [38], Most learners
developed over her/his upbringing an intuition of forces as acting agents [6,
39], both by experiencing and interacting with the world and hearing language
such as “gravity pulls”. These intuitive conceptions are largely incompatible
with the rich physics conceptions of forces, such as in Newtonian mechanics
where Newton’s laws lay out a consistent conceptualization of forces. The
first law states that an object at constant velocity is remaining in this
state of constant velocity unless a force is acting upon it. The second law
states that the acceleration of the object is dependent upon the force and
mass of the object. Finally, the third law states that every force has an
opposite equal force (actio equals reactio).

To probe students’ understanding of Newtonian mechanics, the FCI was developed
[6], originally in 1992, and revised in 1995, containing 30 multiple-choice
items. The FCI, as is typical for physics problems, combines image and text
data. However, as ChatGPT based on GPT 4 only accepts text data as input, we
transcribed the pictures in the FCI, similar to the procedure employed by West
[34]. For example, a typical FCI item would include a picture of the following
scenario:

> “A heavy ball is attached to a thread and moved horizontally around in a
> circle. At one point, the thread suddenly breaks. The whole process is
> viewed from above. Which trajectory best describes the path of the ball
> after the thread breaks? 1. a trajectory curve that nearly follows the path
> of the circle 2. a straight-line trajectory curve that points tangentially
> along the circle at the break point 3. a straight-line trajectory curve that
> points straight away from the circular path at an angle of 45° from the
> tangent at the break point 4. a trajectory curve that initially goes outward
> from the center, but then takes a trajectory curve in the direction of the
> original motion 5. a trajectory curve that points straight outward from the
> center.”333Chat can be accessed here:
> https://chat.openai.com/share/8ae109d2-8b71-4033-8c62-749c33c2582c, last
> access: June 2023; translated with:
> https://www.deepl.com/de/translator#de/en/)

West [34] verified that besides the transcriptions ChatGPT is still capable to
solve the FCI. The transcription of all items that contained a figure can be
found in the Appendix.

###  III.3 RQ1: Probing conceptual understanding of ChatGPT in physics

Probing accuracy of ChatGPT on physics contents and the FCI was performed by
multiple researchers [41, 33, 34]. Some studies examined qualitative physics-
related dialogues with ChatGPT, and short-form physics essays [42, 43].
Gregorcic and Pendrill [42] found for a simple prompt (“A teddy bear is thrown
into the air. What is its acceleration in the highest point?”) that ChatGPT’s
responses (based on GPT version 3.5) were full of contradictions and concluded
that ChatGPT is not good enough as a cheating tool in physics. Yeadon et al.
[43] used ChatGPT (based on version 3.5) to generate responses (300 word
essays) to questions and found that it performed on par with the students.
Plagiarism detection software was largely incapable to distinguish it from
human writing. Kortemeyer [33] then used ChatGPT (version 3.5) on the FCI and
found that it scored correctly 18 out of 30 questions (60%). This would equal
approximately the performance of a beginning learner. To assure that ChatGPT
was not simply trained on the FCI (or at least could generalize beyond mere
textual matching), Kortemeyer changed surface-level information in an item and
found that ChatGPT was not distracted by these changes, and still performed
correctly. It was also found that ChatGPT made several errors that related to
the impetus (I) preconception. Moreover, other errors related to confusion of
individual forces versus net force on an object, unstable concepts, and
logical errors [33]. West [34] could replicate these findings for ChatGPT
based on GPT version 3.5, and found that a substantial improvement to expert
level performance (22 correct out of 23) could be observed when using ChatGPT
based on GPT version 4. In RQ1, we will generate a larger sample of response
sets for ChatGPT (based on GPT version 4) and calculate the accuracy and
standard deviation of the responses against the correct answers. This will
enable us to estimate the degree to which ChatGPT is certain of its answers.
Similar to West [34], we will then compare ChatGPT’s distribution of the
number of correct answers to the distribution of a student cohort.

###  III.4 RQ2: Simulating conceptual understanding of different cohorts

An important capability of LLM for education is to simulate (synthesize) data,
given specific prompting strategies. In general, pre-trained LLM generate
output based on the context provided in the input. The provided input context
determines the output, given that a different sequence of tokens is to be
predicted. It is then possible to adjust the response behaviour of ChatGPT for
the FCI questions. For example, one could request that ChatGPT always answers
randomly for each question. A more informative prompt would be to ask ChatGPT
to answer from the perspective of a different student group, which could be
related to background variables, such as experience. In RQ2, we chose the
experience-related variable, and prompted ChatGPT to generate responses from
the perspective of a) school students ($N$=8), and b) engineering students
($N$=18). We chose these groups to potentially generate variance and see to
what extent differences emerge in ChatGPT’s responses. We then generate for
each cohort a number of response sets in order to analyze the distribution of
correct answers, comparing the cohorts with the real students and the plain
ChatGPT response sets. As a robust (non-parametric) test statistic, suited for
multiple groups, we utilized Kruskal-Wallis rank test [44]. This test does not
assume normality of residuals, though it assumes identically shaped and scaled
distributions for all groups, and ordinal dependent variable. Effect size
(proportion of variance predicted from the independent variable) was
calculated as $(H-k+1)/(N-k)$, where $H$ is the Kruskal-Wallis test statistic,
$k$ is the number of groups (here 3), and $N$ is the total number of
observations. This effect size is called $\eta^{2}$, where $\eta^{2}=.01$ is a
small effect size, $\eta^{2}=.06$ is a medium effect size, and $\eta^{2}=.14$
is a large effect size [45].

###  III.5 RQ3: Simulating students’ preconceptions with ChatGPT

Similar to simulating different student cohorts, it is equally possible to
simulate students with different preconceptions. The FCI is a considerately
designed instrument, based on an assessment of students’ beliefs about force
against a normative theory, namely Newtonian physics. It is recognized that
overcoming preconceptions is among the crucial tasks in physics and science
education, and it is a hard problem. The FCI can help instructors to align
their instruction and assess students’ understanding. The FCI addresses
different preconceptions that students can hold. The authors identify general
categories of preconception, namely: kinematics (K), impetus (I), active force
(AF), action-reaction pairs (AR), concatenation of influences (CI), and other
influences on motion (OI). Category K indicates if learners have a physics
notion on motion description. Category I refers to learner’s notion that force
is some kind of intrinsic quantity to objects that keeps them moving, once it
has been supplied to them. AF then refers to the conception that motion
implies an active force. AR refers the learner’s conception of interactions as
a struggle where the stronger/more forceful party exerts greater force (most
typically exemplified in the crash of a truck and a car). CI aggregates
several preconceptions, such as the dominance principle in the composition of
two unequal forces on a single object. Finally, also OI subsumes several
preconceptions such as the reasoning with centrifugal force and considering it
as a real force.

Given the well-established theory on preconceptions related to the Newtonian
mechanics and forces in particular, we can utilize this information to prompt
ChatGPT to generate (simulate) responses as-if it were a learner with this
preconception. To do so, we deliberately engineered the prompts given to
ChatGPT. We first checked if ChatGPT knew about (could elaborate on) the
respective preconception (K, I, AF, AR, CI, OI). In fact, ChatGPT knew about
all preconceptions and could elaborate what they were. Afterwards, we
engineered a prompt where the items were preceded with the following phrase:
“Respond to the following questions from the perspective of a person who has
the preconception that …”. We then described the preconception in brief or
simply inserted “the impetus preconception” for agreed-upon preconception
[33]. For each preconception, at least 7 different response sets (more
responses were generated for preconceptions with higher observed variability
in the responses) were generated. We then analyze the distribution of correct
answers. Furthermore, we prompted ChatGPT for important items to provide
reasoning to their responses, given that it simulated a student with a
different preconception. We validated if the reasoning was in alignment with
the preconception.

##  IV Findings

###  IV.1 RQ1: Probing conceptual understanding of ChatGPT in physics

Regarding RQ1, we first transformed the response matrix into a binary coded
matrix where correct responses were coded with 1 and incorrect responses were
coded with 0. Based on this binary matrix we calculated the average correct
answers (of overall $N$=14 responses) and the standard deviation. We found
that $M$=0.83 ($SD$=0.22), i.e., ChatGPT scored on average 83% in the FCI
(with some transcribed items). In Table 1 the distributional statistics for
all items with a mean score below 1.0 are displayed. It can be verified that
for many items ChatGPT perfectly solved them. With other items (e.g., A8)
there is quite some variance and solution probability is almost equal to
random guessing. Item A20 was never solved correctly by ChatGPT. The
engineering students ($N$=121) performed on average $M$=0.14 ($SD$=0.20).

Table 1: Item statistics (generated with assistance of ChatGPT). Only items with a mean score below 1.00 are included. Item | Mean | Median | Std.dev. | Variance |  5% |  95%
---|---|---|---|---|---|---
A8 | 0.14 | 0.00 | 0.36 | 0.13 | 0.00 | 1.00
A13 | 0.93 | 1.00 | 0.27 | 0.07 | 0.65 | 1.00
A14 | 0.79 | 1.00 | 0.43 | 0.18 | 0.00 | 1.00
A16 | 0.50 | 0.50 | 0.52 | 0.27 | 0.00 | 1.00
A17 | 0.71 | 1.00 | 0.47 | 0.22 | 0.00 | 1.00
A18 | 0.64 | 1.00 | 0.50 | 0.25 | 0.00 | 1.00
A19 | 0.36 | 0.00 | 0.50 | 0.25 | 0.00 | 1.00
A20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
A21 | 0.57 | 1.00 | 0.51 | 0.26 | 0.00 | 1.00
A29 | 0.57 | 1.00 | 0.51 | 0.26 | 0.00 | 1.00
A30 | 0.71 | 1.00 | 0.47 | 0.22 | 0.00 | 1.00

###  IV.2 RQ2: Simulating conceptual understanding of different cohorts

Next, we prompted ChatGPT to answer the FCI as if it were a) a 10th grade
student in school or b) an engineering student. In Table 2, the means and
standard deviations for the distributions of the groups are depicted. In a
reasonable approximation, we can verify that the scales (i.e., standard
deviations) are similar for the groups, so we meet the assumptions for the
Kruskal-Wallis test. This can also be visually inspected in Figure 1, where
the distributions of correct responses for the different cohorts are depicted.
The Kruskal-Wallis test yields a value for potential group differences of
$H$=1.79, $p$=0.409, $\eta^{2}$=-0.01. Essentially, no differences were found
between the aggregated correct response distributions for the three cohorts.

Figure 1: Distribution of correct responses for different cohorts as prompted and generated by ChatGPT. Table 2: Descriptive statistics for groups. | Mean | Median | Std | Variance | 5% | 95%
---|---|---|---|---|---|---
Group |  |  |  |  |  |
Engineers | 24.28 | 25.0 | 1.02 | 1.04 | 22.0 | 25.0
Unspecified | 24.93 | 25.0 | 1.27 | 1.61 | 23.0 | 27.0
Students | 24.25 | 24.5 | 1.58 | 2.50 | 22.0 | 26.0

###  IV.3 RQ3: Simulating students’ preconceptions with ChatGPT

Figures 2 and 3 show the distributions for the simulated students with
different preconceptions. It can be seen that they widely spread, and
depending on the specific preconceptions more or less accuracy can be observed
in the response sets. Interestingly, students with the AR preconception almost
systematically fail at all items, whereas simulated students with CF solve the
FCI as accurately as without preconceptions 3). Certainly, a much greater
variability as in randomly generated responses can be observed.

Figure 2: Distribution of correct responses for different preconceptions as
prompted and generated by ChatGPT. Figure 3: Distribution of correct responses
for different cohorts as prompted and generated by ChatGPT.

Finally, we qualitatively validated two preconceptions with regards to the
provided reasoning by ChatGPT. As for their intricate and prevalent nature, we
chose centrifugal force (CF) and impetus (I). The centrifugal force is not a
real force as for example gravity, but rather a pseudo force that results from
the chosen frame of reference. In the inertial frame of reference, only the
centripetal force acts on the object attached to the string that keeps it on
the circular track. Once the string is cut, no force (except for gravity and
normal force) is acting on the moving object, which, according to Newton’s
first law keeps on moving with constant velocity (assuming frictionless
movement). The centrifugal force during a circular motion in a stationary
frame of reference (also termed reactive centrifugal force, e.g., in a car)
feels very real for students as they experience this force [46]. Centrifugal
force is assumed in a center of mass frame of reference that moves alongside
the object in the circular track. No net force should act on the object, as in
this frame of reference it is at rest. Students with this preconception would
choose answer 4, i.e., after the string is plugged, the outward force
(centrifugal force) will drive the object outwards. ChatGPT reasons this case
convincingly and correctly. It is noteworthy that no mention of the frame of
reference problem is made.

> Prompt, (see Section III): Responses by ChatGPT (translated by ChatGPT):
> First response: From the perspective of a person who incorrectly believes
> that a centrifugal force exists, they may choose Answer 4: “A trajectory
> that initially goes from the center outwards, but then takes a trajectory in
> the direction of the original movement.” The misconception of centrifugal
> force is based on the feeling that a force is acting during circular motion
> that pushes the body outwards. In reality, it is the centripetal force (a
> force directed inwards) that keeps the body on its circular path. This
> misconception could lead a person to believe that when the string breaks,
> the ball is initially driven outwards from the center (by the supposed
> centrifugal force) before then moving in the direction of the original
> motion because there is no longer any centrifugal force to push it outwards.
> The correct answer would, however, be Answer 2: “A linear trajectory that
> shows tangential along the circle at the breaking point.” Because when the
> string breaks, there is no longer a centripetal force that forces the ball
> into a circular path, and the ball continues its motion in a straight line,
> which is tangential to the original circular path, due to inertia. This
> motion corresponds to Newton’s first law, which states that a body remains
> at rest or moves with constant velocity in a straight line unless a force
> acts on it.

As for the impetus concept, students typically believe (much in agreement with
their lived experiences in a friction-based world) that when an object is
thrown, something is transferred to the object that it can use to fly a
certain time, constantly using up the impetus. Once it is used up, the objects
comes to rest. We would expect students to choose answer 1 for this item.
After all, a circular impetus is given to the object (e.g., by the string).
Once the string is plugged, this circular impetus will cause the object to
continue on the circular path. Again, ChatGPT correctly reasoned about this
preconception and chose the correct answer, responding as-if it were a student
with the impetus preconception.

> Prompt, (see Section III): Responses by ChatGPT (translated by ChatGPT):
> Second response: The theory of impetus is a medieval physical theory that
> was proposed before Newton’s laws of motion. This theory suggests that a
> moving object possesses an “impetus” that drives it further in the direction
> of the original motion, even when no external force acts on it anymore. It
> is believed that the impetus gradually decreases, and the object eventually
> stops. With this misconception, a person might choose Answer 1: “A
> trajectory that closely follows the course of the circular path.” This
> person might believe that after the string breaks, the ball continues to fly
> in the original circular motion because the impetus drives it in that
> direction. The ball would then gradually deviate from the circular path and
> finally stop when the impetus has completely diminished. However, the
> correct answer according to modern laws of physics would be Answer 2: “A
> straight-line trajectory that is tangential to the circle at the breaking
> point.” According to Newton’s first law of motion, in the absence of an
> external force (which was removed by the breaking string), the ball would
> continue its motion in a straight line, which is tangential to the circular
> path it was originally on.

##  V Discussion

The recent advances in AI research, in particular generative AI, provide novel
potentials for physics education researchers to enhance research and teaching,
however, they also introduce novel challenges. To better understand the
potentials and challenges, it is crucial to engage with the advanced
technologies, such as LLM, to evaluate their performance for physics
education-specific problems (as argued by [47], who called for “more careful
and controlled studies” on this matter, particularly with ChatGPT). In this
study, we evaluated the capabilities of an LLM, namely ChatGPT, to generate
synthetic data for the FCI and thus potentially enhance concept inventory
development. We chose the FCI as a use case as this is a well validated and
most widely used concept inventory in physics. Prior studies established that
ChatGPT is well versed to solve the FCI. In our study, we advanced this
research to compare performance with real students (RQ1), and generate
synthetic data for it with regards to different cohorts (RQ2), and varying
preconceptions (RQ3).

In RQ1 we found that ChatGPT (based on GPT 4) solved the FCI (with transcribed
images) accurately. In fact, it solved the FCI much more accurately compared
to real engineering students in a German university. This resonates with prior
studies that established that ChatGPT can accurately solve the FCI [33, 34].
Compared to the prior work by West, where ChatGPT (based on GPT-4) solved 22
out of 23 items correctly, here ChatGPT (based on GPT-4) only correctly solves
about 26 out of 30 items on average. There were eleven items that have not
been solved correctly in each run (see Tab. 1). Six of these eleven items
contained a figure that was translated into text. In comparison, four items
that contained text translated from a figure were solved correctly in each
run. Consequently, the translation from figures to text may cause difficulties
for ChatGPT to interpret. Previously, some of these items were also taken out
from consideration [34]. As a note of interest, ChatGPT can be easily prompted
to output the responses in varying formats (e.g., comma-separated list), and
it solves it much more quickly than real students. We particularly generated
more response sets in order to also retrieve information on the variance
displayed in responding to the FCI. For most items, no variance was found,
because they were solved with 100% accuracy. For other items, however, a mere
random performance was observed.

In RQ2 we sought to utilize ChatGPT (via specific prompting) to simulate
answers from students in different cohorts (10th grade students, and
engineering students). To our surprise, we found no differences between these
cohorts and the unprompted (with regards to cohort) responses. Either the mean
accuracy was similar, and the standard deviation was similar. This struck us
as unexpected, given that it is established that prompting can in fact be used
to modify the responses of ChatGPT and other generative AI-based LLM [48].
However, maybe the provided prompts were not specific enough to affect the
responses in ChatGPT.

Therefore, in RQ3, we specifically prompted ChatGPT to respond to the FCI as-
if it were a student with a certain preconception related to Newtonian force
concept and mechanics more generally. Here, we found large differences in the
response behaviour of ChatGPT. ChatGPT knew about all preconceptions. The
accuracy, given that ChatGPT responses as-if it were a student with almost any
preconception, dropped considerably, sometimes even below random guessing
performance. Interestingly, prompting ChatGPT with preconceptions resulted in
distributions that were equally spread as real students’ distributions. For
two preconceptions we verified that ChatGPT also correctly reasoned if it was
prompted to respond to the items as-if it had a certain preconception, which
it did.

These data augmentation techniques could be of interest to physics education
researchers and educational researchers more generally if they want to pilot
test their instrument, e.g., in the domain of mechanics. Even practitioners
designing classroom exams and policy agents designing state-wide exams could
benefit from these capabilities of generative AI (here: LLM) to assure
validity of their instruments. One caveat would be that users currently will
have to outsource their instruments to a private company. On the other hand,
it can be tiresome to students to function as “lab animals” and be subjected
to pilot testing of research instruments (let alone ethical concerns).
Moreover, precious time of the students is consumed which could be used in
more learning-relevant activities.

##  VI Limitations

While LLM such as GPT 4 provide novel capabilities for research such as data
augmentation, they also introduce novel challenges that researchers have to
engage with. Brown et al. [24] list three: misuse, bias and energy consumption
as important points of concern. For once, LLM such as GPT 4 can be used to
generate human-sounding text and therefore be used to spread certain
narratives. With regards to physics education research, students might
fabricate their essays with ChatGPT, which cannot be easily detected by human
instructors or researchers [43]. In the future, the selection of teaching
materials should be carefully tailored to ensure that tools like ChatGPT are
not undesirably misused as aids. For example, we can observe a similar context
where students increasingly resort to aids such as YouTube videos when the
provided materials exhibit deficiencies [49]. Equally concerning is the
problem of hallucination in LLM such as GPT 4. For example, ChatGPT provides
well-sounding output with sometimes flawed physics [42]. This becomes a
problem when considering that students accept the outputs of Chat-GPT without
critical reflection (copy and paste it) [50]. Instructors need to find means
to prevent tasks where simple copy paste is sufficient. We also found that
important steps in the problem solving process such as explicating on
underlying assumptions and idealizations are oftentimes omitted [51]. This
makes it difficult for students to truly learn the underlying physics.
Kortemeyer [33] prompted ChatGPT on its ability to understand physics and
received the following response:

> “as a language model, I have been trained on a large dataset of text,
> including physics texts. This allows me to understand and generate text
> related to physics concepts, but it does not mean that I have the ability to
> solve physics problems or pass a physics course. I can provide explanations
> and answer questions about physics to the best of my knowledge, but I am not
> a substitute for a human physics expert or a physics education.”

As such, human expert judgements will remain crucially important to verify
responses given by ChatGPT. As a matter of fact, we found the theory-based,
specific prompting with regards to established preconceptions yielded most
variance which could be important for data augmentation. As such, established
theory is crucial to guide prompt engineering and generate meaningful
synthetic data. It was furthermore established that LLM such as GPT 3 and
others incorporate human-like biases, e.g., with regards to gender [52, 24].
Given the fact that they are trained on the Common Crawl of the Internet and
similar unclean data, this is probably not surprising. However, approaches to
mitigate this problem are still not fully developed and thus employing ChatGPT
and other LLM in educational settings should probably be postponed, especially
in contexts of high-stakes testing. Finally, energy consumption of training
these LLM and even requesting a single output are ecologically challenging
[53]. It will likely be not sustainable for every research group to run their
own models. Shared infrastructures will be required to make these applications
sustainable.

##  VII Implications

Our study showed that with adequate prompting strategies ChatGPT can be
utilized to generate synthetic data related to a concept inventory with
meaningful variability in the data set. We submit that similar conclusions
will likely be true for other concept inventories as well, such as climate
change, electricity, and also biological evolution. Further research is needed
to test these predictions. This offers researchers a valuable tool to simulate
field testing their concept inventories and evaluate psychometric properties,
given sufficient sample sizes. We could show that similar distributions as
with real students emerge, and further research could investigate if
convolutions of preconceptions will enable a reconstruction of the true
students’ distribution. We cannot, however, exclude the possibility that real
students also show other response patterns, i.e., have different conceptions
that are not included in our theoretically derived set of preconceptions.
Further research should investigate also the possibility to even generate a
concept inventory in the first place with the help of LLM such as GPT 4 [54],
and assist in other phases and steps of the development process for a concept
inventory [15].

### Data and code availability:

The code for analysis (and further analyses) can be accessed here:
https://colab.research.google.com/drive/1WU_cBfcYMO2dxE9kFVjK9Hlq0mSati8V?usp=sharing

The synthetic data can be accessed upon reasonable request to the authors.

###  VII.1 Translation of figures in FCI items

In this section, we show the translation of figures in FCI items into text so
that ChatGPT was able to interpret it. We did not include the original figures
due to copyright restrictions, but the item numbering is identical to the
original work [6].

Item 5

Which of the following forces act(s) on the ball when it is exactly halfway
through a semicircular channel?

  1. A)

A downward force of gravity

  2. B)

A force exerted by the channel that is directed from the point where the ball
is currently located to the center of the channel

  3. C)

A force in the direction of motion

  4. D)

A force directed from the center point to the point in the channel where the
ball is currently located.

  1. 1.

force A only

  2. 2.

forces A and B

  3. 3.

forces A and C

  4. 4.

forces A, B, and C

  5. 5.

forces A, C, and D

Item 6

Which path would a ball most closely follow after it exits the channel at the
end and moves across the frictionless table top?

  1. 1.

A path that would correspond to the arc of the gutter if it formed a closed
circle.

  2. 2.

A straight line trajectory curve that points tangentially along the circle at
the exit point.

  3. 3.

A trajectory curve that describes an arc of a circle outwards

  4. 4.

A path curve which describes a parabola to the outside

  5. 5.

A trajectory in which the ball is immediately deflected outward in a straight
line

Item 7

A steel ball is attached to a string and is swung in a circular path in a
horizontal plane. At one point, the string suddenly breaks near the ball. If
these events are observed from directly above, which path would the ball most
closely follow after the string breaks?

  1. 1.

A trajectory that almost corresponds to the course of the circular path.

  2. 2.

A straight-line trajectory that points tangentially along the circle at the
tear-off point.

  3. 3.

A straight-line trajectory that points away from the circular path at an angle
of 45° from the tangent line at the break-off point.

  4. 4.

A trajectory that initially goes outward from the center, but then takes a
trajectory in the direction of the original motion.

  5. 5.

A trajectory that points outward in a straight line from the center.

Item 8

Imagine a hockey puck is sliding with constant speed $v_{0}$ in a straight
line from a point ”P” to point ”Q” on a frictionless horizontal surface.
Forces exerted by the air are negligible. You are looking down on the puck.
When the puck reaches point ”Q,” it receives a swift horizontal kick
perpendicular to the previous direction of motion. Had the puck been at rest
at point ”Q,” then the kick would have set the puck in horizontal motion with
a speed $v_{k}$ in the direction of the kick.

Which paths would the puck most closely follow after receiving the kick?

  1. 1.

A straight line in the direction of the collision

  2. 2.

A straight line pointing at a sharp angle to the previous trajectory

  3. 3.

A path in the same direction as the original direction of motion but offset in
parallel in the direction of the impact

  4. 4.

A curved path that slopes in the direction of the shock

  5. 5.

A curved path inclined in the direction of the original direction of motion

Item 12

A ball is fired by a cannon from the top of a cliff in horizontal direction.
Which paths would the cannon ball most closely follow?

  1. 1.

A path that runs in a straight line down to the point of impact.

  2. 2.

A path that is curved toward the point of impact.

  3. 3.

A path which is curved towards the point of impact, but which is horizontal
for a short distance after launch.

  4. 4.

A path that is curved towards the point of impact, but is horizontal for a
certain distance after launch and vertical before impact.

  5. 5.

A path that runs horizontally until almost vertically above the point of
impact and then runs vertically until the point of impact.

Item 14

A bowling ball accidentally falls out of the cargo bay of an airliner as it
flies along in a horizontal direction. As observed by a person standing on the
ground and viewing the plane from the side, which path would the bowling ball
most closely follow after leaving the airplane?

  1. 1.

A curved path that runs backwards.

  2. 2.

A path that runs vertically downward.

  3. 3.

A path that runs forward in a straight line.

  4. 4.

A curved path that runs forward.

  5. 5.

A path that extends forward horizontally to above the point of impact and then
vertically downward.

Items 15, 16 and 17

The figures have not been described as they only carry redundant information
that is also contained in the text.

Item 18 A child swings on a rope tied to a branch (point O). His movement
starts above a point P and then continues downward through the point P.
Consider the following distinct forces:

  1. 1.

A downward force of gravity.

  2. 2.

A force exerted by the rope pointing from A to O.

  3. 3.

A force in the direction of the boy’s motion.

  4. 4.

A force pointing from O to A.

Which of the above forces is (are) acting on the boy when he is at position A?

  1. (A)

1 only.

  2. (B)

1 and 2.

  3. (C)

1 and 3.

  4. (D)

1, 2, and 3.

  5. (E)

1, 3, and 4.

Item 19

Two blocks move to the right. The following table shows the distance covered
by the blocks at intervals of 0.2 s in length units (LU). Identical numbers
indicate identical points in time.

Block a: 1: 2 LU, 2: 4 LU, 3: 7 LU, 4: 11 LU, 5: 16 LU, 6: 22 LU, 7: 29 LU

Block b: 1: 0 LU, 2: 4 LU, 3: 8 LU, 4: 12 LU, 5: 16 LU, 6: 20 LU, 7:24 LU, 8:
28 LU

Do the blocks ever have the same speed?

  1. 1.

No.

  2. 2.

Yes, at instant 2.

  3. 3.

Yes, at instant 5.

  4. 4.

Yes, at instants 2 and 5.

  5. 5.

Yes, at some time during the interval 3 to 4.

Item 20

Two blocks move to the right. The following table shows the distance covered
by the blocks at intervals of 0.2 s in length units (LU). Identical numbers
indicate identical points in time.

Block a: 1: 2 LU, 2: 6 LU, 3: 10 LU, 4: 14 LU, 5: 18 LU, 6: 22 LU, 7 26 LU

Block b: 1: 0 LU, 2: 6 LU, 3: 12 LU, 4: 18 LU, 5: 24 LU

The accelerations of the blocks are related as follows:

  1. (A)

The acceleration of “a” is greater than the acceleration of “b”.

  2. (B)

The acceleration of “a” equals the acceleration of “b”. Both accelerations are
greater than zero.

  3. (C)

The acceleration of “b” is greater than the acceleration of “a”.

  4. (D)

The acceleration of “a” equals the acceleration of “b”. Both accelerations are
zero.

  5. (E)

Not enough information is given to answer the question.

Item 21

A rocket drifts sideways in outer space from point ”a” to point “b” as shown
below. The rocket is subject to no outside forces. Starting at position “b”,
the rocket’s engine is turned on and produces a constant thrust (force on the
rocket) at right angles to the line “ab”. The constant thrust is maintained
until the rocket reaches a point “c” in space.

Which path below best represents the path of the rocket between points “b” and
”c”?

  1. 1.

Point “c” lies to the right above point “b”. The path from “b” to ”c” is
concave.

  2. 2.

Point “c” is right above point “b”. The path from “b” to “c” is straight and
perpendicular to the line connecting “b” to “a”.

  3. 3.

Point “c” lies to the right above point “b”. The path from “b” to “c” is
straight.

  4. 4.

Point “c” lies above and to the right of point “b”. The path curve from “b” to
“c” runs horizontally to the right at the beginning and is convex from then
on.

  5. 5.

Point “c” lies above and to the right of point “b”. The path curve from “b” to
“c” is convex.

Item 23

At point “c” the rocket’s engine is turned off and the thrust immediately
drops to zero.

Which of the paths below will the rocket follow beyond point “c”?

  1. 1.

The path curve runs in a straight line horizontally to the right.

  2. 2.

The path curve runs in a straight line diagonally upwards to the right.

  3. 3.

The path curve runs vertically upwards in a straight line.

  4. 4.

The path curve runs to the right and is concave.

  5. 5.

The path curve runs to the right and is convex.

Item 28 The figure has not been described as it only carries redundant
information that is also contained in the text.

## References

  * Kasneci  _et al._ [2023] E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, _et al._ , Chatgpt for good? on opportunities and challenges of large language models for education, Learning and Individual Differences 103, 102274 (2023).
  * Van De Schoot  _et al._ [2021] R. Van De Schoot, J. De Bruin, R. Schram, P. Zahedi, J. De Boer, F. Weijdema, B. Kramer, M. Huijts, M. Hoogerwerf, G. Ferdinands, _et al._ , An open source machine learning framework for efficient and transparent systematic reviews, Nature machine intelligence 3, 125 (2021).
  * Kuzman  _et al._ [2023] T. Kuzman, I. Mozetic, and N. Ljubešic, Chatgpt: Beginning of an end of manual linguistic data annotation? use case of automatic genre identification, ArXiv, abs/2303.03953 (2023).
  * Surameery and Shakor [2023] N. M. S. Surameery and M. Y. Shakor, Use chat gpt to solve programming bugs, International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290 3, 17 (2023).
  * Sinha  _et al._ [2023] R. K. Sinha, A. Deb Roy, N. Kumar, and H. Mondal, Applicability of chatgpt in assisting to solve higher order problems in pathology, Cureus 15, e35237 (2023).
  * Hestenes  _et al._ [1992] D. Hestenes, M. Wells, and G. Swackhamer, Force concept inventory, The physics teacher 30, 141 (1992).
  * Koponen and Pehkonen [2010] I. T. Koponen and M. Pehkonen, Coherent knowledge structures of physics represented as concept networks in teacher education, Science & Education 19, 259 (2010).
  * Wulff [2023] P. Wulff, Network analysis of terms in the natural sciences insights from wikipedia through natural language processing and network analysis, Education and Information Technologies 10.1007/s10639-022-11531-5 (2023).
  * Liu [2010] X. Liu,  _Using and Developing Measurement Instruments in Science Education: A Rasch Modeling Approach_, Science & Engineering Education Sources (Information Age Publishing Incorporated, Charlotte, NC, 2010).
  * Porter  _et al._ [2014] L. Porter, C. Taylor, and K. C. Webb, Leveraging open source principles for flexible concept inventory development, in  _Proceedings of the 2014 conference on Innovation & technology in computer science education_, ACM Digital Library, edited by Å. Cajander, M. Daniels, T. Clear, and A. Pears (ACM, New York, NY, 2014) pp. 243–248.
  * Nehm and Härtig [2012] R. H. Nehm and H. Härtig, Human vs. computer diagnosis of students’ natural selection knowledge: Testing the efficacy of text analytic software, Journal of Science Education and Technology 21, 56 (2012).
  * Schubatzky  _et al._ [2023] T. Schubatzky, R. Wackermann, C. Wöhlke, C. Haagen-Schützenhöfer, M. Jedamski, H. Lindemann, and K. Cardinal, Entwicklung des concept-inventory ccci-422 zu den naturwissenschaftlichen grundlagen des klimawandels, Zeitschrift für Didaktik der Naturwissenschaften 29, 10.1007/s40573-023-00159-8 (2023).
  * Hake [1998] R. R. Hake, Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, American Journal of Physics 66, 64 (1998).
  * Eaton [2021] P. Eaton, Evidence of measurement invariance across gender for the force concept inventory, Phys Rev Spec Top Phys Edu Res (2021).
  * Adams and Wieman [2011] W. K. Adams and C. E. Wieman, Development and validation of instruments to measure learning of expert–like thinking, International Journal of Science Education 33, 1289 (2011).
  * Bengio  _et al._ [2003] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, A neural probabilistic language model, Journal of Machine Learning Research 3, 1137 (2003).
  * Vaswani  _et al._ [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need: Conference on neural information processing systems, Advances in Neural Information Processing Systems , 6000 (2017).
  * Devlin  _et al._ [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv 1810.04805 (2018).
  * Manning [2022] C. D. Manning, Human language understanding & reasoning, Daedalus 151, 127 (2022).
  * Kaplan  _et al._ [2020] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, arXiv (2020).
  * Wolfram [2023] S. Wolfram,  _What is ChatGPT doing and why does it work?_ (Wolfram Media, 2023).
  * Note [1] More precisely, tokens are used instead of words, which are more general sequences of letters that can be added to form words. However, they have the advantage that also unseen words in the training data can be processed.
  * Note [2] Percentages are taken from %****␣main_PRPER.bbl␣Line␣325␣****https://en.wikipedia.org/wiki/GPT-3#GPT-3.5 (last access 26 May 2023) for training GPT 3.
  * Brown  _et al._ [2020] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, Language models are few-shot learners, arXiv (2020).
  * Rosenbaum [2022] R. Rosenbaum, On the relationship between predictive coding and backpropagation, PloS one 17, e0266102 (2022).
  * Bommasani  _et al._ [2022] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. v. Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang, On the opportunities and risks of foundation models, arXiv (2022).
  * Bender  _et al._ [2021] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, On the dangers of stochastic parrots, FAccT , 610 (2021).
  * Lewkowycz  _et al._ [2022] A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra, Solving quantitative reasoning problems with language models, arXiv (2022).
  * Kojima  _et al._ [2022] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, Large language models are zero-shot reasoners, Advances in neural information processing systems 35, 22199 (2022).
  * Huang and Chang [2023] J. Huang and K. C.-C. Chang, Towards reasoning in large language models: A survey, arXiv (2023).
  * Bowman [2023] S. R. Bowman, Eight things to know about large language models, arXiv (2023).
  * Katz  _et al._ [2023] D. M. Katz, M. J. Bommarito, S. Gao, and P. D. Arredondo, Gpt-4 passes the bar exam, SSRN 10.2139/ssrn.4389233 (2023).
  * Kortemeyer [2023] G. Kortemeyer, Could an artificial-intelligence agent pass an introductory physics course?, Physical Review Physics Education Research 19, 15 (2023).
  * West [2023a] C. G. West, Ai and the fci: Can chatgpt project an understanding of introductory physics?, arXiv preprint arXiv:2303.01067 (2023a).
  * Iwana and Uchida [2021] B. K. Iwana and S. Uchida, An empirical survey of data augmentation for time series classification with neural networks, Plos one 16, e0254841 (2021).
  * Van Dyk and Meng [2001] D. A. Van Dyk and X.-L. Meng, The art of data augmentation, Journal of Computational and Graphical Statistics 10, 1 (2001).
  * Shorten and Khoshgoftaar [2019] C. Shorten and T. M. Khoshgoftaar, A survey on image data augmentation for deep learning, Journal of big data 6, 1 (2019).
  * Hammer [2000] D. Hammer, Student resources for learning introductory physics, American Journal of Physics 68, S52 (2000).
  * diSessa [1993] A. diSessa, Toward an epistemology of physics, Cognition and Instruction 10, 105 (1993).
  * Note [3] Chat can be accessed here: https://chat.openai.com/share/8ae109d2-8b71-4033-8c62-749c33c2582c, last access: June 2023; translated with: https://www.deepl.com/de/translator##de/en/).
  * West [2023b] C. G. West, Advances in apparent conceptual physics reasoning in chatgpt-4, arXiv preprint arXiv:2303.17012 (2023b).
  * Gregorcic and Pendrill [2023] B. Gregorcic and A.-M. Pendrill, Chatgpt and the frustrated socrates, Physics Education 58, 035021 (2023).
  * Yeadon  _et al._ [2023] W. Yeadon, O.-O. Inyang, A. Mizouri, A. Peach, and C. P. Testrow, The death of the short-form physics essay in the coming ai revolution, Physics Education 58, 035027 (2023).
  * Kruskal and Wallis [1952] W. H. Kruskal and W. A. Wallis, Use of ranks in one-criterion variance analysis, Journal of the American Statistical Association 47, 583 (1952).
  * Cohen [1988] J. Cohen,  _Statistical power analysis for the behavioral sciences_ (Erlbaum, Hillsdale, NJ, 1988).
  * Küchemann  _et al._ [2020] S. Küchemann, P. Klein, H. Fouckhardt, S. Gröber, and J. Kuhn, Students’ understanding of non-inertial frames of reference, Physical Review Physics Education Research 16, 010112 (2020).
  * Wang [2023] J. Wang, Chatgpt: A test drive, American Journal of Physics 91, 255 (2023).
  * Liu  _et al._ [2021] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, arXiv (2021).
  * Ruggieri [2020] C. Ruggieri, Students’ use and perception of textbooks and online resources in introductory physics, Phys. Rev. Phys. Educ. Res. 16, 020123 (2020).
  * Krupp  _et al._ [2023] L. Krupp, S. Steinert, M. Kiefer-Emmanouilidis, K. Avila, P. Lukowicz, J. Kuhn, S. Kuechemann, and J. Karolus, Unreflected acceptance – investigating the negative consequences of chatgpt-assisted problem solving in physics education (2023).
  * Kieser and Wulff [ress] F. Kieser and P. Wulff, Using large language models to probe cognitive constructs, augment data, and design instructional materials, in  _Machine Learning in Educational Sciences: Approaches, Applications and Advances_ , edited by M. S. Khine (Springer Nature, in press).
  * Caliskan  _et al._ [2017] A. Caliskan, J. J. Bryson, and A. Narayanan, Semantics derived automatically from language corpora contain human-like biases, Science (New York, N.Y.) 356, 183 (2017).
  * Dodge  _et al._ [2022] J. Dodge, T. Prewitt, R. T. Des Combes, E. Odmark, R. Schwartz, E. Strubell, A. S. Luccioni, N. A. Smith, N. DeCario, and W. Buchanan, Measuring the carbon intensity of ai in cloud instances: Facct ’22, arXiv (2022).
  * Küchemann  _et al._ [2023] S. Küchemann, S. Steinert, N. Revenga, M. Schweinberger, Y. Dinc, K. E. Avila, and J. Kuhn, Can chatgpt support prospective teachers in physics task development?, Phys. Rev. Phys. Educ. Res. 19, 020128 (2023).