# Ultrasensitive optical magnetometry at the microscale

S. Forstner  School of Mathematics and Physics, University of Queensland, St
Lucia, Queensland 4072, Australia  E. Sheridan  School of Mathematics and
Physics, University of Queensland, St Lucia, Queensland 4072, Australia  J.
Knittel  School of Mathematics and Physics, University of Queensland, St
Lucia, Queensland 4072, Australia  Centre for Engineered Quantum Systems,
University of Queensland, St Lucia, Brisbane, QLD 4072, Australia  C. L.
Humphreys  School of Mathematics and Physics, University of Queensland, St
Lucia, Queensland 4072, Australia  G. A. Brawley  School of Mathematics and
Physics, University of Queensland, St Lucia, Queensland 4072, Australia  H.
Rubinsztein-Dunlop  School of Mathematics and Physics, University of
Queensland, St Lucia, Queensland 4072, Australia  Centre for Engineered
Quantum Systems, University of Queensland, St Lucia, Brisbane, QLD 4072,
Australia  W. P. Bowen  School of Mathematics and Physics, University of
Queensland, St Lucia, Queensland 4072, Australia  Centre for Engineered
Quantum Systems, University of Queensland, St Lucia, Brisbane, QLD 4072,
Australia

Recent advances in optical magnetometry have achieved record sensitivity at
both macro- and nano-scale. Combined with high bandwidth and non-cryogenic
operation, this has enabled many applications. By comparison, microscale
optical magnetometers have been constrained to sensitivities five orders-of-
magnitude worse than the state-of-the-art. Here, we report an ambient optical
micro-magnetometer operating for the first time in the picoTesla range, a more
than three order-of-magnitude advance on previous results. Unlike other
ultrasensitive optical magnetometers, the device operates at earth field,
achieves tens of MHz bandwidth, and is integrated and fiber coupled. Combined
with 60 $\mu$m spatial resolution and microWatt optical power requirements,
these unique capabilities open up a broad range of applications including
cryogen-free and microfluidic magnetic resonance imaging, and electromagnetic
interference-free investigation of spin physics in condensed matter systems
such as semiconductors and ultracold atom clouds.

Cryogenic SQUIDs are the favoured magnetometer for most high precision
applications, combining excellent sensitivity with reliability. However,
optical magnetometers are rapidly becoming an attractive alternative due to
their non-cryogenic operation, ultrahigh sensitivity, low power requirements,
and potential for high measurement bandwidth1. At milli-meter scale, atomic
ensemble based magnetometers have achieved record sensitivities down to 160 aT
Hz-1/2 2; while NV (nitrogen vacancy) diamond magnetometers provide the
benchmark at nanoscale, with sensitivity in the range of several nT Hz-1/2 3.
However, many applications in areas such as medical imaging4, defense5, solid
state research6 and space science7, benefit from spatial resolution in the
intermediate microscale. At this scale, optical magnetometers have generally
operated with sensitivity far worse than the state-of-the-art. Significant
efforts have been made to miniaturize and microfabricate atomic vapour cell
based magnetometers over the past decade. However, the smallest such devices
remain constrained to 0.5 mm dimensions1. The only ultraprecise optical micro-
magnetometers reported to date rely on ultracold atoms trapped in optical
traps within bulky and restrictive ultrahigh vacuum systems8; 9.

Figure 1: Fabrication process and experimental schematic. a, Schematic
representation (Side-view) of the magnetometer fabrication steps. b, Schematic
of experimental set-up. FPC: Fiber polarisation controller. Inset: 3D image
showing internal structure of sensor.

In this article we present a microscale cavity optomechanical magnetometer
which uniquely combines picoTesla sensitivity, operation in ambient
conditions, bandwidth in the tens of MHz, and the ability to operate at and
above earth field. Furthermore, these results are achieved in an integrated
all-optical ultralow power device. A peak sensitivity of 200 pT Hz-1/2 is
achieved, three orders of magnitude better than the previous best ambient
optical magnetometer of similar size, with nonlinear frequency mixing
providing nanoTesla sensitivity down to frequencies as low as 2 Hz, limited by
the bandwidth of the measurement electronics rather than the sensor itself.
The six-order-of-magnitude dynamic range achieved is superior to the best NV3
and miniature SQUID magnetometers10; and significantly, exceeds the earth’s
field so that unshielded sensing can be performed. While the 40 MHz bandwidth
surpasses the best previous optical micro-magnetometer by a factor of twenty
five11, and miniature-SQUIDs10 by three orders of magnitude.

Cavity optomechanical magnetometers were proposed recently12. The key
principle is to combine microscale cavity optomechanics with magnetostrictive
magnetometry. The combination of mechanical and optical resonances present in
cavity optomechanical systems provide enhanced mechanical response to applied
forces and optical readout with attometer precision13; 14. In the presence of
a magnetic field, the magnetostrictive material expands, exerting a measurable
force upon the cavity optomechanical system. A proof-of-principle-experiment
in Ref. 12 demonstrated 400 nT Hz-1/2 sensitivity using a grain of
magnetostrictive material bonded via epoxy to the surface of a conventional
microtoroidal cavity optomechanical system. However, the bandwidth and
sensitivity were both critically constrained by poor coupling of the
magnetostrictive expansion to the mechanical resonances of the device.
Furthermore, the sensor was unable to function in the Hz-kHz regime crucial to
many applications, due to low frequency technical noise. Here, these
constraints are overcome through the development of a new fabrication process
allowing the magnetostrictive material to be embedded directly within the
microtoroid structure, and by harnessing nonlinearities inherent in the
magnetostrictive material to mix low frequency signals up to high frequency.

Fig. 1a shows the fabrication process. A ring of photoresist is patterned on
the top surface of a silicon wafer with a thermally grown silica top-layer (2
$\mu$m thick) using standard photolithographic methods. A buffered oxide (HF)
etch is used to transfer the pattern into the silica layer. A xenon difluoride
(XeF2) etch undercuts the silica disk and etches a void in the supporting
silicon pedestal. The silica disk is reflown using a CO2 laser to produce a
silica torus. Finally an appropriately sized piece of the magnetostrictive
material Terfenol-D is positioned and epoxy bonded inside the central void.
The final devices exhibit typical optical quality factors exceeding $10^{6}$,
with multiple relatively broad mechanical modes ($Q\approx 40$) providing
broadband mechanical response over the frequency range from 5 to 40 MHz. This
allows the fundamental thermal noise limit to micromechanical magnetometry to
be reached, a limit typically precluded by several orders of magnitude in
magnetostrictive magnetometers that rely on electrical read-out15.

The sensitivity of the final magnetometer is characterized using the
experimental setup shown in Fig.1b. For linear measurements of the high
frequency sensitivity, the Terfenol-D is magnetized using a permanent bias
magnet to maximize the linear magnetostrictive response16. A pair of solenoids
is then used to create a spatially uniform signal magnetic field across the
device at radio frequencies. Light from a 1550 nm shot noise limited tunable
fiber laser is guided through a polarization controller and evanescently
coupled into the microtoroidal optical cavity via a tapered optical fiber. The
laser frequency is thermally locked to the half maximum of an optical
resonance. Strain in the resonator induced by the magnetostrictive material
shifts the optical resonance frequency, thus modulating the amplitude of the
transmitted light. This transmitted field is detected on an InGaAs photodiode,
with only 50 $\mu$W of off resonant light required at the detector to achieve
good signal-to-noise. Similar to Ref. 12, network/spectral analysis allow the
magnetic field sensitivity to be determined as a function of signal frequency.

Figure 2: High frequency sensitivity. Sensitivity as a function of frequency
in linear mode of operation, resolution bandwidth was 10 kHz. Inset: Sensor’s
response as a function of signal field strength at a resolution bandwidth of
100 Hz. The data was normalized by dividing by the noise floor. Dashed line:
Earth’s average magnetic field.

The measured magnetic field sensitivity of the device is shown in Fig. 2 over
a range from 1 to 40 MHz. As can be seen, the sensitivity exceeds 10 nT Hz-1/2
over the majority of the range, with a peak sensitivity of 200 pT Hz-1/2
achieved at frequencies near 20 MHz, where the dominant mechanical modes are
predominantly radial in nature and thus strongly coupled to the optical
resonance frequency. This peak sensitivity outperforms previously reported
ambient optical magnetometers12 and electrically read-out magnetostrictive
magnetometers15 of comparable size by more than three orders-of-magnitude. The
detection bandwidth of 1 to 40 MHz shown in Fig. 2 is a 25-fold improvement on
that achieved with the best previous NV magnetometer 11, and four orders of
magnitude greater than the most sensitive atomic magnetometers 17; 2. This
high detection bandwidth is highly advantageous for applications such as
microfluidic magnetic resonance imaging (MRI), where it enables observation of
higher frequency spin-precession, and therefore improved sensitivity and
resolution18.

Figure 3: Response in nonlinear mode of operation. a, and b, Spectra of
sensor’s response centred on the mixup frequency of 5.22 MHz under application
of low frequency signals at 500 Hz with 1 $\mu$T and 22 $\mu$T amplitude,
respectively. The resolution bandwidth was 10 Hz. c, and e, Pseudocolor plots
of the measured and modelled sensor response spectrum as a function of signal
strength and detuning from 5.22 MHz. Note: in these plots the strong linear
response to the RF drive at frequencies close to zero detuning has been
excluded. d, Basic structure of magnetic domain pairs used to obtain simple
microscopic model of mix-up process (see Supplementary Information for
details). Model parameters: $\gamma/2\pi=160$ kHz, $\omega_{m}/2\pi=5.2$ MHz ,
$\chi=1.5\times 10^{21}$ s-2 m-1, $c_{0}=3.1\times 10^{7}$ m s-2 T-1,
$c_{1}=3.3\times 10^{18}$ s-2 T-1, $c_{2}=8.2\times 10^{25}$ s-2 T-1 m-1, and
$B_{\rm RF}=17~{}\mu$T.

The unique combination of high sensitivity, high bandwidth, and
electromagnetic interference-free operation in a microscale device, could
facilitate practical applications in areas such as imaging19, material and
circuit characterization20, and medical diagnosis21. It may, further, enable a
range of applications in fundamental science. For instance, there is much
interest in condensed matter physics in the ability to detect single nuclear
spins, and the interaction of pairs or ensembles of spins in semiconductor
systems22; 23. Our devices may allow investigations of such physics with
increased bandwidth and at a new range of frequencies. Similarly, strong spin-
spin interactions in clouds of ultracold atoms allow the formation of spinor
condensates, providing a unique new tool to study the behaviour of quantum
gases24. However, direct laser based interrogation of the spin state is
fundamentally limited by spontaneous emission25. Since spinor condensates can
exhibit MHz magnetic fields with strength in the range of hundreds picoTesla,
our magnetometers provide a route to overcome this limit. Furthermore, since
the size of our devices is comparable to a neuron, they may provide a new
interference-free approach to mapping neuron magnetic fields26; 27, or indeed
the magnetic fields of other comparably sized biological structures.

The dynamic range of the sensor was tested by varying the amplitude of the
signal field over several orders of magnitude. As shown in Fig. 2 (inset), the
response is linear over the full measurement range from 200 pT Hz-1/2 to 150
$\mu$T Hz-1/2, well above earth field and limited by the range of the testing
coils rather than the sensor itself. To our knowledge, no previous
ultrasensitive magnetometer of similar size has been capable of operating
above earth field3; 11; 2.

Many significant applications require sensitivity to magnetic fields in the Hz
to kHz frequency regime, including magnetic anomaly detection (MAD)28,
geophysical surveys29 and magnetoencephalography (MEG)30. This regime is
precluded in the linear mode of operation of our device due to low frequency
technical noise from vibrations, temperature fluctuations, and electronic
noise. Furthermore, the mechanical resonance frequencies of micrometer-scale
structures, such as microtoroids, naturally lie in the MHz-range such that the
mechanical response to the magnetostrictive force is intrinsically weaker at
low frequencies. However, the magnetic domain structure of magnetostrictive
materials make them intrinsically highly non-linear16. This provides a
mechanism through which to mix low frequency magnetic signals up to higher
frequencies, and also has the effect of suppressing ambient low-frequency
noise sources31. Essentially, changes in the magnetic domain structure of the
material due to low frequency fields alter the response of the material to an
applied high frequency field31. To explore such phenomena in cavity
optomechanical magnetometers, a strong RF and weak Hz-kHz magnetic field are
simultaneously applied to the device and the response analysed. An RF source
is used to produce an RF signal at frequency $\Omega$, while the network
analyser produces a second RF signal at frequency $\Omega+\omega$, where
$\omega$ is swept in frequency over a Hz-kHz range. The signal at $\Omega$ is
split on an RF power splitter and applied to both a pair of air solenoids to
generate the strong RF magnetic field, and to a mixer along with the signal at
frequency $\Omega+\omega$, producing a low frequency signal at frequency
$\omega$. The low frequency signal is applied to a second set of independent
air solenoids to produce the low frequency magnetic field. Any nonlinear
mixing between the low and high frequency fields caused by the nonlinear
magnetostrictive response can be analysed, similarly to the case of linear
operation, using spectral and network analysis. Note that electronic filtering
was used to ensure that the detected signal did not arise from the electronic
mix-up process, this was confirmed with control experiments where the magnetic
field was verified using a pickup coil and a magnetoresistive sensor.

Figure 4: Low frequency sensitivity. a, Low frequency sensitivity achieved by
nonlinear mixup as a function of signal frequency. At frequencies above 500 Hz
the sensitivity was determined using a combination of network and spectrum
analysis at 10 Hz resolution bandwidth. At lower frequencies, the network
response was determined individually from each power spectrum for each data
point, at a resolution bandwidth of 1 Hz. The error bar shown for a single low
frequency data-point is representative of the uncertainty of all points. b,
Mixup response as a function of low frequency signal field strength at a
resolution bandwidth of 10 Hz. The data was normalized by dividing by the
noise floor.

Fig. 3a shows a typical optically detected power spectrum upon application of
a strong RF drive field at $5.221$ MHz and a 1 $\mu$T signal field at 500 Hz.
A pair of sidebands are evident, spaced 500 Hz either side of the $5.221$ MHz
drive. This demonstrates that low frequency fields can be mixed-up to high
frequencies in our architecture. Interestingly, as the strength of the low
frequency field is increased above about 2 $\mu$T, a frequency comb of higher
order sidebands appears, as shown in Fig. 3b-c. To our knowledge, such a
frequency comb has not previously been reported in magnetostrictive
magnetometers. To understand the appearance of this comb we developed a simple
microscopic model of the interaction of a pair of domains within the
Terfenol-D with the applied magnetic field, as illustrated in Fig. 3d and
discussed in detail in the supplementary information32. This microscopic model
results in an equation of motion for the dynamics of the domains, neglecting
domain-domain interactions, given by

$\ddot{x}+\gamma\dot{x}+\omega_{m}^{2}x+\chi x^{2}=(c_{0}-c_{1}x-c_{2}x^{2})B$
(1)

where $x$ is the offset of the length of the domain pair away from
equilibrium, $\gamma$ is the dissipation rate of mechanical energy,
$\omega_{m}$ is the mechanical resonance frequency due to the elasticity of
the material, and $B=B_{\rm sig}\cos\omega t+B_{\rm RF}\cos\Omega t$ is the
applied magnetic field with $B_{\rm sig}$ and $B_{\rm RF}$ being the amplitude
of the low frequency signal field and radio frequency drive fields,
respectively. The right-hand-side describes the forcing due to the magnetic
field, and exhibits a linear term with coefficient $c_{0}$, responsible for
our linear magnetometry results, and higher order nonlinear terms with
coefficients $c_{1}$ and $c_{2}$. These higher order terms cause nonlinear
mixing between magnetic fields at different frequencies, and have been used
previously in magnetostrictive magnetometers to achieve low frequency
sensitivity31. Notably, however, our microscopic model predicts a second new
form of nonlinearity, depending purely on the mechanical motion of the
domains, with nonlinear co-efficient $\chi$ (see Eq. (1)). It is this second
nonlinearity that results in the frequency comb evident in our data. Using Eq.
(1) to model the dynamics of our device yields the frequency response given in
Fig. 3e, which shows excellent qualitative agreement with experiment (Fig.
3c).

The sensitivity of the nonlinear mode of operation was determined in a similar
way as that for the linear mode of operation, but this time using the
permanent bias magnet to maximize the nonlinear response of the device. Fig.
4a shows the achieved sensitivity at frequencies between 2 Hz and 20 kHz, with
a peak sensitivity of 150 nT Hz-1/2 over the range 2 Hz to 1 kHz. The
sensitivity deteriorates at higher frequencies, probably constrained by the
bandwidth of the mechanical nonlinearity. Although not shown in the figure,
the sensitivity also deteriorates at frequencies below 0.5 Hz, where we
observe discrete steps in the magnetometer response attributed to Barkhausen
noise from random domain flips in the Terfenol-D. Fig.4b shows the nonlinear
response as a function of the strength of the applied signal field, for a
signal frequency of 1 kHz. As can be seen, it is linear up to a threshold
field strength of 1.7 $\mu$T, above which it drops precipitously. This drop
corresponds to the onset of the frequency comb, with energy being transferred
into higher order sidebands, and constrains the dynamic range in nonlinear
operation.

To conclude, we have developed an optical micro-magnetometer that achieves
sensitivity in the picoTesla range. The magnetometer is all-optical, requires
only microWatts of optical power, and has higher dynamic range and bandwidth
than any previously reported precision microscale magnetometer. By leveraging
inherent mechanical nonlinearities, we achieve nanoTesla range sensitivity at
frequencies as low as 2 Hz. These capabilities provide the opportunity for new
applications such as medical diagnosis with microscale samples using
microfluidic nuclear magnetic resonance and microscale MRI18, and direct
measurement of dynamics in strongly interacting spin systems such as
semiconductors33, superconductors35; 34, and spinor condensates36; 24. With
further improvements in sensitivity possible12, these devices could extend the
use of microscale magnetometers to areas including magnetic mapping of a
single neuron grown on chip37 and low field MRI, allowing portable high
resolution imaging at low power and low cost, and without the need for
cryogenics.

_Acknowledgements_ This research was funded by the Australian Research Council
Centre of Excellence CE110001013 and Discovery Project DP0987146 and by DARPA
via a grant through the ARO. Device fabrication was undertaken within the
Queensland Node of the Australian Nanofabrication Facility.

## References

  * (1) Budker, D. & Kimball, D. F. _Optical Magnetometry_ (Cambridge Univ. Press, New York, 2013).
  * (2) Dang, H. B. Maloof, A. C. & Romalis, M. V. Ultrahigh sensitivity magnetic field and magnetization measurements with an atomic magnetometer. _Appl. Phys. Lett._ 97 151110 (2010).
  * (3) Balasubramanian, G. et al. Ultralong spin coherence time in isotopically engineered diamond. _Nature Mat._ 8 383 (2009).
  * (4) Fagaly, R. L. Superconducting quantum interference device instruments and applications. _Rev. Sci. Instrum._ 77 101101 (2006).
  * (5) Robbes, D. Highly sensitive magnetometers a review. _Sens. Actuators, A._ 129 86-93 (2006).
  * (6) Pannetier, M. Fermon, C. Le Goff, G. Simola, J. & Kerr, E. Femtotesla magnetic field measurement with magnetoresistive sensors. _Science_ 304 1648-1650 (2004).
  * (7) Acuna, M. H. Space-based magnetometers. _Rev. Sci. Instrum._ 73 3717 3736 (2002).
  * (8) Vengalattore, M. et al. High-resolution magnetometry with a spinor bose-Einstein condensate. _Phys. Rev. Lett._ 98 200801 (2007).
  * (9) Wolfgramm, F. Vitelli, C. Beduini, F. A. Godbout, N. & Mitchell, M. W. Entanglement-enhanced probing of a delicate material system _Nature Photon._ 7 28 32 (2013).
  * (10) Baudenbacher, F. Fong, L. E. Holzer, J. R. & Radparvar, M. Monolithic low-transition-temperature superconducting magnetometers for high resolution imaging magnetic fields of room temperature samples. _Appl. Phys. Lett._ 82 3487 (2003).
  * (11) Shin, C. S. et al. Room-temperature operation of a radiofrequency diamond magnetometer near the shot-noise limit. _J. Appl. Phys._ 112 124519 (2012).
  * (12) Forstner, S. et al. Cavity optomechanical magnetometer. _Phys. Rev. Lett._ 108 120801 (2012).
  * (13) Gavartin, E. Verlot, P. & Kippenberg, T. J. A hybrid on-chip optomechanical transducer for ultrasensitive force measurements. _Nature Nanotech._ 7 509 514 (2012).
  * (14) Lee, K. H. McRae, T. G. Harris, G. I. Knittel, J. & Bowen, W. P. Cooling and control of a cavity optoelectromechanical system _Phys. Rev. Lett._ 104 123604 (2010).
  * (15) Osiander, R. et al. A microelectromechanical-based magnetostrictive magnetometer _Appl. Phys. Lett._ 69 2930 (1996).
  * (16) Engdahl, G. _Handbook of Giant Magnetostrictive Materials_ (Academic Press, SanDiego, 1999).
  * (17) Shah, V. Knappe, S. Schwindt P. D. & Kitching, J. Subpicotesla atomic magnetometry with a microfabricated vapour cell. _Nature Photon._ 1 649 - 652 (2007).
  * (18) Ledbetter, M. P. et al. Zero-field remote detection of NMR with a microfabricated atomic magnetometer. _PNAS_ 105 2286-2290 (2008).
  * (19) Grinolds, M. S. et al. Quantum control of proximal spins using nanoscale magnetic resonance imaging _Nature Phys._ 7 687 (2011).
  * (20) Romalis, M. V. & Dang, H. B. Atomic magnetometers for materials characterization. _Mater. Today_ 14 258-262 (2011).
  * (21) Wyllie, R. Kauer, M. Wakai, R. T. & Walker, T. G. Optical magnetometer array for fetal magnetocardiography. _Opt. Lett._ 37 2247-2249 (2012).
  * (22) Zhao, N. et al. Sensing singe remote nuclear spins. _Nature Nanotech._ 7 657-662 (2012).
  * (23) Crooker, S. A. et al. Spin noise of electrons and holes in self-assembled quantum dots. _Phys. Rev. Lett._ 104 036601 (2010).
  * (24) Stenger, J. et al. Spin domains in ground-state bose-Einstein condensates. _Nature_ 396 345-348 (1998).
  * (25) Hope, J. J. & Close, J. D. General limit to nondestructive optical detection of atoms. _Phys. Rev. A_ 71 043822 (2005).
  * (26) De Martino, F. et al. Spatial organization of frequency preference and selectivity in the human inferior colliculus. _Nature Commun._ 4 1386 (2013).
  * (27) Kaufmann, S. et al. Detection of atomic spin labels in a lipid bilayer using a single-spin nanodiamond probe. _PNAS_ 107 18777 (2010).
  * (28) Zhai, J. Xing, Z. Dong, S. Li, J. & Viehland, D. Detection of pico-Tesla magnetic fields using magneto-electric sensors at room temperature _Appl. Phys. Lett._ 88, 062510 (2006).
  * (29) Meyer, H. G. Stolz, R. Chwala, A. Schulz, M. SQUID technology for geophysical exploration. _Phys. Stat. Sol._ 2 1504 1509 (2005).
  * (30) Xia, H. Ben-Amar Baranga, A. Hoffman, D. & Romalis, M. V. Magnetoencephalography with an atomic magnetometer. _Appl. Phys. Lett._ 89 211104 (2006).
  * (31) Dagenais, D. M. Bucholtz, F. & Koo, K. P. Elimination of residual signals and reduction of noise in a low-frequency magnetic fiber sensor. _Appl. Phys. Lett._ 53 1474 (1988).
  * (32) See Supplemental Material http: .
  * (33) Steinert, S. et al. Magnetic spin imaging under ambient conditions with sub-cellular resolution. _Nature Commun._ 4 1607 (2013).
  * (34) Bouchard, L. S. Acosta, V. M. Bauch, E. & Budker, D. Detection of the Meissner effect with a diamond magnetometer. _New J. Phys._ 13 025017 (2011).
  * (35) Jang, J. Budakian, R. & Maeno, Y. Phase-locked cantilever magnetometry. _Appl. Phys. Lett._ 98 132510 (2011).
  * (36) Chang, M. S. et al. Observation of spinor dynamics in optically trapped 87Rb bose-Einstein condensates. _Phys. Rev. Lett._ 92 140403 (2004).
  * (37) Blagoev, K. B. et al. Modelling the magnetic signature of neuronal tissue. _Neuroimage_ 37 137 148 (2007).


# A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in
Programming Education

Jacob Doughty<EMAIL_ADDRESS>Carnegie Mellon
UniversityPittsburghPAUSA ,  Zipiao Wan<EMAIL_ADDRESS>Carnegie Mellon
UniversityPittsburghPAUSA ,  Anishka Bompelli  Carnegie Mellon
UniversityPittsburghPAUSA ,  Jubahed Qayum  Carnegie Mellon
UniversityPittsburghPAUSA ,  Taozhi Wang  Carnegie Mellon
UniversityPittsburghPAUSA ,  Juran Zhang  Carnegie Mellon
UniversityPittsburghPAUSA ,  Yujia Zheng  Carnegie Mellon
UniversityPittsburghPAUSA ,  Aidan Doyle  Carnegie Mellon
UniversityPittsburghPAUSA ,  Pragnya Sridhar  Carnegie Mellon
UniversityPittsburghPAUSA ,  Arav Agarwal  Carnegie Mellon
UniversityPittsburghPAUSA ,  Christopher Bogart  Carnegie Mellon
UniversityPittsburghPAUSA ,  Eric Keylor  Carnegie Mellon
UniversityPittsburghPAUSA ,  Can Kultur  Carnegie Mellon
UniversityPittsburghPAUSA ,  Jaromir Savelka  Carnegie Mellon
UniversityPittsburghPAUSA and  Majd Sakr  Carnegie Mellon
UniversityPittsburghPAUSA

(2024)

###### Abstract.

There is a constant need for educators to develop and maintain effective up-
to-date assessments. While there is a growing body of research in computing
education on utilizing large language models (LLMs) in generation and
engagement with coding exercises, the use of LLMs for generating programming
MCQs has not been extensively explored. We analyzed the capability of GPT-4 to
produce multiple-choice questions (MCQs) aligned with specific learning
objectives (LOs) from Python programming classes in higher education.
Specifically, we developed an LLM-powered (GPT-4) system for generation of
MCQs from high-level course context and module-level LOs. We evaluated 651
LLM-generated and 449 human-crafted MCQs aligned to 246 LOs from 6 Python
courses. We found that GPT-4 was capable of producing MCQs with clear
language, a single correct choice, and high-quality distractors. We also
observed that the generated MCQs appeared to be well-aligned with the LOs. Our
findings can be leveraged by educators wishing to take advantage of the state-
of-the-art generative models to support MCQ authoring efforts.

GPT-4, Large Language Models, LLMs, Learning Objectives, Automatic Generation,
Curricular Development, Course Design Automation, Automated Content Generation

††journalyear: 2024††copyright: rightsretained††conference: Australian
Computing Education Conference; January 29-February 2, 2024; Sydney, NSW,
Australia††booktitle: Australian Computing Education Conference (ACE 2024),
January 29-February 2, 2024, Sydney, NSW, Australia††doi:
10.1145/3636243.3636256††isbn: 979-8-4007-1619-5/24/01††ccs: Social and
professional topics Computer science education††ccs: Social and professional
topics Software engineering education

##  1\. Introduction

Multiple-choice question (MCQ) tests are one of the most popular types of
assessment in education (Butler, 2018; Towns, 2014). However, crafting high-
quality MCQs that accurately target the intended learning objectives (LOs)
requires valuable expertise, is time-consuming and, hence, expensive. This is
especially true in technical domains such as computing education where
developing effective MCQs poses distinct challenges, e.g., those related to
inclusion of pieces of computer code. With changes in technology, growing
interest in programming education, and low barriers for students to share past
assessments, the demand on instructors to author novel high-quality MCQs has
never been higher. Recent developments in large language models (LLMs), such
as generative pre-trained transformers (GPT), show tremendous potential for
addressing this challenge. Leveraging the capabilities of LLMs, educators
could potentially (semi-)automate the generation of MCQ assessments.

We developed a novel LLM-based (GPT-4) pipeline to automate generation of MCQs
for higher-education Python programming courses. The novelty of our approach
lies in making use of a high-level course context and detailed module-level
LOs for producing well-formed high-quality MCQs that use clear language, have
plausible distractors, and are well-aligned with the LOs. Since understanding
the quality and effectiveness of automatically generated MCQs is of utmost
importance we performed a rigorous evaluation of 651 automatically generated
and 449 human-crafted questions. If high-quality automated MCQ generation
proves feasible it could significantly reduce the time and effort educators
currently spend on developing assessments.

To investigate if and how GPT-4 could generate high quality MCQ assessments
for higher education programming courses, we analyzed the following research
questions:

  * RQ1:

To what degree do the generated MCQs meet typical quality requirements?
Specifically, do they:

    * (i)

provide sufficient information in clear language;

    * (ii)

have a single correct answer with

    * (iii)

high-quality distractors; and

    * (iv)

contain syntactically and logically correct code?

  * RQ2:

How well are the generated MCQs aligned with the specified module-level LOs?

By carrying out this work, we provide the following contributions to the
computing education research community. To our best knowledge, this is:

  * C1:

One of the first studies employing and evaluating LLMs in automatic generation
of MCQs for programming classes.

  * C2:

One of the first studies generating MCQs not from short pieces of course
materials, but from LOs.

  * C3:

One of the most extensive (1,100 MCQs) and detailed evaluations of generated
MCQs including alignment with LOs.

##  2\. Related Work

As manually constructing MCQs requires significant effort, researchers have
focused heavily on the task of automated question generation. Most work in
this field focuses on a specific MCQ element, i.e., the _stem_ (question), the
_key_ (correct answer), and the _distractors_ (incorrect options). The most
widely researched task is question answering (QA) (Allam and Haggag, 2012;
Soares and Parreiras, 2020); this is equivalent to key generation, although
most work in question answering do not place themselves within this context.

The automatic generation of stems is related to generating free-form questions
(QG). Initially, these concentrated on reading comprehension tasks, relying on
both traditional NLP methods (Heilman, 2011) and neural networks (Hochreiter
and Schmidhuber, 1997) to yield meaningful questions (Duan et al., 2017). More
recent systems rely on sequence models (Wang et al., 2018) or attention-based
methods like LLMs (Du et al., 2017; Cho et al., 2019; Lopez et al., 2020; Chan
and Fan, 2019) (GPT-2 (Radford et al., 2019), BERT (Devlin et al., 2018)).
Additionally, there have been attempts to tackle QA and QG tasks jointly (Tang
et al., 2017). Kurdi et al. provide a comprehensive overview of automated free
response and MCQ generation for educational purposes (Kurdi et al., 2020).

|  |  | LO Bloom’s Taxonomy Levels |  |
---|---|---|---|---|---
Course Name | Modules | MCQs | RMB | UND | APP | ANL | EVL | CRT | N/A | Overall
Practical Programming with Python | 8 | 194 | 14 | 31 | 20 | 17 | 2 | 29 | 10 | 123
Get Started with Python | 5 | - | 8 | 24 | 9 | 4 | 1 | 5 | 4 | 55
Python Essentials – Part 1 (Basics)111OpenEDG: Python Essentials - Part 1 (Basics). Available at: https://edube.org/study/pe1 [Accessed 2023-07-30] | 4 | 129 | - | 7 | - | - | - | - | 14 | 21
Python Essentials – Part 2 (Intermediate) | 4 | 126 | 1 | 1 | 4 | 2 | - | 7 | 5 | 20
Introduction to Data Science in Python | 4 | - | 3 | 2 | 8 | - | - | 1 | - | 14
Python and Pandas for Data Engineering | 4 | - | 1 | 1 | 2 | - | - | 9 | - | 13
Total | 449 | 27 | 66 | 43 | 23 | 3 | 51 | 33 | 246
Table 1. Dataset of manually created MCQs. The first three columns list the
names of the courses, the number of modules per course, and the number of
MCQs. The remaining columns report the number of LOs from different levels of
Bloom’s taxonomy (RMB – remember, UND – understand, APP – apply, ANL –
analyze, EVL – evaluate, CRT – create, N/A – unassigned).

In contrast to the work on generating stems, current work on automated
distractor generation (DG) focuses on cloze tests (Qiu et al., 2021; Jiang and
Lee, 2017; Ren and Zhu, 2021) and reading comprehension (Gao et al., 2019;
Zhou et al., 2020), as well as MCQs for domains outside of computing such as
biology (Shin et al., 2019). These systems use the stem and/or the key to
generate plausible (i.e., similar) distractors. Often, the systems generate
several more distractors than necessary and select the best through a ranking
system (Liang et al., 2018). Given that DG in technical domains poses distinct
challenges, research recognizes the potential importance of additional
resources such as domain-specific ontologies (Kumar et al., 2023). Recently,
LLMs such as BERT (Kalpakchi and Boye, 2021) or GPT-2 (Offerijns et al., 2020)
have been utilized in DG.

There is existing work on complete end-to-end MCQ generation. Rodriguez-
Torrealba et al. propose an end-to-end pipeline that generates all three
elements of an MCQ, with QG, QA, and DG (Rodriguez-Torrealba et al., 2022),
using Google’s T5 transformer model. They note that evaluation is difficult,
only evaluating 10 questions. Leaf is another T5-powered end-to-end MCQ
generation pipeline (Vachev et al., 2022), but has no evaluation. Unlike these
systems, which focus on each task separately, our work generates the stem,
distractors, and key in a single generation step through GPT-4. Additionally,
while existing systems rely on textual context to generate reading-
comprehension-like questions, (Madri and Meruva, 2023; Ch and Saha, 2018) we
focus on generating a question well-aligned with the provided LO, allowing for
easier large-scale evaluation. Cheung et al. adopt a single-pass generation
approach using ChatGPT222ChatGPT. Available at: https://chat.openai.com/
[Accessed 2023-08-16] to generate MCQs. They rely on medical textbooks to
provide context (Cheung et al., 2023), noting that around 40% of the questions
are usable as-is without prompt engineering. Finally, Nasution uses the same
approach to generate higher education biology MCQs without relying on a
reference text, although their analysis focuses on student usage rather than
instructor usability, which we focus on (Nasution, 2023). Our work is
distinguished from (Nasution, 2023) by rigorous prompt engineering focused on
question quality and alignment, alongside a rigorous evaluation (Section 4).

While the above methods focus on other areas of education, MCQs also play an
essential role in computing education. Hence, there have been attempts on
efficient generation of MCQs in this area as well. We could not find papers
that extend Traynor and Gibson’s early attempt (Traynor and Gibson, 2005) to
automatically generate MCQs for CS1, as it appears the focus rather shifted
towards approaches based on learner-sourcing (Denny et al., 2008). That is
with the exception of the very recent work of Tran et al. who used GPT-3 and
GPT-4 to generate isomorphic variations of pre-existing MCQs (Tran et al., [n.
d.]). Our work is most likely the first attempt to generate _novel_ MCQ
assessments for higher education programming classes using LLMs. However,
there is pre-existing work on answering MCQs with GPT-3 and GPT-4 in computing
education (Savelka et al., 2023c, d, b).

This has been inspired by many other instances where LLMs have already
demonstrated remarkable effectiveness in generating novel language artifacts.
These include course readings (Leiker et al., 2023), code explanations (Sarsa
et al., 2022; Leinonen et al., 2023; MacNeil et al., 2022, 2023), model
solutions (Savelka et al., 2023a; Denny et al., 2022; Piccolo et al., 2023),
feedback (Phung et al., 2023), or responses to help requests (Liffiton et al.,
2023; Sheese et al., 2023; Savelka et al., 2023e; Kazemitabaar et al., 2023).
The above examples are a testament to the capabilities of the state-of-the-art
LLMs in generating grammatically correct, professionally sounding language
which makes the artifacts seemingly indistinguishable from the materials
generated by human experts. However, such automatically generated learning
resources may often lack some of the deep qualities that are necessary for
them to be effective (Prather et al., 2023). Hence, in our work we perform a
rigorous evaluation of the generated MCQs.

##  3\. Data Set

For the experiments in this paper (Section 5), we assembled a dataset of 246
module-level LOs coming from four Python programming333OpenEDG: Python
Essentials - Part 1 (Basics). Available at: https://edube.org/study/pe1
[Accessed 2023-07-30]
OpenEDG: Python Essentials - Part 2 (Intermediate). Available at:
https://edube.org/study/pe2 [Accessed 2023-07-30]
Coursera: Get Started with Python. Available at:
https://www.coursera.org/learn/get-started-with-python [Accessed 2023-07-30]
Sail(): Social and Interactive Learning Platform. Available at:
https://sailplatform.org/courses [Accessed 2023-07-30] and two introductory
data science444Coursera: Introduction to Data Science in Python. Available at:
https://www.coursera.org/learn/python-data-analysis [Accessed 2023-07-30]
Coursera: Python and Pandas for Data Engineering. Available at:
https://www.coursera.org/learn/python-and-pandas-for-data-engineering-duke
[Accessed 2023-07-30] courses. Three of the courses also contained MCQs (529
in total) that we collected in order to compare them to the automatically
generated ones (Section 5). We assumed that these MCQs were created manually.
Two of the authors associated each MCQ with the best aligned LO from the
corresponding course module. We excluded 51 of the MCQs from our experiments
because we could not reasonably assign them to a single corresponding LO. An
additional 29 MCQs were excluded from our study because they required more
than one correct choice whereas we only focus on MCQs with single-choice keys.
Hence, the resulting dataset included 449 human-crafted MCQs. Using the fine-
tuned BERT classifier (Li et al., 2022) (Section 4 has details), each LO was
categorized into one of the six levels of the revised Bloom’s taxonomy
(Krathwohl, 2002). Table 1 shows the distribution of the LOs across courses
and Bloom’s taxonomy levels as well as the number of MCQs collected from each
course. Note that some of the LOs extracted from the courses were not well-
defined (e.g., no action verb) and, hence, it was not possible to categorize
them in terms of Bloom’s taxonomy.

##  4\. MCQ Generation

Figure 1 shows the overall architecture of the MCQ generation pipeline. To
generate an MCQ, we supply only high-level information about the course, the
course unit (module), and the targeted LO. We then combine internal MCQ design
resources with the user’s input into the prompts, which are submitted to
GPT-4. The below sections elaborate on each of the steps involved in the MCQ
generation pipeline in greater detail.

Figure 1. MCQ Generation Pipeline. The diagram describes the automatic MCQ
generation process flowing from left to right. The input provided by the user
is processed—the Bloom’s taxonomy level of the provided LO is predicted which
leads to determination about the appropriate type of the questions and
corresponding MCQ examples. These are combined with the user input and
resources from the design resources to form the system and user prompts that
are submitted to the GPT-4 model outputting the generated MCQ (stem, key,
distractors).

You are a learning engineer support bot focused on creating top quality
multiple-choice question assessments.

A multiple-choice question is a collection of three components aimed at
testing a student’s understanding of a certain topic, given a particular
context of what the student is expected to know. The topic, as well as the
context of the topic, will be provided in order to generate effective
multiple-choice questions. The three components of a multiple-choice question
are as follows: a Stem, a Correct Answer, and two Distractors. There must
always be only one correct answer and only two distractors.

The stem refers to the question the student will attempt to answer, as well as
the relevant context necessary in order to answer the question. It may be in
the form of a question, an incomplete statement, or a scenario. The stem
should focus on assessing the specific knowledge or concept the question aims
to evaluate.

The Correct Answer refers to the correct, undisputable answer to the question
in the stem.

A Distractor is an incorrect answer to the question in the stem and adheres to
the following properties.

  1. (1)

A distractor should not be obviously wrong. In other words, it must still bear
relations to the stem and correct answer.

  2. (2)

A distractor should be phrased positively and be a true statement that does
not correctly answer the stem, all while giving no clues towards the correct
answer.

  3. (3)

Although a distractor is incorrect, it must be plausible […]

  4. (4)

4\. A distractor must be incorrect. It cannot be correct, or interpreted as
correct by someone who strongly grasps the topic.

[…] Use “None of the Above” or “All of the Above” style answer choices
sparingly. These answer choices have been shown to, in general, be less
effective at measuring or assessing student understanding.

Multiple-choice questions should be clear, concise, and grammatically correct
statements. Make sure the questions are worded in a way that is easy to
understand and does not introduce unnecessary complexity or ambiguity.
Students should be able to understand the questions without confusion. The
question should not be too long, and allow most students to finish in less
than five minutes. This means adhering to the following properties.

  1. (1)

Avoid using overly long sentences.

  2. (2)

Avoid code that is longer than 20 lines for questions, and longer than 10
lines for the correct answer and distractors.

  3. (3)

If you refer to the same item or activity multiple times, use the same phrase
each time.

  4. (4)

Ensure that each multiple-choice question provides full context. In other
words, if a phrase or action is not part of the provided topic or topic
context that a student is expected to know, then be sure to explain it briefly
or consider not including it.

  5. (5)

Ensure that none of the distractors overlap. In other words, attempt to make
each distractor reflect a different misconception on the topic, rather than a
single one, if possible.

  6. (6)

Avoid too many clues. Do not include too many clues or hints in the answer
options, which may make it too obvious for students to determine the correct
answer. These options should require students to use their knowledge and
reasoning to make an informed choice. […]

Blooms’ Taxonomy and Action Verbs:

Multiple-choice questions must be well aligned to the learning objectives they
are intended to assess students’ knowledge on. This implies that they must
assess skills at the right cognitive level corresponding to the Bloom’s
taxonomy categorization of the learning objective. Bloom’s Taxonomy offers a
framework for categorizing the depth of learning, and it provides guidance on
selecting appropriate action verbs when writing learning objectives. Here are
the six levels of Bloom’s taxonomy and their definitions:

  * •

_Remember_ \- This level involves retrieving, recognizing, and recalling
relevant knowledge from long-term memory.

  * •

_Understand_ \- At this level, learners construct meaning from oral, written,
and graphic messages through interpreting, exemplifying, classifying,
summarizing, inferring, comparing, and explaining.

  * •

_Apply_ \- This level requires learners to carry out or use a procedure
through executing or implementing it.

  * •

_Analyze_ \- At this level, learners break material into constituent parts,
determine how the parts relate to one another and to an overall structure or
purpose through differentiating, organizing,[…]

  * •

_Evaluate_ \- This level involves making judgments based on criteria and
standards through checking and critiquing.

  * •

_Create_ \- At this level, learners put elements together to form a coherent
or functional whole, or they reorganize elements into a new pattern or
structure through generating, […]

Course Context

Below is a brief description of the Practical Programming with Python course.

Description: Students learn the concepts, techniques, skills, and tools needed
for developing programs in Python. Core topics include types, variables,
functions, iteration, conditionals, data structures, classes, objects,
modules, and I/O operations. Students get an introductory experience with
several development environments, including Jupyter Notebook, as well as
selected software development practices, such as test-driven development,
debugging, and style. Course projects include real-life applications on
enterprise data and document manipulation, web scraping, and data analysis.

Question type: Recall

A recall multiple-choice question often contains minimal code, if at all, in
its stem. They may assess a student’s understanding of basic programming
concepts or include some technical details. It should be conceptual while
containing specific knowledge of the course content and learning objectives.

In the context of an Introductory Programming with Python course, these
questions typically ask about Python syntax and principles, built-in
functions, or standard libraries. They may also evaluate students’
understanding of fundamental programming concepts such as coding conventions
and object-oriented programming (OOP) principles.

Below are some examples:

Example 1:

    {
        "question": "Which of the following methods can be used to remove a single element from a list in Python?",
        "choices": [{"option": "A", "text": "pop()"}, {"option": "B", "text": "delete()"}, {"option": "C", "text": "clear()"}],
        "correctAnswer": "A",
        "explanation": "clear() will remove all elements, you can use del but not delate() to remove element."
    } [...]

Output Format

Output your multiple-choice question in an easy-to-parse json dictionary
format, where the stem is the key, and the correct answer and distractor
choices are values. Be sure to clearly distinguish which choice is the correct
answer and which are distractors. The question generated should have exactly 2
distractors and 1 correct answer (3 choices in total). If there is code in the
stem, please set “code_in_stem” to True. If there is no code in the stem, set
“code_in_stem” to False.

Your return should be the exact json structure of the following example:

    {
        "question": "The stem of the question",
        "choices": [{"option": "A", "text": "Answer Choice A in string type"}, {"option": "B", "text": "Answer Choice B in string type"},
                    {"option": "C", "text": "Answer Choice C in string type"}],
        "correctAnswer": "A",
        "code_in_stem": "True or False",
        "explanation": "The explanation of the choices"
    }

[…] If any of the multiple-choice items contain code, please format the code
snippet as shown below:

    ‘‘‘python
    def test():
        return "Correct Format"
    ‘‘‘

6.2cm(13.75cm,-8.93cm)

6.2cm(13.8cm,-20.0cm)

6.2cm(13.65cm,-13.2cm)

6.2cm(13.82cm,-5.15cm)

6.2cm(13.6cm,-10.5cm)

Figure 2. The System Part of the Prompt. The figure shows extensive excerpts
from the system part of the prompt showing the main constituents: MCQ
Principles, Bloom’s Taxonomy, Course Description, Question Type Examples, and
Output Format. The colored stripes on the left and the colored badges match
the colors of the pipeline constituents from Figure 1. The [...] tokens mark
places where the text has been abridget to fit on the page. The purple text is
dynamic (data dependent).

Generate a top quality Recall multiple-choice question

for the course Practical Programming in Python

on the unit: Python Basics and Introduction to Functions.

Your generated question should target the following learning objective:

Explain what Python is and how to use it to run single-line expressions as
well as
small multi-line programs.

Your generated question should also be at the understand level in Bloom’s
taxonomy. 6.2cm(4.90cm,-3.55cm)  6.2cm(4.55cm,-3.0cm)  6.2cm(4.65cm,-2.5cm)
6.2cm(5.08cm,-2.0cm)  6.2cm(4.7cm,-1.1cm)

Figure 3. The User Message Part of the Prompt. The figure shows an example
user message part of the prompt showing the main constituents: Question Type,
Course Name, Module (unit) Name, Learning Objective, and Bloom’s Taxonomy
Level. The colored stripes on the left and the colored badges match the colors
of the pipeline constituents from Figure 1. The purple text is dynamic (data
dependent).

###  4.1. User Input

The notable feature of the proposed MCQ generation pipeline is that the user
is expected to provide only high-level information about the course and the
module. This is in stark contrast to the large majority of other MCQ
generation systems described in Section 2. While those systems use a piece of
text (e.g., a paragraph from a textbook) to generate an MCQ, our system
utilizes a specific module-level LO to generate an MCQ that is well-aligned
with that LO. This allows us to more carefully align MCQ generation with
assessment of student achievement of the intended LO. Besides the specific
module-level LO and the module it comes from, the user is expected to provide
a course title (see Table 1 for titles of the courses used in this study), a
short course description with the list of course-level LOs, and the list of
course modules. Using this particular context as well as following the best
practices for prompting an LLM (Nasution, 2023; Cheung et al., 2023) to
generate an MCQ are the prominent features of the presented pipeline.

###  4.2. Design Resources

We curated a set of static resources to support various stages of effective
MCQ generation. We focused on the following elements:

  * •

_MCQ principles_ – A set of research-validated and generally accepted
principles that guide authoring of high-quality MCQs. For example, this
includes that distractors should be plausible and limited in number (often
just two). We only provided a concise description here but the _MCQ
Principles_ section of Figure 2 shows an extensive excerpt.

  * •

_Bloom’s taxonomy_ – Definitions of the six levels of the revised Bloom’s
taxonomy (Krathwohl, 2002), i.e., remember, understand, apply, analyze,
evaluate, and create. This taxonomy helps educators articulate LOs that focus
on concrete actions and behaviors, and target distinct levels of cognitive
processes. For LOs to guide the selection of assessments, they must be
measurable, i.e., it should be possible to evaluate whether learners attained
the intended LO. We also include information on the aims and uses of Bloom’s
taxonomy. The _Bloom’s taxonomy_ section of Figure 2 shows an extensive
excerpt.

  * •

_Question type system_ – From an informal analysis of the collected dataset of
human-crafted MCQs (Section 3), we define five types of programming MCQs:
recall, fill-in the blank, identify correct output, analyze (trace) code, and
scenario (see the _LO Mapping to Question Types_ step of the pipeline below).
We include the definitions of these types into the resources. The _QType
Examples_ section of Figure 2 provides an example of the recall type.

  * •

_Question type examples_ – A small set of high-quality MCQ examples for each
of the defined types (example in the _QType Examples_ section of Figure 2).

  * •

_Output format_ – Specification of the output format (JSON). Figure 2 shows
additional details (_Output Format_ section).

Providing an LLM with this type of information to create effective MCQs is one
of the core contributions of this work. Unlike Leaf (Vachev et al., 2022), our
methodology allows for training-free generation of questions and distractors,
ensuring effective generation without necessarily worrying about out-of-domain
generalization. It is reasonable to assume that this holds for mainstream
domains, such as introduction to programming, where plenty of related
materials were included in the dataset used for pre-training of the LLM.
However, this approach may not be directly applicable in highly specialized
domains with less abundant content.

###  4.3. LO Bloom’s Taxonomy Level Classifier

We fine-tuned a binary BERT classifier (Li et al., 2022) for each Bloom’s
Taxonomy category on 21,380 LOs from 5,558 university courses. BERT
(bidirectional encoder representation from transformers) (Devlin et al., 2018;
Vaswani et al., 2017) is a popular LLM notable for its fine-tuning
capabilities on a down-stream task. We used the models to predict the Bloom’s
Taxonomy level (i.e., remember, understand, apply, analyze, evaluate, or
create) of the generated LOs. The prediction is then utilized in the _LO
Mapping to Question Types_ step of the pipeline (see below), and it is also
embedded in the user message part of the prompt as shown in Figure 3.

###  4.4. LO Mapping to Question Types

We use the automatically predicted Bloom’s taxonomy level of the provided LO
(see above) to determine the appropriate MCQ types for the LO (see Table 2).
We map each taxonomy level to one or more question types and a question is
generated for each type:

  * •

_Recall_ – Given minimal to no code, students are asked about basic
programming concepts or technical details.

  * •

_Fill in the Blank_ – Given a code snippet with some sections removed,
students are asked to select an option that successfully replaces the blank to
create syntactically and semantically correct code.

  * •

_Scenario Based_ – Given a scenario or situation, students are asked to
identify appropriate tools, methods, or packages best suited to accomplish the
task prescribed.

  * •

_Correct Output_ – Given a code snippet, students are asked to trace program
execution to determine either intermediate or final outputs.

  * •

_Code Analysis_ – Given a code snippet, students are asked to spot errors or
to build on or use the code in a new manner.

These question types allow us to target specific levels of cognitive
processes, e.g., preventing the generation of MCQs involving complex code
tracing for LOs asking students to remember a concept.

Finally, high-quality MCQ examples are selected based on the question type in
order to ensure that the proper cognitive processes are targeted in accordance
with the provided LO. The corresponding examples of the high-quality MCQs of
that type are retrieved from the design resources.

###  4.5. Prompt

GPT-4 utilizes the so-called system part of the prompt to provide the overall
context of the conversation. From the design resources, we retrieve a set of
MCQ principles and best practices. We provide the definition of the predicted
Bloom’s taxonomy level, examples of the selected question type, the course
description, and the expected output format of the generated MCQ as well.
Figure 2 shows an extensive example of the system part of the prompt.

The user message part of the prompt is the piece of text to which the model is
expected to respond (within the context established by the system part of the
prompt). We include the user-provided course name and the course module, the
specific module-level LO, the predicted Bloom’s taxonomy level, and
subsequently mapped question type into the user prompt. Figure 3 shows an
extensive example of the user message part of the prompt.

###  4.6. MCQ Generation Step

We use GPT-4 (`gpt-4-0613`), which is one of the most advanced LLMs released
by OpenAI as of the writing of this paper (OpenAI, 2023). As for the model
parameters, we set temperature to its default value of 1.0. We keep top_p,
frequency_penalty, and presence_penalty at their default values of 1.0, 0.0,
and 0.0 respectively. We impose a max_token limit of 2,000 tokens (i.e., the
maximum length of a generated MCQ).

As the response to the prompt, consisting of the system part and the user
message part, the GPT-4 model generates an output in the expected JSON format.
The response contains the stem, the key, and the distractors, i.e., all the
needed constituents of an MCQ. One of the choices is marked as correct.

Figure 4. MCQ Evaluation Results. The MCQ evaluation rubric is shown on the
left. On the right, the automatically generated MCQs (GPT) are compared with
the human-crafted ones. The top part of the figure shows the results for the
five rubric items focused on the quality of the MCQs (RQ1). The bottom part
shows the rubric item addressing the LO-MCQ alignment (RQ2). The reported
ratios are based on the counts of the labels after the disagreement resolution
described in Section 5.

##  5\. Experimental Design

To evaluate the quality of the automatically generated MCQs, we generate
several MCQs for each of the 246 LOs in our dataset. We generate a single
question per question type mapped to the LO’s Bloom’s taxonomy level as
described in Table 2. When we could not assign a Bloom’s taxonomy level we
simply generate MCQs of all types. Following this process we obtain a sizeable
dataset of 651 automatically generated MCQs. Table 2 shows the distribution of
the MCQs per both their question type and Bloom’s taxonomy level.

MCQ-Type | RMB | UND | APP | ANL | EVL | CRT | N/A | Total
---|---|---|---|---|---|---|---|---
Recall | 27 | 66 | - | - | - | - | 32 | 125
Fill in the Blank | 27 | 63 | 41 | 23 | - | - | 32 | 186
Scenario Based | - | - | 41 | 21 | - | - | 31 | 93
Correct Output | - | - | 41 | 23 | 3 | - | 33 | 100
Code Analysis | - | - | 41 | 21 | 3 | 49 | 33 | 147
Total | 54 | 129 | 164 | 88 | 6 | 49 | 161 | 651
Table 2. Automatically Generated MCQs. The table shows the distribution of the
automatically generated MCQs per question types (rows) and the levels of
Bloom’s taxonomy (columns).

We developed a rubric consisting of six criteria shown in Figure 4. The first
five criteria of the rubric target RQ1, while the last criterion targets RQ2.
Seven students and six CS instructor annotators (all authors of this paper)
applied the rubric to the generated (651) as well as human-crafted (449) MCQs,
i.e., 1,100 MCQs in total.

Rubric item | Gwet’s AC1 | Fleiss $\kappa$
---|---|---
Sufficient information | 0.96 | 0.35
Correct answer | 0.88 | 0.31
Unique choices | 0.62 | 0.16
No obviously wrong choice | 0.92 | 0.07
Correct code | 0.88 | 0.07
LO alignment | 0.93 | 0.23
Table 3. Inter-rater Agreement. The table shows Gwet’s AC1 and Fleiss $\kappa$
scores for each of the six rubric items (Figure 4).

Each annotator was asked to complete 250 annotations555Due to time, most
annotators completed over 240 annotations, with one only completing 93. We
required annotators to attempt answering the question before applying the
rubric items. The overall inter-rater agreement in terms of Fleiss $\kappa$
was 0.22 which corresponds to a fair agreement. The $\kappa$ statistic is
known to severely underestimate the agreement in situations when one of the
labels is clearly dominant (Zec et al., 2017). This is the case of our data
where for each rubric item there is always a clearly dominant answer. To
address this issue we also compute Gwet’s AC1 which is robust in such
situations (Wongpakaran et al., 2013). Table 3 reports the AC1 and $\kappa$
scores for each of the six rubric items. The measured Gwet’s AC1 range from
0.62 to 0.96. Each MCQ was annotated by at least one student and one
instructor. Overall, 3,076 annotations for the 1,100 MCQs were produced, i.e.,
a little less than 3 annotations per MCQ on average. To resolve disagreements
we used the following rules in the presented order:

  1. (1)

Majority Vote;

  2. (2)

Instructors’ annotations had precedence over students’;

  3. (3)

An annotator who correctly answered the MCQ had precedence over the one who
answered incorrectly;

  4. (4)

The least favorable evaluation.

Then, the automatically generated MCQs were compared to the human-crafted ones
in terms of the six categories from the developed rubric. To test for
statistically significant differences we employ Fisher’s exact test (extended
to multiple categories through approximation).

##  6\. Results

###  6.1. MCQ Quality (RQ1

Figure 4 shows the results of the experiments described in Section 5. The
first five criteria relate to RQ1. From those, we observe that the generated
MCQs appear to be of comparable quality to the human-crafted ones. A notable
issue is that 4.9% of the automatically generated MCQs had multiple correct
answer choices (compared to only 1.1% of the human-crafted MCQs). The quality
of the distractors appears to be another rubric item where the MCQ generation
pipeline struggles. GPT MCQs were more likely to be annotated as having
distractors which gave away the correct answer (4.0% vs 0.9% human). We
performed the Fisher exact test to test for statistical significance. The
presence of the correct answer ($p=0.002$) and the presence of obviously wrong
options ($p=0.002$) criteria had a statistically significant difference
between automatic and human generated questions. This corresponds to the above
described differences related to the presence of multiple correct choices in
the automatically generated MCQs and distractors which gave away the correct
answer. Hence, we conclude that the generated MCQs provide sufficient
information in clear language (RQ1.i) and contain syntactically and logically
correct code (RQ1.iv). That is they do not appear to differ from the human-
crafted MCQs in terms of the mentioned quality criteria. On the other hand,
the generated MCQs are somewhat lacking when compared to the human-crafted
ones when it comes to having a single correct answer (RQ1.ii) and high-quality
distractors (RQ1.iii).

###  6.2. MCQ-LO Alignment RQ2

The results of evaluating RQ2 are also shown in Figure 4—the last rubric item.
We observed that the automatically generated MCQs appear to be noticeably
better aligned with the LOs as compared to the human-crafted ones. This is
confirmed by the Fisher exact test as well ($p<10^{-9}$). It appears that
human generated MCQs were quite often related to the LOs but did not target
the appropriate cognitive level (Bloom’s taxonomy) or simply were too
different to be considered a viable assessment for the LO (20.5% vs 12.0%
auto-generated). Often, the human-crafted MCQs did not relate to the LOs at
all (12.0% vs 4.8% auto-generated). This finding is discussed in great detail
below.

##  7\. Discussion

Generated MCQs were more likely to have multiple correct answers. This was
mostly observed in MCQs generated for the LOs at the _Apply_ and _Create_
cognitive levels of Bloom’s Taxonomy. Consider the example shown in Figure 5.
While the answer choices comprise three different methods of converting a
string to a list and reversing it, all three of them accomplish the same task
with no side-effects. Hence, all of the options are correct. Note that only
the first one was marked as correct by GPT-4. While not too prevalent (4.9% of
generated MCQs), this issue is serious and requires human intervention to be
fixed. Future work should focus on mitigating it.

Given the string s = "Hello, World!" you need to create a piece of code that
will return a list of the words in s but in reversed order while preserving
the original string. Which one of the following code snippets achieves this
goal?

A. reversed_s = s.split(’ ’)
reversed_s.reverse()
B. reversed_s = ”.join(reversed(s))
reversed_s = reversed_s.split(’ ’)
C. reversed_s = s.split(’ ’)[::-1]

Figure 5. An example generated MCQ where all the choices are correct answers.
Only the first choice (A.) was marked as correct by GPT-4.

Compared to the human-crafted ones, the generated MCQs were more likely to
have obviously-wrong choices. This was the most pronounced in the _Fill-in-
the-Blank_ and _Scenario Based_ MCQ type. Of the MCQs that were annotated as
having options that “give away the correct answer”, the majority gave the
answer away in the question stem. This issue is somewhat less pressing than
the one described earlier as it does not render the MCQ completely invalid.
While ideally an instructor would recognize the issue and edit the MCQ this
cannot be relied on. Figure 6 shows an example MCQ where the code snippet
included in the stem gives away the correct answer.

In Python, the _____ loop is used when we want to iterate over a sequence
(like a list, tuple, set, or string) or other iterable objects. Iterating over
a sequence is called traversal.

    for item in iterable:
        # execute some statements

A. for
B. while
C. do

Figure 6. An example generated MCQ where the code snippet included in the stem
gives away the correct answer (A).

GPT-4’s ability to produce effective MCQs was close to human performance. In
MCQs where GPT failed to meet a quality requirement this was usually the only
issue with the question. Based on our analysis, the most serious issue with
the generated MCQs appear to be implausible or plainly incorrect distractors
along with revealing the correct answer in the question stem. Future work
should focus on prompt engineering techniques to mitigate these issues.
Importantly, 81.7% of all the generated MCQs passed all of the evaluation
criteria. This suggests that less than 1 in 5 generated questions would
require instructors’ edits.

The proposed pipeline is designed to generate a single MCQ for a single LO,
and in the vast majority of cases, the MCQ was well-aligned with the LO. We
observed that the alignment of the generated MCQs was vastly superior compared
to the alignment of the human-crafted ones (e.g., Figure 7). This is likely
due to the fact that educators may often focus more on the alignment of an MCQ
with a module’s topic and less on the alignment with an LO. Achieving the
alignment between LOs and assessments is often challenging for educators. This
provides an excellent motivation for a support system such as the one
presented in this paper.

Learning Objective
Discuss the importance of writing comments and how to write correct comments.

Generated MCQ
In Python programming, why is it important to write comments in your code?
A. Comments are functional elements of the code that can affect program
execution.
B. Comments help to improve the readability and maintainability of the code by
explaining the function and intention of parts of the code.
C. Comments are used to debug if there is an error in the program execution.

Human-crafted MCQ
Clean code is easy to maintain.
A. True
B. False

Figure 7. An example where the generated MCQ is better aligned with the LO
than the human-crafted one which only appears to target the module’s topic and
not the LO itself.

During our analysis, we considered Bloom’s Taxonomy, MCQ-type, LO, and course
title as potential factors for MCQ-LO alignment. We observed that relative to
the other courses, MCQs generated from the LOs in the Python Essentials 1
course scored slightly worse on alignment. We found that the failure cases for
GPT MCQ alignment occurred independently of these variables. While our results
are promising, there are some caveats. As we scraped MCQs from the internet,
we needed to manually associate MCQs with LOs. This likely explains some part
of the observed difference in MCQ-LO alignment between the generated and
human-crafted MCQs.

When evaluating the MCQs the human raters were asked to answer the questions.
While such a setup does not replace the evaluation of the questions in real
classroom settings it may provide tentative insights into the difficulty of
the generated MCQs compared to the human-crafted ones. Overall, the human-
crafted MCQs appear to be more challenging than the generated ones. The
student raters answered 71.5% of the generated MCQs correctly (62.6% human-
crafted) while the instructor raters handled 80.1% of the generated questions
successfully (76.3% human-crafted). It is important to emphasize that the
raters were not instructed to put effort into answering the questions
correctly. Hence, the reported rates need to be treated with caution.

##  8\. Implications for Teaching Practice

Our results suggest that LLM-powered tools can generate MCQs that have
comparable quality to MCQs generated by humans while achieving better
alignment with LOs. Programming instructors can use LLM-powered tools to
reduce their workload, enabling them to focus more on student engagement and
curriculum enhancement. Deploying such tools could make updating and revising
assessments more efficient.

Of particular note is the finding that automatically generated MCQs seem to be
better aligned with the LOs than those generated by humans. This could mean
that use of automatically generated MCQs might provide more accurate
representations of students’ mastering of the intended LOs, potentially
leading to more accurate evaluations. Moreover, it indicates that LLM-powered
tools could be used not only for assessment generation but also as an
invaluable tool for curriculum design and planning (Sridhar et al., 2023).

Given the potential of automated MCQ generation, teaching practice could
benefit enormously from incorporating the LLM-based tools into the assessment
design process. As with the introduction of any novel tool, care should be
taken to correctly deploy and use such resource, and to handle its output
responsibly (e.g., editing generated MCQs that have issues).

##  9\. Limitations and Threats to Validity

While we attempted to make the assessment of the quality of automatically
generated MCQs as objective as possible via a well-defined rubric, we
acknowledge potential bias from the limited pool of human raters.
Additionally, the human generated questions were created before the study and,
hence, the instructors authoring the MCQs were not aware of the rubric that we
used to evaluate the questions in this study. It is plausible that if the
rubric was available to them they could have authored the MCQs to better
satisfy the requirements. Finally, as we manually paired human-crafted MCQs to
LOs, it is possible we ourselves caused part of the observed MCQ-LO
misalignment.

Our study has not evaluated the utility of auto-generated MCQs in live
classrooms, and, hence, their pedagogical impact is not known. Although, the
LLM-powered system was able to generate questions that seem comparable to
those written by humans, it remains to be seen how effective these auto-
generated MCQs are at assessing student learning. Additionally, we did not
compare the difficulty of the generated MCQs to that of the human-crafted
ones. We focused on the alignment between an MCQ and a corresponding LO.
However, we did not consider the alignment between an MCQ and the learning
content. These important investigations are left for future work.

Our research focused only on Python programming courses at the higher
education level, which may limit the generalizability of the results to other
programming languages or education levels. While Python is widely used in
introductory programming courses, it is possible that LLMs may perform
differently when generating MCQs for other languages or more specialized
domains. Since the evaluated approach relies on the LLM’s “knowledge” of the
domain it is quite likely that it would not generalize well to highly
specialized domains.

##  10\. Conclusions and Future Work

This paper provides promising evidence suggesting that systems powered by LLMs
can automatically generate high-quality MCQs for Python programming courses.
The generated MCQs are comparable in quality to those created by human
educators (RQ1) and exhibit strong alignment with LOs (RQ2). Overall, this
study demonstrates the feasibility of high-quality automated MCQ generation
that has the potential to significantly reduce the time and effort educators
currently spend on developing assessments

Future work needs to explore the effectiveness of LLM-generated MCQs within
real-world classroom contexts. This will allow a better understanding of the
uptake, utility and discriminative power of generated MCQs. Additionally,
future work needs to be done to better understand the novelty and diversity of
MCQs generated with LLMs. Currently, it is not easy to understand how capable
this system is at generating entire quizzes and entire quiz pools for
particular LOs. Furthermore, we do not assess the refinement capabilities of
GPT-4 for this task. Given that GPT-4 is a chat completion model, providing
further feedback should lead to enhanced generation, and we do not study these
capabilities. There is also the need to understand the effects of different
parameter settings, such as temperature (Agarwal et al., 2023). Lastly,
extending this work to other programming languages and determining the effects
on MCQ quality and alignment with LOs would be an important avenue to explore.

## References

  * (1)
  * Agarwal et al. (2023) Arav Agarwal, Karthik Mittal, Aidan Doyle, Pragnya Sridhar, Zipiao Wan, Jacob Doughty, Jaromir Savelka, and Majd Sakr. 2023\.  Understanding the Role of Temperature in Diverse Question Generation by GPT-4.
  * Allam and Haggag (2012) Ali Mohamed Nabil Allam and Mohamed Hassan Haggag. 2012.  The question answering systems: A survey.  _International Journal of Research and Reviews in Information Sciences (IJRRIS)_ 2, 3 (2012).
  * Butler (2018) Andrew C Butler. 2018\.  Multiple-choice testing in education: Are the best practices for assessment also good for learning?  _Journal of Applied Research in Memory and Cognition_ 7, 3 (2018), 323–331.
  * Ch and Saha (2018) Dhawaleswar Rao Ch and Sujan Kumar Saha. 2018.  Automatic multiple choice question generation from text: A survey.  _IEEE Transactions on Learning Technologies_ 13, 1 (2018), 14–25.
  * Chan and Fan (2019) Ying-Hong Chan and Yao-Chung Fan. 2019.  A recurrent BERT-based model for question generation. In _Proceedings of the 2nd workshop on machine reading for question answering_. 154–162.
  * Cheung et al. (2023) Billy Ho Hung Cheung, Gary Kui Kai Lau, Gordon Tin Chun Wong, Elaine Yuen Phin Lee, Dhananjay Kulkarni, Choon Sheong Seow, Ruby Wong, and Michael Tiong Hong Co. 2023.  ChatGPT versus human in generating medical graduate exam questions–An international prospective study.  _medRxiv_ (2023), 2023–05.
  * Cho et al. (2019) Woon Sang Cho, Yizhe Zhang, Sudha Rao, Asli Celikyilmaz, Chenyan Xiong, Jianfeng Gao, Mengdi Wang, and Bill Dolan. 2019\.  Contrastive multi-document question generation.  _arXiv preprint arXiv:1911.03047_ (2019).
  * Denny et al. (2008) Paul Denny, John Hamer, Andrew Luxton-Reilly, and Helen Purchase. 2008.  PeerWise: students sharing their multiple choice questions. In _Proceedings of the fourth international workshop on computing education research_. 51–58.
  * Denny et al. (2022) Paul Denny, Viraj Kumar, and Nasser Giacaman. 2022.  Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language.  arXiv:2210.15157 [cs.HC]
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017.  Learning to Ask: Neural Question Generation for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1342–1352.
  * Duan et al. (2017) Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017\.  Question generation for question answering. In _Proceedings of the 2017 conference on empirical methods in natural language processing_. 866–874.
  * Gao et al. (2019) Yifan Gao, Lidong Bing, Piji Li, Irwin King, and Michael R Lyu. 2019.  Generating distractors for reading comprehension questions from real examinations. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 6423–6430.
  * Heilman (2011) Michael Heilman. 2011\.  _Automatic factual question generation from text_.  Ph. D. Dissertation. Carnegie Mellon University.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Jiang and Lee (2017) Shu Jiang and John SY Lee. 2017.  Distractor generation for chinese fill-in-the-blank items. In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_. 143–148.
  * Kalpakchi and Boye (2021) Dmytro Kalpakchi and Johan Boye. 2021.  BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset.  _arXiv preprint arXiv:2108.03973_ (2021).
  * Kazemitabaar et al. (2023) Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023.  Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–23.
  * Krathwohl (2002) David R Krathwohl. 2002\.  A revision of Bloom’s taxonomy: An overview.  _Theory into practice_ 41, 4 (2002), 212–218.
  * Kumar et al. (2023) Archana Praveen Kumar, Ashalatha Nayak, Manjula Shenoy, Shashank Goyal, et al. 2023\.  A novel approach to generate distractors for multiple choice questions.  _Expert Systems with Applications_ 225 (2023), 120022.
  * Kurdi et al. (2020) Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020.  A systematic review of automatic question generation for educational purposes.  _International Journal of Artificial Intelligence in Education_ 30 (2020), 121–204.
  * Leiker et al. (2023) Daniel Leiker, Sara Finnigan, Ashley Ricker Gyllen, and Mutlu Cukurova. 2023.  Prototyping the use of Large Language Models (LLMs) for adult learning content creation at scale. In _LLM@AIED_.  https://api.semanticscholar.org/CorpusID:259076210
  * Leinonen et al. (2023) Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023\.  Comparing Code Explanations Created by Students and Large Language Models.  arXiv:2304.03938 [cs.CY]
  * Li et al. (2022) Yuheng Li, Mladen Rakovic, Boon Xin Poh, Dragan Gasevic, and Guanliang Chen. 2022.  Automatic Classification of Learning Objectives Based on Bloom’s Taxonomy. In _EDM_ , Antonija Mitrovic and Nigel Bosch (Eds.). International EDM Society, Durham, United Kingdom, 530–537.  https://doi.org/10.5281/zenodo.6853191
  * Liang et al. (2018) Chen Liang, Xiao Yang, Neisarg Dave, Drew Wham, Bart Pursel, and C Lee Giles. 2018\.  Distractor generation for multiple choice questions using learning to rank. In _Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications_. 284–290.
  * Liffiton et al. (2023) Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023.  Codehelp: Using large language models with guardrails for scalable support in programming classes.  _arXiv preprint arXiv:2308.06921_ (2023).
  * Lopez et al. (2020) Luis Enrico Lopez, Diane Kathryn Cruz, Jan Christian Blaise Cruz, and Charibeth Cheng. 2020\.  Transformer-based end-to-end question generation.  _arXiv preprint arXiv:2005.01107_ 4 (2020).
  * MacNeil et al. (2023) Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023\.  Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book _(SIGCSE 2023)_. ACM, New York, NY, USA, 931–937.  https://doi.org/10.1145/3545945.3569785
  * MacNeil et al. (2022) Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022\.  Generating Diverse Code Explanations Using the GPT-3 Large Language Model _(ICER ’22)_. Association for Computing Machinery, New York, NY, USA, 3 pages.  https://doi.org/10.1145/3501709.3544280
  * Madri and Meruva (2023) Vijaya Raju Madri and Sreenivasulu Meruva. 2023.  A comprehensive review on MCQ generation from text.  _Multimedia Tools and Applications_ (2023), 1–20.
  * Nasution (2023) NEA Nasution. 2023\.  Using artificial intelligence to create biology multiple choice questions for higher education.  _Agricultural and Environmental Education_ 2, 1 (2023).
  * Offerijns et al. (2020) Jeroen Offerijns, Suzan Verberne, and Tessa Verhoef. 2020\.  Better distractions: Transformer-based distractor generation and multiple choice question filtering.  _arXiv preprint arXiv:2010.09598_ (2020).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  (2023).  arXiv:2303.08774 [cs.CL]
  * Phung et al. (2023) Tung Phung, José Pablo Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Kumar Singla, and Gustavo Soares. 2023.  Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models.  _ArXiv_ abs/2302.04662 (2023).
  * Piccolo et al. (2023) Stephen R. Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G. Ridge. 2023\.  Many bioinformatics programming tasks can be automated with ChatGPT.  arXiv:2303.13528 [q-bio.OT]
  * Prather et al. (2023) James Prather, Paul Denny, Juho Leinonen, Brett A Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, et al. 2023\.  The robots are here: Navigating the generative ai revolution in computing education.  _arXiv preprint arXiv:2310.00658_ (2023).
  * Qiu et al. (2021) Xinying Qiu, Haiwei Xue, Lingfeng Liang, Zexin Xie, Shuxuan Liao, and Guofeng Shi. 2021\.  Automatic generation of multiple-choice cloze-test questions for lao language learning. In _2021 International Conference on Asian Language Processing (IALP)_. IEEE, 125–130.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.  (2019).
  * Ren and Zhu (2021) Siyu Ren and Kenny Q Zhu. 2021.  Knowledge-driven distractor generation for cloze-style multiple choice questions. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 4339–4347.
  * Rodriguez-Torrealba et al. (2022) Ricardo Rodriguez-Torrealba, Eva Garcia-Lopez, and Antonio Garcia-Cabot. 2022.  End-to-End generation of Multiple-Choice questions using Text-to-Text transfer Transformer models.  _Expert Systems with Applications_ 208 (2022), 118258.
  * Sarsa et al. (2022) Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022\.  Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. ACM.  https://doi.org/10.1145/3501385.3543957
  * Savelka et al. (2023a) Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr. 2023a.  Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses.  _arXiv preprint arXiv:2306.10073_ (2023).
  * Savelka et al. (2023b) Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023b.  From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-choice Questions for Programming Classes in Higher Education.  _arXiv preprint arXiv:2311.09518_ (2023).
  * Savelka et al. (2023c) Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023c.  Large language models (gpt) struggle to answer multiple-choice questions about code.  _arXiv preprint arXiv:2303.08033_ (2023).
  * Savelka et al. (2023d) Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023d.  Can Generative Pre-Trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 117–123.  https://doi.org/10.1145/3587102.3588792
  * Savelka et al. (2023e) Jaromir Savelka, Paul Denny, Mark Liffiton, and Brad Sheese. 2023e.  Efficient Classification of Student Help Requests in Programming Courses Using Large Language Models.  arXiv:2310.20105 [cs.CY]
  * Sheese et al. (2023) Brad Sheese, Mark Liffiton, Jaromir Savelka, and Paul Denny. 2023.  Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant.  arXiv:2310.16984 [cs.CY]
  * Shin et al. (2019) Jinnie Shin, Qi Guo, and Mark J Gierl. 2019.  Multiple-choice item distractor development using topic modeling approaches.  _Frontiers in psychology_ 10 (2019), 825.
  * Soares and Parreiras (2020) Marco Antonio Calijorne Soares and Fernando Silva Parreiras. 2020.  A literature review on question answering techniques, paradigms and systems.  _Journal of King Saud University-Computer and Information Sciences_ 32, 6 (2020), 635–646.
  * Sridhar et al. (2023) Pragnya Sridhar, Aidan Doyle, Arav Agarwal, Christopher Bogart, Jaromir Savelka, and Majd Sakr. 2023\.  Harnessing llms in curricular design: Using gpt-4 to support authoring of learning objectives.  _arXiv preprint arXiv:2306.17459_ (2023).
  * Tang et al. (2017) Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017.  Question answering and question generation as dual tasks.  _arXiv preprint arXiv:1706.02027_ (2017).
  * Towns (2014) Marcy H Towns. 2014\.  Guide to developing high-quality, reliable, and valid multiple-choice assessments.  _Journal of Chemical Education_ 91, 9 (2014), 1426–1431.
  * Tran et al. ([n. d.]) Andrew Tran, Kenneth Angelikas, Egi Rama, Chiku Okechukwu, David H Smith IV, and Stephen MacNeil. [n. d.].  Generating Multiple Choice Questions for Computing Courses using Large Language Models.  ([n. d.]).
  * Traynor and Gibson (2005) Des Traynor and J Paul Gibson. 2005.  Synthesis and analysis of automatic assessment methods in CS1: generating intelligent MCQs.  _ACM SIGCSE Bulletin_ 37, 1 (2005), 495–499.
  * Vachev et al. (2022) Kristiyan Vachev, Momchil Hardalov, Georgi Karadzhov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. 2022.  Leaf: Multiple-choice question generation. In _European Conference on Information Retrieval_. Springer, 321–328.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2018) Zichao Wang, Andrew S Lan, Weili Nie, Andrew E Waters, Phillip J Grimaldi, and Richard G Baraniuk. 2018.  QG-net: a data-driven question generation model for educational content. In _Proceedings of the fifth annual ACM conference on learning at scale_. 1–10.
  * Wongpakaran et al. (2013) N. Wongpakaran, T. Wongpakaran, D. Wedding, and K. Gwet. 2013\.  A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples.  _BMC Med Res Methodol_ (2013).  https://doi.org/10.1186/1471-2288-13-61
  * Zec et al. (2017) S. Zec, N. Soriani, R. Comoretto, and I. Baldi. 2017\.  High Agreement and High Prevalence: The Paradox of Cohen’s Kappa.  _The Open Nursing Journal_ (2017), 221–218.  https://doi.org/10.2174/1874434601711010211
  * Zhou et al. (2020) Xiaorui Zhou, Senlin Luo, and Yunfang Wu. 2020.  Co-attention hierarchical network: Generating coherent long distractors for reading comprehension. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 9725–9732.