Programming Languages and CS2 courses. They focus on building foundational
knowledge that complements the capabilities of LLMs: “I still want the
students to be doing that work. I still want them to be practising the things
that the tool could do to build the base of knowledge so they can later do the
things that the tool can’t do.” Another interviewee shares this concern: “I’m
very worried that if everybody forgets how to write code and think through
this stuff, we will lose the ability to make new things.” At the same time,
educators seem to be aware of the need to prepare students for industry
expectations, which is likely to include LLM use.

Dan Garcia (UC Berkeley, USA) pursues a similar approach. He recognises the
potential of LLMs as educational tool, or aids, but emphasises the importance
of teaching programming basics first. Garcia encourages the use of LLMs once
students have mastered traditional programming concepts, concluding that “… we
can’t stop teaching kids how to program.” This perspective is shared by Austin
Cory Bart (University of Delaware, USA) who is not changing the learning goals
in an introductory programming class for about 280 students. Nonetheless, they
expect adjustments in subsequent courses as LLMs become more integrated into
programming practices: “But I look at almost every single course after mine
as, Oh, yeah, that’s probably going to need to change learning objectives.”

To conclude, computing educators express the need to balance between
leveraging LLMs for problem-solving while ensuring that students continue to
develop competencies in algorithmic thinking, program comprehension, and
debugging. Educators vary in their approaches, with some maintaining a focus
on teaching fundamental programming concepts, while others see LLMs as
complementary tools to enhance learning. The preservation of core learning
objectives, particularly those related to basic programming, remains a
consistent concern among educators.

####  5.2.3. Towards conversational computing

If computational thinking is the learning goal of a non-majors course, then
using an LLM-based tool such as Github Copilot may be a useful approach as
advocated by Denny et al. (Denny et al., 2023c). For students in non-computing
majors who currently only take one (or just a few) programming course(s) to
learn enough to write simple programs, it may be that using an LLM tool is all
that is needed, making a programming-specific course unnecessary (Piccolo et
al., 2023). It may even be more effective than typical courses at introducing
these students to programming and may also broaden participation in CS courses
in the future.

Michael Caspersen (It-vest & Aarhus University, Denmark) believes that LLMs
are forcing us to rethink what we actually teach our students, and encourages
us to view them as an opportunity. According to Caspersen, LLMs do not add
something qualitatively new, but quantitatively indeed! They emphasise issues
that have always been present. Hence, LLMs may even contribute to increasing
the quality of computing education, thereby making it more attractive to a
broader range of students.

###  5.3. Course activities

LLMs can be useful in generating several kinds of learning activities
including novel variations of programming assignments (Sarsa et al., 2022).
However, this may introduce problems such as ensuring that students have been
provided appropriate content in order to understand novel variations. LLMs can
also generate good explanations of code (Sarsa et al., 2022; Leinonen et al.,
2023a; MacNeil et al., 2023b; Hellas et al., 2023). This provides a mechanism
for novices struggling to understand the run-time behaviour of novel problems
to get auto-generated and hopefully helpful explanations. This use-case
exemplifies the potential for LLMs to ease the burden often felt by teaching
assistants, and could be a first line of help for students (Sarsa et al.,
2022). Another type of learning activity that can be provided by LLMs occurs
in settings where students use them to help generate code solutions. Here,
students can use LLMs in an iterative improvement loop. Students can
continually alter prompts to refine the model output helping them to ‘build-
up’ a solution (Yuan et al., 2022). In their interview Mark Liffiton took this
concept one step further stating “So they’re great educational tools because
they can give something akin to one-on-one tutoring… and I think there’s a ton
of value in there.”. This particular use-case of LLMs is less about what LLMs
can produce, but what they can do for students (and educators) and aligns with
what the Artificial Intelligence in Education (AIEd) community has been
discussing for years (Becker, 2017).

Liffiton has also developed a tool called CodeHelp (Liffiton et al., 2023)
that uses LLMs that he is going to use with his students which can do some of
what ChatGPT can do, but specifically does not solve the problem, and does not
give students complete solutions - a kind of “sanctioned” access to the
educational power of LLMs which could combat the concern of students becoming
over reliant on them, and not actually learning from them. Mark sees this as
adding to the course learning objectives to include working with this new tool
and therefore LLMs. Similarly, Frank Vahid sees LLMs as tutors that will be
available around the clock, and importantly, as tutors that will not judge
students, stating “I’m very hopeful that it will become another TA for the
class.”

An important aspect of teaching is using carefully crafted examples to
illustrate salient points. If the goal is to use a _real_ or _running_
example, it can be tedious to have to deal with the many irrelevant (to the
example at hand) aspects of a problem in order to ensure the example compiles
and runs, in addition to keeping focus on the point desired. LLMs can help
instructors with the tedium of such minutiae (Sheard et al., 2017; MacNeil et
al., 2023b).

Clearly, many of the student-initiated possibilities discussed above come with
academic integrity concerns we discuss in Section 7. Several interviewees were
aware of this, stressing the need for educators to emphasise the importance of
students doing their own work.

Perhaps the most extreme example of how Generative AI might be used in the
introductory programming course comes by way of Dan Zingaro and Leo Porter’s
new textbook (Porter and Zingaro, 2023). The book begins by introducing
students to the GitHub Copilot plugin within the Visual Studio Code IDE before
students have learned to write a single line of Python code. Students create
their first programs by typing English comments and letting Copilot generate
the code. This corresponds to the _sketch model_ from Alves and Cipriano’s
Centaur Programmer (Alves and Cipriano, 2023) where the programmer generates
the outline of the solution and the AI fills in the gaps. As Zingaro and
Porter explain each line of LLM-generated code, they use the opportunity to
teach the related Python syntax and programming concepts. But before they do
this, they introduce functions and use this to motivate top-down design.
Porter will be teaching 700 students using this approach in the upcoming
semester (September 2023).

####  5.3.1. In-class Activities

LLM-based tools have motivated some changes to activities that specifically
happen during classes. Denny et al. (Denny et al., 2023c) found that Copilot’s
performance is substantially improved when it is prompted with individual
problem-solving steps in natural language and encourage explicitly teaching
prompt engineering to students. David H. Smith IV (University of Illinois,
Urbana-Champaign, USA) will be using a tool and specific exercises for
students to practice LLM prompts so they can use LLMs effectively. Leo Porter
also stated that they will be adding a learning goal on prompt engineering as
discussed earlier.

Although not planning on getting into prompt engineering, Briana Morrison is
going to do more live coding with LLMs in the classroom including analysing
why certain code is wrong. Viraj Kumar has also used Generative AI for live
coding. Jérémie Lumbroso (University of Pennsylvania, USA) created guided
sessions using LLMs and shows examples of them giving incorrect answers.
Similarly, Kristin Stephens-Martinez told us that they plan to work out
examples in advance of using prompts that demonstrate a hallucinated answer or
other technically-incorrect response to help students see for themselves that
LLMs are not oracles. Austin Cory Bart is going to demonstrate the use of LLMs
in class for three reasons: First, because LLMs are a great way to introduce
AI topics, also providing a way to bring advanced material into the
introductory course so students can look forward to what is coming later;
Second, because Bart feels that if it is not discussed with students they’ll
just use it anyway, but in a more misguided manner. Finally, Bart stated: “At
some point we have to start incorporating these tools… programmers are going
to be putting these tools into the workflow. And I see that on my own… when
I’m coding this summer having co-pilot auto-complete an entire function that I
just started writing, that’s too much of a game changer for it not to be
addressed.”

Interestingly, Leo Porter is the only interviewee who mentioned pair
programming directly. Porter plans on continuing to use pair programming and
peer instruction in class. Dan Garcia noted that LLMs should be used as
nudgers, hint-givers or help-givers, but not oracles – seeming to match the
desired role of a human pair programmer, and Michael Caspersen noted that LLMs
should be integrated in peer to peer learning. Others in the literature have
speculated on what may come of pair programming in light of LLMs. For
instance, Dakhel el al. noted that when used by experts, Copilot can be an
asset as its suggestions could be ‘comparable to humans’ in terms of quality.
However, it could become a liability when used by novices who may fail to
filter its buggy or non-optimal solutions due to their lack of expertise
(Dakhel et al., 2023). Wermelinger stated that it is not surprising that
GitHub dubs Copilot as ‘your AI pair programmer’, even though the interaction
is far more limited than with a human – notably, Copilot does not provide a
rationale for its suggestions (Wermelinger, 2023). However, Frank Vahid did
note a positive aspect of how many Generative AI tools present their output
noting positively that LLMs generally do not judge the student.

Do we stop teaching long division now that we have calculators? Do we let
students use calculators when they are learning long division? No and no.

However, in the same way that they expect students to learn to use an IDE on
their own time, some instructors have indicated that they will not be
dedicating classroom time to explicitly teach students LLMs. Rodrigo Duran is
not actively encouraging LLM use but at the same time is not actively
discouraging their use. Dan Garcia does not plan on incorporating LLMs into
their course, at least for the time being, choosing to keep their ‘ear to the
ground’ and see what others are doing, and stressing that the fundamentals are
important: “Do we stop teaching long division now that we have calculators? Do
we let students use calculators when they are learning long division? No and
no.” Peter Mawhorter does not plan to allow LLM use in their course out of
fear of harming particular students, expecting that a small percentage of
students – likely those who are marginalised in other ways – will have bad
experiences due to biased, possibly sexist or racist output. It is also
possible that these tools will give different output to different students,
for instance based on the student’s name if provided in starter code or a
prompt.

####  5.3.2. Unsupervised Activities

Student use of LLMs in unsupervised learning activities, self-assessments, and
other, self-directed scenarios have also been the subject of research. MacNeil
et al. (MacNeil et al., 2023b), for example, used GPT-3 and Codex to generate
three types of code explanations (line-by-line explanations, lists of
important concepts, and high-level summaries) from code snippets from an
instructor-developed web software development e-book. They found that Codex
generated less helpful and more verbose explanations than GPT-3. Moreover,
Codex included code in the explanations, even though it was not desired.
Students rated these explanations on a 5-point Likert scale, confirming that
explanations matched the code and were useful for learning in general, whereas
line-by-line explanations were rated as least helpful.

An exploration of the potential of LLMs to generate formative programming
feedback (Kiesler et al., 2023) suggests that ChatGPT performs reasonably well
for some introductory programming tasks and student errors indicating that
students can potentially benefit in unsupervised learning scenarios. However,
it is possible that it will fall on educators to provide guidance on how to
use the generated feedback, as it can contain misleading information for
novices. Several of the approaches discussed in the prior section are aimed at
mitigating this.

Interviewees mentioned unsupervised activities less than in-class activities
in general. This could be at least partially due to the fact that this is the
first academic year where Generative AI could be considered mainstream and
unsupervised activities are more difficult to plan and hypothesise about. Most
of the discussion around unsupervised activities centred around ensuring that
students are doing their own work (even if using Generative AI as a tool for
help) and that students understand the code that these tools produce. Frank
Vahid believes that this could lead to increased instructor emphasis on tools
that analyse student behaviour.

In a high school introductory programming course, Christian Tomaschitz (TU
Wien, Austria) had half of his students use ChatGPT and the other half use an
e-book. ChatGPT was not introduced to students and students registered OpenAI
accounts with their school e-mail addresses. All students had the same exam
which was half open-book (including internet access), half closed-book.
Tomaschitz noticed that the ChatGPT students had the possibility to interact
(with ChatGPT), and these students were faster solving the problems, but it
seemed as if they did not critically reflect the output, and understood less.
Tomaschitz was concerned that the ChatGPT group was over-reliant on the tool
and did not understand the output as well as the e-book group.

Viraj Kumar used GitHub Copilot during class for live coding and told students
they were free to use it, or any other Generative AI for coding. Students were
polled after the course and asked if they were using ChatGPT. Most said that
they were not. The consensus explanation seemed to be that they wanted to
learn programming themselves and their thinking seemed to be that when they go
for technical job interviews they will be expected to code without assistance.

Several interviewees including Leo Porter and Michael Caspersen believe that
LLMs will cause educators to focus more on code reading and less on writing
from scratch. Caspersen noted that students read more than they write when
learning to read natural language, but to-date this has not been the
traditional approach when it comes to programming. Further, LLMs could support
a ‘use, modify, create’ approach for programming. Briana Morrison envisions
more exercises about why certain code is wrong and fewer on writing code from
scratch.

Kristin Stephens-Martinez mentioned that they are concerned about LLM use by
novices pushing more metacognitive practices earlier in the curriculum,
perhaps before students are ready for this. For instance, students will have
to ask themselves ‘am I going to take this shortcut or not?’. Getting students
to recognise when it is a shortcut versus when it actually is not a shortcut
is something that novices typically are not in a position to assess.

###  5.4. Assessment

Although we previously discussed activities that could have been assessed, in
this section we focus specifically on formal assessment. Research on LLMs has
demonstrated their ability to answer typical CS1 assessments that involve
writing simple functions (Denny et al., 2023c; Kiesler and Schiffner, 2023a)
as well as more advanced material. Similarly, it has been shown that they
perform in the upper quartile of real students on CS1 exams (Finnie-Ansley et
al., 2022a). It has also been shown that they perform just as well on CS2
exams as CS1 exams (Finnie-Ansley et al., 2023) suggesting that LLMs may soon
be capable of effectively solving even more advanced problems. Indeed, recent
results with GPT-4 suggest that it can solve most exercises in introductory
programming courses (Savelka et al., 2023a), which is also supported by our
benchmarking work presented in Section 8.

However, evidence also shows that LLMs do not perform as effectively on
computational thinking problems that do not involve code writing (Bellettini
et al., 2023). Similarly while LLMs can often correctly answer more than half
of coding-based multiple-choice problems, they answer double-digit percentages
of multiple-choice problems incorrectly, leading to the hypothesis that either
a combination of natural language and a code snippet, and/or chain-of-
reasoning steps pose a challenge for LLMs (Savelka et al., 2023c). It might be
tempting to think that even though LLMs can do well on relatively simple well-
specified functions, longer and more complex problems are beyond the
capabilities of LLMs – although LLMs have been shown to perform well on more
advanced (coding competition) problems (Li et al., 2022a).

####  5.4.1. Exams

Some instructors are moving towards invigilated exams and assessments, and
these may be worth more marks. For example, one interviewee changed the grade
weighting of their programming assignments from 50% of the course grade to 0%
of the course grade. They added an assessment category, “coding interviews”,
which ensures that students are not using LLM tools as part of the assessment.

Several educators mentioned oral exams. Jean Mehta plans to use 20-30 minute
one-on-one oral exams at the end of every section. This is made possible by a
combination of a flipped classroom with many videos and book/autograder
technology such that there is less need for traditional lectures covering the
material. Michael Kölling said, “I think there needs to be at least some
assessment that includes an oral element because you know, there is at the
moment, the problem is that submission of written work, whether it’s text or
programs, is taken as a proxy for intellectual achievement, right? And what we
actually want to assess is intellectual achievement. We want to see some, you
know, sort of intellectual work having happened there and we take the written
work as evidence of that. And you know, these tools have removed that
connection… The creation of written work is no longer evidence of intellectual
work having happened.”

Similar to personalising assignments, on open-ended writing assignments, Ewan
Tempero requires more specific answers directly related to course materials:
“And it’s not that we used strange terms or unusual terms, but we used
specific terms in a particular way. And so, we expected answers to reflect
that, to demonstrate that, yeah, they did actually understand what the course
material was.”

####  5.4.2. Homework

There may be less summative unsupervised work because instructors no longer
trust that unsupervised work is the students’ own. This may be accompanied by
a tendency to not grade code assignments going forward. Along with increasing
the weight of exams, many teachers are devaluing “homework” assignments
altogether. Mark Liffiton said, “I will sort of be operating in this
assumption that some students may end up just getting a tool to do the work.
And thus, I don’t want to be putting too many points on that and giving an
unfair advantage in those cases.” One instructor decreased the grade weighting
of their programming assignments significantly and added this text to the
assessment description, “As the programming assignments are intended primarily
for practice and learning, your program does not have to be fully correct to
receive credit. The final evaluation of whether you have learned from your
programming assignments is in your ability to solve problems on quizzes and
the exam.”

Some teachers no longer require “writing assignments” in the traditional
sense. Jan Schneider (Goethe Universität, Germany) has a course that used to
include writing a scientific paper, but it has been changed to allow students
to produce it using LLMs. “I mean, the main objective is to help people to
start thinking in a more scientific way. That’s the overall objective…. I will
not ask them to develop a mini paper by scratch where they need to write
everything.”

####  5.4.3. Process over product

Rather than assessing final solutions as products, some educators are
increasingly focusing on assessing learning processes (which is common in
other disciplines, e.g., teacher education) such as: submit-in-stages;
solution reflection / commentary / critique; interviews; portfolios or
learning journals; and presentations. Taking the approach of having students
focus on the learning process through a diary or journal, Christian Tomaschitz
replaced several previous exercises with reflection assignments in a diary for
students to document their learning process.

In relation to more open-ended assignments, Leo Porter said, “… gone are the
days of us really just completely describing the exact behaviour of the
functions… And then it’s going to be a lot more work for us to grade because
we’re going to have to now look at the code or the PDFs of the code. Or look
at the video of them showing how the code works.” This approach also shifts
focus from memorising knowledge (or in a rote manner, the process that leads
to the product without questioning the process) to application of skill and
critical thinking. Jérémie Lumbroso applies this shift to his Discrete
Mathematics course, explaining that “the idea is to focus less on the product
and to focus more on process, on making sure that the students are able to
explain what they are doing.” Some educators have already started using LLMs
as part of the learning process. Briana Morrison explains a possible
assignment thus: “Here’s the problem. Here’s the prompt we gave ChatGPT, or
Copilot, or whatever. Here’s the output it gave us. It’s wrong. Tell us why
it’s wrong.”

Several educators worried that some classes do not lend themselves to
approaches mentioned above. For example, elementary theory courses are not
really amenable to “personalise” a proof. Michael Caspersen, on the importance
of basic competencies, even in the face of powerful LLMs stated “And that
means that if you add a good programmer to large language models, you get two
good programmers. Basically, that’s the equation, right? If you add a mediocre
programmer to large language models, you get just large language models. So,
from that perspective… if you want to really be able to amplify the
capabilities of humans, we need to make sure that the basic competencies
remain.”

Frank Vahid stated: “The biggest challenge is cheating … you’ve gotta learn …
you’ve got to work to learn. You can’t just let tools do your work for you…
and this is true beyond computer science. In English, you’ve got to learn to
write. Even though ChatGPT can do most of your writing for you… you’ve got to
learn to write… that’s how you think. That’s why you’re valuable as a human to
a company… and so the same with computer science. So somehow, we’ve got to get
that message out. It’s just so tempting for students to save so much time.”

###  5.5. Institutional initiatives, policy, and context

Although some institutions (at least initially) have universally banned the
use of LLM tools in student work (Nolan, 2023), others are starting to embrace
them. There is little doubt that this will lead to an array of policies and
initiatives that may be at the university, faculty, or class levels. Given
that public awareness about LLMs occurred mid-academic year for many
institutions (almost universally for North America and Europe in addition to
many other parts of the world) this September marks the first academic year
where LLMs are a nearly ubiquitous topic. Given that institutions are
typically slow to react mid-year – if they react at all, we have yet to arrive
at a steady-state in terms of institutional initiatives. Nonetheless they are
beginning to emerge as educators start thinking about the coming academic
year.

Peter Mawhorter’s local policy is that LLMs are not allowed in class. In
delivering this message to students Mawhorter aims to have a discussion with
students on why that policy exists. On the other side of the coin Leo Porter
is embracing LLMs and is aiming to use them, for instance, in quizzes. However
Porter’s institutional challenge is finding and organising enough computer-
based testing facilities, and one example of how LLMs can have knock-on
effects that could not only affect the course in question, but others via
resource allocation and timetabling. Michael Caspersen set out a middle
ground, starting with basic competencies and gradually building-in the use of
LLMs letting higher levels of the SOLO taxonomy (Biggs and Collis, 2014) come
into focus.

Sven Strickroth (LMU Munich, Germany) noted another challenge that is caused
by institutional policy – significant changes to learning objectives are not
easily possible due to the local accreditation cycle which dictates that this
occurs only every five years.

###  5.6. Other challenges and opportunities

At the end of the interviews, we asked interviewees for their views on the
challenges and opportunities they foresee in terms of the effects that LLMs
will present in the introductory programming course. While some of these have
already been discussed, several have not - some of these are presented here.

####  5.6.1. Challenges

  1. (1)

Jan Schneider noted that everyone (including LLMs) have a lot of blind spots,
and these can be found by writing – either in code or in natural language –
like when other people or a LLM try to understand what we wrote, that is when
blind spots appear, often in the form of ”whoa, there’s a blind spot…
something I didn’t know that I was missing… and I have a big fear that if we
start using these large language models, we will never acknowledge our blind
spots. And we will miss a lot in learning and developing.”

  2. (2)

Mark Liffiton mentioned that it is a challenge now to make sure students are
learning things that they could just have a tool do faster, and the concern of
over-reliance – not learning the things they would have if they did not use
the tool to do it for them, noting that it is like cheating in a way – not
learning the things you would if you had done the work yourself.

  3. (3)

Frank Vahid mentioned that “you need to think”. A human is only useful because
they are clever, and thoughtful, and intelligent. That is why we teach them
programming – because it is a way to help them learn to solve problems – even
though we have calculators we still should know how to do arithmetic, despite
the existence of calculators.

  4. (4)

Michael Caspersen fears that LLMs will enable disciplined students to become
better, but for undisciplined students who are seeking the easy way out –
danger! Caspersen noted that it should not be considered the student’s fault
for taking the easy way out. It should be turned into a challenge for
educators to come up with assessment systems that do not have an easy way out.
Caspersen also noted that in many ways LLMs do not add something qualitatively
new, but emphasise issues that we have been dealing with for a long time. This
was corroborated by Michael Kölling who stated that it is the scale of these
issues exploding that is novel, for instance the illusion of achievement
(which is nothing new) and what that could do to learning at scale. Kölling is
also concerned with intellectual laziness noting that learning is a struggle –
learning only happens when you intellectually struggle with something, and if
LLMs offer an easy way out, then is learning happening? Kölling also mentioned
cheating as a challenge, but that this is obvious, boring, and solvable.

####  5.6.2. Opportunities

  1. (1)

Michael Caspersen sees a big opportunity in terms of rethinking what we are
doing. “We should think deeply about what we actually teaching our students -
and LLMs are doing that… [LLMs] radically challenge our reflections on what to
teach, what to assess, how to assess. So that’s a great opportunity.”

  2. (2)

Michael Kölling sees individualised learning in terms of progress, interests,
feedback, and help as a big opportunity, noting that humans saw similar issues
with books and the printing press. There was concern that people would not
need to remember anything any more. However, we lived. “In fact books made
things better, right?”

  3. (3)

Dan Garcia sees potential in scaling support which benefits educators,
institutions and students, positing “Imagine an LLM that could examine every
exam script and where mistakes were made give a whole concept map of what went
wrong in the notional machine and where and how. I can try this for a few
students but I have over 1,000. This could help scale support for everyone.”

###  5.7. Discussion

Above we have discussed issues raised by the expert instructors that we
interviewed. In addition to those, we would like to discuss a certainly non-
exhaustive set of issues we believe are important for instructors and CS
program designers to consider presently.

The potential biases reinforced by models trained on large datasets (D’ignazio
and Klein, 2020) are concerning. This could be especially important for
instructors who use LLMs to create course materials. Another issue is the
presence of hidden or implicit learning objectives in existing computing
programs – something Leo Porter described regarding problem decomposition in
Section 5.2.1. There is little doubt that there are other such hidden learning
objectives spanning not only knowledge, and skills, such as reading and
tracing code but also dispositions, inter- and intrapersonal competencies, and
other aspects relating to the whole person (Raj et al., 2021) which the
emergence of LLMs might bring to the fore. For example, many programs expect
their graduates to be comfortable developing large programs using (new)
debugging tools, work in a self-directed manner but also perform well in
teams, and overcome challenges to pursue their goals long-term. But curricula
often do not explicitly include these program-level outcomes in the learning
outcomes for a particular course (Becker and Quille, 2019; Kiesler, 2022). The
same is true for course activities and assessments. As individual instructors
respond to the landscape changes induced by LLMs, it becomes more important
than ever to consider and implement the constructive alignment (Biggs, 1996)
of individual course activities and desired program-level outcomes. This is
not a new concern for educators, but one that has been amplified by the rapid
change in teaching and assessment settings we now see happening (or about to
happen).

Related to this is the potential for LLMs to change the workplaces into which
we are graduating students. While a number of researchers have looked at how
current developers may use LLM-based tools (Bull and Kharrufa, 2023; Barke et
al., 2023; Dohmke et al., 2023) and how LLM-tools may be incorporated into
professional software development tools (Ernst and Bavota, 2022), the
participants in these studies have been programmers who learned to code
initially without using LLMs. Although professional developers may benefit
from AI’s human-quality suggestions, novice developers lack the expertise to
recognise and understand buggy or non-optimal solutions (Dakhel et al., 2022).
The use of AI tools could become a liability if inexperienced developers fail
to remove or correct the tool’s incorrect suggestions (Dakhel et al., 2022).
Potentially more dangerous is the fact that students can now execute code that
they do not understand, yet designed through natural language prompts.

It remains to be seen whether students who have LLMs available from the start
will have the discipline and dispositions to develop programming competencies
deeply or whether this will even matter. Reminiscent of the 1990s discussions
of objects-first or objects-later, the opportunity to focus first on top-down
design but have a working code for interesting problems completed by the AI is
not universally recognised as a positive development.

LLMs also impact peripheral and applied computing fields. For example, it was
shown that LLMs can successfully solve 97% of the programming problems in a
bioinformatics course (Piccolo et al., 2023). The authors conclude that the
models perform so well that bioinformatics students in the near future may no
longer need to know how to write (and likely not understand) code. As a
consequence, several questions arise on the effects of LLMs not only on other
applied areas of computing (e.g., data science; digital forensics; security;
and games development) but also on programming as one of the core tiers of
every computing degree. Related to that are questions about how the
introduction of LLMs might affect participation in the computing field.
Perhaps reducing the focus on syntax will make the field more attractive to
traditionally under-represented audiences and increase retention rates.
Additionally the influence of media coverage of computing topics is known to
be a large factor in the decisions pre-university students make in terms of
what courses to pursue at university. It remains to be seen what effects the
intense media hype surrounding LLMs will have on future computing intakes.

We anticipate more changes in learning objectives, course contents, learning
activities, and assessments, which will, in turn, affect whom we teach and why
in the (near) future. This might go hand in hand with long overdue changes in
computing’s signature pedagogy (Shulman, 2005), and its implementation on the
surface, deep, and implicit dimensions. For decades, computing education
researchers have presented excellent research on how to better teach our
discipline, yet much of this has not made it into practice. We believe that
the emergence of LLMs may finally force much-needed (and long ago intended but
not yet fully implemented) change.

Advice for educators: • Acknowledge the existence of LLMs with your classes
regardless of if you embrace them or do not allow their use. • Make clear and
discuss instutional and class policy, what it allows, what it does not allow,
and why it is that way. • Assume that students are using LLMs even when not
permitted. • Do not underestimate the ability of LLMs to produce solutions to
your activities (which may be indistinguishable from student-generated
solutions). • Consider using an LLM tool to help in generating course
materials. If you do this, be aware of possible bias in the output. •
Reconsider your learning objectives in terms of their relevance to preparing
those students who are aiming for careers in the software development industry
(which is increasingly making use of LLMs in day-to-day work). • Reconsider
your learning objectives (e.g., reading and understanding code), learning
activities, and assessments to assure your courses remain constructively
aligned. • Interrogate your learning objectives and ask what might be hidden
or implicit and which LLMs might provide a vehicle for more focus.
Correspondingly, interrogate your learning outcomes and ask which might be
over-emphasised (e.g. code writing) that might need to be balanced with those
that LLMs bring to the fore. • Consider using LLMs in your course if only to
provide a chance for students to receive more feedback, and practice
independently, provided they are equipped to interpret LLM output in a way
that facilitates learning.

###  5.8. Limitations and threats to validity

We used three sources to build a list of educators to invite for interviews,
the SIGCSE mailing list (via a reply to a message about LLMs), the opportunity
for those responding to the instructor survey to volunteer to interview, and
authors’ own networks. Perhaps as a result of this our geographic
representation is skewed. As expected the United States forms the bulk (55%)
of responses. Although our interview pool spanned five continents, we had no
interviewees from Africa and only one interviewee each from Asia (India),
Oceania (New Zealand), and South America (Brazil). Unfortunately, only 14% (3)
of the 22 identified as women.

Additionally, the interviews were semi-structured and although this is a
common approach it can impose interviewer bias, although it also designed to
result in a coherent set of interviews which focus on common topics while
still allowing interviewers to express their own views and experiences.

Finally, we did not attempt to verify claims that interviewees made about
their classes, departments, or institutions, taking interviewee statements at
face value.

##  6\. Ethics

Ethics of algorithms (including AI systems such as LLMs) is concerned with the
societal context around algorithmic systems and how these systems affect both
individuals and society. Our focus must therefore rest on aspects such as the
provenance and quality of training data, the usage of AI systems, and
associated costs (in the widest sense), rather than on how to “integrate
ethics into the system” (Bender et al., 2021; Johnson and Verdicchio, 2023).
Moreover, Mittelstadt et al. (Mittelstadt et al., 2016) point out that
algorithmic systems tend to be large, complex and highly modularised, which
makes it difficult in general to assign responsibilities. Hence, with an
increasingly opaque training procedure of LLMs and a lack of clear
responsibility in terms of authorship (Samuelson, 2020), the use of large
language models naturally raises a number of concerns pertaining to academic
integrity. Furthermore, using a large language model has been shown to affect
the user’s own opinions (Jakesch et al., 2023).

While a full discussion of the ethics of large language models in computing
education is beyond the scope of this paper, we would like to highlight three
crucial aspects: the role and stance of us as professionals in computing
education, the policies brought forward by academic institutions, and the
question of academic integrity in the context of large language models (which
we will discuss in Section 7).

###  6.1. Ethics in LLM literature

In a study on the values encoded in the ML research literature, Birhane et al.
identified a number of ethical values (Birhane et al., 2022), from which the
papers would draw motivation. In a total of 100 highly cited papers, the study
found that _performance_ was clearly the most frequent value and pointed out
that performance is not a neutral term but comes with ethical implications.
For instance, performance is typically measured with respect to specific
benchmarks and data sets, thereby introducing bias—particularly when the
dataset is thought to represent the “real world”.

A full study of all values and their ethical implications found in the
computing education literature is beyond the scope of this paper. However, we
extracted explicitly stated motivations from the papers in our literature
review as a first approximation to a full extraction and coding of values.

In line with Birhane et al. we observed that _performance_ , _generalisation_
, _efficiency_ , _novelty_ , and _scalability_ were often mentioned. Many
papers cited the “impressive” or “human-level” performance of current large
language models. In contrast to what Birhane et al. report, the focus on
performance in our dataset seems to be secondary as a means to highlight the
timeliness of the research. This is particularly the case since the papers in
our study did not propose performance improvements, but rather built on
available performance. Furthermore, while performance in the ML research
community typically relates to specific (and often well-known) benchmarks, we
found that performance in the papers we reviewed usually referred to either
the LLMs’ ability to solving exercises, assignments or passing exams, or the
LLMs’ production of teaching materials, say, such as exercises.

With this focus on application of large language models rather than the design
of a new AI system, we also found that several papers pointed out the
potential to “save time” or help instructors cope with growing class sizes.
While these motivators may be understood as scalability issues, we believe
that there is a difference in how this term would be understood in the paper
surveyed by Birhane et al. Despite ostensible similarities concerning the
underlying values encoded in the research literature, there might be some
notable differences.

The free availability of large language models (i.e. that students can access
LLMs at no cost) was a recurring theme in our surveyed literature. We would
like to highlight this notion as an example of an ethically problematic
assumption for two reasons. On the one hand, availability at “no cost” is
often inaccurate because the costs might be hidden and paid, e.g., through
provision of private data. On the other hand, ChatGPT offers a range of models
with different performance characteristics, not all of which are free. Some
students might therefore have access to more powerful tools than others.
Investigating the assumptions, beliefs, and values held by the computing
education community itself might therefore be well warranted. We call on the
community to do so in future work.

###  6.2. Code of ethics implications

The IEEE Code of Ethics (iee, 2020) comprises three sections. The first
focuses on ethical standards, behaviour and conduct. The second section
focuses on ethical treatment of other people, and the third focuses on
compliance. The AAAI Code of Professional Ethics and Conduct (aaa, 2019) was
adapted from the ACM Code of Ethics and has the same structure. The ACM Code
of Ethics (acm, 2023a) comprises four sections. The first section outlines the
fundamental ethical principles that all computing professions should use to
guide thinking. Section 2 describes the ethical responsibilities of computing
professionals, section 3 covers ethical leadership, and section 4 focuses on
compliance. In the following sections, we use the ACM general ethical
principles to frame the discussion of ethical issues raised by the use of
large language models in computing education.

#### ACM General Ethical Principles

In this section we review policies from major universities around the world on
the use of large language models in education. As of this writing, many
universities do not currently have an official policy publicly available
online. Instead, many universities are still presently working through the
policy implications of large language models through task forces and other
initiatives, such as at the University of Virginia (UVA, [n.d.]), which is a
top-ranked school for computer science in the USA. See Figure 2 for responses
to the question “The policies at my university are clear regarding what is
allowed and what is not allowed in terms of using GenAI tools”, which
illustrates this point.

To structure our review, we consider the first part of the ACM General Ethical
Principles. Parts two through four of these principles were too specific for
most university policies and therefore not as relevant to the present work.
Most universities did not explicitly mention coding in their policies, with
the exception of Yale University (Yal, 2023) and University of Adelaide (Ade,
2023).

To select universities for consideration of their LLM policies, we first found
popular rankings of universities worldwide for computer science programs.
Next, we examined policies at these universities, specifically those from
Canada, USA, UK, and Australia: University of Toronto (UTo, [n.d.]), Duke
University (Duk, 2023), Yale University (Yal, 2023), Massachusetts Institute
of Technology (MIT) (MIT, 2023), University of California Los Angeles (UCLA)
(UCL, [n.d.]), University of Adelaide (Ade, 2023), Monash University (Mon,
2023), and Oxford Brookes University (Oxf, [n.d.]). We do not consider this a
systematic attempt, nor would that be presently possible, given the conditions
described above. Instead, this represents a purposeful sampling of top-ranked
universities around the world to get an overall picture of how universities
are responding to the appearance of LLMs.

The ACM General Ethical Principles are divided into the following sections:

##### Contribute to society and to human well-being, acknowledging that all
people are stakeholders in computing.

Only MIT suggested that the arrival of LLMs into education is an opportunity
to think about student academic well-being (MIT, 2023). No universities
sampled contained anything about general human well-being or the idea that we
are all stakeholders in computing, or by extension, education or society in
general. It seems the ideas in these policies are limited to the specific
implications of using LLMs, rather than the general. We find this to be an
unfortunate oversight because institutions can help guide students’ concerns
beyond themselves and their immediate circumstances to the broader collective
human project of the pursuit of knowledge.

##### Avoid harm.

Use of LLMs can lead to poor outcomes due to over-reliance, ease of breaching
academic integrity, and use of incorrect information leading to poor products.
The universities we sampled all drew attention to the potential to create harm
to others, institutions, and even the students themselves. Harm to others
could come in the form of using the work of others without citation (Duk,
2023). Harm to institutions can come in the form of students using LLM tools
to generate work that is incorrect, offensive, or otherwise inappropriate and
therefore dragging the university into potential altercation. It can also call
into question the legitimacy of the learning outcomes and verification of
them, which can jeopardise accreditation and institutional reputation or the
community’s trust in that institution.

Universities seemed most interested in helping students to understand the
potential harm that students could incur upon themselves through inappropriate
use of AI resources. Isserman writes that “plagiarism isn’t a bad thing simply
because it’s an act of intellectual theft — although it is that. It’s a bad
thing because it takes the place of and prevents learning.” (Isserman, 2003).
One of the most significant concerns is the ease with which LLMs can produce
the output that we ask students to produce. As educators, we are not
interested in the output per se – rather we want students to engage in
activities and processes that result in learning. Using a tool to produce the
required output circumvents that learning, and deprives students from the
opportunity to learn (Isserman, 2003). Many of the policies we sampled
mentioned that students can prevent the growth of critical thinking skills and
competencies in crucial areas by taking shortcuts through inappropriate use of
these models (UCL, [n.d.]; Yal, 2023; Mon, 2023; Oxf, [n.d.]). This is
particularly important when writing code (Yal, 2023). Not only does this harm
their short-term ability to pass the course, but it robs them of their long-
term ability to master the subject matter and preparedness for future work.
Finally, harm could come from inadequate understanding or preparation to self,
others, and institutions in courses where safety concerns are paramount, such
as a chemistry lab. Skipping crucial learning could harm everyone involved
(Ade, 2023). While most computing courses do not share similar safety
concerns, it is possible in industry that inadequate learning or over-reliance
on these tools could expose self, others, and institutions to harm, such as
writing malformed code for self-driving cars.

##### Be honest and trustworthy.

Most universities that we sampled wanted to make it clear that LLM tools are
not reliable and can produce incorrect or fake results. Duke warned faculty
and students that LLM “output is only as good as its input” (Duk, 2023).
Others warned of the now well-known phenomenon of LLM “hallucination” where
they will reply with incorrect data when not enough is available (UCL, [n.d.])
or create sources and facts even when enough data is available (Yal, 2023;
Mon, 2023; UTo, [n.d.]). It could also be that the data is incorrect simply
because it is out of date (Ade, 2023), since many LLMs do not have access to
data created after they were trained. It is clear that universities are
thinking of honesty and trustworthiness in terms of the output of the tool and
when evaluated this way LLMs lack credibility.

##### Be fair and take action not to discriminate.

Almost all university policies and guidelines made it a priority to discuss
biases that exist in AI in general and LLMs in particular. Duke, UCLA, and
Oxford Brookes pointed out that the models themselves often discriminate based
on their training data, which means it receives all the stereotypes and
misinformation from whatever data it is based on (Duk, 2023; UCL, [n.d.]; Oxf,
[n.d.]). Monash University warned faculty and students that LLMs may take data
out of context, cannot predict future events with any amount of accuracy, and
could present certain sensitive data inappropriately (Mon, 2023). Finally, MIT
noted that fairness could be at risk based on who has access to the models
(MIT, 2023). While access to several popular models is currently free, this
may not always be the case, and therefore requiring students to use them could
become discriminatory in the future.

##### Respect the work required to produce new ideas, inventions, creative
works, and computing artefacts.

University policies seemed to mostly skip this criterion. However, there were
a few statements related enough to discuss here. For instance, Duke emphasised
that creative writing or coding meant doing the hard work of developing each
person’s specific voice. Using AI tools could undercut that endeavour and
cheat the student of their ability to develop innovative ideas, computing or
otherwise (Duk, 2023). This could be because, as Adelaide noted, AI tools lack
originality and common sense (Ade, 2023). Related to new ideas and innovation,
UCLA noted in their policy that AI tools will often reflect outdated
information that may fail to represent the progress of social movements since
the training of the model was completed (UCL, [n.d.]). If certain sets of
rights were not legal when the model was trained, one should not expect the
model to suggest that they should be.

##### Respect privacy.

The lack of control over these tools, as well as what data they keep about
their users, was mentioned in most policies. AI tools could invade users’
privacy (Duk, 2023; MIT, 2023), violate FERPA (a student privacy protection
law in the United States) if student records are handed to it (UCL, [n.d.]),
do not assume that users are at least 18 years old (Mon, 2023), and are not
bound by university ethics rules and policies. Furthermore, AI tools will
often take user data and use it to train their models, whether users want that
or not. Monash encouraged students and faculty to consider that their data
could be stored by the model and used in other contexts (Mon, 2023).

##### Honor confidentiality.

Ethical issues identified by universities included threats to confidentiality,
though each one took a slightly different approach. Creating an account and
using AI tools could bother students who are worried about the models stealing
their intellectual property and may therefore be unwilling to use them (UCL,
[n.d.]). The tools also do not respect confidentiality of the people from whom
they have taken the data to train the models, which could result in
unintentional plagiarism for both students and faculty (Mon, 2023). Finally,
these models do not respect confidentiality with regard to legal issues and
data sent to them may be turned over to law enforcement agencies or other
third party vendors and affiliates without user consent (MIT, 2023).

##  7\. Academic Integrity Implications

Recent work on student use of generative AI tools has raised alarms about
academic integrity violations (Prather et al., 2023). Jones et al. summarises
several common practices that are deemed to be cheating, distinguishing
between plagiarism, collusion, and falsification (Jones et al., 2008). We add
contract cheating (as described by Deakin University (dea, 2023)), and use of
unauthorised resources (as described by University of Auckland (auc, 2023)) to
this list of practices that breach academic integrity. These terms are
described by the respective documents as:

Plagiarism::

A student incorporates another person’s or body’s work by unacknowledged
quotation, paraphrase, imitation or other device in any work submitted for
assessment in a way that suggests that it is the student’s original work
(Jones et al., 2008).

Collusion::

The collaboration without official approval between two or more students (or
between student(s) and another person(s)) in the presentation of work which is
submitted as the work of a single student; or where a student(s) allows or
permits their work to be incorporated in, or represented as, the work of
another student (Jones et al., 2008).

Contract cheating::

A student requests another person or service (including, according to Deakin
University, artificial intelligence content production tools) to produce or
complete all or part of an assessment task to submit as their own work (dea,
2023).

Falsification::

Where the content of any assessed work has been invented or falsely presented
by the student (Jones et al., 2008).

Unauthorised resources::

Using software, websites, materials or devices not explicitly permitted (auc,
2023).

We discuss each of these academic integrity concerns with respect to
generative AI.

###  7.1. Plagiarism

Plagiarism is the use of the work of others without appropriate attribution.
This raises the issue of who is the author of work created by generative AI.
There are several possibilities:

  1. (1)

The community that produced the source content used as input to the generative
AI model is the author of the work.

  2. (2)

The generative AI software is the author of the work.

  3. (3)

The user of the generative AI software is the author of the work.

Although some in the literature treat the use of AI tools as plagiarism
(Orenstrakh et al., 2023), we argue that although the community providing
source material has influenced the generated content, this is similar to the
natural process of writing in which authors read source material and use the
information to generate new content, based on existing literature. Generative
AI is almost always creating content _based on_ the training data. In this
case, the community has not authored the work generated by the model, so using
AI generated content would not be considered plagiarism of the original
authors of work that was used as input to the generative AI model. In some
rare cases, GenAI tools can generate the work of someone else exactly, which
would in fact be plagiarism. Since this is extremely rare, we do not consider
it here other than to acknowledge it.

Although it may be tempting to consider generative AI to be the author of the
work in all cases, academic publishers take an opposing view. Examples of
statements include:

> “AI tools cannot meet the requirements for authorship as they cannot take
> responsibility for the submitted work. … Authors are fully responsible for
> the content of their manuscript, even those parts produced by an AI tool,
> and are thus liable for any breach of publication ethics.” (Committee on
> Publication Ethics) (on Publication Ethics, [n.d.])

> “AI does not meet the Cambridge requirements for authorship, given the need
> for accountability. AI and LLM tools may not be listed as an author on any
> scholarly work published by Cambridge.” (Cambridge University Press) (cam,
> [n.d.])

> “Authors should not list AI and AI-assisted technologies as an author or co-
> author, nor cite AI as an author. Authorship implies responsibilities and
> tasks that can only be attributed to and performed by humans.” (Elsevier)
> (els, [n.d.])

> “Artificial Intelligence Generated Content (AIGC) tools — such as ChatGPT
> and others based on large language models (LLMs) — cannot be considered
> capable of initiating an original piece of research without direction by
> human authors. …— these tools cannot fulfil the role of, nor be listed as,
> an author of an article.” (Wiley) (wil, [n.d.])

> “Generative AI tools and technologies, such as ChatGPT, may not be listed as
> authors of an ACM published Work.” (ACM) (acm, 2023b).

Our position is aligned with those of publishers that the user of generative
AI tools should be considered the author of the work. This is consistent with
the view of Pamela Samuelson, who states “The pragmatic answer to the AI
authorship puzzle, …, the [author is the] user who is responsible for
generating the outputs. If anyone needs to be designated as owner of rights in
the outputs, it should be the user.” (Samuelson, 2020) As such, we do not
believe that use of AI-generated content by students should be considered
plagiarism, and should not be referenced or cited as an independently authored
piece of work. The student using the generative AI tool should be treated as
the author of the work.

It should be noted that, in the apparent effort to combat plagiarism, there
have been numerous tools that attempt to detect AI-generated material, such as
CopyLeaks, GPTKit, GLTR, and GPTZero. However, given the probabilistic nature
of generative AI that is used to both generate the assignment in question and
check the assignment, these tools are unreliable and produce many false
positives (Orenstrakh et al., 2023). Orenstrakh et al. found that these
detectors are even worse when evaluating code (Orenstrakh et al., 2023).

###  7.2. Collusion

Collusion occurs when a student works together with another person to create
work that they subsequently claim as their own. This requires both parties to
willingly agree to work together and therefore assumes that both parties have
agency. As generative AI has no agency, we do not consider a student who
submits generated content to be engaged in collusion.

###  7.3. Contract cheating

Contract cheating is traditionally described as a student requesting another
person to produce work that they submit as their own. The definition by Deakin
University extends this view of contract cheating to explicitly include the
use of generative AI tools (dea, 2023) as do some other universities (Mon,
2023). However, we disagree with this position, as it contradicts the view of
the user of generative AI as the author — the position taken by academic
publishers.

We recognise that generative AI models are capable of generating content that
is more extensive than other software tools, but as a matter of principle, we
consider the user to be the author (as discussed previously). generative AI is
a tool which may be used by a student to produce work, much as calculators and
other software tools are used. We therefore do not believe that use of
generative AI software should be treated as contract cheating.

###  7.4. Falsification

Falsification occurs when a student invents or misrepresents data or results.
The possibility that generative AI invents “facts” is well-known, and
typically described by the term hallucination (Ji et al., 2023). Many
university policies recognise that AI tools like LLMs can produce incorrect
data and facts, which we discuss in Section 6.2 under the ACM requirement to
”be honest and trustworthy.” The view of publishers is that an author is
responsible for the accuracy of the content in their work. Examples of policy
from publishers include:

> “Authors are fully responsible for the content of their manuscript, even
> those parts produced by an AI tool, and are thus liable for any breach of
> publication ethics.” (COPE) (on Publication Ethics, [n.d.])

> “Where authors use generative AI and AI-assisted technologies in the writing
> process, these technologies should only be used to improve readability and
> language of the work. Applying the technology should be done with human
> oversight and control and authors should carefully review and edit the
> result, because AI can generate authoritative-sounding output that can be
> incorrect, incomplete or biased. The authors are ultimately responsible and
> accountable for the contents of the work.” (Elsevier) (els, [n.d.])

> “The author is fully responsible for the accuracy of any information
> provided by the [generative AI] tool and for correctly referencing any
> supporting work on which that information depends.” (Wiley) (wil, [n.d.])

> “Authors are accountable for the accuracy, integrity and originality of
> their research papers, including for any use of AI.” (Cambridge) (cam,
> [n.d.])

We take the position that students who use generative AI are responsible for
the content they include in their work. Inaccuracies, citations for non-
existent papers, and other hallucinations that may arise from use of
generative AI are the responsibility of the student author. Therefore, the
category of Falsification is a relevant academic integrity issue for students
using generative AI. As discussed above, this could cause harm to students,
anyone publishing their work, and to the university as a whole.

###  7.5. Use of unauthorised resources

In an educational context, students are required to engage in tasks that may
have specific constraints. For example, the use of calculators in general is
acceptable, and we are comfortable with the notion that using a calculator for
data analysis does not impact authorship, or breach any notion of academic
integrity. However a student in a calculus class may be asked to solve a
differential equation without the use of a calculator, and access to
calculators (or types of calculations) may be restricted in secure assessments
such as exams. Such constraints are typically imposed to ensure that learning
outcomes are met (e.g., that the student can solve a differential equation
without outside assistance). A student who used a calculator for a given
assessment when it was not permitted would be an issue for academic integrity
because the student uses unauthorised resources.

Our position is that this is the appropriate category for use of generative AI
in computing education. Figure 2 shows results to various ethical questions
where it seems that many instructors and students agree that using GenAI tools
to create an entire answer is wrong. In some courses, such as introductory
programming, the use of generative AI can be undesirable as it can solve
problems with minimal intellectual input from students. Not only does this
violate academic integrity, but as discussed above, students who utilise these
resources inappropriately in lower division courses may cause harm to
themselves by preventing their own preparation for upper division coursework.
However, in upper division courses it may be an appropriate productivity tool
that students would be permitted to use, which the survey data from Figure 2
seems to corroborate given responses to questions about generating pieces of
an assignment, help with style, or fixing bugs.

However, this requires teaching staff to be explicit in their course syllabus,
or assessment description, about which resources students are permitted to
use.

Advice for educators: • We encourage educators to teach students about
appropriate ethical use of generative AI throughout the curriculum, and to
allow the use of such tools where it is pedagogically appropriate. See
Appendix D for an example. • When educators assign assessed work for students,
any restrictions on the use of tools such as generative AI should be
explicitly stated. • Students who use generative AI to complete assessed work
should be required to include a statement about how it was used, consistent
with academic publication requirements. • Students who use generative AI when
they are not permitted, or in ways that are restricted, are engaged in
misconduct by using unauthorised resources. Academic consequences as a result
of this behaviour should be made clear in the course syllabus.

###  7.6. Advice for students

Simon et al. (2018) highlights the importance of educating students about
academic integrity, and revealed a wide variety of ways that academic
integrity is communicated to students. Given the disruptive nature of
generative AI, we recommend that a guide for students is developed by teachers
and distributed to students to provide explicit advice about appropriate use
of generative AI tools.

We recommend that any guidelines developed for students should:

  * •

adopt professional practices and standards where possible; and

  * •

adapt professional practices where needed to ensure good pedagogical practices
are maintained.

Publishers typically require acknowledgement where generative AI has been used
in development of a manuscript. We believe this would be useful for teachers,
and for students, to reflect on how generative AI was used, so we recommend
that assessments require students to include a statement about how generative
AI was used in assessment tasks (where permitted).

After considering the relevant findings from this report, we have developed a
resource that provides guidance about the use of generative AI for students.
It is by no means comprehensive or complete and reflects our perspectives on
what students should know before using these tools. Our recommendations are
informed by the risks identified in the literature, the ACM code of ethics,
survey results, and the academic integrity documents that we analysed. We
offer this to teachers as a resource that may be adapted and/or distributed to
students.

Guide for students: See Appendix D for a sample handout or text that could be
adapted and included in a course syllabus.

##  8\. Benchmarking Large Language Models for Computing Education

In this section we focus on the performance of LLMs in the context of
computing education. Teachers are interested in how good LLMs actually are in
conducting tasks such as solving programming problems, explaining code,
generating test questions, and providing feedback to students. However, the
speed at which new models arrive and old models are deprecated is staggering.
As we demonstrate, the currently published literature may underestimate what
the newest models, such as GPT-4, can do. Some recent work has found that
GPT-4 outperforms earlier models in tasks such as visual programming (Singla,
2023), Socratic questioning of novices (Al-Hossami et al., 2023), solving
multiple-choice questions and programming exercises (Savelka et al., 2023a),
and that performance can be close to human tutors for some tasks (Phung et
al., 2023b) while not for others (Al-Hossami et al., 2023).

In addition to being interested in how much the capabilities of LLMs in
computing education related tasks have increased, we are interested in
analysing the suitability of existing benchmarks for computing education as
they might not translate to computing education settings. For instance, we
expect that many students will be able to fix small mistakes made by LLMs
themselves. Similarly, tasks in existing benchmarks might not match those
typically found in computing education courses.

Another issue with current papers lies in our ability to validate results. A
wide variety of parameters, prompts, and evaluation approaches have been used,
and they are not always reported in detail. Furthermore, a slight variation in
a prompt might generate quite different results. In this section we explore
how we assess LLMs in the computing education context. We choose to focus on
the task of generating a solution to a programming problem, because this is a
major task for students and is a focus of existing literature.

First, we review datasets that are available for evaluating LLMs and tag
problems in several of those datasets to assess where they fit in the context
of computing education. Second, we take one of the first papers on LLMs that
appeared in the computing education context (Finnie-Ansley et al., 2022a) and
replicate it using multiple different more recent models. In the replication,
we use the state-of-the-art GPT-4 model to give insight into how rapidly the
performance of models has increased, the GPT-3.5-turbo model that powers the
free version of ChatGPT which many students are likely to use, and GitHub
Copilot which is free for students and educators and can be used as a plugin
for popular IDEs. In addition, we openly release the problem descriptions and
test cases for the dataset used in the original study (Finnie-Ansley et al.,
2022a) and our replication to facilitate future replication.555The data can be
found here: https://osf.io/bu9h3/?view_only=a16f3e474be94188b884aa0dca02041f
Finally, we report on our experiences running two analyses using openly
available datasets, revealing the difficulties we encountered and their
possible effects on results.

###  8.1. Review of empirical datasets

Table 7. Prominent datasets available to evaluate LLM systems and tools.

Name |  Description | Language | Size | Testcases | Solutions |  Link
---|---|---|---|---|---|---
Mostly Basic Python Programs (MBPP) (Austin et al., 2021) |  Natural language descriptions of introductory problems | Python | 1̃000 | Yes | Yes |  https://paperswithcode.com/dataset/mbpp
Search-Based Pseudocode-to-Code (SPOC) (Kulal et al., 2019) |  Pseudocode descriptions of coding problems | C++ | 18356 | Yes | No |  https://paperswithcode.com/dataset/spoc
Blackbox (Brown et al., 2014; Brown et al., 2018) |  Traces of editing and IDE interactions | Java | - | No | No |  https://bluej.org/blackbox/
Deepfix (Gupta et al., 2017) |  Student-generated code with syntax errors | C | 6922 | N/A | N/A |  https://paperswithcode.com/dataset/deepfix
Automated Programming Progress Standard (APPS) (Hendrycks et al., 2021) |  Natural language descriptions of problems of various difficulties | Various | 10000 | Yes | Yes |  https://paperswithcode.com/dataset/apps
HumanEval (Chen et al., 2021) |  Docstring descriptions of (mostly introductory) programming problems | Python | 164 | Yes | No |  https://paperswithcode.com/dataset/humaneval
Grounded CoPilot (Barke et al., 2023) |  Programming tasks provided in an observation study | Python or Rust | 4 | No | No |  https://github.com/michaelbjames/copilot-study
PyFiXV CodeForce (Phung et al., 2023a) |  Buggy submissions to programming contest problems for program repair | Python | 240 | No† | No |  https://github.com/machine-teaching-group/edm2023_PyFiXV/tree/master
ChatGPT_ Bioinformatics (Piccolo et al., 2023) |  Natural language descriptions of introductory bioinformatics problems | Python | 184 | No | Partial |  https://github.com/srp33/ChatGPT_Bioinformatics
EvalPlus (Liu et al., 2023) |  Re-release of the HumanEval code-generation dataset with additional testcases | Python | 164 | Yes | No |  https://github.com/evalplus/evalplus
LeetCode Assistant |  LeetCode problem prompts, responses generated by LLMs, and attempts to repair buggy prompts | Python | 1209 | No† | No |  https://zenodo.org/record/7792965#.ZCvyv-xBwUE
StudentEval (Babe et al., 2023) |  Student-generated prompts for introductory programming problems | Python | 1749 | Yes | Yes |  https://huggingface.co/datasets/wellesley-easel/StudentEval
FalconCode (de Freitas et al., 2023) |  Natural language descriptions of introductory programming problems | Python | 661 | Yes | Yes |  https://falconcode.dfcs-cloud.net/index.php
CS1QA (Lee et al., 2022) |  Naturally-occurring questions asked by students to LLMs with the LLM response | Python | 17698 | N/A | N/A |  https://github.com/cyoon47/CS1QA
AD2022 (Petersen-Frey et al., 2022) |  Problems with a data structures course with student solutions | Python | 16 | Yes | Yes |  https://www.inf.uni-hamburg.de/en/inst/ab/lt/resources/data/ad-lrec
Digital TA (Demidova et al., 2023) |  Small exercises generated by a digital TA and associated solutions | Python | 11 | No | Yes |  https://doi.org/10.5281/zenodo.7799971
Socratic Questioning (Al-Hossami et al., 2023) |  Socratic dialogues with a human instructor | Python | 86 | N/A | N/A |  https://aclanthology.org/2023.bea-1.57/
${\dagger}$ While testcases are not provided, the dataset consists of publicly
available contest problems which have automated testing.

Table 7 presents a set of openly accessible datasets that have been or could
be used to investigate questions about programming exercises or tasks in
computing education contexts. To obtain the datasets in this table, we
reviewed all of the papers in our literature review (Tables 1 and 2) to
identify any data that they used. We do not claim that this list is
exhaustive, but it reflects the datasets in use when we conducted our
literature review. In addition to these datasets, we are aware of one other
attempt to review existing benchmarks for a particular task that might be
performed by LLMs: natural language to code generation (Zan et al., 2023). We
believe the datasets listed in Table 7 represents a more broad set of
applications and illustrates the kinds of questions being investigated and the
breadth of educational contexts being examined.

A review of Table 7 suggests a number of limitations in the data available for
pursuing LLM research in an educational context. Most of the datasets contain
exercises (described in more or less structured natural language). This
reflects a focus on the question of whether LLMs can solve typical programming
problems. In contrast, relatively few datasets contain chat logs where
students interact with an LLM or student-submitted code with syntax errors
(for code repair tasks), but additional publicly available data for questions
beyond code-generation would be beneficial, as that would allow researchers at
small educational contexts, where collecting sufficient data may be difficult,
to engage in LLM work (Kiesler and Schiffner, 2022, 2023b).

Even for code generation tasks, more data would be beneficial. Almost all of
the datasets focus on Python (e.g., providing docstrings as input, Python
starter or solution code, or student chats featuring Python code), with a
smaller number featuring C/C++. Relatively little public data appears to be
available for other languages. Also, as described in the next section, many of
the available datasets focus on small exercises used in introductory
programming, with relatively little data available to examine larger
programming problems or content for more advanced courses.

Finally, as previously identified by Liu et al. (2023), many of the published
datasets do not provide robust evaluation of the exercises they include. Liu
et al. (2023) provide the EvalPlus dataset, which enhances previously
published datasets with additional test cases; they found that the limited
tests available meant that incorrect “solutions” were accepted as passing. We
also found evaluation to be limited, with some datasets requiring manual
intervention to complete evaluation. We provide more detail on issues we
encountered when using these datasets in Section 8.4.

###  8.2. Problem context

Table 8. Fraction of instructor-assigned tags (Introductory (Intro), ObjectOriented (OO), or DataStructures (DS)) assigned to exercises in various datasets. | Intro | OO | DS
---|---|---|---
HumanEval (Chen et al., 2021) | 98.8% | 0 | 1.2%
FalconCode (de Freitas et al., 2023) | 100% | 0 | 0
My AI CS2 (Finnie-Ansley et al., 2023) | 83.3% | 9.3% | 7.4%

To categorize the types of exercises present in the datasets, we manually
tagged two prominent datasets, HumanEval (Chen et al., 2021) and FalconCode
(de Freitas et al., 2023). The datasets were tagged by a single author, an
experienced instructor who has taught introductory programming, data
structures, and advanced systems programming. The author tagged the exercises
as being suitable for (a) an introductory programming course (Intro), as they
use builtin data structures like strings and do not introduce complex
algorithmic logic; (b) an introductory course using classes (OO), as they
introduce classes or methods; or (c) a data structures or algorithms course,
as they use some abstract data types (e.g., trees, queues, graphs) or complex
algorithmic logic (e.g., subsequence matching, linear programming). The
results are presented in Table 8. We found that in these two datasets, the
vast majority of problems only cover material suitable in an introductory
programming setting.

Many of the other datasets in Table 7 are similar to the two datasets we
analysed, in that they appear to focus on introductory material. We requested
access to the data used in Finnie-Ansley et al. (2023)’s paper, as the topic
was a more advanced (CS2) course. This time, the dataset was tagged by two
authors (one who had tagged the previously discussed datasets and a second
experienced instructor); we calculated Cohen’s kappa and found near perfect
agreement (0.94). Again, we found that the majority of the content was
primarily suitable for an introductory course, with relatively few questions
asking about object-oriented code or data structures. In addition, while
reviewing the problems, we found that almost all were examples of small
exercises, with relatively few requiring multiple functions to solve. The
FalconCode (de Freitas et al., 2023) dataset includes a few exceptions to this
general trend.

Finally, we examined the Automated Programming Progress Standard (APPS)
dataset (Hendrycks et al., 2021), as it explicitly advertises that it includes
exercises of various difficulties. This is a large set (10000 problems), so we
manually tagged a sample of 200 exercises and then used keyword searching to
identify the usage of classes and common data structures. This means that our
estimates will underestimate the complexity of the exercises. Many of the
problems in this set are more complex, as expected as they were largely drawn
from programming contests. However, they may not be problems typically seen in
educational contexts. The instructor reviewing the problems would not use many
of these problems in any course, as they introduce issues like floating point
error, exceeding the maximum representable integer, or linear-time pattern
searching. At the same time, relatively few explicitly reference OO topics
(4.1%) or common data structures (6.6%).

Taken together, this analysis suggests that many of the available datasets –
and as a result, the published results – reflect an early introductory
programming context (CS1) with relatively few examples of common CS2 material
(Cipriano and Alves (2023) is a recent exception demonstrating research on OO
topics) and even fewer covering any more advanced topics. Some datasets do
include more challenging tasks, but these may not reflect the kinds of
problems student programmers solve in upper year courses, as they appear to be
inspired by (or were drawn directly from) programming contest sites.

###  8.3. Replication: The robots are coming

Finnie-Ansley et al. (2022a) published the first paper that examined the
performance of large language models in solving introductory programming
exercises. They found that Codex666More specifically, the first version of the
model ‘code-davinci-001’. had better performance than the median student on
the same exam exercises. In order to understand how the performance of large
language models in this task has improved in the past two years, we partially
replicated their study.

Method: We contacted the authors of the original study and received the
problem descriptions and test cases used in the study. The original study had
a total of 30 exercises; 23 used in two exams and seven variants of the
Rainfall-problem (Fisler, 2014). In the original study, the method used for
the Rainfall-problems differed from the method used for the exam questions. In
our replication, we follow the method originally used for the exam problems
for both the exam and the Rainfall problems. We generate up to ten solutions
for each problem, stopping if the LLM creates a solution that passes the
tests, or includes a “trivial formatting error”. As in the original article,
we manually fixed the trivial errors and considered this step as an additional
trial. All models were prompted with the problem description surrounded by
triple-quotes as was done in the original study.

We evaluate three models in our replication: GPT-4, GPT-3.5-turbo, and GitHub
Copilot. At the time of writing, GPT-4 is the state-of-the-art LLM, so it is
interesting to see how it performs on solving the problems. GPT-4 also powers
the paid version of ChatGPT. GPT-3.5-turbo is the model that powers the free
version of ChatGPT, which is likely what many students will be using, and thus
it is also interesting to include. Lastly, GitHub Copilot is free for students
and educators, and is directly embedded into the IDE, so it is possible that
students will be using it too, which is why we decided to include it in the
replication.

Copilot works slightly differently to the other two models, as it cannot be
directly ‘prompted’. For evaluating Copilot, we used the Visual Studio Code
Copilot plugin, and provided the prompt (problem description surrounded by
triple-quotes) in an empty file. We would then wait for Copilot to provide a
suggested completion, and would accept the first suggestion that Copilot
provided. In the rare cases where no suggestion was provided based on just the
problem description, we would write ‘def’ to start the function, after which
Copilot would suggest code if it had not before.

Results (GPT-4): For GPT-4, we used the temperature value of 0.9 similar to
the original study. The results of the replication are presented in Figure 3
(only GPT-4) and Table 9 (all three LLMs). It is clear that GPT-4 outperforms
Codex, which is the model used in the original study. All problems were solved
in under ten attempts, except for the last question in the second test
(T2-Q12). As noted in the original study, some of the Rainfall variants had
somewhat vague problem descriptions, leading to GPT-4 having more trivial
formatting issues with the outputs. In the original study, the results of
Codex were similar to students in the top quartile; in our replication, GPT-4
would have been one of the top students in the class. The only problem that
GPT-4 was not able to solve was the last problem of the second test (T2-Q12),
which involved drawing bar graphs using text where the shape of the bar graph
was determined by values in a dict passed to the function.

Results (GPT-3.5): Similar to GPT-4, we used the temperature value of 0.9 for
GPT-3.5. GPT-3.5 performed only slightly worse compared to GPT-4. It was able
to solve most problems on the first try, although many attempts included
trivial formatting issues. For example, inadvertently leaving out a period at
the end of a printout such as having “print(”The sum is”, sum)” when the tests
expected a period at the end of the string. Compared to GPT-4, GPT-3.5 was
unable to solve one of the Rainfall variants, specifically the one from Simon.
Looking into why GPT-3.5 struggled, the biggest issue for the model was that
the problem description stated that “A day with negative rainfall is still
counted as a day, but with a rainfall of zero.” This was not taken into
account in the code that GPT-3.5 generated as the code for all ten completions
would simply ignore any days with negative rainfall values. Similar to GPT-4,
GPT-3.5 was not able to solve the last question of the second test (T2-Q12).

Results (Copilot): Copilot performed the worst out of the three evaluated
models, successfully solving 20 out of the 23 exam problems and four out of
the seven rainfall variants. We looked into the issues in code for the
problems that Copilot was unable to solve. T1-Q10 involved printing all words
in a given sentence that start with a given character. The problem description
explicitly stated that “The sentence will end with a full-stop.” The code
generated by Copilot would not remove the full-stop from the end of the
sentence, resulting in failure in the edge case when the last word of the
sentence needed to be printed where the tests assumed the full-stop is not
included in the word, but Copilot did not take this into account. T1-Q11
involved sorting four numbers given as a parameter using only the “min()” and
“max()” functions (using of lists, if/elif/else, and loops was forbidden).
While Copilot took the constraints into account, all ten completions had
logical flaws and did not pass all the tests. Finally, similar to GPT-4 and
GPT-3.5, Copilot was unable to solve T2-Q12 which involved printing bar graphs
made of text. For T2-Q12, many completions from Copilot would not compile (the
completion would be incomplete), and when it did, the code was nowhere near
correct. For example, many of the completions would always print the same bar
graph regardless of input.

For the rainfall variants, Copilot was unable to correctly solve the one from
Soloway, the one from Simon, and the one from Guzdial. For the variant from
Simon, the issue was the same as for GPT-3.5 in that negative values were
always ignored entirely, even though the problem description asked for these
to be counted as days with a rainfall of 0. For both the Soloway variant and
the variant from Guzdial, none of the completions generated by Copilot handled
the edge case where the list is empty, leading to division by zero.

One interesting finding was that Copilot would sometimes write just the
function signature with a comment such as “# write your code here” followed by
“pass”, or the completion would not have any code at all but only
suggestions/hints for how to start work on the problem as comments.

Discussion: Overall, all models performed quite well, having performance that
would have allowed them to pass the exams. Unsurprisingly, GPT-4 outperformed
GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has
outperformed other LLMs for various tasks (Singla, 2023; Al-Hossami et al.,
2023; Savelka et al., 2023a; Phung et al., 2023b). The problems where any
model struggled were either complex (e.g., involving printing bar graphs as
text) or had vague problem descriptions, leading to some edge cases being
ignored (e.g., rainfall variants that did not specify what should happen when
the list is empty). As noted in the original study, LLMs somewhat struggle
with trivial formatting issues, such as missing some formatting or having
extra printouts not asked for in the problem description. We noticed that
there was less variation in the completions generated by Copilot compared to
the ones from GPT-4 and GPT-3.5. This might be due to the use of a relatively
high temperature value for these models – previous work has found similar
results and speculated that Copilot likely uses a lower temperature value
(Wermelinger, 2023).

(a) Results of the original “Robots Are Coming” paper that used Codex (Finnie-
Ansley et al., 2022a).

(b) Results of our replication of (Finnie-Ansley et al., 2022a) with GPT-4.

Figure 3. A comparison of the original results and the score achieved by GPT-4 on the two CS1 tests and Rainfall-problem variants presented in (Finnie-Ansley et al., 2022a).  Table 9. The replication results for (Finnie-Ansley et al., 2022a). An asterisk indicates that the solution required trivial modifications, which was counted as an additional attempt (i.e., “2*” means that the problem was solved on the first try, but had trivial mistakes e.g. in formatting of strings such as a missing period or extra unnecessary prints). A ‘-’ means that the problem was not solved within 10 attempts. Problem | Solved on attempt
---|---
| GPT-4 | GPT-3.5 | Copilot
T1-Q1 | 2* | 1 | 2*
T1-Q2 | 1 | 1 | 1
T1-Q3 | 1 | 1 | 1
T1-Q4 | 2* | 2* | 2*
T1-Q5 | 1 | 2* | 1
T1-Q6 | 1 | 2* | 1
T1-Q7 | 2* | 1 | 1
T1-Q8 | 1 | 1 | 1
T1-Q9 | 1 | 1 | 1
T1-Q10 | 1 | 1 | -
T1-Q11 | 3 | 1 | -
T2-Q1 | 1 | 2* | 1
T2-Q2 | 1 | 2* | 1
T2-Q3 | 1 | 1 | 1
T2-Q4 | 1 | 2 | 1
T2-Q5 | 1 | 2 | 4*
T2-Q6 | 1 | 1 | 1
T2-Q7 | 1 | 1 | 1
T2-Q8 | 2* | 2 | 4
T2-Q9 | 1 | 1 | 1
T2-Q10 | 1 | 1 | 1
T2-Q11 | 2 | 1 | 1
T2-Q12 | - | - | -
RF-Soloway | 2* | 2* | -
RF-Simon | 2* | - | -
RF-Fisler | 2* | 2* | 10
RF-Ebrahimi | 2* | 2* | 2*
RF-Guzdial | 2* | 3 | -
RF-Lakanen | 2* | 2 | 3*
RF-Apples | 1 | 1 | 1

###  8.4. Novel analysis

####  8.4.1. APPS

Hendrycks et al. created the Automated Programming Progress Standard (APPS)
dataset (Hendrycks et al., 2021) as a benchmark for program generation. The
dataset consists of 10.000 programming problems of varying difficulty,
manually extracted from the online coding websites Codewars, AtCoder, Kattis,
and Codeforces. The average number of lines for the solution is 18. The
dataset has been divided into a training set and a test set, both containing
5000 problems. The following elements are provided for each problem:

  * •

Problem description, including a description of the expected input and output
of the problem and some concrete examples.

  * •

A JSON file with inputs and corresponding output. On average, each problem has
21.2 test cases.

  * •

A JSON file with metadata, with a difficulty level (introductory,
intermediate, competition) and the url to the website where the problem is
hosted.

  * •

A list of solutions from humans.

Hendryck et al. have tested the dataset in 2021 with GPT-2, GPT-3, and GPT-
NEO. They found that GPT-NEO performed the best by passing almost 15% of test
cases on introductory problems.

##### Method

We aim to use the dataset from this paper and assess how newer models are able
to handle these problems. We use a simplified method, employing the following
steps:

We run the models (GPT-3.5-turbo16k, GPT-4) with the following parameters:
temperature=0.0, max_tokens=4000, and default values for Top P (1), Frequency
penalty (0), and Presence penalty (0). The system prompt we use is as follows:

    You are a highly intelligent coding bot that can easily
    handle any Python programming task. Given a natural
    language instructions you can always implement the
    correct Python solution. Your focus is the solution
    code only. You are not allowed to provide explanations.

    Make sure to use input statements for input, and do not
    give a method definition

    Example (toy) instructions:
    Implement a Python program to print "Hello, World!" in
    the hello.py.

    Example bot solution:
    === hello.py ===
    x=input()
    print(x)
    ===

As the user input, we provide the full problem description as described above.

We process the generated code by running the code and providing the inputs
from the first 25 test cases to each solution. We then compare the output to
the expected output with an exact comparison, after stripping the results.
After noticing that differences in characters for line ends caused problems
(‘“r“n’ versus ‘“n’), we fixed this in the output comparison. We consider a
test case to fail after detecting a timeout of 5 seconds. For each problem, we
store a list of test results and calculate a success rate as the percentage of
passed test cases.

For simplicity, we skipped the problems that had starter code. We also skipped
problem descriptions that could not be read. We ran a sample of 100 interview-
level problems for GPT-4, running 25 test cases for each problem. We manually
assessed the failing test cases, to check if the problems were actual coding
problems, or were caused by formatting mistakes. Minor issues were corrected.

The final runs were conducted in September 2023.

##### Results

Table 10 and Figure 4 show the results. GPT-4 performs quite well overall,
with an average score of 51.5% on test cases. Comparing to GPT-3.5, which
scored 39.2%, the performance has clearly improved. Keeping a strict pass/fail
criterion (all tests should pass), only 36.1% of the GPT-4 solutions pass all
tests, and 21.0% for GPT-3.5. We also observe a large difference between
problem types, with GPT-4 solutions to introductory problems scoring as high
as 72.2% on test case average, but the most difficult, competition level
problems score only 28.7%.

Table 10. Average test case score for APPS problems, ran with max. 25 test cases. |  |  | Test case average | Strict accuracy
---|---|---|---|---
Type | Count | Avg nr of tests | GPT-3.5 | GPT-4 | GPT-3.5 | GPT-4
Introductory | 974 | 8.9 | 57.3% | 72.2% | 44.5% | 62.9%
Interview | 2972 | 15.3 | 38.8% | 52.9% | 18.7% | 35.8%
Competition | 1000 | 9.3 | 22.6% | 28.7% | 5.2% | 10.8%
Overall | 4946 | 12.9 | 39.2% | 51.5% | 21.0% | 36.1%
Figure 4. GPT success rate for different exercise types.

##### Discussion

A major downside of the APPS dataset is that it is a public dataset with
problems from popular online coding websites. There is a large chance that
solutions for it have been included in training recent models.

Overall, APPS is a high-quality dataset with extensive test suites for many of
the problems. However, in the context of computing education, several problems
in this dataset might not be suitable for novice programmers. Even the
introductory set contains some complex problems, and are targeted more at
people participating in programming competitions.

We included an explicit instruction in the prompt to ‘use input statements for
input, and do not give a method definition’. Omitting this in a first test run
showed many tests failing, because the model returned a method definition.
This shows we need to be sure the model creates solutions in the exact same
format as expected.

There are different aspects to be considered when running an analysis of a
model on a certain task. The number of attempts to get a solution from the
model should be specified. The ‘Robots are coming’-replication (Section 8.3)
performed multiple attempts, while we only use one attempt for this dataset.

####  8.4.2. FalconCode

FalconCode is a novel collection of over 1.5 million Python programs from over
2,000 undergraduate students at the United States Air Force Academy (de
Freitas et al., 2023). The dataset is not available online but is provided to
anyone who applies following the instructions at the dataset
website.777https://falconcode.dfcs-cloud.net/index.php The dataset contains
661 introductory Python programming problems used in 4 courses. To understand
how well selected LLMs (GPT-3.5/4) perform on these problems we (i) extracted
the problem statements and unit tests; (ii) performed de-duplication of
problem statements; (iii) utilised the LLMs to generate solutions; (iv) ran
the unit tests provided in the dataset against the outputs of the LLMs; and
(v) analysed the test results.

##### Data extraction

The dataset is provided in the convenient tabular format where the table we
work with has the 661 problems. Among other fields, there are the _prompt_ and
_testcase_ columns that contain the problem statement and unit tests to
evaluate the correctness of the solution. The problem statements are in HTML
format which we preserved. The unit tests are runnable Python code. Hence, we
extracted those into separate Python files.

##### De-duplication

The original dataset lists 661 problems. Apparently, some problems are re-used
across courses. Since the id of the problem appears to be preserved it is
possible to de-duplicate the problems based on the id which reduces the number
of problems to 422. However, it turns out that the problem set is further
reduced to 310 if one de-duplicates based on the problem description (prompt
field). Interestingly, de-duplicating only on the prompt field, disregarding
the id, yields 385 unique problem descriptions which suggests that there are
several problems with the identical id that have different descriptions.
Extracting plain text descriptions from the original HTML format and removing
superficial white space results in 344 unique problem descriptions. For this
study, we opted for de-duplicating on the original (HTML) prompt (just in case
the white space difference could be meaningful). Hence, there are 385 problems
we use in our experiments.

##### Generating solutions

To generate a solution to the given problem, we include the problem statement
(HTML) into the same prompt as used with the APPS dataset experiment
(Subsection 8.4.1). We decided against extracting plain text of the problem
statements because the HTML may encode important information through
formatting that a state-of-the-art LLM such as GPT-3.5/4 may be capable of
leveraging. Note that many of the problem statements were referring to an
external resource (e.g., starter code or a file with data) which has not been
made available as part of the dataset. Hence, we cannot include the resource
into the prompts submitted to the LLMs. We extract the completion of the
submitted prompt and save it into a Python file named with the id of the
problem (needed for the unit tests to discover the solution). In a non-
negligible number of cases, the completion was wrapped in `‘‘‘python ...‘‘‘`
tokens. While this issue could likely be fixed via further prompt engineering,
we simply removed these tokens using a regular expression. Leaving the tokens
in would make the Python file not runnable (syntax error), which would
automatically fail all the unit tests despite the solution being correct.

##### Testing

To test correctness of the automatically generated solutions, we executed the
provided unit tests. The unit tests rely on the solutions being present in the
same directory in a file named with the id of the problem. Additionally, there
is an external _cs110_ grading (testing) library that needs to be installed
through _pip_.888https://pypi.org/project/cs110/

We saved the output of the unit testing for each individual problem into a
separate file. The last line of these had a predictable format: _Unit Test
Returned: #_ , where # is replaced with number from 0 to 100. We utilised
simple regular expressions to extract this final result of the evaluation from
each of the output files. Note that assignments with identical problem
statements could have been associated with different unit tests. Therefore,
for each problem we ran all available unit tests, and took the average result
of the tests as the score.

##### Results analysis

Table 11 shows the performance of GPT-3.5/4 on the 385 FalconCode problems
across three different types of assignments:

  * •

_Skill_ – Small (1–-3 lines) programs focused on specific programming skill.

  * •

_Lab_ – Medium (10–-50 lines) programs focused on utilisation of one or more
skills.

  * •

_Project_ – Larger (50–300 lines) programs solving an open-ended problem.

GPT-4 performed markedly better than GPT-3.5. In the subsequent analysis, we
focus on the better performing GPT-4 model. The overall performance of 45.4%
suggests a rather weak performance of the LLM in handling these introductory
programming problems. In order to understand the causes of the low performance
we analysed each case where GPT-4 did not achieve the full score.
Specifically, we performed a thematic analysis in which causes of each failed
assignment were extracted as codes and then collated into higher-level themes
(Braun and Clarke, 2006). The results of the analysis are shown in Table 12.

Table 11. Results for FalconCode. The middle column describes the raw success rates for each category of problem, and the rightmost column describes the success rates after problems where insufficient information is provided were removed. | Full | Clean
---|---|---
Type | Count | GPT-3.5 | GPT-4 | Count | GPT-3.5 | GPT-4
Skill | 162 | 30.3% | 40.0% | 81 | 57.6% | 80.0%
Lab | 214 | 29.8% | 51.4% | 155 | 47.5% | 71.0%
Project | 9 | 0.0% | 0.0% | 0 | - | -
Overall | 385 | 29.3% | 45.4% | 236 | 51.4% | 74.1%

We first analysed the performance on Projects since it amounted to 0.0%. It
turns out that the problem statement included in the dataset only points to an
external pdf file that contains the actual instructions, e.g.:

> Objective: Create a drone simulation that can scan a battlefield for targets
> and engage them.
>
> Instructions: Read writeup (airstrike.pdf) and use the template file to
> begin work.

Since the pdf file has not been released with the dataset we could not provide
the LLM with adequate instructions to generate a solution. This is the case
for all 9 projects.

Table 12. Reasons for LLM (GPT-4) failures on FalconCode problems. We used this analysis to filter the dataset down to a clean version that is appropriate to use for the evaluation. The clean column signifies which reasons were considered as the failure on the LLM part. These data points were included in the clean dataset. Failure Cause | Clean | Skill | Lab | Project
---|---|---|---|---
Missing Instructions |  |  |  | 9
Missing Starter Code |  | 58 | 7 |
Missing Data File |  | 20 | 46 |
No Unit Tests |  | 2 | 2 |
Incorrect Unit Tests |  | 1 | 4 |
Unexpected Library | ✓ | 1 | 2 |
Incorrect Structure | ✓ |  | 25 |
Incorrect Solution | ✓ | 22 | 28 |
Overall Failed |  | 104 | 114 | 9
Overall Failed (clean) |  | 23 | 55 |

As Skill assignments are supposed to require only small solutions, not
exceeding several lines of code, the performance of 40.0% is rather
unexpected. Our analysis revealed that the main causes of the poor performance
are related to the same cause detected with the Project tasks. There are a
substantial number of situations where the Skill assignment required an
external data file, and even more commonly starter code was needed to complete
the assignment, e.g.:

> You have been provided with a list called list_of_animals. Write a program
> that prints out each of the items in this list (one item per line).

As the starter code has not been included in the dataset, the LLM does not
have the complete information to produce the desired output. We also detected
instances where the LLM used an unexpected library (not part of the Python
standard library) and hence the program would crash, i.e., the unit tests
would fail. In a few cases, the unit tests would be missing or incorrect. We
also identified several instances where GPT-4 generated a genuinely incorrect
solution to the problem statement that provided sufficient information.

The most common cause of failed unit tests for Lab assignments was also a
missing data file and/or starter code (not released with the dataset). Another
common cause was an incorrect structure of the (possibly correct) solution. A
typical example would be a solution containing a function that returns a value
whereas it was supposed to be a script asking a user to provide an input and
print the output to the terminal. In the remaining cases GPT-4 generated a
genuinely incorrect solution.

Based on the above analysis, we report another set of results (Clean) on the
subset of 236 FalconCode problems that provide sufficient information for the
LLM and are associated with valid test cases. The success rate increases from
45.4% to 74.1%. It is worth emphasising that if we would also disregard the
cases where GPT-4 produced the correct solution using an unexpected structure
(e.g., a function returning a value instead of a program asking user for an
input and printing to a terminal) or utilised an unexpected library, the
success rate would increase to 86.8%. Finally, a large portion of the
genuinely incorrect solutions are rather superficial problems, such as not
rounding to a single decimal as demonstrated in the example provided in the
instructions. A simple change to the prompt adding the specific instruction
would certainly fix such issues. Hence, one can conclude that we only observed
minimal amount of cases where GPT-4 would produce a truly incorrect solution
to the problem (most certainly in fewer than 5% of cases).

###  8.5. Discussion

In this section, we provide an overview of the issues we encountered while
performing our replication and analyses. Our experiences could provide
valuable insights to researchers who want to study LLM performance as well as
to teachers who are interested in their performance in an educational context.

##### Higher LLM performance than identified in the literature

Our replication of the Finnie-Ansley et al. (2022a) paper suggests that new
LLM models are significantly more capable than is currently reported in the
literature. Our experiment with the FalconCode dataset further supports this
conclusion, and while success rates are lower on the APPS dataset, those
problems are significantly more challenging than those typically provided in
the CS1 and CS2 courses typically discussed in the literature.

##### Challenges using publicly available datasets

We encountered a number of challenges applying LLMs to publicly available
datasets, even though the datasets themselves are of high quality. In
particular, we anticipate that future researchers will find that very few
datasets will have been produced specifically to support LLM code generation
research, so they are likely to not include critical information like starter
code, data files, or formatting instructions.

Researchers will also encounter challenges even with datasets produced for
code generation tasks. As suggested by Liu et al. (Liu et al., 2023), existing
datasets may need to be augmented to evaluate LLM-generated code accurately.
The number and quality of the test cases provided might not completely cover
all exercise requirements and edge cases, therefore giving a false positive
result. Alternately, test cases can even be incorrect, or too strict,
exceeding what is required in the instructions, lowering the potential
performance of models.

##### Homogeneity in available datasets

Finally, future researchers may struggle to find appropriate datasets. Most
datasets we found could support code generation, using Python, of CS1
problems. To assess model performance on multiple types of tasks, for
different programming languages, or at different levels, will require new
datasets. The community will need to reward the effort of curating and
maintaining such datasets, as providing a complete and well-evaluated dataset
is challenging (as noted above) and is important for enabling research by a
diverse set of groups.

####  8.5.1. Limitations

We focused our efforts on replicating code generation tasks, but there are
many other research questions that are potentially even more challenging to
replicate. Testing generated code can be easily automated by running test
cases, although this might not capture all aspects relevant to computing
educators, such as code quality and suitability of the solution with regards
to which concepts the student has learned so far. These latter aspects could
also be assessed automatically, but we have not attempted to do so in our
study.

Outside of code generation, assessing solutions for other types of exercises
common in CS (e.g. regular expressions, UML-diagrams, automata), LLM-generated
exercises, feedback, and explanations require additional datasets and may
require a qualitative framework that could be difficult to provide or to
transfer to another research team.

Advice for users and creators of LLM CSEd Datasets: • Creators: Include full
and precise problem descriptions, so that the LLM can be given sufficient
information for solving the problem. • Creators: Include full test cases,
ideally in a format where they are easy to run for others, so that LLM
performance can be easily evaluated. • Creators: Include any resources needed
to complete the assignments, e.g., the starter code or data files. • Creators:
Make it easy to update or extend the data set, and report issues. • Users:
Make LLM parameters clear for replication. • Users: Clearly describe the
prompts used with the models, ideally providing example prompts.

##  9\. Conclusions

This report is the output of an ITiCSE Working Group that explored how the
emerging generative AI revolution will impact the future of computing
education. The first time that the group met was in April 2023 – three years
after the release of the ground-breaking GPT-3 large language model; less than
two years after the release of the Codex model (a variant of GPT-3
specifically fine-tuned for coding tasks); less than one year since the
Copilot plug-in (for generating code directly within an IDE) was made
available for free to students worldwide; five months since the release of
ChatGPT (providing a convenient chatbot interface); and just one month after
the release of GPT-4, a powerful multi-modal large language model. Against
this backdrop of rapid advancements, our working group came together at a time
when the computing education community was just beginning to grapple with the
widespread use of generative AI tools by students as well as the general
public. Many urgent questions were being asked about how to adapt to the
challenges and opportunities presented by these new models and tools. In
particular, if students are able to generate solutions to all of their
programming coursework, how will this impact what is taught, how it is taught,
and how students will remain motivated to learn?

Our overarching goal is for this report to serve as a focal point for
researchers and practitioners who are exploring, adapting, using, and
evaluating LLMs and LLM-based tools in computing classrooms. We now return to
the list outlined in Section 2 to summarise our main contributions:

  1. (1)

A review of the literature: We provide a detailed review of the literature on
LLMs in computing education, current as of August 2023. Using a keyword search
of relevant databases and two rounds of forward and backward snowballing, we
synthesise findings from 71 primary articles. Due to rapid changes in the
field and the slow pace of publishing in traditional venues, much of this work
was available only as pre-prints on platforms such as arXiv. We included all
such literature in our review and assessed every article with respect to a set
of quality metrics. The most common type of paper to date involves evaluating
the performance of LLMs when applied to tasks such as programming exercises. A
key finding, which justifies some of the widely voiced concerns around
academic misuse of LLMs, is that current models tend to perform at least as
well as most students on typical introductory-level programming tasks. We also
reviewed papers that discussed possible opportunities and challenges of LLMs,
that studied how end-users (including students) interacted with LLMs, and that
used LLMs to generate high-quality learning resources. Among the risks that
were identified, the most common concern expressed by authors was that
students would become overly-reliant on using LLMs to generate and debug code.

  2. (2)

Prevailing attitudes: To understand how LLMs are currently being perceived and
used, we conducted a survey involving 171 students and 57 instructors from
computing courses spanning 20 distinct countries. We found that, in general,
students and instructors had similar perceptions about LLMs with respect to
questions around experiences, expectations and beliefs. However, they differed
in their perceptions of how clear course policies were about the allowed use
of LLMs, with instructors – somewhat surprisingly – finding these policies to
be less clear than students. Many of the respondents to our survey had very
little experience using generative AI tools at the current time, although we
expect familiarity to grow rapidly in the coming years. Some instructors were
concerned that their students were using such tools inappropriately, and a
small fraction of students refused to use generative AI tools for ethical
reasons and due to concerns about harming their learning. In many cases
perceptions were well-aligned – both students and instructors felt strongly
that there should be some restrictions on the allowed usage of generative AI
tools for coursework.

  3. (3)

New instructional approaches: Although many instructors are only just
beginning to think about the impacts on their teaching, some have already made
concrete changes to their curricula and assessments. In order to document
these recent adaptations, we conducted 22 in-depth interviews with instructors
on five continents who already had concrete plans in place to change some
aspect of their teaching. We found that some instructors were beginning to
place a greater emphasis on ‘process over product’. That is, instead of just
grading a final artefact, there is an evaluation of the processes used by
students when working on a product. In addition, there was also a trend
towards placing a greater emphasis on invigilated assessments such as exams,
with a reduction to the grade weighting placed on unsupervised homework
assignments. We anticipate further changes to learning objectives, course
content and assessment practices in the near future, and we see an important
need to swiftly disseminate best practices that emerge.

  4. (4)

Academic integrity: Policies & recommendations We reviewed academic integrity
policies that mentioned generative AI from major universities around the world
and found that these explicitly addressed many of the principles stated in the
ACM code of ethics. However, it is unclear how students are being educated
about the ethical use of generative AI in the classroom. This appears to be an
important area for future work given the findings from our survey which
revealed that students and instructors have quite different views regarding
the clarity of current policies. Further work in this area is needed to
understand how to effectively embed these principles in computing classrooms.

We follow our review of policies with concrete recommendations for both
students and instructors. We agree with the position articulated by many
publishers that the user of the LLM should be considered the author of the
generated text. This has implications for academic integrity, in that
plagiarism is not usually a concern but instead users would be responsible for
any falsification produced from uncritical use of LLM-generated artefacts. We
encourage instructors to teach students about the ethical use of generative AI
throughout their courses, clearly stipulating any restrictions on use for
assessed work. Should students use such tools for graded tasks, we recommend
they include a statement detailing its usage, and any violations should be
regarded as academic misconduct with the penalties explained clearly.
Institutions and faculty will need to communicate these expectations
explicitly, and thus it is imperative that we provide students with resources
to understand how to use LLMs appropriately. To this end, we have prepared a
sample handout that can be adapted and included in a course syllabus on the
ethics of using generative AI tools for assignment work (see Appendix D).

  5. (5)

Encouraging replication: Given that instructors are naturally interested in
how well LLMs can solve typical tasks that are set for students, a common
thread of work to date has been to evaluate the performance of various models.
However, replicating prior work using newer models is difficult, given that a
wide variety of parameters, prompts, and evaluation approaches have been used,
and not all methods are reported with sufficient detail. Producing a dataset
that contains everything necessary for high-quality LLM research (in
particular, accurate evaluation of the artefacts generated by LLMs) is
challenging and needs to be encouraged by the community. We therefore identify
a seminal paper on LLM evaluation for programming tasks, and prepare and
release the problem descriptions and test cases in order to facilitate future
replication work. Our own replication of this prior work, using a state-of-
the-art model, shows an extraordinary performance improvement over the span of
two years since the original work was carried out.

As we collectively face the changes being ushered in by the AI revolution, it
is clear that LLMs present significant challenges but also new opportunities
for computing educators. We present this report not only as a snapshot of the
current state at this relatively early stage, but also as a call to action: to
encourage broad exploration of the use and impacts of LLMs and LLM-based tools
in computing classrooms, to adapt teaching methods and update academic
integrity policies, and to develop best practice and to share them widely with
the computing education community. The future of computing education is
rapidly evolving, and shaping it must be a collective effort.

###### Acknowledgements.

Thank you to the following:

  * •

Michael Caspersen for joining this effort in the early days and graciously
bowing out when more prestigious matters intervened.

  * •

All of the students and educators who responded our surveys.

  * •

All of the educators who participated in interviews (in alphabetical order by
surname):

    * –

Austin Cory Bart (University of Delaware, USA)

    * –

Michael Caspersen (It-vest & Aarhus University, Denmark)

    * –

James Davenport (University of Bath, UK)

    * –

Rodrigo Duran (Federal Institute of Mato Grosso do Sul Brazil)

    * –

Dan Garcia (UC Berkeley, USA)

    * –

Michael Kölling (King’s College London, UK)

    * –

Viraj Kumar (Indian Institute of Science, Bengaluru, India)

    * –

Mark Liffiton (Illinois Wesleyan University, USA)

    * –

Jérémie Lumbroso (University of Pennsylvania, USA)

    * –

Peter Mawhorter (Wellesley College, USA)

    * –

Jean Mehta (Saint Xavier University, USA)

    * –

Briana Morrison (University of Virginia, USA)

    * –

Leo Porter (University of California San Diego, USA)

    * –

Jan Schneider (Goethe Universität, Germany)

    * –

David H. Smith IV (University of Illinois, Urbana-Champaign, USA)

    * –

Kristin Stephens-Martinez (Duke University, USA)

    * –

Sven Strickroth (LMU Munich, Germany)

    * –

Ewan Tempero (University of Auckland, New Zealand)

    * –

Christian Tomaschitz (TU Wien, Austria)

    * –

Frank Vahid (University of California, Riverside, USA)

    * –

those who wished to be de-identified in this report.

## References

  * (1)
  * Oxf ([n.d.]) [n.d.].  Artificial intelligence.  https://www.brookes.ac.uk/students/academic-development/online-resources/Artificial-intelligence.  Accessed: 06 September 2023.
  * cam ([n.d.]) [n.d.].  Authorship and contributorship.  https://www.cambridge.org/core/services/authors/publishing-ethics/research-publishing-ethics-guidelines-for-journals/authorship-and-contributorship#ai-contributions-to-research-content Accessed: 08 July 2023.
  * wil ([n.d.]) [n.d.].  Best Practice Guidelines on Research Integrity and Publishing Ethics.  https://authorservices.wiley.com/ethics-guidelines/index.html#5 Accessed: 08 July 2023.
  * UVA ([n.d.]) [n.d.].  Generative AI in Teaching and Learning Task Force (GENAI).  https://provost.virginia.edu/subsite/genai.  Accessed: 08 July 2023.
  * UCL ([n.d.]) [n.d.].  Guidance For the Use of Generative AI.  https://teaching.ucla.edu/resources/ai_guidance/.  Accessed: 08 July 2023.
  * els ([n.d.]) [n.d.].  Publishing Ethics — Elsevier Policy.  https://beta.elsevier.com/about/policies-and-standards/publishing-ethics?trial=true#4-duties-of-authors Accessed: 08 July 2023.
  * UTo ([n.d.]) [n.d.].  Using ChatGPT or other generative AI tool on a marked assessment.  https://www.academicintegrity.utoronto.ca/perils-and-pitfalls/using-chatgpt-or-other-ai-tool-on-a-marked-assessment/.  Accessed: 06 September 2023.
  * aaa (2019) 2019\.  AAAI Code of Professional Ethics and Conduct.  https://aaai.org/about-aaai/ethics-and-diversity/.  Accessed: 08 July 2023.
  * iee (2020) 2020\.  IEEE Code of Ethics.  https://www.ieee.org/about/corporate/governance/p7-8.html Accessed: 08 July 2023.
  * acm (2023a) 2023a.  ACM Code of Ethics.  https://www.acm.org/code-of-ethics Accessed: 08 July 2023.
  * acm (2023b) 2023b.  ACM Policy on Authorship.  https://www.acm.org/publications/policies/new-acm-policy-on-authorship Accessed: 08 July 2023.
  * Duk (2023) 2023\.  AI and Teaching at Duke.  https://learninginnovation.duke.edu/ai-and-teaching-at-duke/.  Accessed: 08 July 2023.
  * Yal (2023) 2023\.  AI Guidance.  https://poorvucenter.yale.edu/AIguidance.  Accessed: 08 July 2023.
  * Ade (2023) 2023\.  Artificial Intelligence.  https://libguides.adelaide.edu.au/c.php?g=959585&p=6965069.  Accessed: 08 July 2023.
  * auc (2023) 2023\.  Student Academic Conduct Statute.  https://cdn.auckland.ac.nz/assets/auckland/about-us/about-the-university/policy-hub/Student%20Academic%20Conduct%20Statute%20-%20approved%20by%20Council%202023-03-14(1).pdf Accessed: 08 July 2023.
  * dea (2023) 2023\.  Student Academic Integrity Policy.  https://policy.deakin.edu.au/document/view-current.php?id=107&_ga=2.136370873.847668142.1687377795-888769290.1685566682 Accessed: 08 July 2023.
  * MIT (2023) 2023\.  Teaching & Learning with ChatGPT: Opportunity or Quagmire? Part III.  https://tll.mit.edu/teaching-learning-with-chatgpt-opportunity-or-quagmire-part-iii/.  Accessed: 08 July 2023.
  * Mon (2023) 2023\.  Using Artificial Intelligence.  https://www.monash.edu/learnhq/build-digital-capabilities/create-online/using-artificial-intelligence.  Accessed: 08 July 2023.
  * ACM/IEEE-CS Joint Task Force on Computing Curricula (2013) ACM/IEEE-CS Joint Task Force on Computing Curricula. 2013.  _Computer Science Curricula 2013_.  Technical Report. ACM Press and IEEE Computer Society Press.  https://doi.org/10.1145/2534860
  * Ahmed et al. (2023) Toufique Ahmed, Noah Rose Ledesma, and Premkumar Devanbu. 2023\.  SynShine: Improved Fixing of Syntax Errors.  _IEEE Transactions on Software Engineering_ 49, 4 (2023), 2169–2181.  https://doi.org/10.1109/TSE.2022.3212635
  * Al-Hossami et al. (2023) Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Mahajan, and Mohsen Dorodchi. 2023\.  Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_. 709–726.
  * Albluwi (2019) Ibrahim Albluwi. 2019\.  Plagiarism in Programming Assessments: A Systematic Review.  _ACM Trans. Comput. Educ._ 20, 1, Article 6 (Dec. 2019), 28 pages.  https://doi.org/10.1145/3371156
  * Allen et al. (2019) Joe Michael Allen, Frank Vahid, Alex Edgcomb, Kelly Downey, and Kris Miller. 2019.  An Analysis of Using Many Small Programs in CS1 _(SIGCSE ’19)_. ACM, NY, NY, USA, 585–591.  https://doi.org/10.1145/3287324.3287466
  * Alves and Cipriano (2023) Pedro Alves and Bruno Pereira Cipriano. 2023.  The centaur programmer – How Kasparov’s Advanced Chess spans over to the software development of the future.  arXiv:2304.11172 [cs.HC]
  * Amani et al. (2023) Sara Amani, Lance White, Trini Balart, Laksha Arora, Dr. Kristi J. Shryock, Dr. Kelly Brumbelow, and Dr. Karan L. Watson. 2023\.  Generative AI Perceptions: A Survey to Measure the Perceptions of Faculty, Staff, and Students on Generative AI Tools in Academia.  arXiv:2304.14415 [cs.HC]
  * Apiola et al. (2023) Mikko Apiola, Sonsoles López-Pernas, and Mohammed Saqr. 2023\.  _Past, Present and Future of Computing Education Research: A Global Perspective_.  Springer Nature.
  * Armstrong et al. (2023) Lena Armstrong, Jayne Everson, and Amy J. Ko. 2023\.  Navigating a Black Box: Students’ Experiences and Perceptions of Automated Hiring. In _Proceedings of the 2023 ACM Conference on International Computing Education Research V.1_ (Chicago, IL, USA) _(ICER ’23 V1)_. Association for Computing Machinery, New York, NY, USA, 148–158.  https://doi.org/10.1145/3568813.3600123
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021\.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Babe et al. (2023) Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson. 2023.  StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code.  _arXiv preprint arXiv:2306.04556_ (2023).
  * Balse et al. (2023) Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023.  Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 292–298.  https://doi.org/10.1145/3587102.3588852
  * Bao and Hosseini (2023) Yeting Bao and Hadi Hosseini. 2023.  Mind the Gap: The Illusion of Skill Acquisition in Computational Thinking. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 778–784.  https://doi.org/10.1145/3545945.3569749
  * Barke et al. (2023) Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023\.  Grounded copilot: How programmers interact with code-generating models.  _Proceedings of the ACM on Programming Languages_ 7, OOPSLA1 (2023), 85–111.
  * Becker (2017) Brett Becker. 2017\.  Artificial Intelligence in Education: What Is It, Where Is It Now, Where Is It Going.  _Ireland’s Yearbook of Education_ 2018 (2017), 42–46.
  * Becker et al. (2023) Brett Becker, James Prather, Paul Denny, Andrew Luxton-Reilly, James Finnie-Ansley, and Eddie Antonio Santos. 2023.  Programming Is Hard - Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation _(SIGCSE ’23)_. ACM.
  * Becker et al. (2019) Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice L. Pearce, and James Prather. 2019\.  Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research _(ITiCSE-WGR ’19)_. ACM, NY, NY,USA, 177–210.  https://doi.org/10.1145/3344429.3372508
  * Becker and Fitzpatrick (2019) Brett A. Becker and Thomas Fitzpatrick. 2019.  What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students?. In _Proceedings of the 50th ACM Technical Symposium on Computer Science Education_ (Minneapolis, MN, USA) _(SIGCSE ’19)_. Association for Computing Machinery, New York, NY, USA, 1011–1017.  https://doi.org/10.1145/3287324.3287485