BibTeX

# Evaluating Language Models for Generating and Judging Programming Feedback

Charles Koutcheme<EMAIL_ADDRESS>Aalto UniversityEspooFinland
0000-0002-2272-2763 ,  Nicola Dainese<EMAIL_ADDRESS>Aalto
UniversityEspooFinland 0000-0001-9806-419X ,  Arto Hellas
<EMAIL_ADDRESS>Aalto UniversityEspooFinland 0000-0001-6502-209X ,  Sami
Sarsa<EMAIL_ADDRESS>University of JyväskyläJyväskyläFinland
0000-0002-7277-9282 ,  Juho Leinonen<EMAIL_ADDRESS>Aalto
UniversityEspooFinland 0000-0001-6829-9449 ,  Syed Ashraf
<EMAIL_ADDRESS>Aalto UniversityEspooFinland and  Paul Denny
<EMAIL_ADDRESS>The University of AucklandAucklandNew Zealand
0000-0002-5150-9806

###### Abstract.

The emergence of large language models (LLMs) has transformed research and
practice in a wide range of domains. Within the computing education research
(CER) domain, LLMs have received plenty of attention especially in the context
of learning programming. Much of the work on LLMs in CER has however focused
on applying and evaluating proprietary models. In this article, we evaluate
the efficiency of open-source LLMs in generating high-quality feedback for
programming assignments, and in judging the quality of the programming
feedback, contrasting the results against proprietary models. Our evaluations
on a dataset of students’ submissions to Python introductory programming
exercises suggest that the state-of-the-art open-source LLMs (Meta’s Llama3)
are almost on-par with proprietary models (GPT-4o) in both the generation and
assessment of programming feedback. We further demonstrate the efficiency of
smaller LLMs in the tasks, and highlight that there are a wide range of LLMs
that are accessible even for free for educators and practitioners.

open source, large language models, generative AI, automatic feedback,
automatic evaluation, programming feedback, LLM-as-a-judge

††ccs: Social and professional topics Computing education

##  1\. Introduction

High-quality and timely feedback is essential for students in programming
courses. Some types of feedback, such as whether or not a program runs or
passes a provided test suite, are readily available via simple automated means
(Keuning et al., 2016; Paiva et al., 2022). However, feedback on the causes of
subtle programming errors and suggestions for resolving them can be difficult
to produce (Keuning et al., 2018). Especially in large classes, providing
accurate and personalised explanations of bugs as feedback to students is a
manual and time consuming task for educators, and yet essential for reducing
frustration and aiding learning.

The automated generation of human-like feedback has recently been made
possible thanks to the accessibilty of state-of-the-art generative AI tools,
such as ChatGPT. In particular, API access to powerful large language models
(LLMs) has sparked the development of many programming feedback tools that are
now being deployed in classrooms. These include tools to generate improved
error messages (Leinonen et al., 2023b), aid with real-time debugging
(Kazemitabaar et al., 2023), and to help explain code (Liffiton et al., 2024;
Leinonen et al., 2023a) and tailor next-step hints (Roest et al., 2024). Such
systems have shown success not only in generating feedback, but also in
assessing feedback quality, offering the potential for generating high-quality
feedback through iterative improvement.

Despite the promise of LLM-based feedback generation and evaluation
approaches, the vast majority of research and usage in computing education
contexts has relied on proprietary models such as GPT-4. This reliance on
closed-source LLMs is concerning for several reasons. It requires sending
potentially sensitive data to a third-party with no guarantees on how the data
will be used, a lack of insight into the way models are trained and what
deliberate or inadvertant biases they may contain, and unpredictable licensing
expenses (Kukreja et al., 2024). Open source LLMs, on the other hand, are
freely accessible and open for modification and distribution, and have started
to become viable alternatives. Despite this, very few studies have explored
their capabilities for providing or assessing programming feedback.

In this work, our goal is to investigate the potential for open-source models
to produce high-quality feedback, and to assess the quality of feedback
generated by other LLMs. We focus on feedback consisting of explanations of
bugs or issues in student-written programs and the steps to address these
issues. While prior work suggests that open-source language models offer
competitive alternatives to proprietary models for the generation of feedback,
the extent to which they can be used as judges (validators) of such feedback
remains unknown. Using a publicly available benchmark dataset of student-
written programs, we address the following two research questions:

RQ1:

How do open- and closed-source models compare with respect to the quality of
their generated bug explanations and suggested fixes?

RQ2:

To what extent can open- and closed-source models assess the quality of
programming feedback generated by other models relative to expert human
judgment?

To answer our first research question, we generate explanations of bugs and
their corresponding fixes using five state-of-the-art open-source and two
popular proprietary language models. We manually evaluate this feedback using
a custom rubric that includes the completeness and comprehensibility of the
explanations and the accuracy of the suggested fixes. To answer our second
research question, we use these expert human-generated ground truth labels to
evaluate the performance of the models on the task of judging the programming
feedback.

Our findings suggest that open-source language models are competitive to
proprietary models both for the generation and assessment of programming
feedback. Given the potential benefits that open-source models offer around
transparency, trust and cost, we argue that they should be increasingly used
in computing education contexts.

##  2\. Related work

###  2.1. Using Language Models For Feedback

Automating assessment of programming exercises and providing feedback on the
exercises have been studied for decades within the computing education
research domain (Keuning et al., 2018; Messer et al., 2024; Paiva et al.,
2022). Classically, much of the existing work on automating feedback has
focused on informing students about mistakes in their code, while providing
formative feedback has been less common (Keuning et al., 2018). Providing
suggestions on the location of the issue or hints on how to fix the issue can
improve students’ performance over just pointing out that a test failed (Hao
et al., 2022), but manually creating quality feedback can be very time-
consuming. For example, the authors of (Vihavainen et al., 2012) describe
creating over 4000 hand-written “scaffolding” messages for a programming
course.

The recent emergence of powerful language models has led to researchers
exploring their capabilities for programming feedback (Balse et al., 2023;
Liffiton et al., 2024; Hellas et al., 2023; Kiesler et al., 2023; Pankiewicz
and Baker, 2023; Leinonen et al., 2023b; Phung et al., 2023a, b; Prather et
al., 2023) and, in general, the observations on the quality or utility of
feedback has evolved with the introduction of better language models (Hellas
et al., 2023). As an example, GPT-3 had high variability in the quality of
feedback, at times generating incorrect and inconsistent feedback (Balse et
al., 2023), while GPT-3.5 would often provide meaningful feedback and find
issues in code, but also often hallucinate issues that were not present in the
code (Hellas et al., 2023). Language models are also better at detecting some
types of errors than others (Hellas et al., 2023; Kiesler et al., 2023), being
useful especially for providing feedback on syntax or compilation errors
(Kiesler et al., 2023; Leinonen et al., 2023b; Phung et al., 2023a). Despite
the advances, even the state-of-the-art models like GPT-4 are still not on par
with humans when generating feedback for programming exercises (Phung et al.,
2023b). At the same time, there are increasing amounts of evidence from the
use of language model -powered feedback systems and chatbots in programming
(Liffiton et al., 2024; Hicke et al., 2023; Hellas et al., 2024; Wang et al.,
2024), which can further aid students, at least when the technologies of the
courses are such that they have had sufficient exposure prior to the knowledge
cutoff point of the used language models (Hellas et al., 2024).

The majority of existing work on language models for programming feedback in
the computing education research context has focused on utilizing proprietary
models (mainly from OpenAI), while the use of open-source models has received
only little attention. Calls for increasing use of open-source models have
been voiced (Yan et al., 2023), already due to potential privacy issues
related to sharing student data with language model providers. Work on
utilizing open-source models for the task is also starting to emerge, where
one of the research aspects has been contrasting the performance of open-
source models to the proprietary ones; researchers have already observed that
open-source models are on par with models such as GPT-3.5-Turbo for
programming feedback (Koutcheme et al., 2024b).

In our work, our first research question re-investigates how various language
models, including open-source ones, perform in explaining issues in student
programs and providing fixes, complementing prior studies.

###  2.2. Using Language Models as Judges

The idea of using an LLM to judge the output of other LLMs – LLMs-as-judges –
was first studied in the work of Zheng et al. (Zheng et al., 2023), showing
good promise, but also limitations, e.g., in grading math and reasoning tasks.
Since then, GPT-4 has been used in multiple studies as a judge of the quality
of other LLMs’ generations (Liu et al., 2023; Chiang and yi Lee, 2023), also
in educational contexts (Koutcheme et al., 2024b, a; Scarlatos et al., 2024).
Moreover, the reliance on GPT-4, a proprietary model, has sparked interest in
leveraging other open-source language models to act as judges (Wang et al.,
2023; Zhu et al., 2023; Kim et al., 2023). Yet, recent work (Verga et al.,
2024) has highlighted the limitations of relying on a single language model
for evaluating the quality of other language models’ output, and suggested to
employ a diverse ensemble of smaller models from different LLM families as a
jury for cheaper and less biased evaluations. When answering our second
research question, we test this hypothesis by comparing the usage of single
judges (both open-source and proprietary) and that of a jury of smaller open-
source language models.

##  3\. Methodology

In this section, we describe our methodology for answering our two research
questions. We first introduce the dataset used in our evaluations, then our
methods used for answering RQ1 and RQ2.

###  3.1. Dataset

We use data from the Socratic guidance benchmark (Al-Hossami et al., 2024),
which consists of 57 introductory-level programming assignments requiring
students to write functions. Each of the assignments is accompanied by the
associated test cases, a unique incorrect student solution, the ground truth
descriptions of the bugs in the program, the required bug fixes, and several
conversation threads between a fake student and a teaching assistant. The
ultimate goal for the benchmark is evaluating LLMs’ ability to help students
using the socratic method, i.e., guiding students in finding a solution on
their own, by asking a series of relevant questions that help their reasoning.
However, for this study, we focus solely on identifying the issues in the code
and any required fixes, as it is a fundamental step for effective use of LLMs
for socratic guidance – models incapable of identifying issues in students’
programs would be likely to provide them erroneous guidance. We leave the
problem of evaluating LLMs for socratic guidance for future work.

###  3.2. Generating High-Quality Feedback

Given a student’s incorrect program, our goal regarding RQ1 is to evaluate
LLMs ability to provide two particular types of feedback: explanations of the
bugs in the program and suggested fixes for the found bugs.

#### Feedback generation.

We prompt the models to provide feedback according to following example:

0 You are a CS professor teaching introductory programming using Python.

1 Below are a problem description, test cases, and an incorrect program
written by a student (i.e., it does not pass all test cases).
<problem description>, <test cases>, <student code>
First, list all the bugs in the program that prevent it from passing all unit
tests, then list a series of fixes to address these bugs.

List of bugs and fixes 2

To elaborate, we provide a language model: 0 a system prompt and 1 a
description of the task (with all the necessary contextual information), which
results in output 2.

#### Feedback Language Models.

We consider the following open-source models: Gemma-2B (Google, 2024),
Phi-3-mini (Abdin et al., 2024) (3.8B parameters), Mistral-7B (Jiang et al.,
2023), Llama3-8B (Touvron et al., 2023), Llama3-70B (Touvron et al., 2023). We
choose these models because of their extensive documentation, community
adoption, strong performance on code and language reasoning benchmarks (e.g.,
HumanEval (Chen et al., 2021) and MMLU (Hendrycks et al., 2021)), for their
parameter count, and their ability to follow instructions. This selection
covers the recent state-of-the-art models from various companies across the
most used model-sizes for LLMs. Furthermore, we also evaluate two of OpenAI’s
proprietary flagship models, GPT-3.5-turbo and GPT-4o, well representative of
the current industry standards.

We query proprietary models using the OpenAI Python library, and open-source
ones with the EasyLLM (Schmid, 2023) Python library to simplify querying them
through the HuggingFace Inference API. All models are evaluated using greedy
decoding. Next, we explain the annotation process before detailing the grading
rubric.

#### Annotation.

We use the seven models presented above, and the 57 programs of the benchmark,
which results in 7 × 57 = 399 model outputs. To answer our first research
question, two annotators (two authors of the paper) annotated all 399 model
outputs as follows. First, we selected 11 problems out of the 56 available
problems using the manual annotation subset presented in (Al-Hossami et al.,
2024). Then, the two annotators independently annotated 79 model outputs on an
initial description of each grading criterion on this subset. We then computed
an inter-annotator agreement score using Cohen’s Kappa coefficient. The
resulting annotation process yielded a moderate inter-rater agreement of 0.49.
Discrepancies were discussed and resolved to align the annotators’
understanding. After comparing annotations and discussing conflicts in
understanding, the two authors refined the grading rubric description
(presented shortly). The remaining feedback examples were split between the
two annotators (169 and 151 feedback respectively). The final annotated
dataset formed the basis for evaluating the quality of the feedback generated
by the language models.

#### Grading Criteria.

During the final annotation phase, each expert used the following grading
criteria for evaluating the quality of a single generated bug explanation (E),
and the quality of the generated fixes (F):

  * •

EA - Explanation Accurate: the explanation identifies and mentions all ground
truth bugs in the student program.

  * •

ES - Explanation Selective: the explanation does not mention non-existent (or
non-relevant) bugs or issues.

  * •

EC - Explanation Clear: the generated explanation is easy to understand by a
novice programmer, presented in a readable format, and does not contain
redundant or too little information (i.e., it is not vague about the cause of
the issue). Note: this criterion is independent of the correctness of the
explanations.

  * •

FA - Fixes Accurate: all required bug fixes are explained.

  * •

FS - Fixes Selective: no unnecessary changes are outlined;

  * •

FC - Fixes Clear: fixes are succinct and mention the unique changes to perform
in the code.

These criteria extend prior work (Koutcheme et al., 2024b; Hellas et al.,
2023; Phung et al., 2023c). The first two criteria represent the correctness
of the explanations. The annotators followed the following guidelines: for
each ground truth bug (provided in the dataset), match the bug with the
descriptions in the model output. If some of the generated model descriptions
did not match, the criteria ES was set to false. Then, independently of the
correctness, we looked at whether or not a novice programmer (unaware of the
real issues) could understand the meaning of the provided bug description. We
follow the same strategy for the fixes. Moreover, for the clarity criterion,
we ensure that the fixes provide clear descriptions of changes with snippets
or at least highlight changes in a repaired program (if present).

In addition, as during the experiments we observed that the generated feedback
often included repaired programs, even though we did not prompt the model for
this, we report the correctness of these program repair suggestions (RC -
Repair Correct). We evaluated the correctness of these repairs using the
associated unit tests from the original dataset. Although program repairs are
not the primary focus of this paper, they represent another valuable form of
feedback for students and can later be useful for hint generation.

###  3.3. Automatic Feedback Evaluation

In this subsection, we present the methods we used to automatically evaluate
the quality of LLM-generated feedback using other language models (answering
RQ2). We explored two approaches (a single LLM as a judge, and an ensemble of
LLMs as a jury) on two scenarios, depending on whether a reference answer is
available or not. We first describe how we generate the responses to the
grading criteria using a single LLM as a judge for the two scenarios. We then
outline our ensemble of LLMs and how we obtain the jury annotations.

#### No Reference Answer Available.

Given the feedback generated by a language model, we prompt another language
model (the judge) to grade this feedback (according to the criteria outlined
in Section 3.2) using the prompting strategy shown and described below:

…

3 Below is a list of bugs and their fixes written by a teaching assistant.
<List of bugs and fixes>
Your tasks are as follows:
1\. Compare the TA’s feedback (bug descriptions and fixes) against the
provided ground truth. 2\. Evaluate the quality of the TA’s feedback based on
the following grading criteria:
<grading criteria>
Write your comparisons, then, for each grading criterion, provide your answer
on a separate line using the format "criteria_name: yes/no".

…
<FS - yes/no>
4

Before asking the judge language model to grade the quality of the feedback 3
(using the grading criteria list as part of the prompt), we first ask the
judge to generate its own descriptions of the bugs and issues in the student
program, as described in Section 3.2 (012). This strategy is a form of zero-
shot-chain-of-thought (Kojima et al., 2023) Single Answer Grading (SAG) (Zheng
et al., 2023), i.e., the judge uses its solution to the “problem” to grade the
solution of another language model. This first scenario without a reference
answer is applicable for educators and practitioners interested in evaluating
language models’ feedback abilities on their private datasets without the need
for ground truth annotations.

#### Reference Answer Available.

Educators may also be interested in evaluating their language models on
existing benchmarks containing ground truth annotations of issues. Although
such benchmarks are not abundant (Koutcheme et al., 2024c), we expect more to
come as educational AI advances and becomes more widespread. In consideration
of this scenario, we experiment with providing the judge with the ground truth
descriptions of issues, instead of the more error-prone approach of generating
them using the judge itself. The example below illustrates our prompting
strategy:

You are a CS professor teaching introductory programming using Python.

Below are a problem description, test cases, and an incorrect program written
by a student (i.e., it does not pass all test cases). You are also provided
with the ground truth description of the bugs in that program and the required
fixes.
<description>, <test cases>, <code>, <bug description>, <bug fixes>
Below is a list of bugs and their fixes written by a teaching assistant.
<LLM feedback generation>
Your tasks are as follows: …

This prompting strategy is a form of reference grading (Zheng et al., 2023).
We refer to this strategy as GAG (Ground truth Annotated Grading). We note
that for both strategies (SAG, GAG), we extract the response to each grading
criterion from the final judge model output.

#### Ensemble of Judges.

Prior work suggests that using a single LLM as a judge has limitations. For
example, GPT-4 favours outputs from OpenAI’s GPT line of models (Rajani et
al., 2023). As mentioned in Section 2.2, instead of using a single LLM, Verga
et al. (Verga et al., 2024) showed that using multiple language models from
different families can address many of these issues. We aim to test their
hypothesis in our educational context. We prompt three popular SOTA open
source language models to provide their judgement, and then, we combine the
model decisions using majority voting separately for each criterion. For
example, given three LLMs outputting yes, yes, and no respectively for a given
criterion, the final ensemble result will be yes. We consider this “jury” both
when reference answers are available (GAG) and not available (SAG).

#### Judge and Jury Language Models.

We evaluate both proprietary and open-source models as judges. We use
GPT-3.5-turbo and extend prior work (Koutcheme et al., 2024b) by also
including GPT-4o. We compare these proprietary models against state-of-the-art
open-source language models Phi-3-mini, Llama3-8B, Mistral-7B, and Llama3-70B.
For the jury, we use the three language models: Llama3-8B, Mistral-7B, and
Phi-3-mini. We selected these three strong LLMs due to them being at the top
of the leaderboards for LLMs of their size, and being from different families
of models. As with the feedback generation, we query these language models
using the OpenAI and EasyLLM Python libraries and obtain the outputs using
greedy decoding.

#### Evaluation

To answer RQ2, we compare the evaluation of each judge/jury model for the 399
generated feedback against the ground truth annotations in our dataset,
according to the grading criteria. We then report the performance of each
judge/jury across all of their outputs and for our two scenarios. We report
the weighted average version of the f0.5 score (Koutcheme et al., 2024b) for
each judge/jury as this metric takes into account false positives and
potential class imbalances. We also report the kappa score to complement our
observations.

##  4\. Results

###  4.1. Generating Feedback

Table 1 shows the performance of each language model on various grading
criteria, including both individual and grouped criteria, based on human
evaluations. We make the following observations.

Table 1. Feedback results for various language models based on human
evaluations. Bold: best results for each column.

feedback model | individual | grouped
---|---|---
| EA | ES | EC | FA | FS | FC | RC | EA,ES,EC | FA,FS,FC | FA,FS,FC,RC | ALL | EA,ES,FA,FS
Gemma-2B | 0.32 | 0.02 | 0.67 | 0.32 | 0.02 | 0.65 | 0.07 | 0.02 | 0.02 | 0.00 | 0.00 | 0.02
Phi-3-mini | 0.86 | 0.11 | 0.79 | 0.81 | 0.09 | 0.81 | 0.67 | 0.09 | 0.07 | 0.07 | 0.05 | 0.05
Mistral-7B | 0.68 | 0.25 | 0.91 | 0.63 | 0.25 | 0.86 | 0.40 | 0.23 | 0.19 | 0.11 | 0.09 | 0.21
Llama3-8B | 0.79 | 0.16 | 0.74 | 0.77 | 0.18 | 0.75 | 0.37 | 0.12 | 0.14 | 0.07 | 0.05 | 0.16
Llama3-70B | 0.89 | 0.28 | 0.93 | 0.86 | 0.33 | 0.89 | 0.68 | 0.26 | 0.26 | 0.23 | 0.19 | 0.23
GPT-3.5-turbo | 0.89 | 0.39 | 0.91 | 0.82 | 0.37 | 0.89 | 0.58 | 0.35 | 0.26 | 0.18 | 0.18 | 0.32
GPT-4o | 0.96 | 0.32 | 0.88 | 0.86 | 0.35 | 0.88 | 0.82 | 0.26 | 0.32 | 0.30 | 0.23 | 0.30

Table 2. Judging results.

(a) Detailed f0.5 scores. We show the SAG score and in parenthesis the GAG
score. Legend:
AVGO: (resp. AVGS) average f0.5 overall criteria when judging other models’
(resp. the judge’s own) feedback.

judge | EA | ES | EC | FA | FS | FC | AVGO | AVGS
---|---|---|---|---|---|---|---|---
Phi-3-mini | 0.67 (+0.13) | 0.11 (-0.04) | 0.69 (+0.04) | 0.59 (+0.12) | 0.14 (+0.19) | 0.70 (+0.07) | 0.43 (+0.11) | 0.40 (-0.01)
Mistral-7B | 0.63 (+0.15) | 0.06 (+0.15) | 0.72 (+0.01) | 0.57 (+0.18) | 0.08 (+0.14) | 0.71 (+0.06) | 0.40 (+0.14) | 0.38 (+0.09)
Llama3-8B | 0.64 (+0.11) | 0.05 (+0.19) | 0.72 (+0.00) | 0.58 (+0.08) | 0.07 (+0.19) | 0.71 (+0.03) | 0.41 (+0.10) | 0.34 (+0.13)
Ensemble | 0.63 (+0.17) | 0.05 (+0.05) | 0.72 (+0.00) | 0.55 (+0.17) | 0.06 (+0.19) | 0.71 (+0.06) | 0.39 (+0.13) | /
Llama3-70B | 0.76 (-0.01) |  0.25 (+0.53) | 0.72 (+0.00) | 0.72 (+0.06) |  0.32 (+0.43) | 0.77 (-0.05) | 0.54 (+0.18) | 0.51 (+0.18)
GPT-3.5-turbo | 0.69 (-0.01) | 0.16 (+0.05) | 0.71 (+0.01) | 0.64 (-0.01) | 0.24 (+0.07) | 0.72 (-0.01) | 0.46 (+0.00) | 0.53 (+0.09)
GPT-4o | 0.71 (-0.01) |  0.72 (+0.02) |  0.77(+0.00) | 0.69 (+0.00) |  0.74 (-0.01) | 0.74 (+0.01) |  0.66 (+0.01) |  0.61 (+0.05)

(b) Kappa scores.
SAG (+- GAG diff)

judge | kappa
---|---
Phi-3-mini | 0.01 (+0.08)
Mistral-7B | 0.00 (+0.09)
Llama3-8B | 0.00 (+0.07)
Ensemble | 0.00 (+0.08)
Llama3-70B | 0.09 (+0.31)
GPT-3.5-turbo | 0.04 (+0.01)
GPT-4o | 0.30 (+0.03)

#### Open-Source vs. Proprietary Models.

The GPT-4o model shows very strong performance across nearly all individual
and grouped criteria. Among the open-source models, there is significant
variance in performance across different criteria. For instance, Llama-3-70B
performs on par with or better than GPT-3.5-turbo on the EA, EC, and FA, FC,
RC individual criteria. Llama-3-70B even remains competitive with both
GPT-3.5-turbo and GPT-4o for generating perfect explanations (EA, ES, EC), and
perfect fixes (FA, FS, FC, RC). In contrast, smaller models such as Gemma-2B
perform poorly across the board. However, size alone does not determine
performance, as some small open-source language models, like Phi-3-mini,
despite their smaller size, perform competitively on several criteria, notably
EA, FA, and RC.

#### Strengths and Weaknesses.

Each model has its strengths and weaknesses. However, we notice that most
models struggle with selectivity (i.e., they identify irrelevant or redundant
issues), while they generally produce comprehensible outputs (i.e., well-
formatted responses). When looking at the feedback generations, the stronger
models (e.g. Llama-3-70B, and the GPTs) often added performance suggestions
(e.g. replace a for loop with a built-in function), while the other models
often added incorrect outputs. This indicates a broader challenge in
developing models that can effectively identify and focus on relevant issues
without including redundant or irrelevant information. Improvements in this
area could lead to substantial overall performance gains.

#### Program Repairs.

The RC column in Table 1 shows the proportion of generated correct repairs.
The repair correctness (RC) scores show considerable variation, with GPT-4o
scoring the highest. Interestingly, the correctness of the repairs does not
correlate individually with either the quality of the explanations or with the
accuracy of the fixes. However, the models that are better at writing both
accurate and hallucination-free explanations _and_ fixes (i.e., EA, ES, FA,
FS) are also better at generating high-quality repairs, aligning with previous
observations (Koutcheme et al., 2024a). Similarly as in (Koutcheme et al.,
2024a), GPT-3.5-turbo is the only model that disrupts this relationship.

###  4.2. Evaluating Feedback

Table 2(b) shows the results of the judgment task, detailing the f0.5 scores
and kappa scores for each language model under the two scenarios (SAG, and
GAG). We can make several observations from these results:

#### Open-Source vs. Proprietary Models.

Among the open-source models, Llama3-70B stands out as a better judge than
GPT-3.5-turbo when considering the f0.5 scores, particularly in the
Explanation Accurate (EA), Explanation Selective (ES), Fixes Accurate (FA),
and Fixes Clear (FC) criteria. Notably, without ground truth bug listings,
GPT-4o is the only model performing consistently well among the criteria,
either as the best or near the best model. This is most clear in the
selectivity criteria (ES and FS) where GPT-4o has over double the next-best
f0.5 score. Providing the models with ground truth descriptions of the bugs
and issues in the programs significantly improves their results overall
criteria (almost over 10%) compared to relying on their own explanations of
the issues. Notably, strong models like Llama3-70B and GPT variants did not
benefit as much in completeness (EA), suggesting they already handle this
criterion better. Interestingly, when given ground truth annotations (GAG),
Llama3-70B outperforms GPT-4o across almost all criteria, showing the most
significant improvement from these annotations. However, once again all
models, except GPT-4o, struggle the most with selectivity (ES), although
ground truth annotations significantly boost the score for all models
(especially for Llama3-70B). This indicates that the judges struggle to detect
when the feedback contains hallucinated or non-existent bugs. This could be
due to further hallucination issues and highlights an important area of
improvement.

#### Kappa Scores.

The low kappa scores in the SAG setting indicate that most model annotations
could be due to “chance”. When investigating the reasons for the scores, we
see that most models tend to be overly positive, predicting “yes” for almost
all criteria. This phenomenon aligns with observations in (Koutcheme et al.,
2024b), highlighting a tendency of models to overestimate the quality of
feedback. When provided with ground truth annotations, the results improve to
reach a moderate level of agreement, in particular for the Llama models.

#### Self-Evaluation vs. Evaluation of Others.

Models perform better when evaluating other models’ outputs rather than their
own. This might be due to a bias toward positive evaluations, especially
within models of the same family, as noted in previous research (Zheng et al.,
2023).

#### Ensemble Performance.

Combining multiple models into an ensemble does not improve judgment quality;
instead, it biases the results. The ensemble approach, which averaged the
models Phi-3-mini, Mistral-7B, and Llama3-8B, did not yield better
performance. This contrasts with previous work by Verga et al. (Verga et al.,
2024), possibly due to the absence of few-shot examples and the overall lower
performance of the individual models used here. The previous study involved
models that were competitive with GPT-3.5-turbo, leading to better ensemble
performance.

##  5\. Discussion

#### Teaching and Learning Implications.

Our study aims to show that a wide range of models are accessible to educators
and practitioners, some competitive models even for free. Llama3-70B is shown
competitive against GPT-3.5 for feedback generation and GPT-4o for judging the
quality of generated feedback. Educators might for instance rely on
GPT-3.5-turbo for generating feedback and Llama-3-70B for validating (Phung et
al., 2023c) the quality of this feedback. Llama3-70B’s ability to judge
feedback could be useful for training more performant smaller models, instead
of relying on GPT-4 (Scarlatos et al., 2024). Notably, the size of a language
model no longer correlates directly with performance. For example, the
Phi-3-mini model is competitive with the 7B Mistral and 8B Llama models, while
being able to run on consumer laptops and even phones. The model struggles
with selectivity, which reduces its performance. However, we believe that
fine-tuning techniques might make the results even stronger (Koutcheme, 2023).

Importantly, these open-source models are also easy to access thanks to
libraries such as EasyLLM and APIs offered by companies such as HuggingFace.
For instance, for conducting our experiments, using the EasyLLM package, all
models were freely accessible, except for Llama-70B (which required paying 9
dollars for a month of rate-limited access). We acknowledge that using an
external API for querying open-source language models might beat the purpose
of data privacy. However, several institutions leverage ChatGPT APIs in one
way or another (Liu et al., 2024), and HuggingFace platforms, which are
dedicated to open-source, offer the same data privacy guarantees.

#### Limitations.

Our work has limitations. Our prompts will affect the results and more
specific prompts (or different prompting strategies) might influence model
performance. Also, we only considered introductory programming assignments
written in Python and not other programming languages. Moreover, we only
considered two types of feedback, but other types exist. Our selection of
language models, although considered the recent state-of-the-art, does not
exhaust all possible alternatives to popular models. Our labelling process is
also not perfect, as we only used two raters, which resulted in a moderate
inter-rater reliability (0.49), and our grading criteria did not use actual
students (the intended audience) to rate the clarity of the outputs.
Additionally, for judging feedback, we could have used judge language models
which are specifically designed for evaluation, but we used generic language
models instead. Importantly, due to limitations of the EasyLLM library, the
Phi-3-mini results were aggregated on a subset of the feedback (370 feedback
instead of 399), which limits the interpretation of that specific model’s
judging performance.

#### Future Work.

Our overarching goal is to allow educators to keep track of new language
models available and how they could be useful for their purposes. In the
future, we will conduct a larger scale evaluation of open-source language
models’ ability to generate (and judge) other types of feedback and support.
In particular, we are extending the Socratic benchmark to include next-step
hints (Roest et al., 2024), and are running evaluation of LLMs for being
Socratic guides. We will study how performance for different types of feedback
relates to each other. We will also maintain an online leaderboard, similar to
what is done for instance on the HuggingFace leaderboard, and develop
benchmarks. Beyond the tracking of the model performance, we aim to improve
the ability of small language models (e.g. Phi-3-mini) to be teaching
assistants by using Reinforcement Learning techniques to tackle the
selectivity problem and exploring other generation parameter techniques. The
varied strengths of different models suggest that a combined approach
(ensemble methods) might yield even better results for generation. As our LLM
jury results contrasted those by Verga et al. (Verga et al., 2024), we also
intend to conduct a large-scale study on how the variability, individual
performance, and the number of judges in the LLM jury affect the performance
of the LLM jury in our context.

##  6\. Conclusions

In this paper, we evaluated (1) how language models perform for providing
explanations of issues in programs and generating bug fixes, and (2) how well
different large language models, including open-source ones, perform in
evaluating the quality of other LM-generated feedback. Our paper highlights
that top open-source language models are valid competitors to proprietary
language models both for generating and assessing the quality of programming
feedback. Open-source language models could have benefits to power free tools,
which is important for instance for less funded institutions. As an additional
contribution, we also release the code used for conducting our
experiments111https://github.com/KoutchemeCharles/feed_genju, as well as the
model answers and the annotator’s responses.

###### Acknowledgements.

This research was partially supported by the Research Council of Finland
(Academy Research Fellow grant number 356114).

## References

  * (1)
  * Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, et al. 2024\.  Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.  arXiv:2404.14219
  * Al-Hossami et al. (2024) Erfan Al-Hossami, Razvan Bunescu, Justin Smith, and Ryan Teehan. 2024.  Can Language Models Employ the Socratic Method? Experiments with Code Debugging. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1_ _(SIGCSE 2024)_. Association for Computing Machinery, New York, NY, USA, 53–59.  https://doi.org/10.1145/3626252.3630799
  * Balse et al. (2023) Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023.  Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_. 292–298.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. 2021\.  Evaluating language Models Trained on Code.  arXiv:2107.03374 [cs.LG]
  * Chiang and yi Lee (2023) Cheng-Han Chiang and Hung yi Lee. 2023.  Can Large Language Models Be an Alternative to Human Evaluations?  arXiv:2305.01937 [cs.CL] https://arxiv.org/abs/2305.01937
  * Google (2024) Google. 2024.  Gemma: Our open-source models for machine learning fairness.  https://blog.google/technology/developers/gemma-open-models/
  * Hao et al. (2022) Qiang Hao, David H Smith IV, Lu Ding, Amy Ko, Camille Ottaway, Jack Wilson, Kai H Arakawa, Alistair Turcan, Timothy Poehlman, and Tyler Greer. 2022.  Towards understanding the effective design of automated formative feedback for programming assignments.  _Computer Science Education_ 32, 1 (2022), 105–127.
  * Hellas et al. (2024) Arto Hellas, Juho Leinonen, and Leo Leppänen. 2024.  Experiences from Integrating Large Language Model Chatbots into the Classroom.  _arXiv preprint arXiv:2406.04817_ (2024).
  * Hellas et al. (2023) Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023.  Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In _Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1_. ACM, 93–105.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, et al. 2021\.  Measuring Coding Challenge Competence With APPS.  arXiv:2105.09938 [cs.SE]
  * Hicke et al. (2023) Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. 2023.  AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs.  arXiv:2311.02775 [cs.LG]
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. 2023\.  Mistral 7B.  arXiv:2310.06825 [cs.CL]
  * Kazemitabaar et al. (2023) Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023.  Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In _Proc. of the 2023 CHI Conf. on Human Factors in Computing Systems_. ACM, New York, NY, USA, Article 455, 23 pages.
  * Keuning et al. (2016) Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2016.  Towards a Systematic Review of Automated Feedback Generation for Programming Exercises. In _Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education_. ACM, 41–46.
  * Keuning et al. (2018) Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018.  A Systematic Literature Review of Automated Feedback Generation for Programming Exercises.  _ACM Trans. Comput. Educ._ 19, 1, Article 3 (2018), 43 pages.
  * Kiesler et al. (2023) Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023.  Exploring the Potential of Large Language Models to Generate Formative Programming Feedback.  _arXiv preprint arXiv:2309.00029_ (2023).
  * Kim et al. (2023) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, et al. 2023\.  Prometheus: Inducing Fine-grained Evaluation Capability in Language Models.  arXiv:2310.08491 [cs.CL]
  * Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023.  Large Language Models are Zero-Shot Reasoners.  arXiv:2205.11916 [cs.CL]
  * Koutcheme (2023) Charles Koutcheme. 2023.  Training Language Models for Programming Feedback Using Automated Repair Tools. In _Artificial Intelligence in Education_. Springer Nature Switzerland, 830–835.
  * Koutcheme et al. (2024a) Charles Koutcheme, Nicola Dainese, and Arto Hellas. 2024a.  Using Program Repair as a Proxy for Language Models’ Feedback Ability in Programming Education. In _Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)_ , Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, and Zheng Yuan (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 165–181.
  * Koutcheme et al. (2024b) Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024b.  Open Source Language Models Can Provide Feedback: Evaluating LLMs’ Ability to Help Students Using GPT-4-As-A-Judge. In _Proceedings of the 2024 Innovation and Technology in Computer Science Education, Volume 1_ (Milan, Italy) _(ITICSE ’24)_.  https://doi.org/10.1145/3649217.3653612
  * Koutcheme et al. (2024c) Charles Koutcheme, Nicola Dainese, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny. 2024c.  Benchmarking Educational Program Repair. In _NeurIPS’23 Workshop on Generative AI for Education (GAIED)_.
  * Kukreja et al. (2024) Sanjay Kukreja, Tarun Kumar, Amit Purohit, Abhijit Dasgupta, and Debashis Guha. 2024.  A Literature Survey on Open Source Large Language Models. In _Proceedings of the 2024 7th International Conference on Computers in Management and Business_ (Singapore, Singapore) _(ICCMB ’24)_. Association for Computing Machinery, New York, NY, USA, 133–143.  https://doi.org/10.1145/3647782.3647803
  * Leinonen et al. (2023a) Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023a.  Comparing Code Explanations Created by Students and Large Language Models. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 124–130.  https://doi.org/10.1145/3587102.3588785
  * Leinonen et al. (2023b) Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, et al. 2023b.  Using Large Language Models to Enhance Programming Error Messages. In _Proc. of the 54th ACM Technical Symposium on Computer Science Education V. 1_. ACM, New York, NY, USA, 563–569.
  * Liffiton et al. (2024) Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. 2024.  CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. In _Proc. of the 23rd Koli Calling Int. Conf. on Computing Education Research_. ACM, New York, NY, USA, Article 8, 11 pages.
  * Liu et al. (2024) Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J. Malan. 2024.  Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2_ (Portland, OR, USA) _(SIGCSE 2024)_. Association for Computing Machinery, New York, NY, USA, 1927.  https://doi.org/10.1145/3626253.3635427
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 2511–2522.  https://doi.org/10.18653/v1/2023.emnlp-main.153
  * Messer et al. (2024) Marcus Messer, Neil CC Brown, Michael Kölling, and Miaojing Shi. 2024.  Automated grading and feedback tools for programming education: A systematic review.  _ACM Transactions on Computing Education_ 24, 1 (2024), 1–43.
  * Paiva et al. (2022) José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022.  Automated Assessment in Computer Science Education: A State-of-the-Art Review.  _ACM Trans. Comput. Educ._ 22, 3, Article 34 (2022), 40 pages.
  * Pankiewicz and Baker (2023) Maciej Pankiewicz and Ryan S. Baker. 2023.  Large Language Models (GPT) for automating feedback on programming assignments.  arXiv:2307.00150 [cs.HC]
  * Phung et al. (2023a) Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, et al. 2023a.  Generating High-Precision Feedback for Programming Syntax Errors using language Models.  arXiv:2302.04662 [cs.PL]
  * Phung et al. (2023b) Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, et al. 2023b.  Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors.  _Int. J. of Management_ 21, 2 (2023), 100790.
  * Phung et al. (2023c) Tung Phung, Victor-Alexandru Pădurean, Anjali Singh, Christopher Brooks, José Cambronero, et al. 2023c.  Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation.  arXiv:2310.03780 [cs.AI]
  * Prather et al. (2023) James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, et al. 2023\.  The Robots are Here: Navigating the Generative AI Revolution in Computing Education.  In _Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education_. 108–159.
  * Rajani et al. (2023) Nazneen Rajani, Nathan Lambert, Sheon Han, Jean Wang, Osvald Nitski, et al. 2023\.  Can foundation models label data like humans?  https://huggingface.co/blog/llm-v-human-data.
  * Roest et al. (2024) Lianne Roest, Hieke Keuning, and Johan Jeuring. 2024.  Next-Step Hint Generation for Introductory Programming Using Large Language Models. In _Proceedings of the 26th Australasian Computing Education Conference_ (Sydney, NSW, Australia) _(ACE ’24)_. Association for Computing Machinery, New York, NY, USA, 144–153.  https://doi.org/10.1145/3636243.3636259
  * Scarlatos et al. (2024) Alexander Scarlatos, Digory Smith, Simon Woodhead, and Andrew Lan. 2024.  Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. In _Artificial Intelligence in Education_ , Andrew M. Olney, Irene-Angelica Chounta, Zitao Liu, Olga C. Santos, and Ig Ibert Bittencourt (Eds.). Springer Nature Switzerland, Cham, 280–294.
  * Schmid (2023) Philipp Schmid. 2023.  _EasyLLM: Streamlined Tools for LLMs_.  https://github.com/philschmid/easyllm
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, et al. 2023\.  LLaMA: Open and Efficient Foundation Language Models.  arXiv:2302.13971 [cs.CL]
  * Verga et al. (2024) Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024.  Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models.  arXiv:2404.18796 [cs.CL] https://arxiv.org/abs/2404.18796
  * Vihavainen et al. (2012) Arto Vihavainen, Matti Luukkainen, and Jaakko Kurhila. 2012.  Multi-faceted support for MOOC in programming. In _Proceedings of the 13th annual conference on Information technology education_. 171–176.
  * Wang et al. (2024) Sierra Wang, John Mitchell, and Chris Piech. 2024.  A large scale RCT on effective error messages in CS1. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1_. 1395–1401.
  * Wang et al. (2023) Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, et al. 2023\.  PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization.  arXiv:2306.05087 [cs.CL]
  * Yan et al. (2023) Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, et al. 2023\.  Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review.  _British Journal of Educational Technology_ (2023).
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et al. 2023\.  Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.  arXiv:2306.05685 [cs.CL]
  * Zhu et al. (2023) Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023.  JudgeLM: Fine-tuned Large Language Models are Scalable Judges.  arXiv:2310.17631 [cs.CL]