#  ChaTA: Towards an Intelligent Question-Answer Teaching Assistant using
Open-Source LLMs

Yann Hicke
Cornell University
Ithaca, USA
<EMAIL_ADDRESS>

Anmol Agarwal ∗
Microsoft Research
Bangalore, India
<EMAIL_ADDRESS>
Qianou (Christina) Ma ∗
Carnegie Mellon University
Pittsburgh, USA
<EMAIL_ADDRESS>
Paul Denny
University of Auckland
Auckland, New Zealand
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Responding to the thousands of student questions on online QA platforms each
semester has a considerable human cost, particularly in computing courses with
rapidly growing enrollments. To address the challenges of scalable and
intelligent question-answering (QA), we introduce an innovative solution which
leverages open-source Large Language Models (LLMs) from the LLaMA-2 family to
ensure data privacy. Our approach combines augmentation techniques such as
retrieval augmented generation (RAG), supervised fine-tuning (SFT), and
learning from human preferences data using Direct Preference Optimization
(DPO). Through extensive experimentation on a Piazza dataset from an
introductory CS course, comprising 10,000 QA pairs and 1,500 pairs of
preference data, we demonstrate a significant 30% improvement in the quality
of answers, with RAG being a particularly impactful addition. Our
contributions include the development of a novel architecture for educational
QA, extensive evaluations of LLM performance utilizing both human assessments
and LLM-based metrics, and insights into the challenges and future directions
of educational data processing. This work paves the way for the development of
ChaTA, an intelligent QA assistant customizable for courses with an online QA
platform. 111Relevant code and sample snippets of data can be found at
https://github.com/ChaTa-ai4ed/ChaTA

##  1 Introduction

Online question-answering (QA) platforms like Piazza or EdStem have been
commonly used to deliver timely and effective responses to student inquiries
regarding course content, assignments, and administrative matters [47].
However, responding to the thousands of student questions each semester has a
considerable human cost. This challenge is ubiquitous across disciplines, but
it is particularly relevant in computing courses due to their rapid enrollment
growth [32].

There exists a pressing need for innovative and scalable solutions to provide
immediate and accurate answers to student queries, thereby enhancing the
educational experience without incurring additional costs. Recent approaches
for tackling this problem have revealed several limitations. For example, in
2015 GeorgiaTech launched Jill Watson AI [14]; this model relies on
ontological frameworks and data banks of question-answer pairs [13], which
limits the type of queries that can be answered (logistics or syllabus-
related) and requires significant time to develop and maintain. More recently,
Harvard’s CS50.AI is exploring the use of proprietary large language models
(LLMs) like OpenAI’s GPT-3.5 and GPT-4 [1]. Khan Academy’s Khanmigo is a
similar chatbot tutor, also powered by GPT-4 [5]. These tools send student
queries to third-party providers, thus raising data privacy concerns.

To overcome the above-mentioned limitations, we leverage the adaptability of
open-source LLMs to offer a versatile range of query responses and, crucially,
ensure data privacy. Our current implementation uses LLaMA-2 [48], one of the
most advanced open-source LLMs currently available, and serves as the
foundation of our pipeline, as shown in Figure 1. We enhance the baseline
LLaMA-2 model with a novel combination of techniques and test them on an
anonymized Piazza QA dataset of a CS1 (introductory-level programming) course
(Section 3). We apply retrieval augmented generation (RAG) [25], supervised
fine-tuning (SFT) [10], and an alternative to reinforcement learning with
human feedback (RLHF) [57] to improve the quality of the generated answers in
response to a student’s question (Section 4). We evaluate the performance of
different models using GPT-4 [26, 50] and conduct a manual evaluation of a
subset of the model answers (Section 5). In this work, we present our
preliminary findings and architecture as we progress towards our ultimate
objective of developing ChaTA, an intelligent QA assistant based on a
customized in-house LLM that can be tailored to support any course with an
online question-answering platform.

Our contributions can be summarized as follows:

  * •

We introduce a pipeline that harnesses open-source LLMs for educational QA
tasks. To the best of our knowledge, we are the first to combine RAG, SFT, and
DPO to develop a robust QA system specifically designed for the educational
context.

  * •

We present a comprehensive evaluation of our pipeline, including both LLM-
based and rubric-based human evaluations, on a range of models and baselines.
Additionally, we analyze the impact of different combinations of augmentation
techniques on the performance of the models.

  * •

We investigate the application of LLM-based solutions in the unique context of
educational QA, utilizing a large Piazza QA dataset from an introductory CS
course. We also discuss the challenges, limitations, and future directions
related to data processing and evaluation in this context.

Figure 1: Overview of ChaTA’s pipeline: we combine multiple techniques
including (1) Retrieval-Augmented-Generation (RAG) using course materials, (2)
Supervised Fine-tuning (SFT) using Piazza QA data, and (3) Learning from human
preferences, for which we use Direct Policy Optimization (DPO) as a proxy for
RLHF.

##  2 Related work

Fine-tuning on instruction data: Self-supervised language models (LMs), when
scaled up, are capable of zero-shot [38] or few-shot task completion [7].
Performance on specialized tasks and alignment with user goals can be
significantly enhanced by fine-tuning them on specific datasets comprising
instructions and human-generated text [10].

Fine-tuning on human preferences data: In practice, collecting data based on
human preferences for response quality is often more feasible than gathering
expert demonstrations for instruction tuning. This involves training a reward
model, typically using preference models to predict scores for generated text.
The language model is then fine-tuned to maximize this reward function through
reinforcement learning algorithms like Proximal Policy Optimization (PPO)
[43]. This approach has been previously used to improve the LMs’ performance
in tasks such as translation [22], summarization [46], storytelling [57], and
instruction-following [35]. Nonetheless, RLHF is more complex than supervised
learning, prone to optimization instability, and is sensitive to
hyperparameter adjustments.

Direct Preference Optimization (DPO) [39] is a more streamlined approach that
utilizes a relationship between reward functions and optimal policies to solve
the constrained reward optimization at hand in a single stage of policy
training. Essentially, it treats the problem as a classification task based on
human preference data and is claimed to be more stable and efficient.

Parameter efficient fine-tuning: Training the smallest LLaMA-2 model, i.e.,
LLaMA-2-7B, requires over 100GB of memory for just a batch size of four. To
address such issues, [11] introduced LLM.int8(), an 8-bit matrix
multiplication method, shrinking model sizes significantly by quantizing
parameters. Subsequent work followed to optimize further the required number
of parameters to be stored in memory at the same time [12] which built on top
of Low-rank adaptation methods [17].

Retrieval Augmented Generation: Fine-tuning is less suitable for infusing
knowledge into the model as: it cannot keep up with frequently changing data,
may leak sensitive information, lacks explainability due to lack of
references, and risks catastrophic forgetting. RAG [25, 15] addresses these
issues by retrieving knowledge through a separate search system and grounding
the model’s reasoning and synthesis. With a regularly updated search index,
the knowledge used by the LLM is always up-to-date. By examining the documents
used by the LLM, users can inspect the source of the generated response and
detect potential hallucinations.

AI-based question-answering assistants: Prior works in different domains have
already been taking the approach of fine-tuning open-source LLMs and
augmenting them with different techniques such as RAG [25] and RLHF [57]. For
example, ClinicalGPT [49] fine-tuned Bloom-7B to the medical and healthcare
domain with RAG and RLHF, demonstrating better performance in question-
answering in comparison to baseline LLaMA-7B and ChatGLM-6B evaluated using
GPT-4.

##  3 Dataset

For our experiments, we utilized a pre-anonymized Piazza QA dataset spanning
eight semesters (from 2015 to 2022) of a large-scale introductory programming
course in Matlab and C, taught at a public research university in New Zealand.
The dataset comprises a total of $10,647$ QA pairs. We reserved $717$ QA pairs
from the $2022$ semester for evaluation, and we further randomly selected only
a subset of $236$ pairs due to computational and budget constraints.
Additionally, we obtained the instructional materials for the $2022$ semester,
including the course manual, lab handouts, and assessment dates to perform
information retrieval. See Appendix A for more details about the dataset.

Data Preparation: In preparation for the QA dataset for SFT, we extracted the
single-turn question-answer pairs using a student’s question and the
corresponding answer from the instructor or their peers. In case there were
multiple answers to a question (e.g., a student answer and an instructor
answer), we only kept the final instructor’s answer. Additionally, some posts
included various images such as screenshots of code, tables, graphs, IDE, etc.
We did some preliminary explorations on object character recognition (OCR) on
these images but the quality is still limited because of the diverse image
content, so we ended up filtering out posts with images. In Section 6, we
discuss future work with better quality OCR [6] and multi-turn and
conversational QA support [53].

Additionally, upon analysis of the dataset, we observed the presence of near-
duplicate queries, even within the same semester. Data duplication and train-
test overlap can lead to over-estimate model accuracy in addition to other
disadvantages [24, 36]. Therefore, we performed deduplication where we created
an embedding222using sentence-transformers/all-MiniLM-L6-v2 for each query,
applied agglomerative clustering to group potential duplicates in the same
cluster, and retained only one member of each cluster. This decreased the size
of our dataset by $5.1\%$. In the end, we built a dataset of $10,647$ single-
turn QA pairs without images or duplicates. See appendix A.2 for examples of
duplicates.

Additionally, we also created $1,549$ pairs of answers with proxy human
preferences from the Piazza dataset. We used the edit history of Piazza posts:
some answers have an original response but are then edited for improvement. We
adopted the heuristics of preferring the final answer to the first one to
create this human preference dataset. Refer to Section A.3 for examples of
preference pairs, and Appendix B for more details on the data format for
models.

##  4 Methods

In our experiment, the primary goal is to evaluate the capabilities of LLMs in
the educational domain, especially when they are augmented with RAG, SFT, and
DPO. More details about training and inference parameters are included in
Appendix C.

Baselines: Among many highly proficient LLMs [34, 3, 9, 2, 37], some models
are not publicly available, which often demonstrate better performance
compared to their open-source counterparts. We used the LLaMA-2 family [48] as
our open-source model baselines due to their widespread adoption and support,
specifically LLaMA-2-7B-chat, LLaMA-2-13B-chat, and LLaMA-2-70B-chat. For
closed-source model baseline, we used GPT-4 [34]. 333Before sending inference
queries to the OpenAI API, we ensure that no private or identifiable
information is present in the queries.

Supervised Finetuning (SFT): We used pre-trained LLaMA-2-13B-chat to
initialize our model and perform _instruction-tuning_ with the previously
described QA dataset on a single A100 GPU. We trained the model for three
epochs with 100 warmup steps and set the learning rate to $1e-4$ with a cosine
learning rate scheduler.

Alignment to Human Preferences (DPO): Because DPO [39] bypasses the need for
sampling from the language model during fine-tuning or extensive
hyperparameter tuning, it is much cheaper computationally than RLHF through
PPO. It requires less VRAM to load all three models (policy, value, and reward
models) and also hyperparameter optimization is not as expensive. We used DPO
as our fine-tuning algorithm on human preferences data which performs an
equivalent of RLHF but on a single A6000 GPU. We also trained the model for
three epochs with the DPO objective.

Parameter efficient fine-tuning: Both our supervised-finetuning and our DPO
pipelines make use of the _peft_ huggingface library to load our base models
before finetuning. We load our models in the 4-bit configuration and then
train via the QLora method [12].

Retrieval Augmented Generation (RAG): We extracted text from the PDFs (e.g.,
textbooks and lab manual) using Nougat [6], a Visual Transformer model, which
converts PDFs to markdown files and is particularly impressive in capturing
several instances of LaTeX math, tabular data and code snippets present in the
PDFs.
To improve the retrieval efficiency and to account for the limited context
length of LLMs, we chunk the individual source documents with a maximum limit
of $1000$ characters per chunk, and an overlap of $100$ characters between
adjacent chunks. To keep paragraphs, sentences, and words together, we split
the document using predefined delimiters, including double newlines, single
newlines, spaces, and empty strings, until the resulting chunks are
sufficiently small. We observed that (1) an _embedding-based retriever_ is
more suitable for queries that require a more nuanced understanding of
language such as logistical queries and, (2) _keyword-based retriever_ is more
robust for queries that make reference to specific coding functions and
assignment problem names. To take advantage of both, we constructed the
context using a union of the top-$3$ documents retrieved by the embedding-
based Dense Passage Retriever [19] and used the top-$2$ documents retrieved by
the BM25 ranking function. Note that we did not train the retriever end-to-end
but instead augment the prompt with the retrieved documents. The prompt
construction with the retriever context can be found in Appendix B.

Evaluation: We adopted two methods to evaluate our models: human evaluation
and LLM-based (GPT-4) evaluation. Previous research [44] has shown that
traditional evaluation metrics such as BLEU, Rouge-L, and Cider are not
effective in capturing the usefulness and accuracy of model responses, which
are critical in the context of ChaTA’s usecase. Nonetheless, for the sake of
completeness, we also report scores based on the BertScore-F1 [54] metric in
Section 5.

  * •

Human Evaluation: In our study, we utilized a set of rubric items adapted from
prior work [44, 55] and refined it through an iterative process. Our final
rubrics assign scores of $0$, $0.5$, or $1$ (with higher scores indicating
better quality) to two distinct criteria: _Usefulness_ and _Accuracy_. The
Usefulness category measures the practical value of the model answer to a
human instructor or teaching assistant (TA), taking into account factors such
as the naturalness, coherence, and conciseness of the response. The Accuracy
category, on the other hand, assesses the factual correctness and groundedness
of the information presented in the model answer. For instance, a Usefulness
score of $1$ indicates that the answer is highly useful and would be readily
accepted by a TA with minimal or no revisions, while an Accuracy score of $0$
denotes that the answer is entirely inaccurate and provides false information.
Please refer to Table 6 in Appendix E for a detailed description of the
evaluation rubrics and examples of their application.

For the human evaluation component of our study, we randomly selected a subset
of 20 questions from our evaluation set of $236$ questions. Each question was
accompanied by 9 distinct model answers, resulting in a total of 180 answers
for manual scoring. The 9 models used in this evaluation are listed in Table
5, Section 5. One of the authors conducted the annotation of all 180 answers
using the aforementioned rubrics.

  * •

LLM Evaluation: Recent studies have utilized GPT-4 [34] to assess the quality
of text generated by models [29, 52, 56]. Alpaca Eval [27] demonstrated that
GPT-4 was the best automatic annotator, with the highest agreement with
humans, based on a set of 2.5k human annotations. However, GPT-4 has also been
shown to prefer generations produced by the Instruct-GPT model family [16]. In
our experiment, we prompted GPT-4 to evaluate answers generated on $236$
questions for each of the $9$ models, using the same rubrics as in human
evaluation. Our prompt (included in Appendix E) was adapted from the
AlpacaEval framework [27] and G-Eval [29]444 Original prompt for AlpacaEval
can be found here. We use the G-Eval implementation present here.

##  5 Results

For human evaluation, we evaluated $20$ questions on $9$ different models. For
BertScore and LLM-based evaluation, we evaluated $236$ questions on $9$
models. The scores are summarized in Table 1. Please refer to Table 5 in
Appendix D for example results and ratings of each model.

Table 1: BertScore, Human evaluation scores and LLM evaluation scores (average $\pm$ stdev) as per the rubric. Bold numbers indicate the best performance among open-source and closed-source LLMs. Note that the evaluation has been performed on the chat variant of the LLaMA-2 models. Model | Evaluation by Humans | Evaluation using GPT-4 |  Bert Score F1
---|---|---|---
| Usefulness | Accuracy | Avg | Usefulness | Accuracy | Avg
LLaMA-2-7B |  0.18 ($\pm$ 0.29) |  0.28 ($\pm$ 0.38) | 0.23 |  0.54 ($\pm$ 0.35) |  0.42 ($\pm$ 0.35) | 0.48 |  0.454
LLaMA-2-13B (L-13) |  0.18 ($\pm$ 0.24) |  0.30 ($\pm$ 0.41) | 0.24 |  0.62 ($\pm$ 0.35) |  0.52 ($\pm$ 0.38) | 0.57 |  0.455
L-13+RAG |  0.53 ($\pm$ 0.41) |  0.50 ($\pm$ 0.40) | 0.51 |  0.61 ($\pm$ 0.35) |  0.52 ($\pm$ 0.38) | 0.57 |  0.453
L-13+SFT |  0.13 ($\pm$ 0.28) |  0.10 ($\pm$ 0.21) | 0.11 |  0.26 ( $\pm$ 0.32) |  0.21 ($\pm$ 0.33) | 0.24 |  0.400
L-13+SFT+DPO |  0.23 ($\pm$ 0.38) |  0.33 ($\pm$ 0.44) | 0.28 |  0.60 ($\pm$ 0.36) |  0.50 ($\pm$ 0.39) | 0.55 |  0.453
L-13+SFT+DPO+RAG |  0.50 ($\pm$ 0.43) |  0.58 ($\pm$ 0.41) | 0.54 |  0.63 ($\pm$ 0.36) |  0.53 ($\pm$ 0.39) | 0.58 |  0.449
LLaMA-2-70B |  0.20 ($\pm$ 0.34) |  0.33 ($\pm$ 0.41) | 0.26 |  0.63 ($\pm$ 0.36) |  0.53 ($\pm$ 0.36) | 0.58 |  0.456
GPT-4 |  0.28 ($\pm$ 0.34) |  0.40 ($\pm$ 0.42) | 0.34 |  0.78 ($\pm$ 0.29) |  0.68 ($\pm$ 0.32) | 0.73 |  0.458
GPT-4 + RAG |  0.65 ($\pm$ 0.43) |  0.73 ($\pm$ 0.34) | 0.69 |  0.81 ($\pm$ 0.27) |  0.72 ($\pm$ 0.32) | 0.77 |  0.469

###  5.1 Human evaluation

#### Baseline model performances are similar, improve with size, and GPT-4
performs the best.

According to human evaluation scores, the performance of baseline models
improves with size, with GPT-4 demonstrating the best performance. The LLaMA
models of different sizes (7B, 13B, 70B) do not generate significantly
different answers, although larger models perform slightly better. However,
there is a high standard deviation in the ratings of baseline models,
indicating that their performance varies greatly depending on the type of
query. For instance, models without retrieved-context perform poorly on
logistical questions related to assignments, but can perform well on
conceptual questions such as C code syntax.

#### Different augmentation techniques (SFT, DPO, RAG) have different effects
on performance.

_SFT alone does not improve performance_ , with the LLaMA-2-13b+SFT performing
13% worse than the LLaMA-2-13B base model. Qualitatively, we observed that
LLaMA-2-13B + SFT often hallucinates, which makes its answer neither useful
nor accurate. Fine-tuning is supposed to teach the model to output correct
answers based on it’s knowledge. However, there are some queries (such as
logistical queries) which cannot be answered without relevant context and the
knowledge regarding such queries was absent in the pre-training data. It is
possible that finetuning on such queries infuses the model with the tendency
to guess and hence, might explain the hallucinations.

_DPO appears to regulate hallucination in models_. The LLaMA-2-13B+SFT+DPO
model showed a $17$% improvement in human evaluation scores over the
LLaMA-2-13B+SFT model, with a notable $23$% increase in accuracy. However, its
performance was only marginally better than the LLaMA-2-13B base model, by
$4$%.

_RAG has the most significant impact on performance_ , with all models
incorporating RAG demonstrating improved results compared to those without.
For example, GPT-4+RAG showed a $35$% improvement over GPT-4 without RAG,
while LLaMA2-13b+SFT+DPO+RAG improved by $26$% compared to the
LLaMA2-13b+SFT+DPO model. Qualitatively, we observed answers in RAG-based
models to be much more accurate with less irrelevant information (examples in
Appendix D, Table 4).

#### Model answers: generally helpful, may outperform humans or completely
fail.

The second-best model as per human evaluation, LLaMA2-13B+SFT+DPO+RAG,
performs $15$% worse than GPT-4+RAG in both usefulness and accuracy metrics.
However, when compared to the LLaMA-2-13B baseline, its performance improved
by 30%. With an overall usefulness of $0.50$ and an accuracy rating of $0.58$,
it could already be valuable to instructors as a means of reducing their
workload while also alleviating privacy concerns in comparison to GPT-4.

It is important to note that model answers can sometimes surpass human ground
truth answers in terms of comprehensiveness and positive sentiment. For
instance, one student asked the question _“Do we get penalties when the code
is quite long?”_ The human answer was _“Best is to try to simplify as much as
possible. If it is too long, it is likely not efficient and can be improved
upon”_. In contrast, the response (shown below) from LLaMA2-13b+SFT+DPO+RAG
was more detailed and encouraging:

> _“Hello! Regarding your concern about penalties for long codes, I’m happy to
> clarify that there are no penalties for having long codes in the course.
> However, it’s important to note that long codes can sometimes be difficult
> to read and maintain, so it’s a good idea to strive for conciseness and
> clarity in your coding. This will not only make your codes more manageable,
> but it will also help you avoid unnecessary errors and bugs. So, focus on
> writing clean, efficient, and well-structured codes! If you have any further
> questions or concerns, feel free to ask, and I’ll be here to help.”_

On the other hand, there are also cases where all models fail to generate any
reasonable outputs. For example, given a question that asks the TA to check
their submission result on another platform, none of the models respond
appropriately. It is not a task models should even be expected of performing.

###  5.2 Evaluation using GPT-4

As shown in Table 1, both human and LLM (GPT-4) evaluations agreed that
GPT-4+RAG is the strongest model among all. However, the LLM evaluation rated
the baseline GPT-4 as the second-best model, which differs from the human
evaluation results. This discrepancy may be attributed to the findings of
[16], which suggest that GPT-4 has a tendency to favor models trained on data
derived from InstructGPT, GPT-4, or ChatGPT, even if the content is less
factual or useful.

It is also worth noting that GPT-4 assigned much higher scores to the baseline
models than the human evaluation. While the LLM evaluation did capture some
differences between models (e.g., it also consistently showed that SFT alone
makes LLaMA2-13b perform worse), it did not align well enough with human
evaluation. To quantify the agreement between the human and LLM evaluations,
we calculated the Pearson ($r$) and Spearman ($\rho$) correlation scores.
Across 20 queries for the 9 models (180 responses), we found the Pearson
coefficient between human and LLM-eval to be $0.662$ and Spearman Coefficient
to be $0.659$. These scores indicate a moderate positive correlation between
the two types of evaluation, suggesting that while there is some agreement
between the human and LLM evaluations, there are also many differences. For
detailed analysis on the correlation between different types of evaluation
methods, please refer to Section E.2.

##  6 Limitations and Future work

In this work, our primary goal was to leverage our own in-house infrastructure
to generate responses to student queries. Instead of relying on commercial
models and prompt engineering through API calls, we aimed to maximize the use
of open-source models. This approach enables us to avoid the issues of
unreliability and data privacy that come with using third-party service
providers. Our experiments have shown that using different augmentation
techniques can improve the performance of smaller, in-house models from the
LLaMA-2 family, bringing them closer to the more capable GPT-4 model. However,
there is still significant potential for further improvement in our future
work.

Data: Our work is limited by the dataset filtering and preparation process. In
particular, we may have lost valuable QA data from follow-up interactions that
were not included, due to the challenge of separating questions and answers in
forum-style data. Furthermore, posts containing images, such as screenshots,
were excluded from our dataset. In the future, one could explore multi-turn
and conversational QA, as suggested by recent research [53, 40, 41, 51].
Additionally, one could employ better OCR models [6] to process images in our
data sources.

Training: Our pipeline consists of several stages - RAG, SFT, and RLHF - and
each of these stages can be further developed and enhanced individually. In
this work, we only tested the LLaMa-2 models. In the future, one plan to
experiment with specialized domain-specific models, such as CodeLLaMA, which
have enhanced coding capabilities [42]. We also plan to explore ways to
improve performance using an ensemble of models while reducing inference costs
[8]. As already demonstrated in our work, students’ queries and their
corresponding answers can often be similar across semesters. Therefore, we
intend to retrieve comparable QA-pairs from previous semesters and utilize
them as few-shot examples during the inference process, building upon the
approaches already explored in [30]. Additionally, training a retriever end-
to-end may further improve the quality of the retrieved context [25].

Evaluation: Our human evaluation is currently limited by the small sample size
of 20 responses, which were rated by only one annotator. In the future, we
plan to expand our evaluation to include a larger sample size and multiple
annotators, in order to establish robust inter-rater reliability for our
rubrics. It is important to note that LLMs and existing automatic evaluation
methods may not always align with the answering heuristics of human
instructors, or accurately capture the pedagogical value of an answer. To
address this, we will continue to explore automatic evaluation methods,
drawing on insights from existing work [20, 18, 50, 28].

Furthermore, our experimentation with LLaMA-2 models has so far been limited
to a single course. It has been noted that existing pre-trained models may not
perform as well on languages such as Matlab and C, compared to Python and Java
[33]. To address this, we plan to deploy our system across a range of
university CS courses, in order to gather feedback from TAs and instructors
and experiment with different model configurations.

Deployment & User Study: We are currently developing the ChaTA tool, utilizing
APIs from Piazza, Discord, and Chrome extensions to integrate the collection
of TA feedback seamlessly into their routine QA workflow. We plan to conduct a
user study to assess the productivity and pedagogical value that ChaTA brings
to classes. Additionally, we will explore interface and interaction design to
maximize value and minimize risks in human-AI question answering [4]. For
instance, we could show TAs the uncertainty estimation of the model’s answers
[23] or present the model’s answer as an imperfect peer student’s discussion
[31]. It may also be beneficial to train a model across different
institutions, leveraging the collective teaching knowledge of a subject in a
scalable and secure manner that still maintains data privacy for each school
using distributed machine learning [21, 45].

##  7 Acknowledgement

We acknowledge funding support from Generative AI Innovation Incubator and the
Block Center at Carnegie Mellon University. Any opinions, findings,
conclusions or recommendations expressed in this work are those of the
author(s) and do not necessarily reflect the views of the funding entities.

## References

  * CS [5] MS Windows NT kernel description.  https://www.thecrimson.com/article/2023/6/21/cs50-artificial-intelligence/.  Accessed: 2023-09-16.
  * Anil et al. [2023] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, and Paige Bailey et al.  Palm 2 technical report, 2023.
  * Bai et al. [2022] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional ai: Harmlessness from ai feedback, 2022.
  * Bansal et al. [2021] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld.  Does the whole exceed its parts? the effect of AI explanations on complementary team performance.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , number Article 81 in CHI ’21, pages 1–16, New York, NY, USA, May 2021. Association for Computing Machinery.  ISBN 9781450380966.  doi: 10.1145/3411764.3445717.  URL https://doi.org/10.1145/3411764.3445717.
  * Bidarian [2023] Nadia Bidarian.  Meet khan academy’s chatbot tutor.  _CNN_ , August 2023.  URL https://www.cnn.com/2023/08/21/tech/khan-academy-ai-tutor/index.html.
  * Blecher et al. [2023] Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic.  Nougat: Neural optical understanding for academic documents.  August 2023.  URL http://arxiv.org/abs/2308.13418.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2023] Lingjiao Chen, Matei Zaharia, and James Zou.  Frugalgpt: How to use large language models while reducing cost and improving performance.  _arXiv preprint arXiv:2305.05176_ , 2023.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, and Gaurav Mishra et al.  Palm: Scaling language modeling with pathways, 2022.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _arXiv preprint arXiv:2208.07339_ , 2022.
  * Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_ , 2023.
  * Goel et al. [2021] Ashok Goel, Harshvardhan Sikka, and Eric Gregori.  Agent smith: Teaching question answering to jill watson.  _arXiv preprint arXiv:2112.13677_ , 2021.
  * Goel and Polepeddi [2018] Ashok K Goel and Lalith Polepeddi.  Jill watson.  _Learning engineering for online education: Theoretical contexts and design-based examples. Routledge_ , 2018.
  * Guu et al. [2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  Realm: Retrieval-augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org, 2020.
  * HF [2023] HF.  Hf.  https://huggingface.co/blog/llm-leaderboard, 2023.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Kabir et al. [2023] Samia Kabir, David N Udo-Imeh, Bonan Kou, and Tianyi Zhang.  Who answers it better? an In-Depth analysis of ChatGPT and stack overflow answers to software engineering questions.  August 2023.  URL http://arxiv.org/abs/2308.02312.
  * Karpukhin et al. [2020] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.550.  URL https://aclanthology.org/2020.emnlp-main.550.
  * Kim et al. [2023] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo.  Prometheus: Inducing fine-grained evaluation capability in language models, 2023.
  * Knott et al. [2021] Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten.  CrypTen: Secure multi-party computation meets machine learning.  September 2021.  URL https://scontent.fagc1-1.fna.fbcdn.net/v/t39.8562-6/260321120_621702352397035_7531336426414693679_n.pdf?_nc_cat=106&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=xI-tMErFcxQAX8damhf&_nc_ht=scontent.fagc1-1.fna&oh=00_AfAvpJcL-YE4dkqTV2mheFleKL4UO8kPh3SkpCYW_5VznQ&oe=650B4BFE.
  * Kreutzer et al. [2018] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler.  Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning.  _arXiv preprint arXiv:1805.10627_ , 2018.
  * Kuhn et al. [2023] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  February 2023.  URL http://arxiv.org/abs/2302.09664.
  * Lee et al. [2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  July 2021.  URL http://arxiv.org/abs/2107.06499.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-Tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-Augmented generation for Knowledge-Intensive NLP tasks.  May 2020.  URL http://arxiv.org/abs/2005.11401.
  * Li et al. [2023a] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023a.
  * Li et al. [2023b] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models, 2023b.
  * Liu et al. [2023a] Chengyuan Liu, Divyang Doshi, Ruixuan Shang, Jialin Cui, Qinjin Jia, and Edward Gehringer.  Less but enough: Evaluation of peer reviews through pseudo-labeling with less annotated data.  _JEDM_ , 15(2):123–140, June 2023a.  ISSN 2157-2100, 2157-2100.  doi: 10.5281/zenodo.7304981.  URL https://jedm.educationaldatamining.org/index.php/JEDM/article/view/613.
  * Liu et al. [2023b] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023b.
  * Lu et al. [2022] Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy.  ReACC: A retrieval-augmented code completion framework.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6227–6240, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.431.  URL https://aclanthology.org/2022.acl-long.431.
  * Ma et al. [2023] Qianou Ma, Hua Shen, Kenneth Koedinger, and Tongshuang Wu.  HypoCompass: Large-Language-Model-based tutor for hypothesis construction in debugging for novices.  October 2023.  URL http://arxiv.org/abs/2310.05292.
  * National Academies of Sciences et al. [2018] Medicine National Academies of Sciences, Engineering et al.  _Assessing and responding to the growth of computer science undergraduate enrollments_.  National Academies Press, 2018.
  * Nguyen and Nadi [2022] N Nguyen and Sarah Nadi.  An empirical evaluation of GitHub copilot’s code suggestions.  _2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR)_ , 2022.  doi: 10.1145/3524842.3528470.  URL https://www.semanticscholar.org/paper/cdfe9580f63070f311151444f9df32818cc858bf.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Penedo et al. [2023a] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The RefinedWeb dataset for falcon LLM: Outperforming curated corpora with web data, and web data only.  June 2023a.  URL http://arxiv.org/abs/2306.01116.
  * Penedo et al. [2023b] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023b.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D Manning.  CoQA: A conversational question answering challenge.  _Trans. Assoc. Comput. Linguist._ , 7:249–266, November 2019.  ISSN 2307-387X.  doi: 10.1162/tacl\\_a\\_00266.  URL https://direct.mit.edu/tacl/article/43511.
  * Ross et al. [2023] Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz.  The programmer’s assistant: Conversational interaction with a large language model for software development.  February 2023.  URL http://arxiv.org/abs/2302.07080.
  * Rozière et al. [2023] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code.  August 2023.  URL http://arxiv.org/abs/2308.12950.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shi et al. [2023] Ensheng Shi, Fengji Zhang, Yanlin Wang, Bei Chen, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun.  Sotana: The open-source software development assistant.  _arXiv preprint arXiv:2308.13416_ , 2023.
  * Soin et al. [2021] Arjun Soin, Pratik Bhatu, Rohit Takhar, Nishanth Chandran, Divya Gupta, Javier Alvarez-Valle, Rahul Sharma, Vidur Mahajan, and Matthew P Lungren.  Multi-institution encrypted medical imaging ai validation without data sharing.  _SSRN Electron. J._ , 2021.  ISSN 1556-5068.  doi: 10.2139/ssrn.3973993.  URL https://www.microsoft.com/en-us/research/publication/multi-institution-encrypted-medical-imaging-ai-validation-without-data-sharing/.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Thinnyun et al. [2021] Adrian Thinnyun, Ryan Lenfant, Raymond Pettit, and John R. Hott.  Gender and engagement in cs courses on piazza.  In _Proceedings of the 52nd ACM Technical Symposium on Computer Science Education_ , SIGCSE ’21, page 438–444, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380621.  doi: 10.1145/3408877.3432395.  URL https://doi.org/10.1145/3408877.3432395.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Wang et al. [2023a] Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li.  ClinicalGPT: Large language models finetuned with diverse medical data and comprehensive evaluation.  June 2023a.  URL http://arxiv.org/abs/2306.09968.
  * Wang et al. [2023b] Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou.  Is ChatGPT a good NLG evaluator? a preliminary study.  March 2023b.  URL http://arxiv.org/abs/2303.04048.
  * Wang et al. [2023c] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji.  MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback.  September 2023c.  URL http://arxiv.org/abs/2309.10691.
  * Yidong et al. [2023] Wang Yidong, Yu Zhuohao, Zeng Zhengran, Yang Linyi, Heng Qiang, Wang Cunxiang, Chen Hao, Jiang Chaoya, Xie Rui, Wang Jindong, et al.  Pandalm: Reproducible and automated language model assessment, 2023.
  * Zaib et al. [2022] Munazza Zaib, Wei Emma Zhang, Quan Z Sheng, Adnan Mahmood, and Yang Zhang.  Conversational question answering: a survey.  _Knowl. Inf. Syst._ , 64(12):3151–3195, December 2022.  ISSN 0219-1377, 0219-3116.  doi: 10.1007/s10115-022-01744-y.  URL https://doi.org/10.1007/s10115-022-01744-y.
  * Zhang* et al. [2020] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SkeHuCVFDr.
  * Zhong et al. [2022] Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han.  Towards a unified Multi-Dimensional evaluator for text generation.  October 2022.  URL http://arxiv.org/abs/2210.07197.
  * Zhou et al. [2023] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_ , 2023.
  * Ziegler et al. [2019] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

Appendix

##  Appendix A Dataset Description

###  A.1 Dataset Statistics

Piazza is a widely used online question-answering forum for students and
instructors used in thousands of universities across 90 countries. In a Piazza
dataset, there are six types of user contributions, including question (i.e. a
post created by a student or instructor to ask a question or initiate a
discussion on a particular topic.), i_answer (i.e. answer provided by an
instructor in response to a question), s_answer (i.e. answer provided by a
fellow student in response to a question), followup, followup_response, and
note. Each post has an associate folder label that students and instructors
use to categorize the posts, such as exams, logistics, etc.

In the course data that we analyzed, spanning eight semesters, there were an
average of six instructors and 953 students per semester. A total of $18,181$
posts and $60,884$ user contributions were recorded, from which we extracted
$19,391$ question-and-answer pairs. On average, each contribution, such as a
question, contained $52$ $\pm$ $155$ tokens and $38$ $\pm$ $56$ words, with
the maximum contribution being $5,561$ tokens and $3,688$ words in length.
There can be further question-answering in the follow-up and follow-up
responses — the proportion of each of the six types of user contributions on
the total number of posts, the average number of words, and the total number
of token counts are shown in Figure 2(a) — 25% of the total tokens in the
Piazza dataset is followup and followup responses.

We held out the data from the 2022 semester for evaluation, which has $8$
instructors, $1,096$ students, and a total of $1,870$ posts. The distribution
of posts in the 2022 semester by the different folders and contribution types
is shown in Figure 2(b).

It is important to note that the dataset used in our study presents several
complexities. For instance, a single question may have multiple answers, and
some questions may include images or be duplicated. To address these
challenges, we have established a set of criteria for selecting ground truth
answers. In cases where a query has answers from both the instructor and
fellow classmates, we prioritize the instructor’s response as the ground
truth. If no instructor answer is available, we treat the latest answer
provided by a student as the ground truth. Queries without any answers are
discarded. Additionally, we have excluded all posts that contain images in
either the question or answer body, resulting in the removal of a total of
5,801 images from the dataset. We perform de-duplication based on data
clustering. In the end, we have $10,647$ QA pairs in our dataset, and $717$ of
those from 2022 semester were reserved for evaluation.

Figure 2: (a) Breakdown of post categories and type of contributions in the
2022 semester.(b) The proportion of each type by the number of posts, average
number of words, and the total number of tokens, aggregated across all posts
in all semesters.

###  A.2 Duplication in queries

Some examples of duplicate queries detected by clustering can be found in
Table 2.

Table 2: Examples of duplicate queries present in our dataset Example Cluster 1 |  Example Cluster 2
---|---
• Lab 9 task 2: When I run this code, it says it cannot open input file "blank.bmp." How can I get it to open it? • Lab 9 task 2: When I run task 2 in developer command, it says: could not open input file, "Blank.bmp". I added it to my source files so what is causing this problem? • "Could not open input file: blank.bmp": How do I fix this error for lab 9 task 2? This is in Visual Studios, running it through the developer command prompt works fine. • Lab 9 Prep Task 2: Whenever I try to run the code it says that I "could not open input file blank.bmp" And also I have no idea how to approach this task help would be much appreciated. Edit: No worries this has been solved, you must put the file in the project directory. |  • Where can we access computers with Matlab in order to do our labs before lab times, or to get some Matlab study done? • Which computers in the University have Matlab installed on them? I’m guessing only the ones in the science department? • Which labs in the uni have Matlab installed so that we can check our running times if we’ve done them on our laptops? • Which computers have Matlab installed in uni? Because if work from labs are unfinished it would be convenient to know where you could finish it. • Are there any open computer labs at uni which have MATLAB, apart from the engineering ones?
Example Cluster 3 |  Example Cluster 4
• Are we allowed calculators in this test tonight? • Are we allowed to use calculators for the test? • Are we permitted calculators in the test? • Are we allowed a calculator for the test? • In the test, will we be allowed calculators? • Is calculator usage allowed tonight? |  • fprintf vs sprintf: Can these two be used interchangeably? • General Q: Does it make a difference if you use fprintf or sprintf? • When do we use fprintf and sprintf? Really confused by these two things. • What are the differences between Sprintf and Fprintf? And what does the “f” at the end of both commands correlate to? • What is the difference between fprintf and sprintf? When do we use which one?

###  A.3 Preference data for DPO

Some examples of preference data from our dataset can be found in Table 3.

Table 3: Some examples of preference pairs in our dataset Query: Hello all,
what is the difference between writing ’&’ and ’&&’. E.g ⬇ if(income > 14000
&& income <= 48000) tax = 0.105 * 14000 + 0.175 * (income - 14000); end
Preferred response: && is for scalars & is for vectors. The way in which
MATLAB compares is different for each of the two there is also two more
operators (|| and |). Over-simplifications aside && and || are short circuit
operators, they go left to right. If any of the conditions are false then the
comparing is stopped. So for conditionOne && conditionTwo the condition on the
left if it meets requirements (False here) will not check the condition on the
right. Non-preferred response: && is for scalars & is for vectors
---
Query: How does the Quotient "%" thing work? Tried googling it, did a bunch of
tests in C to try and work it out myself. No luck. How does it calculate
stuff? Preferred response: This is the modulus operator, and gives you the
remainder of a division operation. e.g. 6%2 will give you 0. You’ve already
met the modulus operator as part of the Matlab project, where mod(5,2) would
find the remainder when 5 is divided by 2 (handy for telling if a number is
even or not). There was even a recording on how to use it. If you understand
how mod(5,2) works in Matlab, then in C you have the benefit of a shorthand
for it, i.e. 5%2. Non-preferred response: This is the modulus operator, and
gives you the remainder of a division operation. e.g. 6%2 will give you 0.
Query: How do I get my while loop to start from the second line of data (e.g.
not the header)? Preferred response: Add a fgetl(fid); before your loop
starts. Alternatively, you could try to write some logic to skip it in the
loop. Non-preferred response: Add a fgetl(fid); before your loop

##  Appendix B Data Formats and Model Prompt

For models of the LLaMA-2 family, following is the prompt used by us:

<s>[INST] <<SYS>>
---
You are a helpful, respectful, and honest teaching assistant for an
introductory programming course in Matlab and C. Your current task is to
answer student queries on Piazza. Always answer as helpfully as possible,
while being safe. Your answers should not include any harmful, unethical,
racist, sexist, toxic, dangerous, or illegal content. Please ensure that your
responses are socially unbiased and positive in nature. If a question does not
make any sense, or is not factually coherent, explain why instead of answering
something not correct. If you don’t know the answer to a question, please
don’t share false information. <</SYS>>
{{RAG context (if applicable)}}
Query subject: ‘‘‘{{Subject of the Query}}‘‘‘
Query body: ‘‘‘{{Body of the Query}}‘‘‘
Please answer the query. [/INST]

For RAG, the text of the below format is substituted in the original prompt:
Here are some snippets from the course material & other uploaded content which
might be helpful to generate the response. ### Below is snippet 1: <SNIPPET
DOCUMENT TEXT> ### Below is snippet 2: <SNIPPET DOCUMENT TEXT> ### Below is
snippet K: <SNIPPET DOCUMENT TEXT> Above were the snippets. Now, here is the
query to be answered:

For DPO, the human preferences data was of the following format:

    {
        "instruction": "{subject}\n{question}",
        "output1": (started_off_answer), //the original student or instructor answer
        "output2": (updated_answer), //updated version of the original answer
        "preference": 2
    }

##  Appendix C Experimental Settings

Inference Parameters: All generations were done with the following parameters:

  * •

max_length: $2048$

  * •

max_new_tokens: $1024$

  * •

top_p: $1$

  * •

top_k: $50$

  * •

temperature: $0.3$

The inference on LLaMA-2-70B-chat was done under 4-bit quantization.

Supervised Finetuning: For finetuning the optimizer used was a paged AdamW
with a 32-bit precision setting, which helped optimize memory usage and
computation efficiency. We employed a per-device training batch size of 4 and
an evaluation batch size of 1. To enhance training stability and
effectiveness, gradient accumulation steps were set to 2, and gradient
checkpointing was enabled. Given the size of the LLaMA-2-13B-chat model and
our computational constraints, we leveraged LoRA (Low-Rank Adaptation) layers
with an alpha parameter of 16, a dropout rate of 0.05, and an r parameter of
8. This approach allowed us to adapt the model efficiently without the need
for extensive retraining of the entire network, thus saving both time and
computational resources. The sequence length was fixed at 1024 tokens to
balance between context availability and computational feasibility.

DPO: The setting for DPO was the same as SFT but the beta parameter for the
DPO loss was set at 0.1. The learning rate was increased to 5e-4, and gradient
accumulation steps were doubled to 4.

##  Appendix D Model Results Example Comparison

We provide some examples to show case the potential impact of RAG in Table 4.
Using a same question, we also provide the different model answers and the
corresponding LLM evaluation (using GPT-4) scores and human evaluation scores
(one author’s rating) in Table 5.

Table 4: Some qualitative examples of RAG benefiting the performance Query:
What is the function IsExitUnique() supposed to return when there are no
exits?
---
Ground Truth Answer: Return 0 as the manual clearly says that ‘The
IsExitUnique() function should return true if there is exactly one exit
position in the cave array, and false otherwise.‘
Model considered: GPT-4
Relevant content within the context retrieved:
# Task Five: “The one and only” (10 marks)
Next, a valid cave must have only one exit. Define a function called
IsExitUnique() which checks that there is a single ‘exit’ position in the
2-dimensional array:  ⬇ int IsExitUnique(int cave[CAVE_SIZE][CAVE_SIZE]) If
there is exactly one ‘exit’ position, then the function should return true,
otherwise it should return false. Consider the following example:  ⬇ char
layout[200] = "1111111111100000000110000000011000000001
1EEEEEEEE110000000011000000001100000000110000000011111111111"; int
cave[CAVE_SIZE][CAVE_SIZE] = {0}; InitialiseCave(cave, layout);
printf("Unique? \%d\n", IsExitUnique(cave)); \\\ 0 This is because there are
many ‘exit’ positions on row 4 of the cave (see ‘E’ characters).
Answer without RAG:
Thank you for your question. Based on the provided information, it seems that
the ‘IsExitUnique‘ function is meant to check if exits are unique in a given
context. However, without more context or details about the function, it’s
difficult to provide a specific answer to your question. If there are no
exits, the function could return ‘true‘ (since there are no duplicate exits),
or it could return a specific value (e.g., -1) to indicate that there are no
exits at all. The expected behavior would depend on the function’s
implementation and the requirements of the programming assignment. If you
could provide more information about the function or the context in which it
is being used, I would be happy to help you further.
Answer with RAG:
The ‘IsExitUnique‘ function is designed to check if there is exactly one
’exit’ position in the 2-dimensional array. If there is exactly one ’exit’
position, then the function should return true (or 1 in the case of C
programming), otherwise, it should return false (or 0 in C). In the case where
there are no exits in the 2-dimensional array, the function should return
false (or 0). This is because the condition of having exactly one ’exit’ is
not met.
Remember, the function is not checking for the presence of an exit, but rather
it is checking if there is exactly one exit. If there are no exits or more
than one exit, the function should return false.
Query: Can we push a boulder to crash the treasure boxes?
Ground Truth Answer: The boulder will continue to roll in the same direction
as long as there is empty space ahead of it. It will stop when it hits
anything that is not empty space (such as a wall).
Model considered: LLAMA-2 Chat 13B (SFT + DPO)
Relevant content within the context retrieved:
# Task Nine: “Time to get moving” (10 marks)  • The player cannot move into or
through the walls of the cave. When a boulder rolls into a hole, both the hole
and the boulder disappear (i.e. ’empty space’ appears at the previous location
of the hole) • A boulder, once pushed, will continue to move in a straight
line until it hits an obstacle (either a hole, a treasure, another boulder or
a cave wall) • You can assume that the boulder will never be pushed through
the exit. Answer without RAG:
As a helpful & responsible teaching assistant, I’m here to assist you with
your query. However, I would like to point out that crashing treasure boxes
with a boulder may not be the most appropriate or safe approach, as it could
cause harm to others or damage the environment. Instead, I suggest we focus on
finding alternative & more constructive solutions to gain access to the
treasure boxes. In the context of programming, it’s important to prioritize
safety and ethical considerations in our code and actions. We should always
strive to create programs and solutions that are socially unbiased, positive,
& respectful of others.
Answer with RAG:
As your helpful TA, I’m here to answer your query about player pushing in the
Boulder Break game. To answer your question, unfortunately, it is not possible
to push boulders to crash the treasure boxes in the game. When you push a
boulder, it will roll in a straight line until it hits an obstacle, such as a
hole, another boulder, or a cave wall. It will never move through treasure
chests or other objects in its path. However, you can use boulders to fill in
holes that may be blocking your way to the treasure chests! If you push a
boulder into a hole, it will disappear, allowing you to access the area
beyond. Remember, the goal of the game is to help the adventurer collect
treasure and escape the cave by pushing boulders into holes that may block the
exit. Good luck, and have fun playing Boulder Break!
Table 5: Comparison between average scores given by LLM (GPT-4) and human to responses by different models. We demonstrate the rubrics using an example query related to the C Project of the course: _“Can you put an if statement inside another if statement?”_. Few of the model responses have been compressed for readability. Model |  Response |  LLM |  Human
---|---|---|---
LLaMA-2-7B-chat |  Hello! I’m here to help you with your query. To answer your question directly, the short answer is no, you cannot put an ‘if statement‘ inside another ‘if statement‘ in Matlab/C. In Matlab, you can only have one ‘if‘ statement per ‘if-else‘ block. If you try to put another ‘if‘ statement inside the first ‘if‘ statement, you will get a compiling error… |  $0$ |  $0$
LLaMA-2-13B-chat |  Of course! I’d be happy to help. The answer to your question is yes, you can put an if statement inside another if statement in both Matlab and C. This is called nesting if statements. In Matlab, you can use the elseif keyword to nest if statements. [example omitted]. In C, you can use the if keyword to nest if statements. [example omitted]. I hope this helps! Let me know if you have any other questions. |  $1$ |  $0.5$
LLaMA-2-70B-chat) |  Hello! I’m happy to help you with your question. Yes, it is possible to put an if statement inside another if statement. This is called a nested if statement. Here’s an example: [example omitted]. In this example, the inner if statement will only execute if the condition1 is true. If condition1 is false, the code inside the inner if statement will not be executed. It’s important to note that nesting if statements can make the code harder to read and understand, so it’s generally recommended to use them sparingly and only when necessary. I hope this helps! Let me know if you have any further questions. |  $1$ |  $1$
GPT-4 |  Subject: Nested if statements in Matlab and C Answer: Yes, you can put an if statement inside another if statement. This is called "nested if statements." Both Matlab and C support nested if statements. Here’s an example in both languages: Matlab: [example omitted] C: [example omitted] In both examples, we have an outer if statement checking if ‘x‘ is positive, and inside that, we have another if statement checking if ‘y‘ is positive. This allows us to handle different cases based on values of ‘x‘ and ‘y‘. |  $1$ |  $0.5$
GPT-4 + RAG |  Yes, you can definitely put an "if" statement inside another "if" statement. This is known as nested if statements. Here’s a simple example: [example omitted] In this example, the inner "if" statement (checking condition2) is only executed if condition1 is true. This can be useful when you need to check for multiple conditions that depend on each other. Remember to always properly indent your code when using nested if statements, as it can quickly become difficult to read and understand the flow of your program if you don’t. |  $1$ |  $1$
LLaMA-2-13B-chat+RAG |  Hello! As your helpful teaching assistant, I’m here to help you with your query. Yes, it is possible to put an if statement inside another if statement in C programming. This is called nested if statements. Here’s an example: [example omitted] In this example, the inner if statement (condition2) will only be executed if the outer if statement (condition1) is true. I hope this helps! If you have any more questions or need further clarification, please don’t hesitate to ask :-) |  $1$ |  $1$
LLaMA-2-13B-chat+SFT |  Yes. |  $0.5$ |  $0.25$
LLaMA-2-13B-chat+SFT +DPO |  Certainly! Yes, it is possible to put an if statement inside another if statement in MATLAB and C. This is known as nested if statements. In MATLAB, you can use the ‘if‘ statement within another ‘if‘ statement like this: [example omitted] In C, you can use the ‘if‘ statement within another ‘if‘ statement like this: [example omitted] It’s important to note that when using nested if statements, you should always make sure that the inner if statement is fully enclosed within the outer if statement, as shown in the examples above. Additionally, it’s worth noting that using nested if statements can make your code more complex and difficult to read. It’s usually a good idea to break up your code into separate if statements if possible, to make it easier to understand and maintain. |  $1$ |  $0.75$
LLaMA-2-13B-chat+SFT +DPO +RAG |  Yes, you can put an if statement inside another if statement in C programming. This is called nesting if statements. Here’s an example: [example omitted] In the above example, the inner if statement is nested inside the outer if statement. The outer if statement checks the condition ‘condition1‘, and if it is true, the inner if statement is executed. If the inner if statement is false, the outer if statement will execute the code inside the ‘else‘ block. You can use this syntax to add more complexity to your code and make decisions based on multiple conditions. I hope this helps! Let me know if you have any more questions. |  $1$ |  $1$

##  Appendix E Evaluation Rubrics, Examples and Evaluation Prompt

###  E.1 Rubric and Prompt

Evaluation prompt: To evaluate our generations automatically we queried GPT-4
using the following prompt format. It includes the prompt template followed by
the definitions of the corresponding evaluation criteria that can be included
in a prompt, as well as the evaluation steps to be followed for each
respective criterion.

⬇

# Evaluation prompt template based on G-Eval

EVALUATION_PROMPT_TEMPLATE = """

You will be given one answer to a question written by a student on a Question-
Answer platform for a Computer Science undergraduate course. You will also
have access to the ground truth answer given by a human teaching assistant.
Your task is to rate the answer on one metric. Please make sure you read and
understand these instructions very carefully. Please keep the ground truth
answer given by the teaching assistant in mind while reviewing, and refer to
it as needed.

Evaluation Criteria:

{criteria}

Evaluation Steps:

{steps}

Example:

Question:

{question}

Ground Truth Answer:

{GroundTruthAnswer}

Answer:

{answer}

Evaluation Form (scores ONLY):

- {metric_name}

"""

# Metric 1: Usefulness

USEFULNESS_SCORE_CRITERIA = """

Usefulness (0-2) - judge whether a response would be useful to a Teaching
Assistant in answering a student’s question.

Here is the scale:

0 - A score of 0 means that the response is not useful at all. \

A Teaching Assistant would simply reject this answer\

because it is not a natural response, is irrelevant to the question, or is too
verbose.

1 - A score of 1 means that the response is partially useful. \

A Teaching Assistant needs to edit this answer, but it still sounds natural
and \

relevant so editing will not take long.

2 - A score of 2 means that the response is useful as is. \

A Teaching Assistant can use this response as is.

"""

USEFULNESS_SCORE_STEPS = """

1. Read the question carefully.

2. Read the response carefully.

2. Read the ground truth answer carefully.

3. Consider whether the response would be useful to a Teaching Assistant in answering a student’s question.

4. Assign a usefulness score from 0 to 2.

"""

# Metric 2: Accuracy

ACCURACY_SCORE_CRITERIA = """

Accuracy (0-2) - determine whether this response provides a factually correct
answer to the question.

Here is the scale:

0 - A score of 0 means that the response is completely inaccurate. The answer
is entirely incorrect or provides false information.

1 - A score of 1 means that the response is partially accurate. The answer
lacks some correct information or contains incorrect or unnecessary
information.

2 - A score of 2 means that the response is accurate. The answer is completely
accurate, providing correct information and a valid solution.

"""

ACCURACY_SCORE_STEPS = """

1. Read the question carefully.

2. Read the response carefully.

3. Read the ground truth answer carefully.

4. Consider whether this response provides a factually correct answer to the question.

5. Assign an accuracy score from 0 to 2.

"""

Listing 1: Python code for defining the evaluation prompt template.

Evaluation rubric: Our evaluation rubric for human evaluation and examples are
summarized in Table 6. Note that an answer can still be useful even if it is
factually inaccurate. Table 6 and Table 5 are based on a query asked on Piazza
related to the C Project of the course: _“Can you put an if statement inside
another if statement?”_ The actual answer to the above query on Piazza is:

Can you? Yes. Should you? Sometimes.
For the project, it is perfectly okay.
Generally, it is considered bad practice as:

⬇

if (A) {

if (B) {

// Some code

}

}

can always be written as:

⬇

if (A && B) {

// Some code

}

But, there is not really an actual issue with it outside of readability.

Table 6: Human evaluation scoring criteria. We demonstrate the rubrics using an example query related to the C Project of the course: _“Can you put an if statement inside another if statement?”_. Few of the model responses have been compressed for readability. Note: L-13 is shorthand for LLAMA-2-13B-chat. Category |  Score |  Scoring Criteria |  Example answer |  Explanation
---|---|---|---|---
Usefulness |  $0$ |  Useless:A response that would be rejected by a TA due to its lack of relevance, unnatural language, verbosity, or other factors |  L-13 + SFT: Yes. |  Despite being correct, the response is too brief to provide any meaningful information to a TA.
$0.5$ |  Partially useful: A response that requires editing by a TA, but remains natural and relevant, thus requiring minimal effort to revise. |  L-13: … The answer to your question is yes, you can put an if statement inside another if statement in both Matlab and C. This is called nesting if statements. In Matlab, you can use the elseif keyword to nest if statements. [example omitted]. In C, you can use the if keyword to nest if statements. [example omitted]. |  The response provides a direct answer and examples, but contains irrelevant information about MATLAB and lacks discussion of trade-offs. A TA may need to revise the content.
$1$ |  Useful: A response that would be accepted by a TA with little or no editing, as it provides a natural and coherent answer to the question. |  L-13+SFT+DPO: Certainly! Yes, it is possible to put an if statement inside another if statement in MATLAB and C. This is known as nested if statements. In MATLAB, you can use the ‘if‘ statement within another ‘if‘ statement like this: [example omitted]. In C, you can use the ‘if‘ statement within another ‘if‘ statement like this: [example omitted]. Additionally, it’s worth noting that using nested if statements can make your code more complex and difficult to read. It’s usually a good idea to break up your code into separate if statements if possible, to make it easier to understand and maintain. |  The response provides a clear and concise answer to the question, including examples and additional information about the potential complexity of using nested if statements. While it contains extra information about MATLAB, this can be easily removed with minor edits.
Accuracy |  $0$ |  Completely inaccurate: A response that is entirely incorrect or provides false information. |  L-7: Hello! I’m here to help you with your query. To answer your question directly, the short answer is no, you cannot put an ‘if statement‘ inside another ‘if statement‘ in Matlab/C. In Matlab, you can only have one ‘if‘ statement per ‘if-else‘ block. If you try to put another ‘if‘ statement inside the first ‘if‘ statement, you will get a compiling error… |  The response is completely inaccurate as it provides false information, stating that nesting if statements is not allowed in MATLAB/C, which is incorrect.
$0.5$ |  Partially accurate: A response which lacks some important information or contains incorrect or unnecessary information. |  L-13: The answer to your question is yes, you can put an if statement inside another if statement in both Matlab and C. This is called nesting if statements. In Matlab, you can use the elseif keyword to nest if statements. [example omitted]. In C, you can use the if keyword to nest if statements. [example omitted]. |  The response is considered partially accurate as it provides correct information about C, but also contains irrelevant information about MATLAB.
$1$ |  Accurate: A response that is completely accurate, providing correct information and a valid solution. |  GPT-4+RAG: Yes, you can definitely put an “if” statement inside another “if” statement. This is known as nested if statements. [example omitted]. In this example, the inner "if" statement is only executed if condition1 is true. This can be useful when you need to check for multiple conditions that depend on each other. Always properly indent your code when using nested if statements, as it can quickly become difficult to read and understand the flow of your program. |  The response is considered accurate as it provides correct information, including examples, and does not contain any irrelevant information about MATLAB.

###  E.2 Correlation between LLM, Human, and BertScore Evaluations

In this section, we examine the relationship between the scores assigned by
LLMs, humans, and BertScore. To do this, we use three different correlation
coefficients: the Pearson correlation coefficient ($r$), the Spearman
correlation coefficient ($\rho$), and the Kendall-Tau correlation coefficient
($\tau$). The Pearson correlation coefficient measures the linear relationship
between two variables, while the Spearman correlation coefficient measures the
monotonic relationship between two variables. The Kendall-Tau correlation
coefficient measures the strength of the association between two variables and
is based on the number of concordant and discordant pairs of observations.

We report these correlation coefficients between the different evaluation
metrics in Table 7, using data from $20$ queries for $9$ models, resulting in
a total of $180$ responses.

Metric-1 | Metric-2 | Pearson | Spearman | Kendall-Tau
---|---|---|---|---
Human Evaluation (Usefulness) | LLM Evaluation (Usefulness) | 0.586 | 0.613 | 0.567
Human Evaluation (Accuracy) | LLM Evaluation (Accuracy) | 0.646 | 0.65 | 0.591
Human Evaluation (Overall) | LLM Evaluation (Overall) | 0.662 | 0.659 | 0.565
BertScore-F1 | LLM Evaluation (Overall) | 0.383 | 0.363 | 0.28
Table 7: Correlations of different metrics on our evaluation dataset.

From the table, we can see that there is a moderate positive correlation
between the Human Evaluation metrics (Usefulness, Accuracy, and Overall) and
the corresponding LLM Evaluation metrics. This suggests that the LLM
Evaluation tends to agree with the Human Evaluation to a certain extent.

To further understand the relationship between the LLM and Human Evaluations,
we examined the confusion matrices for the Usefulness and Accuracy metrics,
which can take on 3 values: 0, 0.5, and 1. These confusion matrices are shown
in Figure 3. The diagonal of the matrices represents the fraction of examples
where the LLM and Human Evaluations agreed on the ratings. We can see that
there is a significant fraction of examples in the lower triangle of both
matrices, indicating that LLM Evaluation tends to be more lenient and assign
higher scores than the Human Evaluation.

Figure 3: Confusion matrices showing the difference between LLM-based and
Human Evaluation ratings

We also investigated the relationship between BertScore [54], an automated
similarity-based metric, and Human Evaluation. As shown in Figure 4, the
correlation between BertScore and the Human Evaluation is lower than the
correlation between the LLM Evaluation and the Human Evaluation. This suggests
that the LLM Evaluation aligns more closely with the Human Evaluation than
metrics such as BertScore.

Figure 4: The relationship between the BertScore-F1 values and the scores
assigned by human evaluators for different examples. From the plot, it is
clear that there is no apparent correlation between the two metrics. This
suggests that the BertScore-F1 values do not consistently align with the human
evaluation ratings.