# Large Language Models for Education:
Grading Open-Ended Questions Using ChatGPT

Gustavo Pinto  UFPA & Zup InnovationBelémPABrazil<EMAIL_ADDRESS>,
Isadora Cardoso-Pereira  Zup InnovationRecifeBrazil<EMAIL_ADDRESS>,  Danilo Monteiro Ribeiro  Zup InnovationRecifeBrazil
<EMAIL_ADDRESS>,  Danilo Lucena  Zup InnovationJoão PessoaBrazil
<EMAIL_ADDRESS>,  Alberto de Souza  Zup InnovationSão PauloBrazil
<EMAIL_ADDRESS>and  Kiev Gama  Universidade Federal de
PernambucoRecifeBrazil<EMAIL_ADDRESS>
(2023)

###### Abstract.

As a way of addressing increasingly sophisticated problems, software
professionals face the constant challenge of seeking improvement. However, for
these individuals to enhance their skills, their process of studying and
training must involve feedback that is both immediate and accurate. In the
context of software companies, where the scale of professionals undergoing
training is large, but the number of qualified professionals available for
providing corrections is small, delivering effective feedback becomes even
more challenging. To circumvent this challenge, this work presents an
exploration of using Large Language Models (LLMs) to support the correction
process of open-ended questions in technical training.

In this study, we utilized ChatGPT to correct open-ended questions answered by
42 industry professionals on two topics. Evaluating the corrections and
feedback provided by ChatGPT, we observed that it is capable of identifying
semantic details in responses that other metrics cannot observe. Furthermore,
we noticed that, in general, subject matter experts tended to agree with the
corrections and feedback given by ChatGPT.

ChatGPT, Open-ended Questions, Automated grading

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: The 37th Brazilian Symposium on Software
Engineering – Education Track; 25 - 27 September, 2023; Campo Grande,
Brasil††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Software and its
engineering Software design tradeoffs

##  1\. Introduction

The software industry regularly presents challenges to development teams.
Companies and teams continually strive to improve operational efficiency while
reducing costs and maintaining or increasing productivity. In this context,
software developers must continuously enhance their skills to remain relevant
in their careers (Begel and Simon, 2008). The process of studying and
training111For the context of this work, we use the terms “training” and
“studying” interchangeably. is integral to developers’ work routine,
supporting their continuous improvement and playing a crucial role in
enhancing technical skills.

During the training process, problem-solving activities or exercises are
essential for two main reasons. Firstly, they help solidify and self-assess
the understanding of new concepts. Secondly, they indicate whether the
individual’s current knowledge is sufficient for performing tasks (Tomporowski
et al., 2015; Smith, 1982; Makransky et al., 2016). The feedback that
developers receive regarding their exercises is just as important as solving
them (Vollmeyer and Rheinberg, 2005; Thurlings et al., 2013). Good feedback is
crucial for the learning process of software developers, as it provides
insights into their progress and areas for improvement. However, offering
effective feedback can be challenging, requiring broad subject knowledge,
availability, and the ability to identify knowledge gaps (Gielen et al.,
2010). Moreover, providing feedback promptly adds to the complexity of this
task (Thurlings et al., 2013).

In the context of a software producing organization, it is necessary for the
individuals who provide feedback to find time to do that. Thus, they need to
allocate hours they usually dedicate for tasks related to software development
and allocate part of their schedule for this grading and feedback activity,
which reduce their operational efficiency. This can have a significant impact,
especially for companies that invest in their employees’ learning. This is
specifically the scenario experienced at Zup Innovation.

Due to Zup’s size, with several thousand employees, of which approximately 90%
work in engineering teams, theoretical exams predominantly consist of closed-
ended questions, either single-choice or multiple-choice. However, the usage
of closed-ended questions presents a significant limitation in the evaluation
process, given their limited _feedback_. Aiming to accelerate the grading and
feedback process, this work presents an investigation into the use of ChatGPT
as a supplementary evaluation method. Our goal is to understand whether
ChatGPT could be considered as a mechanism for grading open-ended questions in
the training process employed by Zup.

To accomplish this, the study began by creating a set of open-ended questions
on two topics of interest to Zup: (1) web application caching and (2) stress
and performance testing. We asked two experts in this field to answer three
questions each. With the responses from these experts, we conducted a pilot
experiment involving six developers from the engineering team at Zup. In this
pilot, we administered an online questionnaire and asked the developers to
respond to the six open-ended questions. We used ChatGPT to correct the pilot
questions and evaluated the quality of the prompts for answer grading. After
receiving feedback from the pilot participants, we randomly invited 100 more
people from the engineering team, of whom 50 had completed at least one
technical training, while the other group had started but not completed any
training. Of these, 16 and 24 individuals from each group, respectively,
completed the questionnaire (N=40). In this work, we bring the following
contributions:

  * •

We explored the use of ChatGPT in the domain of open-ended question grading
and feedback.

  * •

We assessed the responses of experts (2 people) and non-experts (40 people)
using ChatGPT.

  * •

We compared the grading provided by ChatGPT using a typical metric,
identifying and explaining any potential inconsistencies.

##  2\. Why ChatGPT?

The recent technological advancements, such as the significant improvement in
computational power and the enormous amount of data stored in structured and
unstructured formats, have greatly benefited the field of Machine Learning
(ML), especially Deep Learning (DL). DL, in particular, has revolutionized
various domains of knowledge, such as image and speech recognition (Hernández-
Blanco et al., 2019).

The early DL models for long sequences (such as texts) processed inputs
sequentially (Li et al., 2020), which required larger models, more time, and
computational power for training. Additionally, these models struggled to
relate different parts of the sequences, resulting in limitations in learning
(Ma et al., 2019). However, with the introduction of Transformers models,
parallel processing of long-term sequences became possible, enhancing the
learning capacity and accelerating the process (Vaswani et al., 2017). These
improvements culminated in _Large Language Models_ (LLMs), which provided
significant advances in text processing and natural language understanding
(Zhao et al., 2023).

Transformers Models. Transformers models are a widely used deep neural network
architecture in Natural Language Processing (NLP). Unlike previous neural
network models that processed elements sequentially, Transformers introduced
the attention mechanism (Vaswani et al., 2017).

The attention mechanism allows the model to assign different weights to
specific parts of the input during training. Instead of solely relying on
sequential order, the model can focus on parts of the sequence that are more
relevant to the task at hand, capturing complex and long-range dependencies.
For instance, when given a paragraph as input, while previous models "read"
the paragraph sequentially, the attention mechanism allows Transformers to
assign higher importance to specific words. This way, the model can capture
long-range relationships between words, even when there are several words
between them.

Moreover, the attention mechanism enables Transformers to process data in
parallel, dividing the input sequence into multiple parts and performing
attention operations independently. This leads to efficient and simultaneous
processing of information, resulting in smaller models compared to traditional
ones. This characteristic is one of the main reasons why these models can
efficiently and scalably process and generate text on a large scale (Vaswani
et al., 2017; Zhao et al., 2023).

Large Language Models. LLMs are Transformers models with millions, billions,
or even more parameters, extensively trained on large textual datasets, such
as libraries of books, web articles, and conversations on social networks.
Through this diverse training data, these models acquire a deep understanding
of the structure, grammar, and semantic context of human language.

Previously, training and utilizing these models for specific tasks required
considerable computational resources and advanced technical knowledge to
develop, deploy, and enhance the models. However, platforms like ChatGPT and
similar ones have democratized access to LLMs, enabling individuals without ML
expertise to interact intuitively with these models through chat interfaces,
such as virtual assistants.

This mode of interaction has brought about an alternative learning paradigm in
the field of ML: prompt-based learning. Instead of refining the model
traditionally, through providing more data and adjusting parameters, tasks are
reformulated as textual prompts. An appropriate prompt can shape the model’s
behavior, directing the desired output without the need for conventional fine-
tuning (Liu et al., 2023). As a result, LLMs demonstrate impressive ability to
perform complex tasks, even when trained with few examples (few-shot learning
(Brown et al., 2020)) or no examples (zero-shot learning (Kojima et al.,
2022)). This qualifies them for a wide range of NLP activities, including
automatic translation, text generation, and document summarization (Zhao et
al., 2023).

ChatGPT in Education. ChatGPT is a specific implementation of an LLM based on
the GPT (_Generative Pre-trained Transformer_) architecture (Brown et al.,
2020). This tool has the potential to promote improvements in learning and
teaching experiences at various levels, from school education to university
and professional development.

One of the advantages of ChatGPT, along with other LLMs, is the ability to
offer personalized learning, taking into account the preferences, abilities,
and individual needs of each student. This personalization can contribute to
making the learning experience more effective and engaging (Kasneci et al.,
2023).

##  3\. Research Questions

This work aims to provide answers to two research questions:

  * RQ1:

Considering the responses of experts, what is the quality of the grading
provided by ChatGPT?

  * RQ2:

Considering the responses of non-experts, what is the quality of the grading
provided by ChatGPT?

To answer RQ1, we asked experts to respond to six open-ended questions. These
questions were corrected using ChatGPT. For comparison purposes, we also asked
ChatGPT to answer the same questions; and we also corrected the questions
answered by ChatGPT with ChatGPT. After refining the responses to the six
questions, to answer RQ2, we asked a larger group of developers, but without
expertise in that specific domain, to respond to the questions.

##  4\. Method: Questionnaire

In this section, we will present in detail the process of creating, testing,
and implementing the questionnaire. We will discuss the steps involved in
formulating the questions, as well as the methods used to ensure the validity
and reliability of the results. Additionally, we will describe the process of
testing the questionnaire with a pilot group and how the results of this test
were used to improve the final experiment.

###  4.1. Questionnaire design

To create the questionnaire, we selected two topics for which we had experts
available in the Education team at Zup: (1) caching and (2) stress and
performance testing . This decision was made so that we could generate prompts
that compared the responses of the experts with the responses of the
participants in the experiment. This way, we could reduce confirmation bias in
the similarity calculation performed by ChatGPT (Witteveen and Andrews, 2019).
For the selection of questions, we chose three questions for each topic. We
opted for a small number of open-ended questions to avoid overburdening the
research participants, as they were using their work hours to answer the
questions. The questions were organized in ascending order of difficulty,
starting with the easiest and ending with the most difficult. The chosen
questions (Q _n_), along with the responses of the experts (R _n_), were:

_Caching_

  1. (Q1)

Explain in your own words what you understand about _caching_ in REST
applications.

  2. (R1)

“ _Caching_ is a technique that allows storing frequently accessed data in
memory to reduce the complexity cost of querying them. Nowadays, there are
several ways to apply _caching_ in REST APIs, starting on the server-side
through techniques of local and distributed caching. It is also possible to
enable caching on the client-side, where through the use of HTTP protocol
headers, policies are defined to govern the _caching_ behavior, such as using
versions, expiration time, and also specifying which clients can store the
data, referring to the end user’s browser and/or CDNs.”

  3. (Q2)

Explain briefly how the two types of _caching_ work: client-side caching and
server-side caching.

  4. (R2)

“ _Server-side caching can be served locally, causing a portion of the
server’s memory heap to be used for storing data that has high network or
computation cost and is frequently accessed. This strategy should be used in
scenarios where the system architecture is monolithic or there is a
restriction that only one instance of the system is used. Another way to
provide caching at the application layer is through the use of distributed
cache providers, favoring a global point of access that is shared among
instances, facilitating data synchronization with the source of truth._ ”

  5. (Q3)

Explain cache invalidation in REST applications and present a way to address
it.

  6. (R3)

“ _Cache invalidation is an operation that aims to keep the caching lean and
consistent. To provide these guarantees, invalidation policies must be used.
Some examples are Least Recently Used (LRU), which aims to remove from the
cache the data that has not been accessed recently. Another example of a
policy is Least Frequently Used (LFU), which aims to remove from the cache the
data that is least accessed. There are also providers that work with
expiration policies, where the data enters with a duration time, and upon
reaching a certain time, they are automatically removed from the cache._ ”

_Stress and performance testing_

  1. (Q4)

Explain the concept of load and stress testing.

  2. (R4)

“ _Load testing means verifying how an application or system behaves under an
expected workload, which can be small, moderate, or large. Additionally, this
workload is applied for a certain interval of time, such as minutes or hours,
to validate the system’s stability and detect possible problems in resource
usage, such as memory, CPU, disk, or connections to a database, for example.
It is important to understand that load testing does not exceed the expected
or designed capacity for an application or system. On the other hand, stress
testing is related to verifying how an application or system behaves when
subjected to a very high and intense workload, usually a workload higher than
expected or specified in the requirements. The idea here is to subject the
application beyond its designed capacity in order to detect problems or
bottlenecks in resource or internal component usage. The goal is to discover
how the system behaves under extreme pressure, such as traffic spikes,
excessive resource usage, hardware failures, or abnormal conditions._ ”

  3. (Q5)

What are the main metrics used to evaluate the performance of an application
during a load test?

  4. (R5)

“ _Generally, for a web application, including REST APIs, the main metrics we
collect and evaluate are: response time, throughput (number of operations per
unit of time), and error rate. There is a well-known method called the ‘RED
Method,’ which basically recommends evaluating these 3 metrics for request-
based services. For non-request-based applications, such as batch processing
or streaming services, other metrics like CPU, memory, or network are also
usually collected and evaluated._ ”

  5. (Q6)

What are the best practices for conducting load tests on applications that
expose REST APIs?

  6. (R6)

“ _There are several important practices when conducting load tests, such as
defining the use cases to be validated and the expectations of the expected
workload. It is also important to define which metrics are relevant for the
test, as they will help identify performance issues and bottlenecks (here, the
‘RED Method’ can be adopted). Another point is to run load tests against an
application in production or a similar production-like environment, such as a
staging environment; this way, we will obtain numbers close to the reality of
the system. And last but not least, applying the tests with a realistic data
set whenever possible._ ”

The questionnaire used in the study was implemented through the TypeForm
platform. It was designed to be anonymous, and all questions were mandatory.
However, due to the implementation of privacy policies at Zup, the
questionnaire did not include demographic questions. Instead, it contained
only six open-ended questions. Before each group of questions, we presented
the purpose of the study and introduced the concepts of caching and stress and
performance testing.

###  4.2. Questionnaire Pilot

Before making the survey available and collecting responses from a larger
population, we first conducted a pilot with two objectives: 1) to evaluate the
understanding of the presented questions, and 2) to assess the quality of the
prompts used for question grading in ChatGPT.

To conduct the pilot, we sent an invitation message to a virtual space with
developers from Zup who discuss topics related to caching and to stress and
performance testing222At Zup, the use of such spaces for forming discussion
groups on relevant topics is common.. In the message, we introduced the
objectives of the work along with the link to the online form. In total, six
participants responded to the questions. According to TypeForm statistics, 65
people opened the form, while 49 started to answer it. Out of those, only six
completed the responses to all questions. The average time for completing the
activity was 38 minutes. With the first set of responses, we initiated the
process of engineering the prompt for grading (further details in Section
4.4). After the gradings were completed, we shared the responses together with
the evaluation from ChatGPT with the same virtual space where we invited the
individuals, as a way to request feedback on the evaluation performed by
ChatGPT. Two people provided feedback, which helped us improve the provided
prompt (e.g., “All my responses followed the same feedback pattern. It
presents a correct and general approach on the topic, mentioning some
important points. However, it could provide more details and real-world
examples. Sometimes, it asks for more technical details, and in others, it
requests clear explanations for all audiences.”).

###  4.3. Actual Questionnaire

After conducting the pilot and refining the prompts based on participants’
feedback, we sent the questionnaire to a larger population of developers at
Zup. Initially, we selected 100 individuals who had already participated in at
least one of the training sessions offered by the Education team. We made this
decision to target the questions towards developers who are more accustomed to
study activities during their work environment. Out of the 100 participants,
we selected 50 who had successfully completed at least one training, meaning
they achieved the minimum required average to pass the course; similarly, we
selected 50 who had not completed any training. Subsequently, we individually
invited each participant through a message sent via the private enterprise
chat platform. Similar to the pilot, in the message we expressed our interest
and the study’s objective, along with the link to the online questionnaire. As
in the pilot, the questionnaire was anonymous and no questions regarding
demographic information were asked. Over the course of a week, we sent
reminder messages to those who had not indicated whether they had responded to
the questionnaire or not. According to TypeForm statistics, 90 individuals (53
from the group who completed at leas one training and 37 from the group who
did not) opened the form, while 76 (43+33) started to respond to it. In the
end, we obtained 40 (16 +24) responses. The average time for completing the
activity was 42 minutes.

###  4.4. Prompt Engineering

The term _prompt_ refers to a set of instructions provided to a LLM (Language
Model) as a way to customize or refine its capabilities. The prompt defines
the context of the conversation, informs the LLM about what information is
relevant, and specifies the type of output expected. The quality of the output
generated by an LLM is directly related to the quality of the prompt provided
by the user.

Prompt engineering (Liu and Chilton, 2022), on the other hand, refers to the
process of adjusting or improving the responses generated by LLMs (Zhao et
al., 2023). In other words, prompt engineering involves training LLMs via
prompts. The process of prompt engineering involves modifying the prompts to
obtain more desirable results. This fine-tuning technique optimizes human-
computer interaction by refining text generation based on the users’ needs and
expectations.

####  4.4.1. Prompt Engineering Phases

We describe our prompt engineering steps below.

Prompt V1. In the first prompt version, we used ChatGPT as an oracle for open-
ended questions. Additionally, we included technical aspects that, although
relevant to the company’s context, were not necessary for answering the open-
ended questions. The initial prompt is as follows:

Suppose you are an expert in creating web applications using REST APIs. In
addition to being an expert, you grade exams on this topic. Consider the
following question and answer: Q: Explain in your own words what you
understand about caching in REST applications? R: {} What score would you give
to this answer on a scale from 0 to 10? Return the answer in a JSON format,
with a variable ’score’ for your rating, and another variable ’explanation’
for the justification of this score. Your explanation should have at least 20
words. In the explanation, identify any knowledge gaps and explain how to
minimize them using real-world examples. If no response is provided, indicate
’No response was given,’ and assign a score of zero.

In the prompt above, we illustrated a question about _caching_. We used the
approach of creating a prompt for each question, only changing the context
(first paragraph) and the question itself. The response is provided to the
prompt in a way that can be parameterized (indicated by {} in the prompt).
However, when evaluating this first prompt on the pilot data, we observed that
the scores given by the model were consistently high, even for simple answers,
indicating a possible model miscalibration.

Prompt V2. As an attempt to increase the model’s evaluation rigor, we removed
the first sentence from the prompt and replaced it with the following
instruction: “I need you to grade the exams using the highest grading standard
you can.” The rest of the prompt remained unchanged. Furthermore, the first
prompt was designed to be specific to the context of programming in Java.
However, it was observed that the questions were theoretical and did not
require knowledge of a specific programming language. Therefore, this
information was also removed. With these modifications, it was possible to
observe an increase in the evaluation rigor of the questions.

Prompt V3. Although the V2 version of the prompt increased the rigor of
question evaluation, it still has an important limitation: relying solely on
the knowledge base and interpretation of ChatGPT, which is known to provide
incorrect information (Ray, 2023). In order to minimize this limitation, we
designed a new prompt by providing the expert’s answer and then we asked to
compare the student’s answer with the expert’s answer.

Prompt V4. After the first round of pilot evaluation, the scores with the
explanations provided by ChatGPT were shared with the pilot participants.
Among the feedback received, it was suggested that the explanation would be
more illustrative if it included real-world examples. Therefore, we added a
final instruction at the end of the prompt, indicating that “Whenever
possible, the explanation should include real-world examples.” The final
prompt generated and used for the correction of the remaining open-ended
questions can be seen below:

I need you to grade the exams using the highest grading standards possible.
Consider the following question and the answer provided by an expert: Q:
Explain in your own words what you understand about _caching_ in REST
applications? R Expert: _Caching_ is a technique that allows storing data
frequently accessed in memory to reduce the complexity cost of querying it.
Nowadays, there are various ways to apply _caching_ in REST APIs, starting on
the server-side through local and distributed cache techniques. It is also
possible to enable client-side _caching_ , where HTTP protocol headers define
policies that govern _caching_ behavior, such as versioning, expiration time,
and defining which clients can store the data, referring to the end-user’s
browser and/or CDNs. Now, consider the student’s response provided below. R
Student: {} What grade would you give to the student’s response, considering
the expert’s answer, on a scale from 0 to 10? Return the response in a JSON
format, with a variable ’grade’, containing your grade, and another variable
’explanation’ with the explanation for this grade. Your explanation must have
at least 20 words. In the explanation, identify knowledge gaps and explain how
to minimize these gaps using real-world examples. If no response is provided,
inform ’No response provided’, and give a grade of zero.

####  4.4.2. Prompts Execution

In this article, we utilized ChatGPT through its
API333https://platform.openai.com/docs/api-reference. We used the GPT-4 model
with the following configurations: a maximum of 100 tokens and a temperature
of 0.6; the temperature is a hyperparameter that affects the probability of
token distributions in the model. It is possible to adjust the temperature to
control the diversity and creativity of the generated texts. The temperature
value ranges from 0 to 1. While a high temperature (e.g., 0.7+) leads to more
diverse results, a low temperature (e.g., 0.2) tends to produce more
deterministic results.

The value of 0.6 was chosen to obtain slightly different corrections for
similar questions. The maximum token value was set to make the GPT’s
corrections more concise and direct.

###  4.5. Quality Metrics

Like other LLMs, ChatGPT has a well-known limitation regarding false or
distorted perceptions of information generated by the model itself. These
“hallucinations” (a technical term used to describe this limitation) can cause
the LLM to generate answers that appear correct when, in fact, they are not.

To minimize this limitation and complement the grading provided by ChatGPT, we
also calculated the cosine similarity metric between the response provided by
the expert and the response provided by the study participant. The cosine
similarity metric is widely used to measure similarity between vectors in
multidimensional spaces (Li and Han, 2013; Lahitani et al., 2016; Singh and
Singh, 2021). By calculating the cosine similarity between the participants’
responses and a reference standard (expert’s response), we can obtain an
objective measure of how close the responses are to the desired pattern.

To calculate this metric, we used the implementation available in the sbert
library444https://www.sbert.net/. In general, to calculate cosine similarity,
we first convert the responses into numerical vectors. Then, we multiply these
vectors with each other and sum the results. Finally, we divide the
multiplication result by the product of the sizes of the vectors. The final
value obtained ranges from -1 to 1, where 1 indicates that the responses are
very similar, while -1 indicates that the responses are very different. In
summary, this metric measures the projection of the participant’s response
onto the direction of the expert’s response, providing a measure of their
similarity.

##  5\. Results

We organize the results in terms of the research questions. First, to answer
RQ1, we present an analysis of the correctness of experts’ responses (Section
5.1), comparing them to ChatGPT’s responses. Next, to address RQ2, we present
an analysis of the correctness of students’ responses (Section 5.2).

###  5.1. Grading experts’ responses

We started by evaluating the responses provided by experts (as presented in
Section 4) using ChatGPT. To conduct this evaluation, we asked ChatGPT to
grade the responses given by the experts. To do this, we had to edit the
_prompt_ to remove the lines indicating the expert’s evaluation. As a result,
the expert’s response was corrected as if it were a student’s answer. The
scores assigned by ChatGPT to the experts’ responses are presented in the
Experts” column of Table 1. After collecting and grading the experts’
responses with ChatGPT, we asked ChatGPT to answer the six open questions. To
do this, we used the following prompt: “Using the highest grading scale you
can, provide an answer of up to 100 words for the following question: QN”. The
limit of 100 words was established since the average word count in the
experts’ responses was 114 words. These responses provided by ChatGPT were
then also corrected by ChatGPT itself. The result of this correction is
available in the “ChatGPT” column of Table 1.

Table 1.  Evaluation of experts’ and ChatGPT’s responses, graded by ChatGPT. The column # Words” presents the size of the response provided by the expert (or ChatGPT), measured by the number of words. The column “Cos Sim” presents the result of the cosine similarity metric. | Experts | ChatGPT |
---|---|---|---
Question | # Words | Grade | # Words | Grade | Cos Sim
Q1 | 107 | 8 | 94 | 9 | 0.8410
Q2 | 109 | 7 | 105 | 9 | 0.7338
Q3 | 94 | 7 | 82 | 8.5 | 0.7492
Q4 | 178 | 8.5 | 93 | 9 | 0.8546
Q5 | 88 | 9 | 61 | 9 | 0.5429
Q6 | 110 | 8.5 | 93 | 9 | 0.8046

Grading of experts’ responses by ChatGPT. As one could see, the answers to the
first group of questions about caching (Q1, Q2, and Q3) consistently received
lower scores compared to the second group. As highlighted in Section 4, the
responses related to caching were provided by one expert, while the responses
concerning stress testing and performance were provided by another expert.

Upon analyzing the feedback provided by ChatGPT, we observed comments that
might have influenced the received scores. For instance, when considering the
first set of answers about caching, ChatGPT indicated that the responses
presented basic concepts but did not adequately explain or provide examples.
In Q2, which received a score of 7 and asked about the difference between
client-side and server-side caching, the expert explained server-side caching
but did not mention the second approach, client-side caching. ChatGPT
identified this gap and highlighted it in its correction. “ _The student
explained server-side caching well, including monolithic and distributed
strategies, but did not mention client-side caching. In client-side caching,
data is stored on the user’s device, such as browsers and APIs, speeding up
page loading. For example, saving images and CSS from the site of the last
visit._ ”

Another interesting example to highlight is the explanation provided for Q5,
which inquired about the main load testing metrics. In this explanation,
ChatGPT indicated that: “ _The student’s response correctly addressed the
response time, throughput, and error rate metrics, mentioned the RED Method,
and cited examples of other situations. However, it could have explicitly
explained the relationship of these metrics with the user experience and
mentioned more specific metrics for other applications, such as latency
indexes for real-time systems_”. In other words, ChatGPT correctly identified
the items provided in the responses but did not recognize a potential
relationship among these items—even though this relationship was not
questioned in the prompt.

Perception of ChatGPT’s grading by the experts. Next, we sent the corrections
provided by ChatGPT to the experts who wrote the responses. When we inquired
whether the experts agreed with the evaluations, they generally concurred with
the assessments. In particular, regarding question Q3, where there was a
higher disagreement between the experts’ and ChatGPT’s responses, the expert
who provided the answer commented that: “ _[in this question] he was precise
and noticed that my response was incomplete. I believe the score was higher
than it should have been. I would give myself a 5._ ” However, for Q4, a point
of disagreement was observed as the expert mentioned that “ _This explanation
doesn’t make much sense since what she says is missing from the response is
actually there._ ” referring to the difference between load testing and
performance testing.

Corrections of ChatGPT’s responses by ChatGPT. The gradings provided for the
expert responses questioned whether ChatGPT could provide even more elaborate
answers. The column labeled “ChatGPT” shows the result of this investigation.
As evident, in general, the responses provided by ChatGPT received higher
scores when compared to the responses provided by the experts. Particularly,
the difference was greater in the group of questions about _caching_ : 2
points of difference for question Q2 and 1.5 points of difference for question
Q3. For example, ChatGPT’s response to Q2 addressed both server-side and
client-side caching concepts: “ _Client-side and server-side caching are
strategies to temporarily store data to improve performance and loading speed.
In client-side caching, browsers store information locally on the user’s
device, such as HTML, CSS, and JavaScript files. This reduces the need to
repeatedly retrieve resources from the server, saving loading time and
bandwidth. On the other hand, server-side caching involves storing
information, such as request responses, directly on the server. In these
cases, if the information is already cached, server responses are faster,
optimizing the user experience._ ” However, for the second group of questions
about stress and performance testing, a smaller variation was observed between
the expert’s grading and ChatGPT’s grading: only 0.5 points for Q4 and also
0.5 points for Q6.

Comparison of expert and ChatGPT gradings. Although the gradings provided by
ChatGPT for the expert responses offer an initial means of comparison, as
discussed in Section 4.5, relying solely on ChatGPT’s corrections may be
insufficient, as it can exhibit hallucinations, meaning it provides false or
distorted insights generated by the model itself (Ji et al., 2023). To
complement ChatGPT’s corrections, we employ the Cosine Similarity metric,
presented in Table 1, under the column labeled “Cos Sim.” When closer to 1,
there is a higher likelihood that the responses provided by the experts and
ChatGPT are similar. As evident, in general, the responses exhibited a high
degree of similarity ($>$0.7). However, only response Q5 showed a lower
similarity value, even though both responses received a score of 9, according
to ChatGPT’s correction. Upon evaluating these two responses, it was observed
that the expert and ChatGPT approached slightly different topics, although
both were correct. For instance, in Q5, the expert commented: “ _Usually, in a
web application, including REST APIs, the main metrics we collect and evaluate
are: response time, throughput (number of operations per unit of time), and
error rate. In fact, there is a well-known method called the ’RED Method,’
which basically recommends evaluating these 3 metrics for request-based
services and applications._ ” On the other hand, ChatGPT’s response was more
generic and did not mention the existence of the ’Red Method’: “ _The main
metrics used to evaluate the performance of applications during a load test
include response time, throughput, the number of simultaneous users, resource
utilization (CPU, memory, disk, and network), and system errors or failures.
These metrics help identify bottlenecks, determine scalability, and ensure
system stability under different load conditions._ ”

###  5.2. Grading students’ responses

After the individual evaluation of expert responses, we proceeded with the
evaluation of the study participants’ responses. As highlighted in Section
4.4.1, in the final version of the prompt, we instructed ChatGPT to compare
the student’s response with the expert’s response. This approach aimed to
minimize known hallucination biases often observed in LLMs like ChatGPT (Ji et
al., 2023; Ray, 2023). Furthermore, based on the feedback from the experts,
the initially provided responses were adjusted by the authors of this article
to incorporate the insights highlighted by ChatGPT. The following Table 2
provides a summary of the evaluations from the study participants.

Difference in scores between participant groups. Although the participants of
the survey did not undergo any training on caching or stress and load testing,
our initial hypothesis was that participants who had completed at least one
technical training would have less difficulty answering the questions
(indicated by higher scores) compared to participants who had not completed
any technical training. In particular, for questions Q1 and Q2, it was
possible to observe that participants who completed at least one training
scored at least one point higher than those who did not complete any training
(for Q1, median of 6 for the completed group, while median of 5 for the non-
completed group; for Q2, the completed group scored 7 at the median, while the
non-completed group scored 6). However, for questions Q4 and Q5, the
difference between the groups was small (around 0.29 points in Q4 and around
0.1 points in Q5, on average). On the other hand, for questions Q3 and Q6,
participants who did not complete any training received higher scores compared
to those who completed a training (for Q3, median of 5 points for the
completed group and 7 points for the non-completed group, while for Q6, the
average was again 5 points for the completed group and 7 points for the non-
completed group).

Table 2. Assessments of study participants’ responses. Question | Group | Avg. | Median | Std. Dev.
---|---|---|---|---
Q1 | Completed | 5,50 | 6 | 1,71
Non-Completed | 4,96 | 5 | 0,92
Q2 | Completed | 7,29 | 7 | 0,47
Non-Completed | 5,65 | 6 | 0,69
Q3 | Completed | 5,21 | 5 | 1,86
Non-Completed | 6,04 | 7 | 1,94
Q4 | Completed | 5,29 | 6 | 2,26
Non-Completed | 5,00 | 5 | 2,72
Q5 | Completed | 5,21 | 6 | 2,82
Non-Completed | 5,31 | 6 | 2,66
Q6 | Completed | 5,21 | 5 | 1,86
Non-Completed | 6,04 | 7 | 1,94

$\begin{array}[]{ccc}\textsc{Q1}&\textsc{Q2}&\textsc{Q3}\\\
\includegraphics[width=138.76157pt]{figs/cache1_en.pdf}&\includegraphics[width=138.76157pt]{figs/cache2_en.pdf}&\includegraphics[width=138.76157pt]{figs/cache3_en.pdf}\\\
\textsc{Q4}&\textsc{Q5}&\textsc{Q6}\\\
\includegraphics[width=138.76157pt]{figs/perf1_en.pdf}&\includegraphics[width=138.76157pt]{figs/perf2_en.pdf}&\includegraphics[width=138.76157pt]{figs/perf3_en.pdf}\\\
\end{array}$

Figure 1.  Comparison of responses provided by ChatGPT and those obtained by
calculating the cosine similarity metric. The red line indicates the question
grade value indicated by ChatGPT, while the blue line indicates the resulting
metric value.

Reliability in ChatGPT’s corrections. As a way to complement the corrections
provided by ChatGPT, we calculated the cosine similarity metric, which, in
summary, evaluates the degree of proximity between two sentences in a vector
space (further details in Section 4.5). Figure 1 presents a comparison between
ChatGPT’s correction (normalized to range from 0 to 1) and the result of the
cosine similarity metric calculation. In general, it can be observed that, for
most questions, the lines follow a similar pattern: when one line tends to go
up or down, the other line tends to follow suit. The main difference in this
trend can be observed in Q5 (Figure 1-(e)). In this particular question, on
average, ChatGPT’s scores had a mean of 5.25 points, while the mean of the
similarity metric was 0.38. Figure 2 presents the distribution of corrections
using ChatGPT and the cosine similarity metric. As observable, although the
distribution of some responses has slightly different shapes, the values of
mean and median remain close, with Q5 being the exception.

Divergences observed in ChatGPT’s corrections. To better understand the
divergence observed in Q5, we investigated which responses had the greatest
discrepancy in scores. We found a total of 10 responses that had more than 3
points of divergence. Upon analyzing these responses, we noticed that ChatGPT
was able to detect details in the answers that potentially influenced the
generated scores. For example, one participant answered Q5 as follows: “
_response time, quantity of data transferred, and success rate (throughput)
and error rate_ ”. However, ChatGPT noticed the confusion the participant made
between success rate and throughput: “ _The student’s response addresses the
main metrics, with some inaccuracies. He mentions response time, throughput,
and error rate but confuses the quantity of data transferred with throughput.
To clarify, throughput generally refers to the number of operations per unit
of time and not the amount of data transferred. […]_ ”. In other words,
although the participant used the correct terms, the semantics of the terms
were incorrect. This may have misled the cosine similarity metric, even though
ChatGPT was able to detect and correct it. Other cases of divergence occurred
when the participant answered the question in a limited or incomplete manner.
In these cases, again, the response contained certain terms used in the
expert’s response but lacked details or elaboration.

$\begin{array}[]{ccc}\textsc{Q1}&\textsc{Q2}&\textsc{Q3}\\\
\includegraphics[width=130.08731pt,clip={true},trim=30.11249pt 40.15pt
30.11249pt
50.18748pt]{figs/q1-box.pdf}&\includegraphics[width=130.08731pt,clip={true},trim=30.11249pt
40.15pt 30.11249pt
50.18748pt]{figs/q2-box.pdf}&\includegraphics[width=130.08731pt,clip={true},trim=30.11249pt
40.15pt 30.11249pt 50.18748pt]{figs/q3-box.pdf}\\\
\textsc{Q4}&\textsc{Q5}&\textsc{Q6}\\\
\includegraphics[width=130.08731pt,clip={true},trim=30.11249pt 40.15pt
30.11249pt
50.18748pt]{figs/q4-box.pdf}&\includegraphics[width=130.08731pt,clip={true},trim=30.11249pt
40.15pt 30.11249pt
50.18748pt]{figs/q5-box.pdf}&\includegraphics[width=130.08731pt,clip={true},trim=30.11249pt
40.15pt 30.11249pt 50.18748pt]{figs/q6-box.pdf}\\\ \end{array}$

Figure 2. Distribution of ChatGPT corrections and the cosine similarity
metric.

##  6\. Discussion: LLMs for Education

The popularization of AI assistants could significantly impact traditional
education. For example, students can use AI assistants to get answers to
specific questions, clarify concepts or receive real-time feedback on their
work. They are also auxiliary equity tools, considering the individual
journeys of students. Neurodivergent people often cannot finish reading long
texts, or they can at a high emotional cost and cognitive overload, something
many neurotypical people do not imagine exists. ChatGPT and the like can help
extract key points from readings, minimizing such effort. Teachers can benefit
from the aid of these tools in creating teaching materials and personalizing
instruction to meet individual student needs.

However, AI assistants should not replace the key role of teachers. Education
goes beyond the transmission of information and involves social interactions,
the development of socio-emotional skills and the ability to apply knowledge
in real contexts. These aspects are fundamental and cannot be replaced by
technology.

Finally, it is necessary to consider the ethical and privacy issues related to
the use of AI assistants in education. It is important to ensure that student
data is protected and that decisions related to education are not based solely
on algorithms, but rather on a balanced approach that takes into account the
expertise of teachers and the well-being of students.

##  7\. Limitations

As any empirical study, this work also has several limitations and threats to
validity. First, to conduct the study, we used a population of 40 developers.
Although they are professionals in the field, we can hardly generalize their
responses to other groups of developers from different countries with
different experiences.

Another threat is related to the feedback provided by ChatGPT. In some
responses, we noticed that ChatGPT seems to have been too strict. For example,
the question "What are the main metrics used to evaluate the performance of an
application during a load test?" does not address potential relationships
among these metrics. However, we frequently observed responses where ChatGPT
provided feedback indicating this absence. Due to its proprietary
implementation, the authors are not aware of the reasons why ChatGPT did not
stick to answering what was asked.

Still, there is limitation related to the metric we used to compare ChatGPT’s
responses. Although there are other well-known metrics like BLEU (Papineni et
al., 2002) and METEOR (Banerjee and Lavie, 2005), we decided not to report
them in this work due to inconsistent results we obtained. For example, a
significant number of answers were evaluated as zero, which indicates that the
student’s answer does not match the reference answer. Moreover, BLEU is a
metric designed to evaluate translations of texts, and we understand that the
scenario of the article is not necessarily the most suitable for its use.
Other studies have observed that using these metrics is not appropriate for
educational purposes as they do not correlate with human evaluation (Scialom
and Staiano, 2019).

##  8\. Related Work

Although very recent, there is a growing number of studies using ChatGPT for
educational purposes. For instance, the study conducted by Moore et al. (Moore
et al., 2022) explored the use of ChatGPT-3 to assess whether student-
generated questions can be useful in the learning process. The results suggest
that the model can be a powerful tool to assist teachers in pedagogical
assessments, providing an innovative and effective approach to evaluate
students’ knowledge.

Zhu, Liu, and Lee (Zhu et al., 2020) investigated the impact of automated
assessment technologies on the review of scientific arguments presented by
students. The results revealed that automated reviews were positively
correlated with an increase in students’ grades. Furthermore, when the
automatic reviews were contextualized with the students’ responses, they
proved to be even more effective in aiding learning by providing personalized
and specific feedback.

Additionally, Bernius, Krusche, and Bruegge (Bernius et al., 2022) evaluated
the use of LLMs to generate feedback for open-ended questions in courses with
a large number of students. The results demonstrated a decrease of up to
$85\%$ in the effort required by teachers for evaluating these questions.
Furthermore, $92\%$ of the evaluations generated by LLMs were considered of
high quality by instructors. Moreover, the majority of students rated the
quality of this feedback as equivalent to that provided by instructors.

These studies demonstrate how the use of ChatGPT and automated assessment
technologies can have a positive impact on the learning process of students.
These tools offer the opportunity to provide immediate, personalized, and
detailed feedback, allowing students to improve their performance and
understanding of concepts. However, none of these works addressed the use of
tools like ChatGPT for the correction and feedback of open-ended questions in
the field of software engineering, which is the objective of this study.

##  9\. Conclusions

This study investigated the use of ChatGPT as a complementary strategy for
correcting and providing feedback on open-ended questions. For this purpose,
we recruited two experts and 40 developers to answer a set of six questions on
two different topics: (1) caching and (2) stress and performance testing. The
responses from these individuals were corrected by ChatGPT. Based on this
data, several observations were made:

  * •

In general, experts agreed with the corrections provided by ChatGPT. Among the
six feedbacks given by ChatGPT, there was only one instance of disagreement
from the expert;

  * •

The cosine similarity metric is not always suitable to be used as a proxy for
ChatGPT scores, as it loses contextual information that ChatGPT is able to
identify.

###  9.1. Future Work

For future work, we are interested in using grading rubrics annotated by
experts with specific weights for certain items that deserve emphasis in the
responses. This way, we can direct corrections to the most relevant response
items. We also hope to expand our analysis to cover other LLMs, as well as
different versions of ChatGPT. Moreover, we plan to explore other teaching
materials, such as books, as our database for open questions (and their
answers). This approach may further enrich the analysis and offer a more
comprehensive view on the effectiveness and applicability of ChatGPT in
education. Finally, we still want to understand to what extend does ChatGPT
recognizes its own answers and, thus, give better grades to them.

### Artifacts Availability

The data analyzed in this study is available online at:
https://tinyurl.com/chatgpt-for-edu.

### Acknowledgements

We thank all the Zuppers who answered the questionnaire and the reviewers who
provided relevant suggestions for improvements. This work is partially
supported by FAPESPA (#053/2021) and CNPq (#308623/2022-3).

### AI Tooling

Ultimately, although this work used ChatGPT as a subject under study, we did
not use ChatGPT to generate text content for this paper. However, this paper
was initially submitted in Portuguese, but later translated into English with
the support of ChatGPT (using the prompt “The text below is written using the
LateX markup language. Translate it to English, so it keeps the original LaTeX
markup. Do not perform any other textual adjustments.”).

## References

  * (1)
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_. 65–72.
  * Begel and Simon (2008) Andrew Begel and Beth Simon. 2008.  Novice Software Developers, All over Again. In _Proceedings of the Fourth International Workshop on Computing Education Research_ (Sydney, Australia) _(ICER ’08)_. Association for Computing Machinery, New York, NY, USA, 3–14.  https://doi.org/10.1145/1404520.1404522
  * Bernius et al. (2022) Jan Philip Bernius, Stephan Krusche, and Bernd Bruegge. 2022\.  Machine learning based feedback on textual student answers in large courses.  _Computers and Education: Artificial Intelligence_ 3 (2022), 100081\.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Gielen et al. (2010) Sarah Gielen, Elien Peeters, Filip Dochy, Patrick Onghena, and Katrien Struyven. 2010.  Improving the effectiveness of peer feedback for learning.  _Learning and instruction_ 20, 4 (2010), 304–315.
  * Hernández-Blanco et al. (2019) Antonio Hernández-Blanco, Boris Herrera-Flores, David Tomás, and Borja Navarro-Colorado. 2019\.  A systematic review of deep learning approaches to educational data mining.  _Complexity_ 2019 (2019).
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _Comput. Surveys_ 55, 12 (2023), 1–38.
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023\.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ (2022).
  * Lahitani et al. (2016) Alfirna Rizqi Lahitani, Adhistya Erna Permanasari, and Noor Akhmad Setiawan. 2016.  Cosine similarity to determine similarity measure: Study case in online essay assessment. In _2016 4th International Conference on Cyber and IT Service Management_. IEEE, 1–6.
  * Li and Han (2013) Baoli Li and Liping Han. 2013.  Distance weighted cosine similarity measure for text classification. In _Intelligent Data Engineering and Automated Learning–IDEAL 2013: 14th International Conference, IDEAL 2013, Hefei, China, October 20-23, 2013. Proceedings 14_. Springer, 611–618.
  * Li et al. (2020) Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. 2020\.  A survey on text classification: From shallow to deep learning.  _arXiv preprint arXiv:2008.00364_ (2020).
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023\.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _Comput. Surveys_ 55, 9 (2023), 1–35.
  * Liu and Chilton (2022) Vivian Liu and Lydia B Chilton. 2022.  Design guidelines for prompt engineering text-to-image generative models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–23.
  * Ma et al. (2019) Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. 2019.  Universal text representation from bert: An empirical study.  _arXiv preprint arXiv:1910.07973_ (2019).
  * Makransky et al. (2016) Guido Makransky, Malene Warming Thisgaard, and Helen Gadegaard. 2016.  Virtual simulations as preparation for lab exercises: Assessing learning of key laboratory skills in microbiology and improvement of essential non-cognitive skills.  _PloS one_ 11, 6 (2016), e0155895.
  * Moore et al. (2022) Steven Moore, Huy A Nguyen, Norman Bier, Tanvi Domadia, and John Stamper. 2022.  Assessing the Quality of Student-Generated Short Answer Questions Using GPT-3. In _Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption: 17th European Conference on Technology Enhanced Learning, EC-TEL 2022, Toulouse, France, September 12–16, 2022, Proceedings_. Springer, 243–257.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  BLEU: A Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ (Philadelphia, Pennsylvania) _(ACL ’02)_. Association for Computational Linguistics, USA, 311–318.  https://doi.org/10.3115/1073083.1073135
  * Ray (2023) Partha Pratim Ray. 2023\.  ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.  _Internet of Things and Cyber-Physical Systems_ (2023).
  * Scialom and Staiano (2019) Thomas Scialom and Jacopo Staiano. 2019.  Ask to learn: A study on curiosity-driven question generation.  _arXiv preprint arXiv:1911.03350_ (2019).
  * Singh and Singh (2021) Ritika Singh and Satwinder Singh. 2021.  Text similarity measures in news articles by vector space model using NLP.  _Journal of The Institution of Engineers (India): Series B_ 102 (2021), 329–338.
  * Smith (1982) Robert McCaughan Smith. 1982\.  _Learning how to learn: Applied theory for adults_.  Open University Press Great Britain.
  * Thurlings et al. (2013) Marieke Thurlings, Marjan Vermeulen, Theo Bastiaens, and Sjef Stijnen. 2013.  Understanding feedback: A learning theory perspective.  _Educational Research Review_ 9 (2013), 1–15.
  * Tomporowski et al. (2015) Phillip D Tomporowski, Bryan McCullick, Daniel M Pendleton, and Caterina Pesce. 2015.  Exercise and children’s cognition: The role of exercise characteristics and a place for metacognition.  _Journal of Sport and Health Science_ 4, 1 (2015), 47–55.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vollmeyer and Rheinberg (2005) Regina Vollmeyer and Falko Rheinberg. 2005.  A surprising effect of feedback on learning.  _Learning and instruction_ 15, 6 (2005), 589–602.
  * Witteveen and Andrews (2019) Sam Witteveen and Martin Andrews. 2019.  Paraphrasing with large language models.  _arXiv preprint arXiv:1911.09661_ (2019).
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023\.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).
  * Zhu et al. (2020) Mengxiao Zhu, Ou Lydia Liu, and Hee-Sun Lee. 2020.  The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing.  _Computers & Education_ 143 (2020), 103668.


TUCAN collaboration

# First ultracold neutrons produced at TRIUMF

S. Ahmed  University of Manitoba, Winnipeg, MB, Canada  E. Altiere  University
of British Columbia, Vancouver, BC, Canada  T. Andalib  University of
Manitoba, Winnipeg, MB, Canada  B. Bell  TRIUMF, Vancouver, BC, Canada  McGill
University, Montreal, QC, Canada  C. P. Bidinosti  University of Winnipeg,
Winnipeg, MB, Canada  University of Manitoba, Winnipeg, MB, Canada  E. Cudmore
TRIUMF, Vancouver, BC, Canada  Carleton University, Ottawa, ON, Canada  M. Das
University of Manitoba, Winnipeg, MB, Canada  C. A. Davis  TRIUMF, Vancouver,
BC, Canada  B. Franke  TRIUMF, Vancouver, BC, Canada  M. Gericke  University
of Manitoba, Winnipeg, MB, Canada  P. Giampa  TRIUMF, Vancouver, BC, Canada
P. Gnyp  TRIUMF, Vancouver, BC, Canada  Coburg University, Coburg, Germany  S.
Hansen-Romu  University of Manitoba, Winnipeg, MB, Canada  University of
Winnipeg, Winnipeg, MB, Canada  K. Hatanaka  Research Center for Nuclear
Physics (RCNP), Osaka University, Osaka, Japan  T. Hayamizu  University of
British Columbia, Vancouver, BC, Canada  B. Jamieson  University of Winnipeg,
Winnipeg, MB, Canada  D. Jones  University of British Columbia, Vancouver, BC,
Canada  S. Kawasaki  KEK, Tsukuba, Japan  T. Kikawa  Kyoto University, Kyoto,
Japan  M. Kitaguchi  Department of Physics, Nagoya University, Nagoya, Japan
Kobayashi-Maskawa Institute for the Origin of Particles and the Universe
(KMI), Nagoya University, Nagoya, Japan  W. Klassen  University of Manitoba,
Winnipeg, MB, Canada  University of Winnipeg, Winnipeg, MB, Canada  A. Konaka
TRIUMF, Vancouver, BC, Canada  E. Korkmaz  University of Northern British
Columbia, Prince George, BC, Canada  F. Kuchler  TRIUMF, Vancouver, BC, Canada
M. Lang  University of Manitoba, Winnipeg, MB, Canada  L. Lee  TRIUMF,
Vancouver, BC, Canada  University of Manitoba, Winnipeg, MB, Canada  T.
Lindner  TRIUMF, Vancouver, BC, Canada  University of Winnipeg, Winnipeg, MB,
Canada  K. W. Madison  University of British Columbia, Vancouver, BC, Canada
Y. Makida  KEK, Tsukuba, Japan  J. Mammei  University of Manitoba, Winnipeg,
MB, Canada  R. Mammei  University of Winnipeg, Winnipeg, MB, Canada
University of Manitoba, Winnipeg, MB, Canada  TRIUMF, Vancouver, BC, Canada
J. W. Martin  University of Winnipeg, Winnipeg, MB, Canada  University of
Manitoba, Winnipeg, MB, Canada  R. Matsumiya  TRIUMF, Vancouver, BC, Canada
E. Miller  University of British Columbia, Vancouver, BC, Canada  K. Mishima
J-PARC, Tokai, Japan  T. Momose  University of British Columbia, Vancouver,
BC, Canada  T. Okamura  KEK, Tsukuba, Japan  S. Page  University of Manitoba,
Winnipeg, MB, Canada  R. Picker  TRIUMF, Vancouver, BC, Canada  Simon Fraser
University, Burnaby, BC, Canada  E. Pierre  TRIUMF, Vancouver, BC, Canada
Research Center for Nuclear Physics (RCNP), Osaka University, Osaka, Japan  W.
D. Ramsay  TRIUMF, Vancouver, BC, Canada  L. Rebenitsch  University of
Manitoba, Winnipeg, MB, Canada  University of Winnipeg, Winnipeg, MB, Canada
F. Rehm  TRIUMF, Vancouver, BC, Canada  Coburg University, Coburg, Germany  W.
Schreyer<EMAIL_ADDRESS>TRIUMF, Vancouver, BC, Canada  H. M. Shimizu
Department of Physics, Nagoya University, Nagoya, Japan  S. Sidhu  Simon
Fraser University, Burnaby, BC, Canada  TRIUMF, Vancouver, BC, Canada  A.
Sikora  University of Winnipeg, Winnipeg, MB, Canada  J. Smith  TRIUMF,
Vancouver, BC, Canada  University of British Columbia, Vancouver, BC, Canada
I. Tanihata  Research Center for Nuclear Physics (RCNP), Osaka University,
Osaka, Japan  B. Thorsteinson  University of Winnipeg, Winnipeg, MB, Canada
S. Vanbergen  TRIUMF, Vancouver, BC, Canada  University of British Columbia,
Vancouver, BC, Canada  W. T. H. van Oers  TRIUMF, Vancouver, BC, Canada
University of Manitoba, Winnipeg, MB, Canada  Y. X. Watanabe  KEK, Tsukuba,
Japan

###### Abstract

We installed a source for ultracold neutrons at a new, dedicated spallation
target at TRIUMF. The source was originally developed in Japan and uses a
superfluid-helium converter cooled to $0.9\text{\,}\mathrm{K}$. During an
extensive test campaign in November 2017, we extracted up to $325\,000$
ultracold neutrons after a one-minute irradiation of the target, over three
times more than previously achieved with this source. The corresponding
ultracold-neutron density in the whole production and guide volume is
$5.3\text{\,}{\mathrm{cm}}^{-3}$. The storage lifetime of ultracold neutrons
in the source was initially $37\text{\,}\mathrm{s}$ and dropped to
$24\text{\,}\mathrm{s}$ during the eighteen days of operation. During
continuous irradiation of the spallation target, we were able to detect a
sustained ultracold-neutron rate of up to $1500\text{\,}{\mathrm{s}}^{-1}$.

Simulations of UCN production, UCN transport, temperature-dependent UCN yield,
and temperature-dependent storage lifetime show excellent agreement with the
experimental data and confirm that the ultracold-neutron-upscattering rate in
superfluid helium is proportional to $T^{7}$.

Ultracold neutrons, spallation, superfluid helium

##  I Introduction

Ultracold neutrons (UCNs) with energies of a few hundred nanoelectronvolts can
be trapped by material bottles, magnetic fields, and gravity for hundreds of
seconds. That makes them an ideal tool to precisely measure fundamental
properties of the neutron, e.g. its electric dipole moment Pendlebury  _et
al._ (2015); Serebrov  _et al._ (2015), lifetime Pattie  _et al._ (2018);
Serebrov  _et al._ (2018), decay correlations Mendenhall  _et al._ (2013),
interaction with gravitational forces Abele  _et al._ (2009), and charge
Siemensen  _et al._ (2015).

However, the precision of such experiments is limited by the low UCN densities
that can be delivered by current sources. Typically, less than two dozen UCNs
per ${\mathrm{cm}}^{3}$ are detected after filling an experiment Bison  _et
al._ (2017); Ito  _et al._ (2018). The UCN source that has been operating the
longest, but still is one of the most intense ones, is installed at Institute
Laue-Langevin, Grenoble, France. It reflects cold neutrons on moving blades
mounted on a “UCN turbine”, slowing them to ultracold velocities Steyerl  _et
al._ (1986). All newer sources rely on a superthermal process: cold neutrons
scattering on a cold converter can induce solid-state excitations and lose
almost all of their energy Golub and Pendlebury (1975). The low temperature of
the converter suppresses the inverse process of upscattering.

So far, superthermal sources have been realized with two converter materials:
solid deuterium at temperatures around $5\text{\,}\mathrm{K}$ and superfluid
helium (He-II) at temperatures below $1\text{\,}\mathrm{K}$. Solid deuterium
offers a rich spectrum of solid-state excitations, offering a high UCN-
production cross section, but also high absorption cross sections Morris  _et
al._ (2002). Conversely, superfluid helium has a lower UCN-production cross
section, but can have much lower absorption.

Several superthermal sources with deuterium converters are currently
operational, at Los Alamos National Laboratory Ito  _et al._ (2018), Paul
Scherrer Institut Lauss (2014) (both using spallation neutron sources), and at
University of Mainz Kahlenberg  _et al._ (2017) (using a reactor neutron
source).

A superfluid-helium converter is used at Institut Laue-Langevin Piegsa  _et
al._ (2014) (using a cold-neutron beam from a reactor source) and has been
used at the Research Center for Nuclear Physics Masuda  _et al._ (2012) (RCNP,
using a spallation neutron source). The latter source has been moved to TRIUMF
and installed at a new, dedicated spallation neutron source Ahmed  _et al._
(2018) in 2017.

##  II Production and losses of ultracold neutrons in superfluid helium

The dispersion relations of free neutrons and of phonons in superfluid helium
cross at an energy $E$ of $1\text{\,}\mathrm{meV}$, allowing a neutron with
that energy to excite a single phonon and lose virtually all of its energy and
momentum. Detailed measurements of the scattering function of superfluid
helium show that multi-phonon scattering allows the same process at slightly
higher energies Schmidt-Wellenburg  _et al._ (2015); Korobkina  _et al._
(2002). The UCN-production rate $P$ in the superfluid is given by the cold-
neutron flux $\Phi(E)$ and the total scattering cross section $\sigma(E)$
given by these processes:

$P=\int\Phi(E)\sigma(E)dE.$ (1)

The UCN-loss rate $\tau^{-1}$ in the superfluid, defined as the inverse of the
storage lifetime $\tau$, is given by the rates of upscattering in superfluid
helium $\tau_{\mathrm{up}}^{-1}$, absorption in helium
$\tau_{\mathrm{abs}}^{-1}$, wall loss $\tau_{\mathrm{wall}}^{-1}$, and beta
decay $\tau_{\beta}^{-1}$:

$\tau^{-1}=\tau_{\mathrm{wall}}^{-1}+\tau_{\mathrm{up}}^{-1}+\tau_{\mathrm{abs}}^{-1}+\tau_{\beta}^{-1}.$
(2)

The wall-storage lifetime is determined by the material, cleanliness, and
roughness of the walls and is typically on the order of tens to hundreds of
seconds.

The upscattering lifetime is strongly dependent on the temperature $T$ of the
superfluid and roughly follows

$\tau_{\mathrm{up}}^{-1}\approx
B\cdot\left(\frac{T}{$1\text{\,}\mathrm{K}$}\right)^{7},$ (3)

with $B$ between
$0.008\text{\,}{\mathrm{s}}^{-1}0.016\text{\,}{\mathrm{s}}^{-1}$ Leung  _et
al._ (2016). So, to suppress the upscattering rate to a similar level as the
wall-loss rate, the superfluid helium has to be cooled to a temperature around
$1\text{\,}\mathrm{K}$.

The absorption lifetime is dominated by the high neutron-absorption cross
section of 3He. In natural helium—with a 3He abundance of
$1\text{\times}{10}^{-6}$—the absorption lifetime would be less than
$100\text{\,}\mathrm{ms}$. Isotopically purified helium—available with 3He
abundances below $1\text{\times}{10}^{-12}$ Hendry and McClintock (1987)—can
increase the absorption lifetime to several thousand seconds.

The ultimate limit of storage lifetime is given by the lifetime of free
neutrons of $\tau_{\beta}=$880.2\pm 1.0\text{\,}\mathrm{s}$$ Tanabashi  _et
al._ (2018).

##  III Description of the source

Figure 1: UCN source and guide geometry at TRIUMF. When the target is
irradiated, spallation neutrons are moderated and converted to ultracold
neutrons in the cryostat, see Fig. 2. After a period of accumulating UCNs in
the source, the UCN valve is opened and UCNs can reach the detectors. The
radiation shielding encasing the cryostat and pumps is not shown. Figure 2:
Detailed simulation model of the source. Spallation neutrons, produced by
irradiating the target with protons, are moderated in heavy water and
converted to ultracold neutrons in the superfluid helium. Red dots indicate
the temperature sensors used to determine the temperature of the superfluid.
The temperature profile on the UCN guide is indicated as well. All dimensions
are given in millimeters.

The UCN source developed at RCNP uses $8\text{\,}\mathrm{L}$ of isotopically
purified superfluid helium, cooled to about $0.9\text{\,}\mathrm{K}$ with a
3He cooling circuit. Cold neutrons are provided by two-stage moderation in
liquid heavy water at room temperature and solid heavy water cooled to
$20\text{\,}\mathrm{K}$. For a more detailed description refer to Masuda  _et
al._ (2012).

In 2017, we moved the source to TRIUMF and installed it at a new, dedicated
spallation neutron source. TRIUMF’s cyclotron provides a
$483\text{\,}\mathrm{MeV}$ proton beam of which up to
$40\text{\,}\mathrm{\SIUnitSymbolMicro A}$ of beam current can be diverted
onto a tungsten spallation target surrounded by lead blocks Ahmed  _et al._
(2018). The UCN source is placed above the target and surrounded by graphite
blocks serving as additional neutron reflectors. To conform to Canadian safety
standards we had to add pressure reliefs on the cryostat and UCN guide and had
to add more radiation shielding, requiring a $4.5\text{\,}\mathrm{m}$ longer
UCN guide than at RCNP, see Fig. 1.

The UCN-production volume filled with superfluid helium has a cylindrical
shape and is attached to a vertical UCN guide, see Fig. 2. Heat is conducted
from the production volume to a 3He-cooled heat exchanger through a single
$2\text{\,}\mathrm{mm}$-diameter hole in the guide wall and a
$0.05\text{\,}\mathrm{mm}$-wide gap along its circumference. The temperature
of the superfluid helium is measured by four Cernox sensors placed in the
superfluid between the UCN guide and the heat exchanger.

The combined height of UCN-production volume and vertical UCN guide is
$1.25\text{\,}\mathrm{m}$, with the lower $0.62\text{\,}\mathrm{m}$ filled
with superfluid helium. Right above the liquid surface, a short, narrower
section of the vertical guide blocks superfluid film flow to reduce heat load.
Above the cryostat, the UCN guide continues horizontally in a vacuum jacket to
transition from cryogenic to room temperature. It ends with a burst disk for
pressure relief and a gate valve (VAT 17.2 series) with a protective ring
improving UCN transmission in the open state.

Downstream of the valve, the UCN guide follows a horizontal
$45\text{\,}\mathrm{\SIUnitSymbolDegree}$ kink to avoid radiation leaking
through a direct line of sight to the experimental area. Finally, it
penetrates through $3\text{\,}\mathrm{m}$ of additional shielding and drops
down to allow the UCNs to penetrate a $0.1\text{\,}\mathrm{mm}$-thick
aluminium foil and to enter the main detector. The total volume of the UCN-
production volume and UCN guides is $60.8\text{\,}\mathrm{L}$. The foil
separates the helium-filled UCN guide from the detector vacuum to reduce
contamination of the source. The main detector uses photomultiplier tubes to
detect scintillation light produced by UCNs captured in 6Li-enriched glass
Jamieson  _et al._ (2017). A secondary 3He proportional counter with its own
aluminium window is mounted to a $5\text{\,}\mathrm{mm}$ pinhole in the guide,
see Fig. 1 and serves as a monitor detector for measurements of transmission
through additional guides that will be presented in a separate publication.

##  IV Ultracold-neutron yield

A typical measurement of UCN yield starts with an irradiation of the target
with a certain proton-beam current and for a certain duration $t_{i}$, with
the UCN valve closed. During this time, UCNs accumulate in the source,
reaching a number

$N=P\tau_{1}\left[1-\exp\left(-\frac{t_{i}}{\tau_{1}}\right)\right],$ (4)

determined by the production rate $P$ and the loss rate in the source
$\tau_{1}^{-1}$. The loss rate

$\tau_{1}^{-1}=f_{1}\tau_{\mathrm{He}}^{-1}+(1-f_{1})\tau_{\mathrm{vapor}}^{-1}+\tau_{\mathrm{wall,1}}^{-1}+\tau_{\beta}^{-1}$
(5)

is the sum of losses in liquid helium $f_{1}\tau_{\mathrm{He}}^{-1}$, in
helium vapor $(1-f_{1})\tau_{\mathrm{vapor}}^{-1}$, on the guide walls
$\tau_{\mathrm{wall,1}}^{-1}$, and due to decay $\tau_{\beta}$. Since the
source is only partially filled with superfluid helium, the loss rate is
corrected by the fraction of time $f_{1}$ that detectable UCNs spend in the
superfluid. These components are difficult to disentangle in experiment,
instead we estimated them in simulation, see section VI.

Figure 3: Rate in the detector during two typical measurement cycles with a
beam current of $1\text{\,}\mathrm{\SIUnitSymbolMicro A}$, an irradiation time
of $60\text{\,}\mathrm{s}$, and with the valve opened for
$120\text{\,}\mathrm{s}$. The dashed lines indicate the start of irradiation
and the valve actuation times in the first cycle. The red line is a fit as
explained in section VI.

Once the irradiation period ends the valve opens and the accumulated UCNs can
reach the detector. The rate in the detector quickly peaks after a few
seconds, see Fig. 3, and then drops exponentially with a time constant

$\tau_{2}^{-1}=f_{2}\tau_{\mathrm{He}}^{-1}+(1-f_{2})\tau_{\mathrm{vapor}}^{-1}+\tau_{\mathrm{wall,2}}^{-1}+\tau_{d}^{-1}+\tau_{\beta}^{-1}.$
(6)

With the valve open, the loss rate to the detector $\tau_{d}^{-1}$ has to be
included. The fraction of time UCNs spend in the superfluid $f_{2}$ and the
wall losses $\tau_{\mathrm{wall,2}}$ are now different compared to equation
(5). The valve stays open for two to three minutes and then the cycle repeats.

We determined the total number of detected UCNs by integrating the rate in the
detector while the valve was open and subtracting the background rate, which
we estimated before the irradiation started while the valve was closed. During
irradiation, the background rate in the detector increased proportionally to
the beam current by $2.5\pm
0.5\text{\,}{\mathrm{s}}^{-1}\text{\,}{\mathrm{\SIUnitSymbolMicro A}}^{-1}$.
More detailed studies of cross-talk and pile-up in the detector showed that
those effects distort the measured rate by less than $1\text{\,}\mathrm{\char
37\relax}$. For details refer to Andalib (tion). To check that the detected
neutrons are indeed ultracold neutrons we performed an experiment with a
nickel foil replacing the aluminium foil. In this configuration, the rate in
the main detector did not increase above the background, confirming that the
vast majority of detected neutrons had energies below
$245\text{\,}\mathrm{neV}$, the Fermi potential of nickel.

Figure 4: Number of UCNs extracted from the source after irradiating the
target for $60\text{\,}\mathrm{s}$ with different beam currents. At currents
below $1\text{\,}\mathrm{\SIUnitSymbolMicro A}$, the UCN yield is proportional
with current (dashed line). At higher currents, the yield drops due to the
increased heat load; the labels indicate the peak helium temperatures reached
during irradiation.

The UCN-production rate is expected to be proportional to the beam current.
Consequently, for lower beam currents the UCN yield increases linearly with
current. However, at higher beam currents the increased heat load on the
superfluid raises its temperature and UCN-upscattering rate, reducing the UCN
yield, see Fig. 4. The highest number of extracted UCNs was $325\,000$ after
irradiating the target for $60\text{\,}\mathrm{s}$ with
$10\text{\,}\mathrm{\SIUnitSymbolMicro A}$. Dividing this number by the total
guide volume of $60.8\text{\,}\mathrm{L}$ yields a UCN density of
$5.3\text{\,}{\mathrm{cm}}^{-3}$. At the nominal beam current of
$1\text{\,}\mathrm{\SIUnitSymbolMicro A}$ the yield was $47\,500$,
corresponding to a UCN density of $0.78\text{\,}{\mathrm{cm}}^{-3}$.

Figure 5: Number of UCNs extracted from the source after irradiating the
target for various periods with different beam currents. The dashed lines
extrapolate the data for irradiation times up to $60\text{\,}\mathrm{s}$ with
exponential saturation curves. The saturation time constant (labels) decreases
with higher beam currents.

The saturating number of UCNs in the source can be directly observed by
measuring the UCN yield after different irradiation times, see Fig. 5. The
saturation time constant decreases at higher beam currents, again due to the
increasing temperature and upscattering rate of the superfluid. For currents
above $1.5\text{\,}\mathrm{\SIUnitSymbolMicro A}$ and irradiation times above
$60\text{\,}\mathrm{s}$ the yield starts to drop again due to the further
increasing temperature.

Furthermore, instead of operating the source in “batch mode”, with the valve
opening after the irradiation period, we can also continuously irradiate the
target while leaving the valve open. At beam currents of
$1\text{\,}\mathrm{\SIUnitSymbolMicro A}$ or less, such a configuration will
lead to a constant stream of
$1500\text{\,}\mathrm{UCN}\text{\,}{\mathrm{s}}^{-1}\text{\,}{\mathrm{\SIUnitSymbolMicro
A}}^{-1}$ reaching the detector. During irradiation with higher beam currents,
the temperature of the superfluid slowly increases and we observe a decreasing
rate.

##  V Storage lifetime

Figure 6: UCN yield after irradiating the target for $60\text{\,}\mathrm{s}$
with $1\text{\,}\mathrm{\SIUnitSymbolMicro A}$ while varying the delay between
end of irradiation and opening the UCN valve. An exponential fit to the data
up to a delay time of $120\text{\,}\mathrm{s}$ determines the storage
lifetime.

The number of UCNs that can be accumulated directly depends on their storage
lifetime in the source $\tau_{1}$ (equation 4), making it a crucial
performance parameter. To determine the storage lifetime, we ran cycles where
we opened the valve with different delays after the irradiation ended. A
typical storage-lifetime measurement consisted of nine cycles with valve delay
times of
$0\text{\,}\mathrm{s}170\text{\,}\mathrm{s}20\text{\,}\mathrm{s}120\text{\,}\mathrm{s}50\text{\,}\mathrm{s}80\text{\,}\mathrm{s}30\text{\,}\mathrm{s}20\text{\,}\mathrm{s}5\text{\,}\mathrm{s}$;
an exponential fit through the delay-dependent UCN yield determines the
storage lifetime (see Fig. 6). Although fitting a sum of two exponentials
provides a better fit since it takes into account the longer storage lifetimes
of low-energy UCNs, we opted for a single exponential fit as the short-term
storage lifetimes determine the performance of the source for short
irradiation times.

Figure 7: Storage lifetime of UCNs in the source on different days after
irradiating the target with $1\text{\,}\mathrm{\SIUnitSymbolMicro A}$ for
$60\text{\,}\mathrm{s}$. An exponential fit shows that it dropped by
$2.1\text{\,}\mathrm{\char 37\relax}$ per day. Uncertainties are smaller than
the markers.

Since the source volume is connected to a long UCN guide sealed with O-rings,
we expected residual gas to contaminate the source every time we open the UCN
valve. To determine the impact of this contamination, we regularly measured
the storage lifetime over a period of eighteen days, see Fig. 7. The drop in
storage lifetime also directly impacted the UCN yield as expected from
equation 4.

##  VI Comparison with simulations

To simulate UCN storage and transport, we built a detailed model of the
production volume and UCN guides for the Monte Carlo simulation PENTrack
Schreyer  _et al._ (2017), including the burst disk, actual shape of the UCN
valve in open and closed state, pinhole, foil, and main detector. PENTrack
uses Fermi potentials to model interaction of UCNs with materials; the
imaginary part of the potential determines the loss of UCNs. We set the losses
in the foil according to measurements performed by Atchison  _et al._ (2009).
We modeled the main detector with its two scintillator layers Jamieson  _et
al._ (2017) and their corresponding Fermi potentials and absorption cross
sections, as stated in Ban  _et al._ (2016).

Table 1: Fermi potentials and diffuse-reflection probabilities used for materials in the PENTrack simulation. Material | Fermi potential (neV) | Diffusivity
---|---|---
He-II | $18.8-0.5\hbar BT^{7}i$ | 0.16
He vapor | $-0.5\hbar\tau_{\mathrm{vapor}}^{-1}i$ | 0
Production volume (NiP) | $213-0.100i$ | 0.05
$213-0.120i$
Foil (aluminium) | $54.1-0.00281i$ | 0.20
Guides (stainless steel) | $183-0.100i$ | 0.03
$183-0.140i$
GS30 scintillator | $83.1-0.000123i$ | 0.16
GS20 scintillator | $103-1.24i$ | 0.16

We assumed that the spectrum of produced UCNs is proportional to $\sqrt{E}$
and that the upscattering rate in superfluid helium follows
$\tau_{\mathrm{He}}^{-1}=B\cdot\left(\frac{T}{$1\text{\,}\mathrm{K}$}\right)^{7}$,
with $B$ between
$0.008\text{\,}{\mathrm{s}}^{-1}0.016\text{\,}{\mathrm{s}}^{-1}$ as measured
by Leung  _et al._ (2016). By tuning the imaginary Fermi potentials of guides
and production volume to
$0.100\text{\,}\mathrm{neV}0.140\text{\,}\mathrm{neV}$, see Table 1, we
roughly matched the simulated storage lifetime in the source with the measured
storage lifetime $\tau_{1}$. The resulting simulated wall-loss lifetime
$\tau_{\mathrm{wall,1}}$ was $32\text{\,}\mathrm{s}38\text{\,}\mathrm{s}$.

We also included the upscattering rate in the helium vapor above the liquid
$\tau_{\mathrm{vapor}}^{-1}=\langle v\rangle n\sigma_{\mathrm{He}}$. This rate
depends on the average atomic velocity $\langle v\rangle$ given by the vapor
temperature, the vapor density $n$ given by the saturated vapor pressure of
the liquid and the vapor temperature, and the thermal-neutron-scattering cross
section of 4He $\sigma_{\mathrm{He}}=$0.76\text{\,}\mathrm{b}$$. We assumed
that the vapor has the same temperature gradient as measured by several
temperature sensors on the outer UCN-guide surface (see Fig. 2). To include
the temperature gradient in the simulation, we split the guide volume from the
liquid surface to the foil into $10\text{\,}\mathrm{cm}$-long sections and
assigned each an averaged UCN-upscattering rate in this section. Any time-
dependent effects, like rapid pressure and temperature changes when the UCN
valve is opened and the pressure difference between source and UCN guide is
equalized, are not captured with this simple model.

Figure 8: Comparison of fall time $\tau_{2}$ (top) and rise time
$\tau_{\mathrm{rise}}$ (bottom) in experimental data and simulations with
different diffuse-reflection probabilities. The boxes indicate the second and
third quartile of the experimental data, the empty circle its average. The
best match is found with a diffuse-reflection probability of
$3\text{\,}\mathrm{\char 37\relax}$.

To match the simulated UCN transport more accurately to measured data, we fit
both the simulated and measured rate of UCNs in the detector after opening the
valve at $t=0$ with the function

$R(t)=R_{0}\left[1-\exp\left(-\frac{t-\Delta
t}{\tau_{\mathrm{rise}}}\right)\right]\exp\left(-\frac{t-\Delta
t}{\tau_{2}}\right)+R_{B}.$ (7)

An example is shown in Fig. 3. Then, we tuned the probability that a UCN is
diffusely reflected by the guide walls (following Lambert’s law) to match the
rise time $\tau_{\mathrm{rise}}$ and the fall time $\tau_{2}$ to the
experimental data, see Fig. 8. The delay between opening the valve and the
first UCNs being detected in the detector, $\Delta t$, is constant in all
scenarios. The parameter $R_{B}$ is the background rate in the experimental
data and zero in the simulated data.

The experimental fall time can be matched with diffuse-reflection
probabilities of $3\text{\,}\mathrm{\char 37\relax}5\text{\,}\mathrm{\char
37\relax}$, the rise time is best matched with $3\text{\,}\mathrm{\char
37\relax}$. This value is similar to values reported for a range of UCN guides
Daum  _et al._ (2014); Wlokka  _et al._ (2017); Atchison  _et al._ (2010) and
we chose it for all subsequent simulations. The time constants slowly change
with increasing valve delay times, presumably due to a slow change in the
energy spectrum while the UCNs are stored in the source. The simulation also
correctly models this behavior.

To estimate UCN production, we built detailed target, moderator, and UCN-
converter geometries for the Monte Carlo software MCNP6.1 Goorley  _et al._
(2012), taking into account material impurities determined from assays and
fill levels of moderator vessels, see Fig. 2. With this model, we simulated
the complete source: the proton beam hitting the target; secondary neutrons,
protons, photons, and electrons; and neutron moderation in graphite and heavy
water. In contrast to liquid heavy water, there is no detailed data on
thermal-neutron scattering in solid heavy water available. Instead, we relied
on a free-gas model with an effective temperature of $80\text{\,}\mathrm{K}$,
as this seems to be the minimum effective neutron temperature achieved with
solid-heavy-water moderators Rush  _et al._ (1966). From the simulated cold-
neutron flux in the UCN-production volume and equation (1) we determined a
production rate of $20\,600\pm
200\text{\,}{\mathrm{s}}^{-1}\text{\,}{\mathrm{\SIUnitSymbolMicro A}}^{-1}$
for UCNs with energies up to $233.5\text{\,}\mathrm{neV}$.

Figure 9: Storage lifetime (top) and UCN yield (bottom) at different
temperatures after irradiating the target with
$1\text{\,}\mathrm{\SIUnitSymbolMicro A}$ for $60\text{\,}\mathrm{s}$ (filled
circles). Due to large temperature uncertainties, a range of simulated data
(empty squares and triangles) fits the experimental data (see text). The lines
are interpolations of simulated data to guide the eye. Figure 10: Measured UCN
rates and temperatures (grey area) while the target is continuously irradiated
with the UCN valve open. Due to large temperature uncertainties, a range of
simulated data (empty squares and triangles) fits the experimental data (see
text). The lines are interpolations of simulated data to guide the eye.

Figure 9 shows the storage lifetime and yield at different temperatures of the
superfluid. In the yield measurement the higher temperatures were reached
during an interruption of cooling, in the storage-lifetime measurement by
using heaters. Especially at lower temperatures, the four temperature sensors
showed large discrepancies. Comparisons with the vapor pressure in the UCN
guide, based on the vapor-pressure formula from Clement  _et al._ (1955),
suggested that the temperature sensors might underestimate the temperature due
to poor heat conduction through the small gaps in the guide. Unfortunately,
the vapor-pressure measurement was very noisy and had an offset that had to be
corrected, increasing the uncertainties at low temperatures even further. The
horizontal error bars in Fig. 9 show the range from the lowest temperature
measured by the temperature sensors to the highest temperature derived from
both the temperature sensors and the vapor pressure, including the uncertainty
in the offset correction, during each measurement.

Due to the large uncertainties, simulations with a range of wall-loss and
helium-upscattering parameters fit the data well, see Fig. 9. However, when
setting $B$ to $0.008\text{\,}{\mathrm{s}}^{-1}$ (triangles), the storage
lifetime and yield at higher temperatures is slightly overestimated.
Simulations without vapor upscattering show significant differences at higher
liquid temperatures—at $1.5\text{\,}\mathrm{K}$ the simulated storage lifetime
in vapor, $(1-f_{1})\tau_{\mathrm{vapor}}$, is reduced to roughly
$50\text{\,}\mathrm{s}$ while it is too large to have a significant effect at
$0.9\text{\,}\mathrm{K}$.

Figure 10 shows the UCN rate in the detector while we continuously irradiated
the target with different beam currents with the UCN valve open. At beam
currents above $1\text{\,}\mathrm{\SIUnitSymbolMicro A}$ the temperature of
the superfluid increases slowly, reducing the UCN rate. Again due to the large
temperature uncertainties, a range of simulation parameters can match the
data, but with $B=$0.008\text{\,}{\mathrm{s}}^{-1}$$ the simulations slightly
overestimate the UCN rate at higher temperatures.

Unfortunately, the discrepancies between the temperature sensors and the vapor
pressure prevent a more accurate determination of the upscattering parameter.
Measurements of heat transport in superfluid helium also agreed with the
expected trends in the temperature sensors, once correcting for offsets, and
sensor calibration dominated the uncertainty Rehm (2018). We are currently
preparing an improved analysis of data with more accurately measured and
controlled temperatures and pressures.

##  VII Conclusions

We successfully operated a superfluid-helium source for ultracold neutrons at
a new spallation source at TRIUMF. Although we were able to extract three
times more UCNs than ever before, thanks to an increased beam current on the
spallation target, we achieved only half of the previously best storage
lifetime, most likely due to contamination of the source while it was moved,
the burst disk added to the UCN guide, and the new UCN valve not optimized for
UCN storage.

Simulations including the temperature-dependent upscattering in superfluid
helium and helium vapor confirm that the former follows
$\tau_{\mathrm{He}}^{-1}=B\cdot\left(\frac{T}{$1\text{\,}\mathrm{K}$}\right)^{7}$,
matching the experimental UCN yield and storage lifetime with $B$ between
$0.008\text{\,}{\mathrm{s}}^{-1}0.016\text{\,}{\mathrm{s}}^{-1}$. Upscattering
in helium vapor plays a significant role at liquid temperatures above
$1\text{\,}\mathrm{K}$.

This research provides the prerequisites for future developments: a next-
generation source with cooling power and ultracold-neutron flux increased by
two orders of magnitude, and an experiment to measure the electric dipole
moment of the neutron with a sensitivity of
$1\text{\times}{10}^{-27}\text{\,}\mathrm{\text{ $e$ }}\text{\,}\mathrm{cm}$.
The excellent match of simulations and experiment makes us confident that we
can predict the performance of this future source and experiment very well.

Further operation of the current prototype source will focus on tests of
components for these future installations, e.g. UCN guides, valves,
polarizers, storage volumes, and vacuum windows to mitigate degradation due to
contamination.

###### Acknowledgements.

We would like to thank B. Hitti and C. Dick for supplying us with liquid
helium; C. Marshall, T. Hessels, and S. Horn for engineering support; C.
Remon, D. Morris, and the TRIUMF Controls Group for support with controls and
DAQ; and the TRIUMF Operations Group for delivering beam to the spallation
target. This work is supported by the Canada Foundation for Innovation (CFI),
the Canada Research Chairs program, the Japan Society for the Promotion of
Science (JSPS), the Natural Sciences and Engineering Research Council of
Canada (NSERC), and Research Manitoba. Computations were performed on the
Cedar supercomputer of WestGrid and ComputeCanada.

## References

  * Pendlebury  _et al._ (2015) J. M. Pendlebury _et al._ , Phys. Rev. D 92, 092003 (2015).
  * Serebrov  _et al._ (2015) A. P. Serebrov _et al._ , Phys. Rev. C 92, 055501 (2015).
  * Pattie  _et al._ (2018) R. W. Pattie _et al._ , Science 360, 627 (2018).
  * Serebrov  _et al._ (2018) A. P. Serebrov _et al._ , Phys. Rev. C 97, 055503 (2018).
  * Mendenhall  _et al._ (2013) M. P. Mendenhall _et al._ (UCNA Collaboration), Phys. Rev. C 87, 032501 (2013).
  * Abele  _et al._ (2009) H. Abele _et al._ , Nucl. Phys. A 827, 593c (2009), pANIC08.
  * Siemensen  _et al._ (2015) C. Siemensen _et al._ , Nucl. Instrum. Methods Phys. Res. A 778, 26 (2015).
  * Bison  _et al._ (2017) G. Bison _et al._ , Phys. Rev. C 95, 045503 (2017).
  * Ito  _et al._ (2018) T. M. Ito _et al._ , Phys. Rev. C 97, 012501 (2018).
  * Steyerl  _et al._ (1986) A. Steyerl _et al._ , Phys. Lett. A 116, 347 (1986).
  * Golub and Pendlebury (1975) R. Golub and J. Pendlebury, Phys. Lett. A 53, 133 (1975).
  * Morris  _et al._ (2002) C. L. Morris _et al._ , Phys. Rev. Lett. 89, 272501 (2002).
  * Lauss (2014) B. Lauss, Physics Procedia 51, 98 (2014), eSS Science Symposium on Neutron Particle Physics at Long Pulse Spallation Sources, NPPatLPS 2013.
  * Kahlenberg  _et al._ (2017) J. Kahlenberg _et al._ , Eur. Phys. J. A 53, 226 (2017).
  * Piegsa  _et al._ (2014) F. M. Piegsa _et al._ , Phys. Rev. C 90, 015501 (2014).
  * Masuda  _et al._ (2012) Y. Masuda _et al._ , Phys. Rev. Lett. 108, 134801 (2012).
  * Ahmed  _et al._ (2018) S. Ahmed _et al._ , Submitted to Nucl. Instrum. Methods Phys. Res. A (2018), arXiv:1810.01001 [physics.ins-det] .
  * Schmidt-Wellenburg  _et al._ (2015) P. Schmidt-Wellenburg _et al._ , Phys. Rev. C 92, 024004 (2015).
  * Korobkina  _et al._ (2002) E. Korobkina _et al._ , Phys. Lett. A 301, 462 (2002).
  * Leung  _et al._ (2016) K. K. H. Leung _et al._ , Phys. Rev. C 93, 025501 (2016).
  * Hendry and McClintock (1987) P. Hendry and P. McClintock, Cryogenics 27, 131 (1987).
  * Tanabashi  _et al._ (2018) M. Tanabashi _et al._ (Particle Data Group), Phys. Rev. D 98, 030001 (2018).
  * Jamieson  _et al._ (2017) B. Jamieson _et al._ , Eur. Phys. J. A 53, 3 (2017).
  * Andalib (tion) T. Andalib, Ph.D. thesis, University of Manitoba (In preparation).
  * Schreyer  _et al._ (2017) W. Schreyer _et al._ , Nucl. Instrum. Methods Phys. Res. A 858, 123 (2017).
  * Atchison  _et al._ (2009) F. Atchison _et al._ , Nucl. Instrum. Methods Phys. Res. A 608, 144 (2009).
  * Ban  _et al._ (2016) G. Ban _et al._ , Eur. Phys. J. A 52, 326 (2016).
  * Daum  _et al._ (2014) M. Daum _et al._ , Nucl. Instrum. Methods Phys. Res. A 741, 71 (2014).
  * Wlokka  _et al._ (2017) S. Wlokka _et al._ , (2017), arXiv:1701.07431 [physics.ins-det] .
  * Atchison  _et al._ (2010) F. Atchison _et al._ , Eur. Phys. J. A 44, 23 (2010).
  * Goorley  _et al._ (2012) T. Goorley _et al._ , Nuclear Technology 180, 298 (2012).
  * Rush  _et al._ (1966) J. J. Rush, D. W. Connor, and R. S. Carter, NSE 25, 383 (1966).
  * Clement  _et al._ (1955) J. R. Clement, J. K. Logan, and J. Gaffney, Phys. Rev. 100, 743 (1955).
  * Rehm (2018) F. Rehm,  _Heat conductivity in superfluid helium and ultra cold neutron source cryogenics_ , Bachelor’s thesis, University of Coburg (2018).