# Computer Aided Design and Grading for an Electronic Functional Programming
Exam

Ole Lübke**footnotemark: * Konrad Fuger††footnotemark: †
Fin Hendrik Bahnsen‡‡footnotemark: ‡ ,§§§F. H. Bahnsen was with the Institute
of Embedded Systems, TUHH, when the presented work was created. Katrin
Billerbeck¶¶footnotemark: ¶ Sibylle Schupp11footnotemark: 1 11footnotemark: 1
Institute for Software Systems22footnotemark: 2 Institute of Communication
Networks55footnotemark: 5 Center for Teaching and Learning Hamburg University
of Technology (TUHH) Hamburg, Germany 33footnotemark: 3 Institute for
Artificial Intelligence in Medicine University Medicine Essen Essen, Germany
0000-0001-5962-6583 0000-0002-8472-1483 0000-0002-5204-4713 {ole.luebke,
k.fuger, fin.bahnsen, katrin.billerbeck<EMAIL_ADDRESS>
###### Abstract

Electronic exams (e-exams) have the potential to substantially reduce the
effort required for conducting an exam through automation. Yet, care must be
taken to sacrifice neither task complexity nor constructive alignment nor
grading fairness in favor of automation. To advance automation in the design
and fair grading of (functional programming) e-exams, we introduce the
following: A novel algorithm to check Proof Puzzles based on finding correct
sequences of proof lines that improves fairness compared to an existing, edit
distance-based algorithm; an open-source static analysis tool to check source
code for task relevant features by traversing the abstract syntax tree; a
higher-level language and open-source tool to specify regular expressions that
makes creating complex regular expressions less error-prone. Our findings are
embedded in a complete experience report on transforming a paper exam to an
e-exam. We evaluated the resulting e-exam by analyzing the degree of
automation in the grading process, asking students for their opinion, and
critically reviewing our own experiences. Almost all tasks can be graded
automatically at least in part (correct solutions can almost always be
detected as such), the students agree that an e-exam is a fitting examination
format for the course but are split on how well they can express their
thoughts compared to a paper exam, and examiners enjoy a more time-efficient
grading process while the point distribution in the exam results was almost
exactly the same compared to a paper exam.

Keywords: Electronic Examination, Automated Grading, Constructive Alignment

##  1 Introduction

A strong argument for electronic exams is their high potential for automation,
which can decrease the effort of conducting an exam, especially during
grading. Yet, care must be taken to sacrifice neither task complexity [14] nor
constructive alignment (CA) [4] in favor of automation. For example, an exam
consisting only of multiple choice tasks would be easy to grade automatically,
but it would also be impossible to assess the ability of students to _create_
a solution rather than _select_ a given one. On the other hand, it is possible
to integrate arbitrary program logic into an e-exam, which does not only help
with automating grading (checking the correct answer through means ranging
from simple equality checks, over specialized, self-written algorithms, to
possibly even artificial intelligence [27]), but can also be used for task
design (e.g., randomization to impede cheating, sensibly showing or hiding
input fields depending on previously-given answers, execution of student code
to allow students to test their answers in a controlled way). Following the
recent introduction of large scale electronic examinations [29] and the
development of the extensible Your Open Examination System for Activating and
emPowering Students (YAPS) [2] at Hamburg University of Technology (TUHH), we
investigate in this paper how to leverage that opportunity to improve the
examination in our functional programming (FP) course. We provide a complete
tour of how we successfully designed an e-exam for the FP course at TUHH, both
from a technological and an educational perspective.

As a first step we analyzed our pre-e-exam FP courses and past exams to
identify potentials for automation and improvement. The analysis was driven by
the learning objectives of the course and how past exam tasks aligned with
them. We found that RegExs are a good candidate for automatically checking
answers that consist of short source code snippets. More elaborate programming
tasks could be evaluated using software tests. One particular LO (Students are
able to interpret compiler warnings and errors) had low coverage, but in the
e-exam we could alleviate that by integrating live compiler feedback. Overall,
we found that many of our existing tasks could reasonably be transformed to an
e-exam version.

We implemented our ideas in YAPS, extending the system where necessary.
Because we teach FP using the example of Haskell but YAPS only offered a C/C++
compiler, we extended it with a common template for Haskell programming tasks.
The template facilitates the quick creation of new tasks, and features a tool
to analyze student code for task-relevant features (e.g., usage of pattern
matching) that we developed specifically for the e-exam. Another extension is
checking answers with RegExs. We found that crafting RegExs that are flexible
enough to accept all valid solutions, but also strict enough to reject any
wrong answers, is a challenging and time-consuming process. Therefore, we
devised a small, high-level specification language for RegExs tailored to
common patterns found in Haskell code, and a tool that compiles these
specifications to actual RegExs. Furthermore, YAPS features a way for students
to enter proofs (Proof Puzzles) that are then graded by an algorithm based on
edit distance between correct solution and given answer. This algorithm
produces results that differ from our own, manual evaluation. Consequently, we
developed a new algorithm based on the length of correct proof sequences that
is in line with our judgment. Finally, while a paper exam is very flexible in
terms of input (i.e., anything that can be written or drawn), an e-exam is not
because it requires specific input in specific input fields (e.g., a numeric
input field only accepts numbers). Because we did not want to take that
flexibility away from students, we extended YAPS with a comment field for each
task that students can freely use to their liking, e.g., for taking notes, or
noting assumptions they made in case they deem the task ambiguous.

For evaluation, we manually analyzed the degree of automation of the e-exam
(focusing on automated grading), and asked the students that participated in
the exam for their opinion. Additionally, we report our own experiences and
reflect on sources of errors in automated grading. Grading of almost all tasks
features at least some automated elements; some tasks are graded fully
automated. For almost all tasks correct answers can be awarded the full amount
of points automatically. Yet, awarding a sensible amount of points for
partially correct solutions remains a challenge in most cases. For us, the
most common mistakes in automated grading were related to tasks checked by
RegExs and programming tasks. Currently, experience and manual testing are the
best defense against errors, but our RegEx creation tool is a first step
towards more (but less error-prone) automation in that regard as well. The
results from the student poll are encouraging, as most students state that an
e-exam is the right format for the FP course. Yet, they are split on how well
they can express their thoughts in comparison to a paper exam.

Altogether, we contribute the following to advancing computer aided design and
grading for FP e-exams:

  * •

An algorithm to check Proof Puzzles (cf. Section 5.3).

  * •

A small static analysis tool to check Haskell code for exam task relevant
features (cf. Section 5.5).

  * •

A language to specify RegExs on Haskell snippets (cf. Section 5.6).

In addition, this paper is a complete experience report which highlights
challenges when moving from a pen-and-paper to an e-exam, and provides answers
to the following issues:

  * •

How can exam quality be ensured? By analysis of the LOs of each task. The
e-exam should assess the same LOs as the paper exam (cf. Section 4).

  * •

How can fair grading be ensured? By critically examining results from
automated grading (cf. Section 5.3), giving students room to express their
ideas (cf. Section 5.4), and (for now) resorting to manual assessment if
necessary (cf. Sections 4.3 and 6.1).

  * •

How can programming tasks with compiler support be realized in a way that
feels natural to students, does not leak secret information about the task,
and facilitates automation? By employing a reusable template where the main
function cannot be edited by students (cf. Section 5.1).

  * •

How can certain restrictions in programming tasks be checked? By syntactic
analysis of the submitted code (cf. Section 5.5).

  * •

How can an empty answer be distinguished from the absence of an answer? By
combining ambiguous input fields with a suitable yes/no question (cf. Section
5.2).

  * •

How can the absence of errors be ensured in complex RegExs that are used to
check answers? By employing a higher-level, domain-specific language that
compiles to RegExs (cf. Section 5.6).

The paper is organized as follows: Section 2 discusses related work, and
Section 3 introduces constructive alignment (CA) and the YAPS examination
software. Sections 4, 5 and 6 contain the above-mentioned analysis,
realization, and evaluation respectively. Section 7 summarizes and provides an
outlook on future work.

##  2 Related Work

While electronic assessment and automated grading have been an active research
area for decades [9, 22], the design of entire e-exams seems to get less
attention. The report on introducing e-exams in two (Java-based) programming
courses by Rajala et al. [26] appears closest to our work. They also establish
a categorization of tasks, focus on complete exam designs, and use questions
similar to ours to collect student feedback. The most notable difference is
the method to ensure exam quality: the authors asked two (otherwise
uninvolved) other researchers for their opinion. Instead, we suggest to make
sure all LOs of the course are assessed in the exam through analysing each
task. For additional quality assurance, both methods could be combined. Other
differences include that Rajala et al. require source code submitted in the
exam to compile (which we decided against), check correctness by comparing the
program output (instead, we use property tests), and they neither restrict
which code can be executed, nor the input language. These differences can
likely be attributed to different LOs in the respective courses and the
different programming languages and paradigms.

Bloom’s Taxonomy [14] is a tool widely used to assess task complexity, and can
therefore also be used to assess an entire exam. Still, Sheard et al.
developed a scheme for classifying exam tasks using seven different features,
targeted specifically at introductory computer science courses [28]. Our
analysis is different in that its aim is not to characterize a single exam,
but to make sure exam quality did not degrade from one to the next exam. It
uses the LOs covered by each task as the only feature to make sure all tasks
are still constructively aligned [4].

An older study focussing on the transition from paper to e-exam has been
conducted by Stergiopoulos et al. for a course in Electronic Physics [30].
However, the set of investigated task types is limited to yes/no questions,
multiple choice, and calculations (with numeric input that must lie in a
specific range).

Regarding challenges when introducing e-exams, Kuikka et al. surveyed
educators at Turku University of Applied Sciences for which requirements for
introducing e-exams they see [16]. In contrast, we focus on challenges when
those requirements are met and an actual e-exam is created.

During the COVID-19 pandemic, educators worldwide had to move their classes
online and shared their experiences. Often, e-exams are also part of these
reports, but with varying level of detail regarding task design and automated
grading. Loftsson and Matthíasdóttir report on how they combined different
online teaching tools to transform a first semester (Python-based) programming
course [17]. They evaluate the changes in depth based on student surveys and
exam results, but, except for the employed tools, no details on task design
and automated grading were given. Kappelmann et al. focus on how students can
be engaged in online teaching in an introductory (Haskell-based) FP course,
and also provide details on task designs and automated grading [13]. For
checking programming tasks, they also use property testing, and add unit tests
as well as a novel IO mocking library. The latter can be seen as another way
of controlling the execution of student code (we prevent main from being
changed), yet there is no mention of restricting the input language for
certain tasks. For checking proof tasks, they include an actual proof checker
instead of Proof Puzzles.

The Proof Puzzles in YAPS are a similar to Proof Blocks [24, 25]: students are
presented with building blocks containing lines of the proof as well as
distractors, and are supposed to construct the proof via drag-and-drop. In
Proof Blocks, the proof structure is encoded as a directed acyclic graph
(DAG), and the points to award for a given solution are calculated using an
edit distance measure. Proof Puzzles also rely on edit distance for awarding
partial points. We found that this algorithm may produce unfair results and
consequently developed a new one. One the one hand, our algorithm is based on
pre-defined correct block sequences which can be seen as non-branching DAGs.
On the other hand, we use a different mechanism for awarding partial points.
Generally, points are awarded for each correct block in a sequence. Apart from
the sequences, entry points are specified to allow for reentering the correct
proof, which allows for a fair distribution of points even when some mistakes
were made. An alternative to such block-based automatic proof grading is using
a theorem prover [13, 11]. For instance, Kappelmann et al. employed Check Your
Proof (CYP) 666https://github.com/noschinl/cyp in an FP exam [13]. The input
language of CYP is intentionally close to Haskell, and it supports checking
equational reasoning, proofs by structural induction, proofs by
extensionality, case analysis, and computation induction. For the exam, they
did not require the students to strictly adhere to the syntax of CYP. Still,
similar to many tasks in our e-exam, correct solutions, if given in the
correct syntax, can automatically be verified as such. Additionally,
establishing a common syntax for proofs in the course also facilitated the
manual grading of partially correct submissions. Another example of a theorem
prover used in an exam is given by Jacobsen et al., who integrated
Isabelle/HOL [21] in a course on automated reasoning and report similar
experiences with respect to grading. In comparison to theorem provers, Proof
Puzzles or Blocks allow for fully automated proof grading, but they cannot
reason about the proof itself. Instead, they rely on pre-defined structural
information, such as the DAG representation. Yet, the two options do not
necessarily exclude each other, as shown by McCartin-Lim et al. [18]. They
developed a graphical, graph-based user interface where proof assumptions and
assertions need to be selected by students and dragged to the graph pane,
where they can be connected with directed edges. In the background, a theorem
prover checks for each each assertion whether its proof is complete, and the
outcome is indicated visually.

Automatically grading programming tasks is a vast field that has recently been
reviewed by Paiva et al. [22]. While it is often easy to determine whether a
given program is correct through traditional testing, it is much harder to
assign a reduced amount of points to partially correct solutions. Our approach
is to execute multiple tests and use a set of rules that map test results to
points. However, manual inspection is still required for incorrect programs
for two reasons: Programs that do not compile cannot be evaluated this way
(but we also want to grade those), and there are cases where the assigned
amount of points is too low because the tests did not cover a detail that we
do want to award points for. Static analysis is a more formal way to measure
the difference between a given and correct solution based on a graph
representation derived from the abstract syntax tree (AST) and its usefulness
has been shown in numerous studies [22]. We also use static analysis in the
automated grading process, but not for assigning reduced points. Instead, it
is employed to check whether a given solution can be a viable solution at all.
Some of our programming tasks restrict the input language by
forbidding/requiring the use of certain language features or functions. Since,
to our knowledge, no tool exists that reports on the functions and language
features used in a Haskell program, we devised our own.

For grading smaller tasks that do not require writing a full program or
function, we use RegExs. RegExs are a known tool for automated assessment,
with applications such as parsing output of student programs [20, 23],
extracting information from student code [20, 31], or even grading free-form
text answers [12]. In this work, we are mostly concerned with generating
correct RegExs that represent valid solutions to a given task because writing
complex RegExs that accept different answers is tedious and error-prone.
Existing work on ensuring the correctness of RegExs has recently been reviewed
by Zheng et al. [32]. While there are numerous approaches to testing or even
verifying existing RegExs, methods for creating RegExs are mostly concerned
with learning them either from examples or from a natural language
description. Another line of research is concerned with making RegExs easier
to understand for humans, e.g., by abstraction [7]. Our approach to generate
RegExs can be seen as a reverse abstraction process. We use a more abstract
language that is compiled to RegExs instead of abstracting RegExs.

##  3 Constructive Alignment & YAPS

The concept of _Constructive Alignment_ by Biggs [4] is related to
constructivist learning theories, i.e., knowledge is seen to be _constructed_
by each learner, as opposed to being _transferred_ from teacher to learner.
Thus, in order to facilitate successful learning a learning environment must
be designed that allows the desired LOs to be achieved through appropriate
learning activities. The key to this is the definition of LOs at an
appropriate level as well as the design of matching examination tasks. This is
because, according to the principle of _testing drives learning_ , the
cognitive level of the examination tasks also determines the learning
activities of the students. Thus, if what is to be learned is not reflected in
the examination tasks, well-intentioned motivational teaching often fails due
to a lack of student engagement.

To achieve higher cognitive learning objectives in an e-exam in engineering
science, a technical environment is needed that offers much more than the
construction of multiple choice tasks. Students should be able to analyze
presented issues, to develop and justify independent solutions, and not just
select predetermined answers. For such demanding examination scenarios that
assess deeper understanding [14], the examination system YAPS was developed at
TUHH, which allows for a more creative and technically appropriate
construction of examination tasks [2].

Several key design decisions were made in the development of YAPS: It is
licensed open source and is cost-efficient for the university with regard to
the required hardware resources. In Germany, there is no real alternative to
self-hosting of the examination system used for reasons of data protection.
This criterion alone excludes many candidates for other examination software.
YAPS is characterized by a contactless and state-based operating concept that
maps the testing procedure of TUHH, i.e., steps like having each student
confirm they are fit to take the exam, checking student ID cards and whether
they are actually registered, are supported by YAPS and can be performed while
keeping a distance between each other (which was especially important during
the COVID-19 pandemic, when exams at TUHH were still conducted in person).
Finally, YAPS is designed to be extensible and built with well-known
technologies (e.g., Typescript, Docker).

Other, similar softwares such as Autolab777https://github.com/autolab/Autolab
[19], INGInious888https://github.com/UCL-INGI/INGInious [6] or
ArTEMiS999https://github.com/ls1intum/Artemis [15] can also be self-hosted,
are extensible, and the latter even supports Haskell out of the box. However,
YAPS is already available at TUHH and well-integrated with the processes
attached to conducting an exam. Its lack of support for Haskell can easily be
remedied through its extensibility.

##  4 Analysis of our Pre-E-Exam FP Course

Changes to the examination of a course should be reflected in the teaching
activities, and vice versa. This ensures that the teaching activities prepare
the students to take the exam, and that the exam assesses the knowledge and
skills taught in the course. To be able to review this mutual dependency, in
this section we analyze our FP course and exams before the e-exam.

First, through describing and listing the teaching activities of the course,
we capture its current state to identify any exam-related parts that may need
to be updated. Then, we examine previous exams with regard to CA and task
categories to identify potentials for automation and improvement of task
design. Finally, having established candidates for change in both the teaching
activities and in the exam, we devise a plan to reach our goal: an updated FP
course with an e-exam that reduces effort through automation without reducing
exam quality.

###  4.1 Course Description

Table 1: LO of our FP course as specified in the module handbook [8]
Knowledge-based
---
$K_{1}$ |  Students apply principles, constructs, and simple design techniques of FP.
$K_{2}$ |  Students demonstrate ability to read Haskell programs, and explain Haskell syntax.
$K_{3}$ |  Students interpret warnings and find errors in programs.
$K_{4}$ |  Students apply fundamental data structures, data types, and type constructors
$K_{5}$ |  Students employ strategies for unit tests of functions and simple proof techniques for partial and total correctness.
$K_{6}$ |  Students distinguish laziness from other evaluation strategies.
Skill-based
$S_{1}$ |  Students break a natural language description down in parts amenable to a formal specification and develop a functional program in a structured way.
$S_{2}$ |  Students assess different language constructs, make conscious selections both at specification and implementation level, and justify their choice.
$S_{3}$ |  Students analyze given programs and rewrite them in a controlled way.
$S_{4}$ |  Students design and implement unit tests and can assess the quality of their tests.
$S_{5}$ |  Students argue for correctness of their program.

The course is based on the textbook “Programming in Haskell” by Graham Hutton
[10] and mainly targeted at first semester computer science students. The
teaching activities of the course are aligned with the LOs summarized in Table
1. Overall, there are three weekly activities: First, the lecture lays the
foundation for the other activities. To a limited extent, it also features
practical elements like executing code examples and short in-class exercises.
Second, so-called programming labs provide an opportunity to get hands-on
experience with FP. During the labs, students solve small programming tasks
with the support of student tutors, as it is common in other courses as well
[3]. They can also help out each other and exchange different approaches
towards solving the tasks. At the end of each lab session, the tutors discuss
the solutions with each student, making sure that misunderstandings are
detected and cleared up as early as possible. Third, there are homework
exercise sheets that, in contrast to the labs, are supposed to be solved
alone. They feature more complicated tasks that require and foster a deeper
understanding. To check their solutions for correctness, students upload their
code files to an autograding system, so they can get feedback anytime. Usually
the students need to provide a number of examples and tests for each function,
and the given feedback reports on the correctness of functions, examples, and
tests alike. For failing tests also the test input, expected and actual output
are given. Finally, the solutions are discussed in a dedicated lecture hall
exercise session.

During the last month of the lecture period, we introduce a few changes to
what is described above, to help the students prepare for the exam. The last
two tasks of each programming lab are designed to revisit topics covered
earlier in the lecture period. During the last lab session, students are
presented with the opportunity to solve an old exam, so they get an idea of
what to expect during the real one.

###  4.2 Existing Exams

The exam is usually divided into eight main tasks with subtasks, where each of
the main tasks is dedicated to a certain topic from the lecture:  1. Types and
Type Classes  2. List Comprehension  3. Pattern Matching  4. Recursion  5.
Higher-order Functions  6. User-defined Types  7. Evaluation  8. Reasoning and
Testing.  To ensure we preserve CA while transforming the exam, and to get an
overview of what types of task we have, we analyzed the tasks of an exemplary
old exam. The results are summarized in Table 2. During the analysis we
noticed that there are no warning messages in the exam, so the first part of
$K_{3}$ is not checked at all.

Table 2: LO and types of existing exam tasks Task |  LO | Type
---|---|---
1a |  $K_{1}$, $K_{2}$, $K_{4}$ | snippet
1b |  $K_{1}$, $K_{2}$, $K_{4}$ | multiple choice
2a |  $K_{1}$, $K_{2}$, $K_{3}$ | single choice, snippet
2b |  $K_{1}$, $K_{2}$, $S_{2}$, $S_{3}$ | code
3a |  $K_{1}$, $K_{2}$, $K_{3}$, $K_{4}$ | single choice, snippet
3b |  $K_{1}$, $K_{2}$, $K_{4}$, $S_{3}$ | code
4a |  $K_{1}$, $K_{2}$ | text
4b |  $K_{1}$, $K_{2}$, $S_{1}$, $S_{2}$, $S_{3}$ | code
5a |  $K_{1}$, $K_{2}$ | snippet
5b |  $K_{1}$, $K_{2}$ | text
5c |  $K_{1}$, $K_{4}$, $S_{1}$, $S_{2}$ | code
6a i |  $K_{1}$, $K_{2}$, $K_{4}$, $S_{1}$ | snippet
6a ii |  $K_{1}$, $K_{2}$, $K_{3}$, $K_{4}$ | single choice
6b |  $K_{1}$, $K_{4}$, $S_{1}$ | code
7a |  $K_{1}$, $K_{2}$, $K_{6}$ | text
7b |  $K_{1}$, $K_{2}$, $K_{6}$ | snippet
7c |  $K_{1}$, $K_{2}$, $K_{6}$ | text
8a |  $K_{1}$, $K_{2}$, $K_{5}$, $S_{1}$, $S_{2}$, $S_{4}$ | code
8b |  $K_{1}$, $K_{2}$, $K_{5}$, $S_{5}$ | text

Regarding the types of tasks, we found that all tasks can be presented in one
of the following five categories:  1. Single choice: Select one option from
two or more.  2. Multiple choice: Select zero or more options from two or
more.  3. Snippet: Write a short piece of source code, e.g., a type, an
expression, or even just a number.  4. Code: Write a larger piece of source
code, e.g., a complete function or user-defined type.  5. Text: Write a text,
e.g., some explanation or justification.

To make sure the e-exam assesses the same LOs as we did previously, we took
the arguably most simple approach to design it: “translating” each task
directly. With the first two categories we already use task types that can
directly be integrated into an e-exam. Integrating snippet and code tasks is
more difficult, but the answers must follow the rules of Haskell by design, so
at least partially automated assessment is possible for them. Text answers are
comparatively unstructured, and students may answer in either English or
German, which makes those tasks ineligible for automated assessment.

###  4.3 Consequences

The analysis of the teaching activities showed that the later lab sessions
with recapitulation tasks are suited best for familiarizing the students with
the e-exam format. Therefore, we decided to move those tasks to YAPS instead
of using text editor and terminal as usual. In addition, the exemplary exam
that students can solve during the last lab appointment is also in electronic
form. This way, all students have the opportunity to learn and try out how to
use YAPS, and we can also test new ideas in a comparatively risk-free
environment. Furthermore, our tutors are not only there to help out with any
issues that may arise, but also to collect valuable feedback directly from the
students, which we could not get any other way (especially observations on how
the system is used and insights from student-to-student/tutor conversations).
Through this feedback we can detect problems (e.g., task designs that are
difficult to navigate) early and prevent them from occurring in the real exam.

Regarding the exam itself, we found ways to transform each of the task
categories we identified to an e-exam. Single and multiple choice tasks can
naturally remain. For snippet tasks we found that RegExs can in all cases be
used to detect correct solutions, yet a RegEx match is a Boolean decision
(i.e., either the given answer matches the RegEx or not), so awarding a
reduced amount of points for partially correct solutions is not possible. This
is why we also allow for a manual assignment of points in case there was no
match. Alternatively, snippet tasks could be checked using a compiler. While
this does not solve the problem of grading partially correct solutions, it
would reduce the workload because no RegExs need to be created. On the other
hand, RegExs are arguably more flexible because they could also be used to
detect common errors (by additionally specifying multiple erroneous RegExs),
hence enabling awarding reduced amounts of points. Tasks that require writing
a larger amount of source code can also remain, but additional work is
required for the automated assessment here as well. In YAPS, programming tasks
are worked on in an integrated code editor with syntax highlighting and the
possibility to compile and execute the code, where the output of both
compilation and execution is shown to the students. For automated assessment
we mainly employ randomized property tests via QuickCheck [5] and a new static
analysis tool that provides information on used language features and
functions. Welcome side effects of integrating a code editor are that
programming tasks can now be executed in a more familiar way, and through
error and warning messages from the compiler, LO $K_{3}$ is now fully
included, while in the paper exam it was not. One important decision to make
with automated programming tasks is whether only compiling code is accepted as
an answer, or whether (partial) points are awarded also for non-compiling
programs. We decided for the latter because we believe that a solution that is
correct except for, e.g., a single wrongly-indented line, should still be
worth points. Still, non-compiling code cannot be tested, so this reduces the
degree of automation. Plain text tasks are the hardest to automate and there
is no general way to do this. Therefore, we examined each task individually
and found that there actually is a way to transform many of the text tasks in
our exam in a similar way. We often ask students to make a decision and then
justify it, so at least the decision part can be modeled as a single or
multiple choice task, and only the reasoning is left for manual evaluation.
Tasks in which we ask students to prove, e.g., a certain property of a
function are a special case though. Here we make use of the proof puzzle task
type available in YAPS, with a new evaluation algorithm tailored to our needs.

##  5 Realization

In the following we explain in detail how we realized the plan derived from
the course analysis. First, we focus on the extensions to YAPS: Haskell
compiler integration, RegEx tasks, proof puzzle evaluation, and the addition
of a comment field for all tasks. Then, we introduce the code analysis tool
for Haskell programming tasks and the RegEx generator program, together with
its higher-level RegEx specification language.

###  5.1 Haskell Compiler Integration

To facilitate creating new programming tasks, we devised a template that
provides a common layout and structure, so only task-specific text and code
needs to be added using the following workflow:  1. Write the task description
into exercise.html. This is the page that students see when selecting the
task.  2. Write a short main function that executes the student code into
main.hs. Students cannot modify this file, hence students cannot execute
arbitrary code. They are limited to the functions evaluated in main, but we
also control the types of these functions. Most importantly, these functions
are never IO actions in our tasks, so we can leverage the type system of
Haskell to prevent students from doing anything dangerous (the code is
additionally isolated using containerization).  3. Write any task-related code
or comments into functions.hs. This is the file in which students implement
their solutions. It is a good idea to summarize the task in a comment here, so
students do not have to switch back and forth between the task description and
their solution.  4. Write tests for the student code into main_test.hs.
Students cannot see this file.  5. Write the code to execute the tests, parse
and interpret their output, and report the results back to YAPS into
evaluate.py. Students cannot see this file.

This structure works for almost all tasks without modification, but is still
flexible enough to allow for deviations. One example of that are tasks where
students need to write tests themselves. Testing tests is more difficult than
testing “normal” functions because we cannot write randomized property tests
for them. Instead, we omit main_test.hs and provide two (or more)
implementations of the function under test: one that is correct, and one (or
more) that is faulty. The code from main.hs is executed with the correct
version during the exam. During evaluation, we link the student test with each
of the versions of the function under test. Then we check whether the test
passes for the correct version and fails for the faulty ones.

###  5.2 Checking Answers with Regular Expressions

Figure 1(a) shows for two instances of the RegEx task type how they appear in
the exam. During the evaluation, the given text input is matched against the
specified RegEx and the results are displayed to the examiner as in Fig. 1(b).
The first answer is found to be correct. The second answer is only partially
correct, so partial points are awarded through the numeric input field to the
right. This input field is hidden when the RegEx matched to prevent any
accidental modifications of the awarded points.

(a) Exam view

(b) Evaluation view

Figure 1: RegEx task

Still, there is one problem with that implementation: what if the correct
answer is no answer? Envision a task “Given a certain list comprehension, does
it compile, and if so, what is the resulting list?”. We usually design such
tasks as a combination of a single choice (for the decision) and a RegEx task
(for the list). If the list comprehension is not valid, the correct answer
would be to leave the list input field empty, so we cannot distinguish between
“no list” and “no answer.” As a remedy, we allow RegEx tasks to optionally
depend on a single choice task. The answer to the RegEx part is only evaluated
if an answer to the single choice task was given. Additionally, the input
field is hidden as soon as the student selects the negative option of the
single choice part. This resolves the ambiguity, because now we clearly know
whether a task is answered.

###  5.3 New Algorithm to Evaluate Proof Puzzles

An example of the proof puzzle task type is given in Fig. 2. Students can drag
the elements from right to left (and vice versa) to construct their proof.
Such a task is defined as follows: First, we list all the available items, and
optionally assign a weight to them. Then, we specify possible solutions as
sequences of selected items, and assign each solution a number of points. If a
student produces one of these solutions exactly, they get the full amount of
points.

Figure 2: Proof puzzle task

For clarity, we introduce the following notation: Let $\Sigma^{*}$ be the set
of all (Unicode character) strings. An _item_
$(t,w)\in\Sigma^{*}\times\mathbb{Q}$ consists of the displayed text $t$ and
the associated weight $w$. Let $\mathbb{S}_{I}$ be the set of all sequences of
items in $I\subseteq\Sigma^{*}\times\mathbb{Q}$. A _solution_
$(s,p)\in\mathbb{S}_{I}\times\mathbb{Q}$ of length $n$ consists of a sequence
of items $s=\iota_{1},\dots,\iota_{n}$, and the associated number of points
$p$. We denote the $i^{th}$ item of $s$ as $\iota_{i}=(t_{i},w_{i})$.

In the original version of this task type, the automatic evaluation is based
on the edit distance between the given solution attempt and the specified
solution. The edit distance is defined as the weighted number of _insert_ and
_remove_ operations required to transform one sequence of items into another.
For an item $(\\_,w)$, the cost of inserting/removing it is equal to $w$.
Algorithm 2 (Appendix) shows the complete procedure for calculating the number
of points to award for a solution attempt. It can be summarized as follows:
For each possible solution, calculate the edit distance to the given solution
attempt. Then, subtract the edit distance from the maximum amount of points
that can be awarded for that solution. The resulting number of points is the
maximum of these differences.

During testing, this algorithm produced reasonable results. Yet, after
conducting the exam, we found that sometimes the resulting points were not in
line with how we manually evaluated these tasks in the past, and that
sometimes the results were even unfair, e.g., in the following case (cf.
Figure 4, Appendix): Student A has correctly identified the inductive step of
the proof and the next step, and gets 0.5 points. Student B has only correctly
identified the base case and gets 2.0 points. We could not find a different
assignment of weights to the items that produces better results, so we decided
to devise a new algorithm tailored to our needs.

The new algorithm is based on finding correct sequences of items and awards
points according to the item weights. There is no subtraction of points, but
sequences have clearly-defined entry points (otherwise just using all items in
a random order could yield the full amount of points). As the entry points, we
use the first items of the predefined solutions. The new algorithm is given in
Algorithm 1 and replaces lines 3 and 4 in Algorithm 2 (Appendix). It proceeds
as follows: First, initialize the result with 0 (line 11), and look for the
first entry point of a sequence in the given solution. The subroutine to
search for the next sequence start (line 1) takes the current indices in the
lists of solution and given answer items as input (both initially 0). It
iterates over the solution sequence, and checks whether the current item is an
entry point, and if it is also part of the given answer, a new synchronization
point between solution and given answer is found. The subroutine returns the
indices of the common item in the solution and in the given answer. If such a
synchronization point cannot be found, the subroutine returns infinity for
both indices, which terminates the main loop (line 13). The main loop iterates
over the solution items as well, checking for each item whether it is still in
the sequence or breaks it. If it is part of the sequence, its weight is added
to the result, and the indices in the lists of solution and answer items are
incremented. Otherwise, search for the next entry point and skip all items in
between.

Algorithm 1 resolves the unfair grading explained above. Student A now gets
1.5 points, while student B gets 1.0 (cf. Figure 5, Appendix). In case of
proof by induction tasks, we usually specify the first line of the base case
and the first line of the inductive step as entry points for sequences.

Algorithm 1 New proof puzzle evaluation

1:$I\subset\Sigma^{*}\times\mathbb{Q}$ (items),
$s\in\mathbb{S}_{I}\times\mathbb{Q}$ (solution), $a\in\mathbb{S}_{I}$
(solution attempt)

2:Amount of points to award for $a$

3:function FindNextSequenceStart($solIdx,ansIdx$)

4: for $solIdx<|s|$ do

5: if $\textsc{IsSequenceStart}(s_{solIdx})\ \text{{and}}\ s_{solIdx}\in a$
then

6: return $(solIdx,\ \text{index of}\ s_{solIdx}\ \text{in}\ a)$

7: end if

8: $solIdx\leftarrow solIdx+1$

9: end for

10: return $(\infty,\infty)$

11:end function

12:

13:$r\leftarrow 0$

14:$(solIdx,ansIdx)\leftarrow\textsc{FindNextSequenceStart}(0,0)$

15:for $solIdx<|s|\ \text{{and}}\ ansIdx<|s_{a}|$ do

16: if $s_{solIdx}=a_{ansIdx}$ then

17: $r\leftarrow r+w_{solIdx}$

18: $(solIdx,ansIdx)\leftarrow(solIdx+1,ansIdx+1)$

19: else

20: $(solIdx,ansIdx)\leftarrow\textsc{FindNextSequenceStart}(solIdx,ansIdx)$

21: end if

22:end for

23:return $r$

###  5.4 Comment Field for Students

Another extension we added to the YAPS framework is a comment field for all
exercises. We frequently found in paper exams that students add additional
thoughts about exercises beyond their answers. Often these are explanations of
how the exercise was understood or assumptions on which the answer of the
student was based. While we aim to phrase all exercises in a way that neither
an interpretation nor additional assumptions are necessary, we still wanted to
give students the option to express these so that their answers can be
evaluated in the right context. Therefore, we added an all-purpose text field
at the bottom of each exercise with an explanatory text describing its
purpose. Whatever is written into this field persists in addition to all
actual answers so that we can read it during the correction of the exam.
Another use case we found for this text field is that students wrote notes for
themselves during the exam that they would ordinarily scribble in the margins
of a paper exam.

###  5.5 Analyzing Student Code for Task-Relevant Features

Many of our programming tasks have certain restrictions, e.g., students must
(not) use a certain language feature, or are only allowed to use certain
functions. To be able to check these constraints automatically we devised a
new tool101010https://collaborating.tuhh.de/cda7728/check-hs-task-restrictions
that itself is written in Haskell. It takes a source code file as input, and
outputs certain information on each function, that is defined in the input
file, in JavaScript Object Notation (JSON). For example, with the well-known
implementation of quicksort, given in LABEL:lst:quicksort, the program outputs
the JSON shown in LABEL:lst:quicksort-out. For each function, the program
provides the following information: the name of the function, its arguments,
called and locally declared functions, and whether it uses pattern
matching/guarded equations/list comprehensions/case expressions. To extract
that information, we use the haskell-
src111111https://hackage.haskell.org/package/haskell-src package to build and
traverse the AST of the given source file. The program itself is split into a
library that provides the described functionality, and an application that
uses this library. Altogether, the entire program consists of less than 500
lines of code of which most are used for pattern matching on the constructors
of the sum types that represent the AST and advancing the search in depth-
first manner.

Listing 1: Example of quicksort in
Haskell131313https://wiki.haskell.org/Introduction#Quicksort_in_Haskell

⬇

quicksort :: Ord a => [a] -> [a]

quicksort [] = []

quicksort (p:xs) = (quicksort lesser) ++ [p] ++ (quicksort greater)

where lesser = filter (< p) xs

greater = filter (>= p) xs

Listing 2: Code analyzer output for LABEL:lst:quicksort

⬇

{ ”functions”: [{

”name”: ”quicksort”,

”patMatch”: true,

”guards”: false,

”listComprehension”: false,

”hasIf”: false,

”hasCase”: false,

”args”: [ ”p”, ”xs” ],

”calledFns”: [ ”quicksort”, ”++”, ”filter”, ”<”, ”>=” ],

”declaredFns”: [ ]

}]}

The haskell-src library parses the given file to the HsModule type which
features a list of declarations. We filter those declarations for functions
(HsFunBind) and constants (HsPatBind), and then check for the features and
functions used. The HsFunBind constructor holds the clauses of the function
([HsMatch]), and the clauses provide access to guards (HsGuardedRhss), as well
as access to the patterns used in the left hand side. Any pattern that is not
a name (HsPVar) is considered a use of pattern matching. Names, however, are
added to the list of arguments (subpatterns are also considered, e.g., (x:xs)
is considered a use of pattern matching and adds the arguments x and xs). To
collect the remaining information, the right hand sides (HsRhs) and local
declarations (HsDecl) are traversed similarly.

###  5.6 Generating Flexible Regular Expressions

To increase automation during exam creation we developed a second tool141414https://collaborating.tuhh.de/cda7728/gen-hs-task-regexs in Haskell that facilitates writing the RegExs we need to automatically check many of the tasks. It reads a custom, specialized specification language that we call Haskell Task  Specification Language (HTRSL), and outputs JavaScript compatible RegExs. For an example, consider the input ("Num" \\\ a) "=>" "[" a "]" "->" ["String" | "[" "Char" "]" ]. It represents the function type (Num a) => [a] -> String that can be written in different ways, e.g., omitting the parentheses, writing [Char] instead of String, or using a different name for the type variable a. All of these variations need to be accepted by the generated RegEx. The result is shown in LABEL:lst:regex-out and illustrates how large these expressions can become. RegEx of that size are difficult to understand for humans, so writing them by hand is rather error-prone.

Listing 3: Example RegEx generator output

⬇

^\s*(?:\\((?=(?:[^\\(\\)]*)\\)))?\s*Num\s*\s(?<a>[_a-z][_a-
zA-Z0-9’]*)\s*(?:(?<=\\((?:[^\\(\\)]*))\\))?\s*=>\s*\\[\s*\k<a>\s*\\]\s*->\s*(?:String\s*|\\[\s*Char\s*\\]\s*)$

Therefore, the HTRSL provides an additional layer of abstraction tailored to
common patterns in Haskell expressions. Its grammar is given in
LABEL:lst:htrsl-grammar (Appendix) in labelled Backus-Naur form: A HTRSL file
is a semicolon-separated list of specifications. Each specification is a
description. A description is a list of description items. Allowed items are:

  * •

Literals: Match the string given in double-quotes literally.

  * •

Identifiers: Match any identifier ([_a-z][_a-zA-Z0-9’]*). If reused in the
same description, only matches the same identifier that was found at the first
occurrence.

  * •

Mandatory whitespace: Match any whitespace (\s+), represented as \\\ in the
description.

  * •

Alternatives: Match any of the alternatives that are given in square brackets
and separated by |.

  * •

Optional parentheses: Match what is specified inside the parentheses, either
surrounded by parentheses or not. Cannot be nested.

We use the BNF Converter151515https://hackage.haskell.org/package/BNFC to
generate a parser for the language. This way, we obtain an abstract syntax
tree that we can traverse to generate the desired RegExs.

In the future, we plan to add support for automatically testing the generated
RegExs. Currently, this is done manually by testing each RegEx for some
positive and negative examples. This can be automated as well by adding the
test strings to the specification, so the tool can directly check whether the
generated RegEx matches all positive examples, and rejects all negative ones.

##  6 Evaluation

For evaluating the e-exam we aim to answer the following three questions: What
is the degree of automation? Are students satisfied? Are examiners (we)
satisfied? The first question is targeted at evaluating how far we have come
with respect to our initial motivation: reducing the effort of conducting an
exam through automation. We believe that, by design, we did not sacrifice CA,
yet CA compliance comes at the cost of less automation, and we wanted to know
how high that cost actually is. While reducing the effort of conducting the
exam mostly benefits us, the examiners, we also wanted to provide the students
with a more comfortable and familiar way of taking an exam, especially when it
comes to programming tasks. Finally, we reflect on our own experiences, from
creating the exam to grading it, to answer the last question.

###  6.1 Automated Grading

Table 3: Categorization of tasks by degree of automation Task | Task Type | Category
---|---|---
1a | RegEx | Automated if correct
1b | Multiple Choice | Fully automated
2a | Single choice + RegEx | Automated if correct
2b | Programming | Automated if correct
3a | Single Choice + RegEx | Automated if correct
3b | Programming | Automated if correct
4a | Single Choice + Text | Partially automated
4b | Programming | Automated if correct
5a | Programming | Automated if correct
5b | Text | Not automated
5c | Programming | Automated if correct
6a | Programming | Automated if correct
6b | Multiple Choice | Automated
6c | Programming | Automated if correct
7a | Single Choice + Text | Partially automated
7b | RegEx | Automated if correct
7c | Multiple Choice + Text | Partially automated
8a | Programming | Automated if correct
8b | Proof Puzzle | Fully automated

To evaluate the degree of automation of the grading process, we divide all
tasks in the e-exam into the following categories:  1. Fully automated: human
intervention is not required in any case.  2. Automated if correct: human
intervention is only required if the given answer was incorrect (to
potentially award partial points)  3. Partially automated: human intervention
may even be necessary for correct answers.  4. Not automated: human
intervention is required in any case.  The results are given in Table 3 for an
example exam (we do not rule out that other exams have, e.g., a task 1c). They
show a direct mapping between task type and category: multiple/single choice
or proof puzzle → fully automated, RegEx or Programming → automated if
correct, text → not automated. When two task types are combined, the resulting
degree of automation is the one that is worse. Combinations of text tasks with
one that is at least “automated if correct” result in a “partially automated
task”. We see that most tasks are “automated if correct”, which is our second-
best category. Yet, just from the categorization we cannot determine how much
this reduces the effort required for grading the exam in practice. Still, all
tasks except for one feature at least some degree of automation, and overall
that makes it likely to reduce the required efforts. In other words, the
results can be summarized as follows: Correct solutions can in most cases be
awarded the full amount of points automatically, but awarding partial points
is difficult.

###  6.2 Student View

To capture the view of the students, we conducted a short poll immediately
after the exam. When time for the exam was up, the exam browser automatically
redirected the students to the poll website. On the one hand this allowed us
to capture the opinion of the students without delay and reduced external
influence. On the other hand, we thought that students may be unwilling to
fill in an elaborate questionnaire just after finishing an exam. Therefore, we
decided to only ask three questions and provide a text input field for
additional feedback.

$1$$1.5$$2$$2.5$$3$$3.5$$4$$4.5$$5$$10$$15$$20$$25$$30$$35$ (a) Statement 1

$1$$1.5$$2$$2.5$$3$$3.5$$4$$4.5$$5$$8$$10$$12$$14$$16$$18$$20$$22$$24$ (b)
Statement 2

$1$$1.5$$2$$2.5$$3$$3.5$$4$$4.5$$5$$8$$10$$12$$14$$16$$18$$20$$22$ (c)
Statement 3

Figure 3: Poll results

Rate the following statements from 1 to 5 (1 = fully disagree, 5 = fully
agree)  1. An e-exam is a right format for the lecture “Functional
Programming”.  2. The programming tasks with compiler support feel like a
natural way to answer programming tasks.  3. In an e-exam, I can express my
thoughts as good as I could in a paper exam.

Of the 77 exam participants, 73 answered the poll. The results are shown in
Fig. 3. Figure 3(a) shows that the majority of the participants
($36+10=46\equiv 63\%$) generally agrees that an e-exam is the right
examination format for the course. Most ($24+14=38\equiv 52\%$) also found
working with the integrated compiler natural, yet comparatively many students
are neutral on this question ($15\equiv 21\%$), as shown in Fig. 3(b). For the
third question, Fig. 3(c) shows that less than half of the participants
($21+13=34\equiv 45\%$) agreed that they can express their thoughts as they
could in a paper exam. Almost a third of the students ($8+15=23\equiv 32\%$)
disagrees. 29 students even provided additional feedback that ranges from
constructive criticism to outright ecstasy (“It was SIIIICKKK!!! Also perfect
amount of time for the tasks. I finished in the last 5 seconds”). Other
critical remarks were:

  * •

“It’s sometimes a little bit complicated for color-blind people to see whether
I marked a task already as green or it’s still orange.” In YAPS, students can
mark tasks green when they finished them, unseen tasks are gray, and tasks
where some input was given appear orange. This choice of colors apparently is
problematic for some and should be changed.

  * •

“Maybe it would be better to make more friendly function names” Sometimes the
function names are a very short abbreviation that at first sight may appear as
a random sequence of letters. Naming functions this way is a habit that stems
from creating paper exams where horizontal space for code is much more limited
than in the YAPS code editor, so we should consider using the additional space
and use more descriptive function names.

  * •

“In my opinion it’s nicer to write the proof (last task) as text instead of
drag and drop, as the drag and drop fields all look very similar, and it’s
more like a task to find the correct field, than to actually think about the
proof” This is a hint that the proof puzzle may not be the optimal way to
assess the ability of the students to construct a proof, and we should
investigate other ways.

  * •

“Please get new computer mice. That is really necessary” (translated from
German). When designing the e-exam we did not think about hardware, apart from
how the screen size impacts the visual presentation of the exam, as it was
provided by our university anyway. This statement shows that hardware does
play a role and should be taken into account.

  * •

“C and G are difficult to differentiate on the small laptop screen”
(translated from German). This goes into a similar direction as the previous
statement. While we checked how all tasks look on the laptop screens that are
used during the exam, we did not see this problem. Possibly using a different
font can help here.

Overall we think the results from the poll are encouraging, but there are some
details that need improvement. Especially the free text student feedback shows
how we can improve the exam, and that it is important to incorporate student
feedback when designing and improving an e-exam.

###  6.3 Examiner View

In this section we report our own experiences with the complete e-exam
workflow from creation to grading, and highlight successes as well as
difficulties.

When creating a new e-exam, we begin with writing the tasks and solutions
outside YAPS. Where possible, we also write tests for the solutions to ensure
they are indeed correct. During that phase we employ literate Haskell with
Markdown through markdown-unlit161616https://github.com/sol/markdown-unlit to
combine text, code, and tests in the same file. This workflow has two main
advantages: we can render the exam in PDF format and split between a PDF that
contains all tasks, and one that contains the solutions, and we can directly
execute the exam code and tests. For a paper exam we would instead write the
exam in LaTeX(with tests in a separate file), which can be more time-intensive
because we need to pay more attention to the resulting layout.

When the exam tasks themselves are finalized, we implement them in YAPS. The
task texts can mostly be simply pasted into Hypertext Markup Language (HTML)
templates, answers for single or multiple choice tasks are easily configured
in JSON files, and RegExs are generated automatically by our tool. This part
of implementing the exam is a rather tedious process, but the templates and
tool support help to minimize redundancies. Yet, we manually perform the same
steps multiple times, which likely could be automated. More creative work is
required for proof and programming tasks. Here, we need to find sensible
distractors (proof tasks) and write suitable property tests (programming
tasks). This is the most time-consuming part of the entire exam creation
process. Finally, we perform a few ($<5$) rounds of testing the finished exam
to make absolutely sure everything works as intended. For testing, we mostly
enter correct answers and check whether they are detected as such. Usually
there are few mistakes, e.g., typos, but sometimes also problems with the
tests for programming tasks occur.

Table 4: Comparison of paper exam (fall 2020) and e-exam (fall 2021) results Task | Paper Exam | E-Exam
---|---|---
| max | avg | max | avg
1a | 7 | 4 | 6 | 4
1b | 5 | 3 | 6 | 3
2a | 6 | 4 | 6 | 4
2b | 6 | 3 | 6 | 3
3a | 10 | 6 | 10 | 6
3b | 3 | 1 | 3 | 2
4a | 6 | 4 | 6 | 4
4b | 4 | 3 | 4 | 2
5a | 3 | 1 | 3 | 2
5b | 5 | 1 | 5 | 1
5c | 5 | 1 | 5 | 1
6a | 6 | 3 | 6 | 3
6b | 6 | 2 | 6 | 4
7a | 6 | 4 | 6 | 4
7b | 6 | 3 | 6 | 4
7c | 5 | 1 | 5 | 1
8a | 5 | 2 | 6 | 1
8b | 6 | 2 | 5 | 2

On the day of the exam, the required technical infrastructure is already set
up by a team from our university [29]. On the other hand, conducting an e-exam
may require more organizational overhead because we only have 100 laptops
available, so for large exams we need to do multiple rounds. This introduces a
new set of difficulties because we need to make sure no information about the
exam is given from, e.g., the first round to the second. One way to do this is
to provide different exams, which increases the required effort substantially.
Here, the randomization features of YAPS are very useful to create task
variations with little effort. When time for the exam is up, YAPS prevents any
further input from students, and collects all given answers (or rather
collects them continuously during the exam to make sure no data is lost). We
can then trigger the automated evaluation, which may take some minutes, so we
come back later.

Grading is then done in YAPS as well. We almost exclusively look at those
tasks where mistakes were found by the automated grading to manually
distribute points. Additionally, we sample a few answers detected as correct
to check whether the automated grading worked as intended, which usually is
the case. If there is a mistake, however, we can correct it and execute the
automated evaluation again. Grading an exam with 77 participants took a single
person approximately two work days. The last paper exam with 136 participants
took two people approximately three work days. Unfortunately, “grading effort”
is hard to measure, and the numbers are difficult to compare because of the
very different numbers of participants. Still, it is a good sign that the exam
could be graded by a single person in a comparatively short amount of time.
Assuming that with twice as many participants (154) the required amount of
time would double as well, a single person could grade as many e-exams as two
people who grade the same exam in paper, in the same timeframe.

Table 4 shows the results per task for our last paper exam, and the e-exam we
report on in this paper. For each task, the maximum achievable number of
points, and the average amount of points scored by students are shown. Rows
with better/worse results (in comparison to the paper exam) are colored
blue/red. Overall, the students performed very similarly. The few, minor
differences we see cannot be clearly attributed to a specific phenomenon. For
instance, tasks 2b, 3b, 4b, 5a, 5c, 6b, and 8a are programming tasks, but the
students sometimes performed better, worse, or the same. When deciding for the
Proof Puzzle, we thought that the proof task (8b) may become noticeably easier
because in the paper exam the proof had to be created entirely from scratch.
However, the results suggest that in this exam creating the proof was similar
in difficulty

In summary, creating an e-exam takes more time than creating a paper exam, but
with maturing templates and tools we can confidently expect that this overhead
becomes smaller. Conducting and grading, however, takes less time now. The
exam results were almost exactly the same compared to a previous paper exam.
Altogether, we conclude that the approach to create an e-exam from a paper
exam by direct “translation” of each task was successful.

##  7 Summary & Future Work

We showed how a traditional paper exam with complex proof and programming
tasks can be transformed to an e-exam with automated grading, but without
sacrificing CA. Through careful analysis of the course and previous exams we
can ensure that exam quality does not degrade during the transformation. For
realizing the e-exam we built upon existing software that we extended with a
Haskell compiler, tasks that can be checked with RegExs, a new algorithm to
automatically evaluate proofs, and a general purpose comment field for
students. Additionally, we introduced two new tools that substantially support
automation: one that analyzes student code for task-relevant features, and one
that generates suitable RegExs from a more high-level description language. We
achieved that almost all task can be graded automatically at least in part,
and students as well as examiners are largely satisfied with the resulting
e-exam.

In the future we want to investigate how a more fine-grained automated grading
can be achieved because currently awarding reduced amounts of points for
partial solutions is often not possible (at least not as reliably as we
require it for an exam). Moreover, because creating an e-exam requires more
effort than creating a paper exam, we also want to explore how this process
can be automated further. Our vision is that most of the e-exam can be
generated automatically from the initial literate Haskell Markdown file that
contains the task texts, solutions, and point distribution.

## References

  * [1]
  * [2] F. H. Bahnsen & G. Fey (2021): _YAPS - Your Open Examination System for Activating and emPowering Students_.  In: 2021 16th Int. Conf. Comput. Sci. Educ. (ICCSE), pp. 98–103, 10.1109/ICCSE51940.2021.9569549.
  * [3] A. Bieniusa, M. Degen, P. Heidegger, P. Thiemann, S. Wehr, M. Gasbichler, M. Sperber, M. Crestani, H. Klaeren & E. Knauel (2008): _HtDP and DMdA in the Battlefield: A Case Study in First-Year Programming Instruction_.  In: FDPE ’08: Proc. 2008 Int. Workshop Funct. Declar. Program. Educ., pp. 1–12, 10.1145/1411260.1411262.
  * [4] J. Biggs (1996): _Enhancing Teaching through Constructive Alignment_.  High. Educ. 32(3), pp. 347–364, 10.1007/BF00138871.
  * [5] K. Claessen & J. Hughes (2000): _QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs_.  In: ICFP ’00: Proc. Fifth ACM SIGPLAN Int. Conf. Funct. Program., pp. 268–279, 10.1145/351240.351266.
  * [6] G. Derval, A. Gego, P. Reinbold, B. Frantzen & P. Van Roy (2015): _Automatic Grading of Programming Exercises in a MOOC Using the INGInious Platform_.  In: Proc. Eur. MOOC Stakehold. Summit 2015, pp. 86–91.
  * [7] M. Erwig & R. Gopinath (2012): _Explanations for Regular Expressions_.  In: Fundam. Approaches Softw. Eng., Lect. Notes Comput. Sci. 7212, pp. 394–408, 10.1007/978-3-642-28872-2_27.
  * [8] Hamburg University of Technology (2022): _Module Manual Bachelor of Science (B. Sc.) Computer Science – Cohort: Winter Term 2021_.  Available at https://studienplaene.tuhh.de/po/E/mhb_CSBS_kh_w21_von_20220524_v_0_en.pdf.
  * [9] J. Hollingsworth (1960): _Automatic Graders for Programming Classes_.  Comm. ACM 3(10), pp. 528–529, 10.1145/367415.367422.
  * [10] G. Hutton (2016): _Programming in Haskell_ , second edition.  Cambridge University Press, 10.1017/CBO9781316784099.
  * [11] F. K. Jacobsen & J. Villadsen (2023): _On Exams with the Isabelle Proof Assistant_.  In: Proc. 11th Int. Workshop Theorem Proving Compon. Educ. Softw., Electron. Proc. Theor. Comput. Sci. 375, pp. 63–76, 10.4204/EPTCS.375.6.
  * [12] J. C. S. Kadupitiya, S. Ranathunga & G. Dias (2016): _Automated Assessment of Multi-Step Answers for Mathematical Word Problems_.  In: 2016 Sixt. Int. Conf. Adv. ICT Emerg. Reg. (ICTer), pp. 66–71, 10.1109/ICTER.2016.7829900.
  * [13] K. Kappelmann, J. Rädle & L. Stevens (2022): _Engaging, Large-Scale Functional Programming Education in Physical and Virtual Space_.  In: Proc. Tenth Elev. Int. Workshop Trends Funct. Program. Educ., Electron. Proc. Theor. Comput. Sci. 363, pp. 93–113, 10.4204/EPTCS.363.6.
  * [14] D. R. Krathwohl (2002): _A Revision of Bloom’s Taxonomy: An Overview_.  Theory Pract. 41(4), pp. 212–218, 10.1207/s15430421tip4104_2.
  * [15] S. Krusche & A. Seitz (2018): _ArTEMiS: An Automatic Assessment Management System for Interactive Learning_.  In: SIGCSE’18: Proc. 49th ACM Tech. Symp. Comput. Sci. Educ., pp. 284–289, 10.1145/3159450.3159602.
  * [16] M. Kuikka, M. Kitola & M.-J. Laakso (2014): _Challenges When Introducing Electronic Exam_.  Res. Learn. Technol. 22, 10.3402/rlt.v22.22817.
  * [17] H. Loftsson & Á. Matthíasdóttir (2021): _Moving Classes in a Large Programming Course Online: An Experience Report_.  In: Second Int. Comput. Program. Educ. Conf. (ICPEC 2021), Open Access Ser. Inform. (OASIcs) 91, pp. 2:1–2:13, 10.4230/OASIcs.ICPEC.2021.2.
  * [18] M. McCartin-Lim, B. Woolf & A. McGregor (2018): _Connect the Dots to Prove It: A Novel Way to Learn Proof Construction_.  In: SIGCSE’18: Proc. 49th ACM Tech. Symp. Comput. Sci. Educ., pp. 533–538, 10.1145/3159450.3159609.
  * [19] D. Milojicic (2011): _Autograding in the Cloud: Interview with David O’Hallaron_.  IEEE Internet Comput. 15(1), pp. 9–12, 10.1109/MIC.2011.2.
  * [20] D. S. Morris (2003): _Automatic Grading of Student’s Programming Assignments: An Interactive Process and Suite of Programs_.  In: 33rd Annu. Front. Educ., 3, pp. S3F–1 – S3F–6, 10.1109/FIE.2003.1265998.
  * [21] T. Nipkow, M. Wenzel & L. C. Paulson (2002): _Isabelle/HOL_.  Lect. Notes Comput.Sci. 2283, 10.1007/3-540-45949-9.
  * [22] J. C. Paiva, J. P. Leal & Á. Figueira (2022): _Automated Assessment in Computer Science Education: A State-of-the-Art Review_.  ACM Trans. Comput. Educ. 22(3), pp. 34:1–34:40, 10.1145/3513140.
  * [23] V. Pieterse (2013): _Automated Assessment of Programming Assignments_.  In: CSERC’13: Proc. 3rd Comput. Sci. Educ. Res. Conf., pp. 45–56.
  * [24] S. Poulsen, S. Kulkarni, G. Herman & M. West (2022): _Efficient Partial Credit Grading of Proof Blocks Problems_ , 10.48550/arXiv.2204.04196.
  * [25] S. Poulsen, M. Viswanathan, G. L. Herman & M. West (2022): _Proof Blocks: Autogradable Scaffolding Activities for Learning to Write Proofs_.  In: ITiCSE ’22: Proc. 27th ACM Conf. Innov. Technol. Comput. Sci. Educ. Vol. 1, pp. 428–434, 10.1145/3502718.3524774.
  * [26] T. Rajala, E. Kaila, R. Lindén, E. Kurvinen, E. Lokkila, M.-J. Laakso & T. Salakoski (2016): _Automatically Assessed Electronic Exams in Programming Courses_.  In: ACSW’16: Proc. Australas. Comput. Sci. Week Multiconf., pp. 1–8, 10.1145/2843043.2843062.
  * [27] J. Schneider, R. Richner & M. Riser (2022): _Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers_.  Int. J. Artif. Intell. Educ. 33(1), pp. 88–118, 10.1007/s40593-022-00289-z.
  * [28] J. Sheard, Simon, A. Carbone, D. Chinn, M.-J. Laakso, T. Clear, M. de Raadt, D. D’Souza, J. Harland, R. Lister, A. Philpott & G. Warburton (2011): _Exploring Programming Assessment Instruments: A Classification Scheme for Examination Questions_.  In: ICER’11: Proc. Seventh Int. Workshop Comput. Educ. Res., pp. 33–38, 10.1145/2016911.2016920.
  * [29] D. Sitzmann, K. Kruse, D. Gallaun, N. Kubick, B. Reinhold, M. Schnabel, L. Thoms, H. Barbas & D. Meiling (2022): _Aufbau eines mobilen Testcenters für die Hamburger Hochschulen im Rahmen des Projekts MINTFIT E-Assessment_.  Hochschullehre 8, pp. 113–129, 10.3278/HSL2208W.
  * [30] C. Stergiopoulos, P. Tsiakas, D. Triantis & M. Kaitsa (2006): _Evaluating Electronic Examination Methods Applied to Students of Electronics. Effectiveness and Comparison to the Paper-and-Pencil Method._ In: IEEE Int. Conf. Sens. Netw. Ubiquitous Trust. Comput. (SUTC’06), 2, pp. 143–151, 10.1109/SUTC.2006.65.
  * [31] L. C. Ureel II & C. Wallace (2019): _Automated Critique of Early Programming Antipatterns_.  In: SIGCSE ’19: Proc. 50th ACM Tech. Symp. Comput. Sci. Educ., pp. 738–744, 10.1145/3287324.3287463.
  * [32] L.-X. Zheng, S. Ma, Z.-X. Chen & X.-Y. Luo (2021): _Ensuring the Correctness of Regular Expressions: A Review_.  Int. J. Autom. Comput. 18(4), pp. 521–535, 10.1007/s11633-021-1301-4.

## Appendix

(a) Student A

(b) Student B

Figure 4: Example of unfair grading with old algorithm

Left column: student solution, middle column: our solution, right column:
points

(a) Student A

(b) Student B

Figure 5: No unfair grading with new algorithm

Left column: student solution, middle column: our solution, right column:
points

Algorithm 2 Default proof puzzle evaluation

1:$I\subset\Sigma^{*}\times\mathbb{Q}$ (items),
$S\subset\mathbb{S}_{I}\times\mathbb{Q}$ (solutions), $a\in\mathbb{S}_{I}$
(solution attempt)

2:Amount of points to award for $a$

3:$r\leftarrow 0$

4:for all $(s_{i},p_{i})\in S$ do

5: $d\leftarrow\textsc{EditDistance}(s_{i},a)$

6: $r_{i}\leftarrow\max\\{0,p_{i}-d\\}$

7: if $r_{i}>r$ then

8: $r\leftarrow r_{i}$

9: end if

10:end for

11:return $r$

Listing 4: HTRSL grammar

⬇

entrypoints [Spec] ;

separator Spec ”;” ;

SNoTests. Spec ::= Desc ;

D. Desc ::= [DescItem] ;

separator nonempty DescItem ”” ;

DLit. NoParensDescItem ::= String ;

DName. NoParensDescItem ::= Ident ;

DSpace. NoParensDescItem ::= ”\\\” ;

DNAlt. NoParensDescItem ::= ”[” [NoParensDescAlt] ”]” ;

DND. DescItem ::= NoParensDescItem ;

DPar. DescItem ::= ”(” [NoParensDescItem] ”)” ;

separator nonempty NoParensDescItem ”” ;

DNAltElem. NoParensDescAlt ::= [NoParensDescItem] ;

separator nonempty NoParensDescAlt ”|” ;

comment ”–”;

comment ”{-” ”-}”;

  *[CA]: constructive alignment
  *[e-exam]: electronic exam
  *[YAPS]: Your Open Examination System for Activating and emPowering Students
  *[TUHH]: Hamburg University of Technology
  *[FP]: functional programming
  *[RegExs]: regular expression
  *[LO]: learning objective
  *[RegEx]: regular expression
  *[e-exams]: electronic exam
  *[LOs]: learning objective
  *[DAG]: directed acyclic graph
  *[DAGs]: directed acyclic graph
  *[CYP]: Check Your Proof
  *[AST]: abstract syntax tree
  *[JSON]: JavaScript Object Notation
  *[]:
  *[HTRSL]: Haskell Task   () Specification Language
  *[HTML]: Hypertext Markup Language


# Improved methodologies to study the performance of the ANET Compact Neutron
Collimator

O. Sans-Planell  F. Cantini  M. Costa  E. Durisi  F. Grazzi  E. Mafucci  V.
Monti  R. Bedogni  Y. Li  L. van Eijck

###### Abstract

The ANET project aims at developing 2D compact neutron collimators for neutron
imaging applications. The results of the ANET collimator performances,
presented in this communication, are based on data collected at the FISH
beamline at TU-Delft. Two independent methods to evaluate the neutron
radiography resolution are described and discussed, as well as a comparison of
the beam divergence with or without the ANET collimator.

###### keywords:

Neutron radiography , Compact Neutron Collimator , Siemens Star , Gadolinium
Knife-edge

###### PACS:

0000 , 1111

###### MSC:

0000 , 1111

††journal: Nuclear Physics A

[inst1]organization=Universita degli Studi di Torino,addressline=Via Pietro
Giuria 1, city=Torino, postcode=10125, state=, country=Italy

[inst2]organization=INFN Sezione di Torino,addressline=Via Pietro Giuria 1,
city=Torino, postcode=10125, state=, country=Italy

[inst3]organization=INFN Frascati National Laboratories,addressline=Via Enrico
Fermi 54, city=Frascati, postcode=00044, state=, country=Italy

[inst4]organization=Consiglio Nazionale delle Ricerche, Istituto dei Sistemi
Complessi,addressline=Via Madonna del Piano 10, city=Sesto Fiorentino,
postcode=50019, state=, country=Italy

[inst5]organization=INFN Sezione di Firenze,addressline=Via Madonna del Piano
10, city=Sesto Fiorentino, postcode=50019, state=, country=Italy

[inst6]organization=Delft University of Technology,addressline=Mekelweg 15,
city=Delft, postcode=2629 JB, state=, country=Netherlands

[inst7]organization=INFN Sezione di Pavia,addressline=Via Agostino Bassi 6,
city=Pavia, postcode=27100, state=, country=Italy

Neutron Beam Divergence measurement

Numerical method to derive neutron image resolution with a Siemens Star

New data-driven approach to defining the MTF threshold on neutron radiography
measurements

##  1 Introduction

The ANET project aims at developing 2D Compact Neutron Collimators (CNC) for
Neutron Imaging applications. Its design and operational principles have been
described in [1], while the first results on its performance have been
reported in [2].
This paper illustrates two improved methodologies to determine the image
resolution from measured data. A PSI Siemens-Star and a Gd knife-edge assembly
are used as reference samples [3]. The applicability of the herein exposed
methods is general and not restricted to a specific beam-line or collimation
system. Data have been collected at the FISH beamline [4] at the HOR research
reactor, owned by Technische Universiteit Delft. In the following sections,
after having illustrated the experimental setup, a detailed description of the
analysis methods is given. The ANET CNC effectiveness, in terms of beam
divergence reduction and consequent resolution improvement, is also
demonstrated.

##  2 The experimental setup

The FISH beam-line provides, at the standard detector position (5.5 $m$ from
the shutter), a thermal neutron flux of 3$\cdot$10${}^{6}cm^{-2}s^{-1}$, with
a rectangular field of view of 50 x 100 $mm^{2}$, resulting in an asymmetric
divergence for the horizontal and vertical axis: the quoted divergence angles
are 0.176 and 0.207 degrees respectively.
This is a very interesting working case for the application of the ANET CNC
due to its intrinsic 2D symmetric geometry. For this measuring campaign, a
neutron imaging system with a 100$\mu m$ thick LiF/ZnS(Ag) scintillator and a
16-bit ANDOR camera had been installed at 232$cm$ from the beam shutter, as
shown in figure 1. The distance of the ANET CNC from the shutter has been
constrained by hardware limitations on the beam-line. To measure the
horizontal and vertical divergence of the neutron beam, with and without the
ANET CNC, the procedure explained in [5] has been applied. It requires the
acquisition of neutron radiographies of a reference sample at several
distances from the detector, in an appropriate range, as specified in the
following.

Figure 1: Scheme of the set-up used for the measurements (not in scale). The
ANET collimator is mounted on top of a Stewart Platform and raised at the beam
axis height by a calibrated spacer. The samples are installed on top of X-Y
remotely controlled moving stages

The ANET CNC has been operated following the procedure detailed in [2], in
which a dynamic pattern is used. The exposure time for each radiography has
been tuned to 30 seconds without the collimator and 300 seconds with the ANET
CNC. The acquisition time has been set to reach approximately one-half of the
full dynamic range of the ANDOR camera. For each reference sample, eight
different radiographs were taken, from 10$mm$ to 105$mm$ away from the
scintillator. To evaluate the impact of the ANET CNC in terms of resolution
and beam divergence, two independent analysis methods have been applied. They
are described in the following sections.

##  3 The ”unfolded” Siemens Star method

In figure 2 two radiographs are shown, which show ”by-eye” the improvement in
resolution produced when the ANET CNC is placed in front of the sample.

Figure 2: Comparison between two Siemens star measurements at FISH, without
(left) and with (right) the ANET CNC. The distance of the sample from the
detector is at 20$mm$.

By choosing the appropriate ImageJ Look-Up-Table (LUT) it is possible to
highlight the blurry areas and separate them from the well-defined regions of
the Siemens star. The right image in which ANET is included in the set-up
shows an evident improvement in spatial resolution with respect to the image
on the left, with FISH alone.
The precision to appreciate the resolution on the Siemens star by optical
means is limited to $\pm 25\mu m$. Further, it’s not always easy to
distinguish at which point the threshold is, especially when considering the
difference between the vertical and horizontal divergences in a beam-line with
a rectangular shutter or pin-hole such as that present at FISH. In order to
improve the evaluation of spatial resolution, a new numerical approach has
been taken.
By means of the software FIJI [6], the single image has been radially re-
sliced from the centre to the most external circle, starting at 45∘ angle. The
procedure is represented in figure 3.

Figure 3: Representation of the process of ”unfolding” the Siemens star using
FIJI. The reslicing is done clockwise from the 45∘ line.

The image contains, on the right, a representation of the Siemens star
”unfolded”, where the top pixel row is equivalent to the 45∘ radius used to
start the re-slicement. Dividing the image in four 90∘ slices allows to
separate the two components of the divergence. Figure 4 shows the 4 images
representing the resolution in the 4 different sectors of the Siemens star.

Figure 4: Visualisation in four angular slices of the ”unfolded” Siemens star.
The greyscale image is displayed using the ”jet” LUT in order to highlight the
difference between the noise region and the actual spikes of the Siemens star.

To improve the best-possible optical precision of $\pm$25$\mu m$, the
following method is adopted: for each single image, the standard deviation of
every vertical array of points is taken (from left to right). The standard
deviation is a measure of the amount of dispersion of a set of values, and
thus the expectation will be constant and small on the first points, in the
inner area of the Siemens star (inner rings, leftmost area in each figure),
where only noise is present. The standard deviation value will increase with
the higher contrast (outer rings, right-most area in each figure). This is
shown in figure 5, where the Standard deviation is plotted against the
resolution.

Figure 5: Left: Sample of the standard deviation of a measure at 10$mm$ from
the detector, without the ANET CNC present. The red dotted line separates the
noise region and the signal. Using a 2$\sigma$ threshold, the value for the
resolution is 76$\mu m$. Right: Histogram of the noise created by the standard
deviation in the noise region. The noise is fitted with a Gaussian curve in
order to obtain the 2$\sigma$ threshold.

In the figure, the initial region is almost constant and it is possible to
distinguish the valleys made by the Siemens star ring markers. The decrease in
the standard deviation at those points is due to the fact that the points in
the area corresponding to the incisions are more uniform than that of the
surroundings, thus decreasing the variation. The quoted value for the spatial
resolution is found at the intersection between the constant (on average) area
and the sloped one.
To determine the value of the resolution, a threshold to distinguish signal
from noise has been chosen. What has been noticed from the experimental data
is that, in the region where there is only noise, the pixel counting has a
relatively small standard deviation (low contrast), while in the signal
region, the pixel count distribution has a larger standard deviation (high
contrast). In figure 5 the green line represents the threshold, tuned to the
region where only noise is present, set at 2$\sigma$ from the average value of
the noise. The vertical red dotted line separates the noise and signal
regions. A systematic study to evaluate the sensitivity of the result with
respect to the choice of the threshold has been done, varying from 2$\sigma$
to 5$\sigma$, observing a maximum variation on the final L/D of less than 5%.
This possible systematic error is taken into consideration for the error
calculation. For the horizontal resolution calculation, the average of top (45
to 135 deg) and bottom (225 to 315 deg) images are used, while for the
vertical resolution the left (135 to 225 deg) and right (-45 to 45 deg) are
used.
The procedure to calculate the effective L/D factor is done by repeating the
measurement and varying the distance of the sample from the scintillator, as
detailed in [5].
Finally the divergence angle is extracted through equation 1.

$\theta_{div}=arctan\big{(}\frac{1}{L/D}\big{)}$ (1)

The horizontal and vertical divergence angles measured with and without the
ANET CNC are reported in table 1

Horizontal | Horiz+ANET | Vertical | Vert+ANET
---|---|---|---
$0.215\pm 0.011$ | $0.144\pm 0.007$ | $0.325\pm 0.022$ | $0.162\pm 0.008$
Table 1: Horizontal and Vertical divergence angles with and without the ANET
CNC, measured using the ”unfolded” Siemens method.

##  4 The Gadolinium knife-edge technique

The gadolinium knife-edge is a reference sample, composed by a 100$\mu m$
thick gadolinium sheet with two polished edges and mounted on an aluminium
frame. With a fixed field of view, The edge images from this high contrast
edge device allow a good estimate of the spatial resolution by means of the
Modulation Transfer Function (MTF) analysis method[7].
The measurements, as explained in the previous section, have been performed at
8 different distances from the scintillator, ranging from 10$mm$ up to
105$mm$.

Figure 6: Two examples of radiographs with the ANET collimator, at 10$mm$
(top) and 100$mm$ (bottom) from the detector.

The gadolinium knife-edge sample has been placed orthogonal to the beam, in
order to properly measure the vertical and horizontal spatial resolutions.
Figure 6 shows the variation in image blurring from two measurements (with the
ANET CNC) at respective distances from the detector of 10$mm$ (top) and
100$mm$ (bottom).
For each distance, two images have been taken, and the MTF has been
calculated.
The MTF method relies on the setting of a threshold identifiyng the so-called
limit resolution, i.e. the resolution at which it becomes impossible to
separate a line’s pair. Nevertheless, it is well known, there is not a
universal standard criteria to set the threshold value. [8, 9, 10, 11]. In the
following we propose a procedure to determine in an empirical way the proper
threshold, relying on an independent measurement of the limit resolution, i.e.
the detector blurring. In our case, the previously described measurement with
the Siemens star attached to the scintillator delivers a precise value of the
limit resolution and it will be used to fix the proper threshold. The
procedure has been the following:

  1. 1.

Set a MTF threshold value in the physical range.

  2. 2.

Calculate the MTF for each radiography on the dataset obtained by moving the
sample at different distances from the scintillator

  3. 3.

Calculate the values for the L/D and the detector contribution using the
method described in [5]

This procedure has been iteratively performed for several threshold values in
the range from 0.05 up to 0.25, leading to the graph in figure 7.

Figure 7: Detector contribution to the spatial resolution measurement as a
function of the MTF threshold.

There is a clear dependence between the detector blurring contribution and the
choice of the threshold. The measurement with the Siemens star in contact with
the scintillator leads to a value for the limit resolution of 50$\mu m$, which
in fig 7 corresponds to a threshold value of 0.08.
Applying this threshold, the graph in figure 8 is generated: a clear
improvement in the resolution when including the ANET CNC in the FISH set-up
is visible.

Figure 8: Resolution fit for the vertical measurement of the divergence with
and without the ANET CNC.

Consequently, an improvement of the beam divergence, is expected: the
horizontal and vertical divergence angles measured with and without the ANET
CNC are reported in table 2

Horizontal | Horiz+ANET | Vertical | Vert+ANET
---|---|---|---
$0.200\pm 0.008$ | $0.148\pm 0.006$ | $0.354\pm 0.013$ | $0.160\pm 0.006$
Table 2: Table to test captions and labels.

##  5 Results

The paper presents the performance of the ANET CNC in terms of spatial
resolution and beam divergence. Two different reference samples and analysis
methods have been used. The L/D values are extracted for each sample, one for
the vertical and one for the horizontal axis. The results are summarized in
figure 9 where the comparison between the divergences measured at FISH with
and without the ANET CNC are shown.

Figure 9: Calculated divergence angle on every configuration using the
gadolinium knife-edge method and the Siemens star. The errors have been
evaluated through a Monte-Carlo error propagation.

The divergence measurements obtained using the Siemens star are compatible
with those obtained using the gadolinium knife-edge.
The 2D structure of the ANET CNC tends to make uniform the vertical and
horizontal divergence. It is important to remark that, despite the ANET CNC
applying the same theoretical correction on both axes, the effect is more
relevant on the vertical axis with respect to the horizontal one, being the
horizontal one originally more collimated
The ANET CNC has proven to be a valid instrument not only to improve, but also
to make uniform the divergence of a neutron beam. In this contribution, the
effect on the divergence has been studied, with two novel methods that allow
to estimate the resolution quantitatively with improved control of the
inherent parameters.

## References

  * [1] R. Bedogni, M. Costa, E. Durisi, F. Grazzi, A. Lega, E. Mafucci, L. Menzio, V. Monti, O. S. Planell, L. Visca, Design of a compact neutron collimator, Journal of Instrumentation 16 (08) (2021) P08055.
  * [2] O. Sans-Planell, M. Costa, E. Durisi, E. Mafucci, L. Menzio, V. Monti, L. Visca, F. Grazzi, R. Bedogni, S. Altieri, First results with the ANET compact thermal neutron collimator, Journal of Instrumentation 16 (11) (2021) P11025.
  * [3] C. Grünzweig, G. Frei, E. Lehmann, G. Kühne, C. David, Highly absorbing gadolinium test device to characterize the performance of neutron imaging detector systems, Review of Scientific Instruments (2007) 78.
  * [4] Z. Zhou, J. Plomp, L. van Eijck, P. Vontobel, R. P. Harti, E. Lehmann, C. Pappas, Fish: A thermal neutron imaging station at hor delft, Journal of Archaeological Science: Reports 20 (2018) 369–373.
  * [5] A. Kaestner, Z. Kis, M. Radebe, D. Mannes, J. Hovind, C. Grünzweig, N. Kardjilov, E. Lehmann, Samples to determine the resolution of neutron radiography and tomography, Physics Procedia 88 (2017) 258–265.
  * [6] M. D. Abràmoff, P. J. Magalhães, S. J. Ram, Image processing with imagej, Biophotonics international 11 (7) (2004) 36–42.
  * [7] G. D. Boreman, Modulation transfer function in optical and electro-optical systems, Vol. 4, SPIE press Bellingham, WA, 2001.
  * [8] S. Williams, A. Hilger, N. Kardjilov, I. Manke, M. Strobl, P. Douissard, T. Martin, H. Riesemeier, J. Banhart, Detection system for microimaging with neutrons, Journal of instrumentation 7 (02) (2012) P02014.
  * [9] A. Saad, M. Saab, D. Gatinel, Repeatability of measurements with a double-pass system, Journal of Cataract & Refractive Surgery 36 (1) (2010) 28–33.
  * [10] J. R. Rogers, Three-bar resolution versus mtf: how different can they be anyway?, in: An Optical Believe It or Not: Key Lessons Learned, Vol. 7071, SPIE, 2008, pp. 38–45.
  * [11] J. H. Kinney, M. Nichols, U. Bonse, X-ray tomographic microscopy using synchrotron radiation, APS Users Organization Steering Committee (1992) 99.