# Toward Finding and Supporting Struggling Students in a Programming Course
with an Early Warning System

Belinda Schantong  Chemnitz University of Technology Germany
<EMAIL_ADDRESS>,  Dominik Gorgosch  Chemnitz
University of Technology Germany<EMAIL_ADDRESS>and  Janet Siegmund  Chemnitz University of Technology Germany
<EMAIL_ADDRESS>
###### Abstract.

Background: Programming skills are advantageous to navigate today’s society,
so it is important to teach them to students. However, failure rates for
programming courses are high, and especially students who fall behind early in
introductory programming courses tend to stay behind.

Objective: To catch these students as early as possible, we aim to develop an
early warning system, so we can offer the students support, for example, in
the form of syntax drill-and-practice exercises.

Method: To develop the early warning system, we assess different cognitive
skills of students of an introductory programming course. On several points in
time over the course, students complete tests that measure their ability to
develop a mental model of programming, language skills, attention, and fluid
intelligence. Then, we evaluated to what extent these skills predict whether
students acquire programming skills. Additionally, we assess how syntax drill-
and-practice exercises improve how students acquire programming skill.

Findings: Most of the cognitive skills can predict whether students acquire
programming skills to a certain degree. Especially the ability to develop an
early mental model of programming and language skills appear to be relevant.
Fluid intelligence also shows predictive power, but appears to be comparable
with the ability to develop a mental model. Furthermore, we found a
significant positive effect of the syntax drill-and-practice exercises on the
success of a course.

Implications: Our first suggestion of an early warning system consists of few,
easy-to-apply tests that can be integrated in programming courses or applied
even before a course starts. Thus, with the start of a programming course,
students who are at high risk of failing can be identified and offered
support, for example, in the form of syntax drill-and-practice exercises to
help students to develop programming skills.

###### Key words and phrases:

Programming education, Empirical study, Cognitive abilities

##  1\. Introduction

The ability to program is becoming more and more important every day in a
world that is becoming more and more digitalized. Some even see programming as
the 4th literacy, as abilities that are so vital to the understanding of our
current and future society that everyone will need to acquire them Román-
González et al. (2017); Vee (2013). Naturally, there is much research on
programming education practices, yet we still struggle with teaching students
programming skills. The failure rate for introductory programming courses
varies around 30 % and has not changed much since many years Luxton-Reilly
(2016); Bennedsen and Caspersen (2007), despite decades of dedicated research.
To address this issues, one line of research studies how cognitive skills are
related to programming skills, so that (programming) training can be tailored
to how students think and develop programming skills.

For example, Ambrosio and others found that general intelligence and spatial
reasoning are highly related to students’ overall performance in a programming
course Ambrósio et al. (2011). Additionally, attention to detail also seems to
be somewhat related to students’ performance, but not mathematical reasoning.
Endres and others found that both, reading skill and spatial skill, correlate
with programming skill after an 11-week programming course Endres et al.
(2021).

Román-Gonzáles and others used an integrated test battery to assess several
cognitive skills, but instead of programming skills, they evaluated how these
skills correlate with computational thinking. They also did not recruit
university students, but students of middle and high school Román-González et
al. (2017). They found medium (reasoning and spatial skills) to high (problem-
solving) correlations, concluding that "computational thinking could be
fundamentally linked with general mental ability (particularly with fluid
intelligence); and to a lesser extent with different cognitive aptitudes, such
as logical reasoning and spatial ability." (Román-González et al., 2017, p.
10). This could be replicated by Stewart and others with a younger population
(i.e., 4th and 5th grade elementary school students) Stewart et al. (2021). In
the end, Román-Gonzáles and others have a successfully developed a test to
measure computational thinking on the level of 5th to 10th grade students and
are able to identify computationally talented students already in middle
school Román-González et al. (2018). However, struggling students cannot be
identified with this test.

Helmlinger and others also evaluated reasoning skills on different modalities
(verbal, numeric, figural), and found that only figural reasoning had a
relationship with programming experience, such that students with programming
experience performed significantly better in figural inductive reasoning tasks
than students with no programming experience Helmlinger et al. (2020).

Prat and others assessed an entire battery of cognitive skills and computed a
stepwise regression model of how these skills can predict programming skills
Prat et al. (2020). They found that numerical skills seem to have no influence
on acquired programming skills, but, depending on the concrete
operationalization of programming skills, either general cognitive abilities
(specifically fluid reasoning, working memory updating and working memory
span) or language aptitude seem to be relevant Prat et al. (2020).

These results are in line with recent neuro-imaging studies, in which a
considerable contribution of language-processing areas in the brain are
observed Siegmund et al. (2014, 2017); Floyd et al. (2017); Lee et al. (2016).
Additionally, the contribution of spatial abilities has also been shown Huang
et al. (2019). Thus, there is strong evidence of connection of several
cognitive skills to programming.

Not only cognitive skills have been used to assess programming skills or to
predict whether someone will acquire skills, also early programming tests have
been developed and applied. For example, Ahadi and others evaluated how the
development of a suitable mental model111In terms of the neo-Piagetian model,
which consists of a sensorimotor, preoperational, and concrete operational
stage Lister (2011) for basic programming concepts (i.e., how values are
assigned to and moved between variables) is related to the success or failure
of a programming course Ahadi et al. (2014). They found that students who do
not develop a suitable mental model at the beginning of a programming course
often cannot overcome these difficulties.

Inspired by this line of research, we set out to develop an early warning
system for students who are at high risk of failing a programming course.
Since learning to program has its difficulties, it is important to intervene
early enough so students do not fall behind early and stay behind, but instead
can close their potential mental gap (or even avoid its emergence). To this
end, we assess several of the cognitive skills and early programming skills
that have been subject of previous studies. Our goal is to combine the results
in a statistical model that predicts the programming performance of students
at the end of a programming course.

With our early warning system, we would have an instrument that would help us
to identify struggling students at a point where we would still be able to
implement counter measures, such as tailored assignments Thurner et al. (2017)
or training of skills that are related to programming and can be transferred
to programming skills Endres et al. (2021).

Another approach are syntax drill-and-practice exercises, since novice
programmers often have problems with learning the syntax of a programming
language, a phenomenon known since the 80s Soloway et al. (1983). These
problems persist to this day, even for students who otherwise show good
performances in programming. For example, Denny and others found that nearly
half of all submissions by students in the top quartile contain syntax errors
Denny et al. (2011). Based on these findings, several studies used syntax
drill-and-practice exercises to reduce the cognitive load caused by
uncertainty about the syntactical rules of a programming language Edwards et
al. (2018, 2020); Ly et al. (2021); Sullivan et al. (2021). We integrated
syntax drill-and-practice exercises into the second part of a programming
course to evaluate whether they can improve programming skills of students at
the end of the course.

In a nutshell, we found that especially the development of a suitable early
mental model and language skills are potentially useful predictors for our
early warning system. Fluid intelligence might also have some predictive
power, but not attention. In summary, we make the following contributions:

  * •

Empirical evidence for the role of cognitive skills in learning to program,
strengthening the result of previous studies

  * •

Differentiation of results of previous studies, showing ceiling effects in the
cognitive skills of students of a programming course

  * •

Course-specific syntax drill-and-practice exercises that improve programming
skills

  * •

The experimental design and raw data is available
(https://anonymous.4open.science/r/9252151/) for others to build on and
replicate our results, and we explicitly encourage researchers of other
programming courses to do so.

##  2\. Experiment

###  2.1. Objective Definitions

To guide our study design, we formulated 6 research questions. For each of
these questions, programming skill is the dependent variable, which is
operationalized as the performance (in terms of score) in the final exam of
the programming course. In Section 2.2, we provide more details on this exam,
as well as all independent variables.

RQ1: Can the development of an early mental model predict programming skill by
the end of a programming course?

Ahadi and others have shown that a set of eight simple programming tasks can
predict whether students without programming experience successfully complete
a programming course Ahadi et al. (2014). These programming tasks, being
provided a few weeks into the course, evaluated whether students have learned
the concept of variables as having a state, rather than being a box that
contains information. The easiest task was to identify the value of a variable
after a few assignment statements, the most difficult task asked students to
implement a swap of values between two variables. The cognitive prerequisites
to successfully complete these tasks can be mapped to the neo-Piagetian stages
of learning, with the easiest task mapping to the sensorimotor stage, and the
most difficult to the concrete operational stage Lister (2011). To investigate
whether these tasks could be integrated into our early warning system, we
replicated the study by Ahadi and others, comparing the students’ performance
in these tasks at the beginning of the course to their performance in the
final exam.

RQ2: Can language skills predict programming skill?

RQ3: Can selective and sustained attention predict programming skill?

RQ4: Can fluid intelligence predict programming skill?

Since all these cognitive skills have been identified in some studies as
predictor for learning to program, we include these in our study. The
advantage of using purely cognitive skills that do not require knowledge of
programming is that we can assess them even before a programming course
starts, so that we might be able to identify possibly struggling students even
before they struggle.

RQ5: What skills are suitable for an early warning system?

Previous studies have shown varying levels of predictive power for all the
skills mentioned in RQ1 to RQ4. Some correlations are promising (e.g., between
the Computational Thinking test and problem-solving ability with r = 0.67
Román-González et al. (2017), or between language aptitude and programming
learning rate with r = 0.56 Prat et al. (2020)). However, this means that only
some portion of the variance in the acquired programming skill can be
accounted for. To increase the predictive power of these skills, we aim at
combining them (i.e., the development of a suitable mental model (RQ1),
language skills (RQ2), selected and sustained attention (RQ3), and fluid
intelligence(RQ4)) into one model.

RQ6: Can syntax drill-and-practice exercises improve programming skill?

To provide an actionable plan for educators, we integrate syntax drill-and-
practice exercises to our programming courses. Previous studies have shown the
success of such exercises in improving the learning outcome of a programming
course Edwards et al. (2018, 2020); Ly et al. (2021); Sullivan et al. (2021).
Given the consistent finding of language skills as predictor for programming
skill, and that learning syntax is difficult Denny et al. (2011) and coupled
to language skills, this seems to be a promising option as intervention for
students who appear to have fallen behind in a programming course.

###  2.2. Material

In this section, we present the operationalization of our variables, arranged
by research question. We start with the dependent variable, the programming
skill.

####  2.2.1. Programming Skill

As indicator for programming skill, we use the points that students receive in
the final exam. The format was a take-home exam. Students received
instructions to implement three programs. We defined four tasks in total, and
students could choose which of Task 3 or 4 to implement.

  * •

Task 1: Implementation of a double linked list for integer values, including
an algorithm that successively deletes sub-lists whose sums equal a prime
number.

  * •

Task 2: Implementation of a tree structure with weighted nodes, including an
algorithm that searches for all nodes within a certain range of weight, while
also considering the weight of their child-nodes.

  * _and either one of:_

  * •

Task 3: Implementation of an algorithm that finds the shortest path on the
board of a game while considering different types of terrain and different
movement-speed assigned to units in the game.

  * •

Task 4: Implementation of the simulation of a social network, including
methods to identify friendship-circles within the network.

The time limit to complete the tasks was six hours. Students could implement
their solutions in either Java or Python; both languages were used in the
programming course. Since this was an open-book exam, students were allowed to
use materials and code from the course and the internet, but needed to work
alone.

While the conditions of the exam are less controlled than in a closed-book
exam under observation, the measurement of programming skills is much more
realistic. Thus, although there might be some outliers among the students
about how they conduct the exam (e.g., seek help of colleagues), we believe
that, for the majority of students, we have a valid operationalization of
programming skill.

####  2.2.2. RQ1: Can the development of an early mental model predict
programming skill by the end of a programming course?

To evaluate whether and how students have developed an early mental model of
programming, we used the Early-mental-model test (Early-MM) created by Ahadi
and others Ahadi et al. (2014). It consists of eight tasks that test the
student’s understanding of assignment statements, one of the first concepts
required to learn in programming, with the most difficult one requiring to
implement a value swap between two variables. For each task, students get
either 0 or 1 point, so the minimum of this test would be 0 points and the
maximum 8 points.

In Figure 1, we show the easiest and most difficult tasks with correct
solutions. Students who can implement the swap correctly must have fully
understood the concept of variables having a state, whereas students who fail
to answer the easiest question correctly have not yet understood how variables
work. However, since variables are the building blocks of most programs,
students will struggle with keeping up understanding higher concepts. Thus,
these tasks may be a suitable predictor for programming skills.

(a) The easiest task was to name the values of variables a and b (a = 3 and b
= 2 is correct).

(b) The most difficult task was to implement a swap. A correct solution would
be temp = first; first = second; second = temp

Figure 1. Tasks of the Early-MM Ahadi et al. (2014).

####  2.2.3. RQ2: Can language skills predict programming skill?

Since language skills are reliably found to predict programming skill, we
selected several language skill tests to have different options to explore a
relationship with programming skills. All three tests were provided by a local
institute specialized in test development (will be disclosed after reviewing).

##### C-Test (English)

A C-Test is an integrative written test to measure overall fluency in a
specified language and is usually used to assess the participant’s reference
level in the Common European Framework of Reference for Languages (CEFR)222The
CEFR describes foreign language proficiency at three divisions, each divided
into two levels: Basic users (levels A1-A2), independent users (levels B1-B2)
and proficient users (levels C1-C2), with the highest level (C2) being similar
to a native speaker. For further explanation, see Council of Europe. (2001).
Ajideh and Mozaffarzadeh (2012). The C-Test specifically ranked participants
from the CEFR levels B1 to C1. For comparison, the level expected from high-
school graduates is B2, with high-performing students being able to reach C1
Keller et al. (2020). The C-Test has four items which consist of a cloze text
in the target language that is constructed according to specific rules. Each
text has 25 gaps. We show an example with solution in Figure 2. The
participant’s score is computed by counting the correctly solved gaps, leading
to a maximum score of 100 points. Participants had 5 minutes to solve each
item.

(a) As shown to students.

(b) Version with solutions (not shown to students).

Figure 2. One item in the C-Test.

##### Receptive and Productive Vocabulary Test (English)

The vocabulary tests assess vocabulary size and indicate the test person’s
reading level in the target language. They assess the participant’s vocabulary
knowledge in five different ranks, which are based on high-frequency
vocabulary lists. The first rank assesses knowledge among the 1000 most
frequent words in the target language, the second rank among the most frequent
2000, and so forth, up to rank five Institute for Test Research and Test
Development Leipzig (2019). A participant is considered to have passed a rank
if they reach at least 80 % of the possible points in the rank. From the
passed vocabulary ranks, the person’s reading level in the CEFR can be
inferred, in this case, ranging from level A2 to B2, so narrowly reaching the
level expected from high-school graduates Keller et al. (2020). We used both,
a receptive and a productive vocabulary test. Receptive (or passive)
vocabulary competence refers to the words a person is able to understand while
reading, which is usually higher than the productive vocabulary competence,
which refers to words that can be actively used Institute for Test Research
and Test Development Leipzig (2019). Points are awarded for both tests by
counting the correctly solved tasks. Participants had a time limit of 30
minutes per test. Both tests use similar, but different items:

  * •

An item of the receptive vocabulary test (R-Voc) consists of a drop-down menu
with six words from the current vocabulary level, and three phrases that
define one of the words from the drop-down menu, as shown in Figure 3(a). The
test person matches the words with the corresponding phrases. The tests
consisted of 10 items per rank with up to 3 points per item, leading to
maximum 30 points per rank and a total possible score of 150.

  * •

An item of the productive vocabulary test (P-Voc) consists of a cloze text
that is one or two sentences long. There is one gap for the target word, as
shown in Figure 3(b). To disambiguate an item, up to the first three letters
of the missing word are given. The test consists of 18 items per rank with 1
point per correctly solved item, leading to a maximum of 18 points per rank
and a total possible score of 90.

(a) For question 1a, ’concept’ needs to be selected from the drop-down menu.
Institute for Test Research and Test Development Leipzig (2019)

(b) The solution is ’school’, thus ’chool’ needs to be added. Institute for
Test Research and Test Development Leipzig (2019)

Figure 3. Possible items of the R-Voc (a) and the P-Voc (b).

####  2.2.4. RQ3: Can selective and sustained attention predict programming
skill?

The d2 Test of Attention measures selective and sustained attention, and
visual scanning speed, allowing for a neuropsychological estimation of
individual attention and concentration performance Brickenkamp et al. (2016).
For this study, we used the digital version of the d2-R (Revision). In Figure
4, we show an excerpt of the test. Participants were instructed to react as
quickly as possible to every symbol containing a d with two lines, no matter
where the lines were. The test consists of 14 items, with a time limit of 20
seconds per item. One item consists of 6 rows, each row consisting of 10
symbols. Including instructions, the test can take up to 10 minutes to
complete, but typically takes less time. The results of the d2-R are measured
on a T scale, so the standardized mean is 50, with a standard deviation of 10.
The test’s reference groups consists of Europeans, male and female, aged 18 to
55 Brickenkamp et al. (2016).

Figure 4. The task is to react as quickly as possible to any d with two lines.
The target symbols are highlighted with thicker boxes for purpose of
illustration.

####  2.2.5. RQ4: Can fluid intelligence predict programming skill?

The DESIGMA-Advanced is a test for fluid intelligence using figural matrices.
In Figure 5, we show one item of the test. The missing figure is a circle,
which participants can compose with the elements of the second column. With
this composition approach, distractor elements are not necessary, so the
probability that a participant can guess the solution by process of
elimination is reduced Becker et al. (2014).

Figure 5. DESIGMA-A: Participants need to compose the missing figure of the
matrix. In this example, the correct solution is a circle, which can be
composed by the elements of the second column. Becker et al. (2014)

There are 38 items in the entire test, and the time limit is 90 seconds per
item, meaning the test takes at max 57 minutes to complete. The results of the
DESIGMA-A are measured on the IQ scale, so it has a standardized mean of 100
and a standard deviation of 15.

####  2.2.6. RQ6: Can syntax drill-and-practice exercises improve programming
skill?

We designed syntax drill-and-practice exercises for the following programming
constructs: conditions, for-, while- and do-while-loops. We introduced syntax
errors in each of the constructs. We defined 5 categories of errors: brackets
(mismatched, unbalanced, or abused usage of brackets, braces and parenthesis),
semicolon (e.g., an unnecessary semicolon, or an incorrect character instead
of a semicolon), loop counter (e.g., i+ instead of i++), keywords (e.g.,
capitalized keywords) and several other errors that do not fit into the
categories (e.g., a misplaced else if). In Figure 6, we show examples for
different constructs and different error categories.

The tasks were summarized per programming construct, leading to 23 to 25 tasks
per construct (98 tasks in total), with varying categories of errors. For each
construct, we expected students to take about 15 minutes to complete, but
there was no time limit for the exercises. Students should identify and fix
errors so that a program would compile; each task contained one error. The
responses were assessed automatically, giving the participants direct
feedback. Students could obtain either 0 or 1 point for a task, leading to a
maximum of 98 points. We presented the programming constructs in fixed order,
but the individual tasks per construct were randomized.

(a)

(b)

(c)

(d)

Figure 6. Syntax drill-and-practice exercises. The errors are: (a) Line 6 -
Incorrect Assignment Operator (Category: Other), (b) Line 8 - Incorrect
separators in the head of a for-loop (Category: Semicolon), (c) Line 14 -
Missing closing bracket (Category: Bracket), (d) Line 13- incorrect
capitalization of a keyword (Category: Capitalization).

###  2.3. Participants

As participants, we recruit students from a programming course that spans two
semesters. The first part of the course is referred to as Algorithms and
Programming (AP) and the second part is called Data Structures (DS). Due to
the pandemic and varying study regulations, we have a high fluctuation of
participants for the different research questions. Specifically, 111 students
participated for RQ1, and for the remaining RQs, we have between 35 and 50
participants. The age ranged between 18 to about 43. Students who started AP
have been programming for an average of about 1.8 years (sd = 1.8) before
taking the course. Most students are enrolled in a major that is rooted in the
department of computer science at our university, with some of the majors
being more interdisciplinary. All students are non-native English speakers. A
detailed breakdown of students based on each research question is available on
the project’s Web site.

###  2.4. Conduct

After consulting with the instructors of AP, the Early-MM test was released to
the students shortly after the start of the first semester. It was given in
the form of an online survey, which was available to the students over the
course of two months. As compensation, students received 2 bonus points for
their assignments in AP for participation (independent of the performance).

In DS, the experiment was linked with assignments that were mandatory to pass
for the students to be allowed to take part in the exam. Students would need
to get at least 50 % of the points that were achievable in the assignments to
pass. Students could choose whether they wanted to participate in the
experiment or not. If they chose to participate, they would get slightly
easier assignments to make up for the extra time they would use for the
experiment, but they still had access to the normal version of the assignments
in case they needed it, for example, in preparation for the exam. In addition,
students who participated in all tests were offered two bonus points to count
to the 50 % of points, independent of their performance in the tests. Few
students exploited this by just clicking through tests, which we removed from
our analysis (we present details in the results section).

The course spanned fourteen weeks. Assignments were distributed weekly
starting in the third week of the course. The students had two weeks to
complete each assignment, including the tasks for the experiment. The tests
and syntax drill-and-practice exercises were distributed as follows:

  * •

Week 7: Syntax drill-and-practice exercises

  * •

Week 9: d2-R and C-Test

  * •

Week 10: DESIGMA-A, R-Voc and P-Voc

One could argue that applying the cognitive tests to be used as early warning
systems so late in the course does not make much sense. However, since in this
experiment, we are developing the early warning system (not applying it), and
since the cognitive skills as measured by these tests are not subject to
change considerably over the course of several months, the results would still
tell us whether these tests are useful for an early warning system. In that
case, they must be applied early on, possibly even before the start of a
course.

The final exam took place five weeks after the end of the fourteen-week
course.

##  3\. Threats to Validity

###  3.1. Construct Validity

With measuring cognitive constructs, it is always challenging to find a
suitable operationalization that is also fast enough to apply. With this
compromise, we have used for most cognitive skills established tests that are
fast to apply; however, we found that they cannot differentiate well among the
high-skilled population of students. Thus, we might misinterpret the role of
certain cognitive skills as predictor for programming skill. To mitigate this
threat, we take it into account when discussing the results.

For programming skills, one could also argue whether the final exam is a good
operationalization. However, since the exam takes place in a realistic setting
with sufficient time, we believe that our measurement of programming skill is
valid (e.g., evaluation apprehension is reduced).

###  3.2. Internal

Since participation was voluntary, we have a high risk of selection bias.
Given the number of tests, distributed over two semesters, we have only a
relatively small number of students who participated in all tests and the
final exam. Thus, our results for the early warning system should be
considered in light of this higher motivation.

Additionally, students did all the tests at home, online, and without
supervision. Thus, we cannot be sure that all students followed the protocol
for all tests. To mitigate this threat, we removed outliers that indicate when
students violated the instructions (e.g., were considerably faster than the
expected time a test takes).

###  3.3. External Validity

It is difficult to generalize the results to other settings, such as different
programming courses, different majors, or different education systems.
Nevertheless, since our sample is representative for many universities in our
country, they have an implication for many students who want to learn
programming.

###  3.4. Statistical Conclusion Validity

Unfortunately, due to the nature of the long-term experiment, and also the
Covid-19 pandemic, our sample on which we intended to compute the integrated
model with the cognitive skills is too small; only 18 student completed all
the tests. For the other tests, the data is rather fragmented, so we also
could not select a useful subset of tests for cognitive skills. Thus, we
cannot reasonably compute a statistical model. Instead, we only did a manual
analysis of the correlations and co-correlations of our variables to not come
to false conclusions.

##  4\. Results

Since we have highly fragmented data, (i.e., not all students completed all
cognitive tests and the exam), we decided to present the data description per
research question in 2 steps. For completeness, we first present the data of
all students who completed a test that is used for a certain RQ. Second, we
present the data for the students who participated in an RQ and who completed
the final exam. For example, for RQ1, 111 students took the Early-MM test, but
only 32 of these took the final exam. For outlier removal, we report the
concrete procedure for each of the research questions separately, since there
was no unified procedure due to the diverse nature of the tests.

Before diving into the research questions, we start by presenting data on the
final exam. A maximum of 100 points could be reached, and it was taken by 102
students. The mean was 66.61 with a standard deviation (sd) of 18.16. The
median was 68.5 points. Since the number of students who took part in the
tests for the RQs fluctuated, we present the concrete mean of the participants
in the exam per research question in Table 1. The exam results were not
normally distributed, as assessed by the Shapiro-Wilk-Test (p = 0.004). Thus,
we always the Spearman rank correlation as non-parametric alternative to the
Pearson correlation when comparing test results.

Table 1. On the left: Mean results for each test with all participants. On the right: Mean values of the exam and the tests, as well as correlation values between the exam and each test, with participants who did took in both. Test |  Participants (All) | Mean Test (SD) |  Participants (Test + exam) | Mean Exam (SD) | Mean Test (SD) |  Correlation with exam
---|---|---|---|---|---|---
Final exam |  102 | 66.61 (18.16) |  |  |  |
Early-MM |  111 | 6.17 (1.86) |  32 | 75.09 (15.22) | 6.97 (1.03) |  0.23
C-Test |  50 | 76.90 (14.62) |  36 | 73.75 (14.79) | 80.11 (12.03) |  0.20
R-Voc |  38 | 129.76 (18.09) |  27 | 73.65 (14.99) | 130.74 (14.41) |  0.28
P-Voc |  35 | 61.77 (11.45) |  26 | 72.81 (14.69) | 62.38 (11.85) |  0.27
d2-R |  49 | 57.12 (8.99) |  36 | 73.75 (14.79) | 57.11 (9.27) |  0.09
DESIGMA-A |  35 | 115.97 (12.50) |  25 | 72.42 (14.75) | 118.72 (10.32) |  0.19

###  4.1. RQ1: Can the development of an early mental model predict
programming skill by the end of a programming course?

The mean score for all 111 students who participated in the Early-MM test was
6.17 (sd = 1.86), with a maximum of 8 points. The majority of the students
received at least 75 % of the points, indicating a ceiling effect. In Figure
7, we visualize the results. Considering the responses by task (Fig. 7(b)),
the majority of the students is able to correctly solve them, with the second
task being the easiest. Thus, most of the students in the course appear to
have mastered the preoperational stage, but may not have fully reached the
concrete operational stage. Interestingly, Tasks 6 and 7 (providing a natural-
language description of source code) appeared to be most difficult, but not
Task 8, which was intended to be the most difficult tasks (i.e., implement a
swap).

Of these 111, only 32 students finished the programming course by taking the
final exam of the DS course. The mean for these 32 is 6.97 points (sd = 1.03),
so the ceiling effect is even more pronounced. Of these 32, all scored 5 or
more points, with the majority scoring 7 or 8 points (23; cf. Fig. 7(a)).
Regarding the points per task, the first 3 tasks have been answered correctly
by all participants. Furthermore, the students are performing approximately 10
% better, compared to all participants who completed the Early-MM test.

The correlation with the exam is r = 0.23 (p = 0.204). Thus, the performance
in the Early-MM test has a weak correlation with the performance in the final
exam of the course.

The Early-MM test can predict the success of the programming course to a small
degree.

(a)

(b)

Figure 7. (a) sum of points for all participants and those who took part in
the final exam. (b) number of correct solutions per task.

###  4.2. RQ2: Can language skills predict programming skill?

To answer RQ2, we used three different tests: The C-Test, the R-Voc and the
P-Voc (cf. Section 2.2).

50 students completed the C-Test. The mean score was 76.90 points (a maximum
of 100 points could be reached), with a standard deviation of 13.89. The
median was 81. From these 50 participants, 36 took the final exam. The mean
for these 36 is 80.11 points (sd = 12.03) and the median is 84. Most students
(31 of 50) are in the C1 level, which indicates a ceiling effect.

The R-Voc was completed by 39 students. We removed the data of one student,
because they only received 24 points (from a maximum of 150 points), which is
more than two standard deviations below the mean. The remaining 38 students
received on average 129.76 points (sd = 18.09), and the median was 134.5
points. 27 students took part in the final exam. For these 27, the mean was
130.74 points (sd = 14.41) and the median was 134. Most students (28 of 38)
are in the B2 level, indicating a ceiling effect.

The P-Voc had 35 participants. Out of 90 points, the mean was 61.77 points (sd
= 11.45) and the median was 62 points. Out of the 35 participants, 26
completed the exam. They have a mean of 62.38 points (sd = 11.85) and the
median was 61 points. The ceiling effect here is less pronounced: While on
average, the participants still reached far more than half the points, only 7
students passed the highest rank and are at the B2 level.

Noteworthy is also that, when considering only the students who completed the
final exam, the test scores are a bit higher.

To answer RQ2, we correlate the points of each of the three language-skill
tests with the points in the final exam. For all tests, the correlations are
in the weak range, starting with 0.20 for the C-Test (p = 0.251), 0.28 for
R-Voc (p = 0.159), and 0.27 for P-Voc (p = 0.186).

Language skills can predict the success of the programming course to a small
degree.

###  4.3. RQ3: Can selective and sustained attention predict programming
skill?

49 students participated in the d2-R. For our participants, the mean was 57.12
(sd = 8.99). 36 finished the DS course by taking the final exam. Of these 36,
the mean was 57.11 (sd = 9.27). As with the language skill tests, the students
performed above average (the standardized mean is 50 points, sd = 10),
although the higher performance was not so pronounced as for the language-
skill tests or the Early-MM test.

The correlation between the d2-R and the exam is 0.09 (p = 0.584).

Selected and sustained attention cannot predict the success of the programming
course.

###  4.4. RQ4: Can fluid intelligence predict programming skill?

We received the response of 35 students for the DESIGMA-A. One participant
took the test twice, and we only kept the response of the first attempt. The
mean was 115.97 (sd = 12.51). 25 students finished the DS course by taking the
final exam. Of these 25, the mean was 118.72 (sd = 10.32).

As for the language tests and d2 tests, the students performed above average,
even one standard deviation above, showing their higher than normal
intelligence as measured with this test. As for the Early-MM and the language
skills, the performance of the students who finished the DS course is a bit
higher than for all students who completed the DESIGMA-A.

The correlation between the DESIGMA-A and the exam is 0.19 (p = 0.373).

Fluid intelligence can predict the success of the programming course to a
small degree.

###  4.5. RQ5: What skills are suitable for an early warning system?

Table 2. Correlation matrix of all tests. **$p<.01$, *$p<.05$. Exact p-values provided on the project’s Web site. | d2-R | DESIGMA-A | C-Test | R-Voc | P-Voc | Early-MM | Exam
---|---|---|---|---|---|---|---
d2-R | 1.00 | 0.12 | -0.16 | 0.20 | -0.14 | -0.14 | 0.29
DESIGMA-A | 0.12 | 1.00 | -0.17 | 0.41 | 0.20 | 0.36 | 0.23
C-Test | -0.16 | -0.17 | 1.00 | 0.52* | 0.83** | -0.05 | 0.13
R-Voc | 0.20 | 0.41 | 0.52* | 1.00 | 0.68** | -0.06 | 0.24
P-Voc | -0.14 | 0.20 | 0.83** | 0.68** | 1.00 | -0.11 | 0.24
Early-MM | -0.14 | 0.36 | -0.05 | -0.06 | -0.11 | 1.00 | 0.28
Exam | 0.29 | 0.23 | 0.13 | 0.24 | 0.24 | 0.28 | 1.00

To answer RQ5, we calculated the correlations for all tests from RQ1 to RQ4.
The plan was to integrate the results into a stepwise regression model.
However, only 18 students completed all tests and the final exam, which is not
sufficient for computing a stepwise regression model. Instead, we take a
detailed look at each correlation one by one. In Table 2, we present the
correlation matrix of all tests and the final exam. The mean score for the 18
students in the exam was 73.03 (sd = 16.63), which is higher than the mean for
all students (66.61).

Early-MM has no considerable correlations with the other tests, except
DESIGMA-A, which means the uniqueness it provides as predictor appears rather
high. The correlation with the exam is low (0.28), but still among the highest
when compared to the other predictors. Thus, even though there is a ceiling
effect, the Early-MM might be a good candidate for the early warning system.

The language tests correlate highly with each other ($r>=0.52$), especially
C-Test and P-Voc, suggesting that these two tests do not add much uniqueness
as predictors. Thus, one of these tests would suffice for the early warning
system. Of these two, the P-Voc has a higher correlation with the exam, but
the C-Test has a lower correlation with the DESIGMA-A. Since both tests take
about the same amount of time to apply, either of these tests might be a good
candidate for the early warning system.

The d2-R only has low correlations with the other tests, so from that
perspective, is a good candidate for the early warning system. Interestingly,
taking only the students into account who completed all the tests as well as
the final exam, the correlations changes drastically, from almost zero to an
(cf. Table 1) almost medium correlation. Thus, it may not be a good predictor
for all students of a course, but only for the ones who are highly motivated.
Hence, the d2-R may not be suitable for our early warning system, as it cannot
differentiate among the weaker students, which we want to identify with it.

The DESIGMA-A has a medium correlation with Early-MM, but a low correlation
with the exam. Additionally, it has a medium correlation with R-Voc. Low
correlations with the other tests also make it a good predictor regarding
uniqueness. In contrast to Early-MM, it can be applied before a programming
course starts, since it does not require any prior training.

Possible candidate tests for an early warning system are the Early-MM and/or
the DESIGMA-A, plus one language-skill test, such as the C-Test or the P-Voc.

###  4.6. RQ6: Can syntax drill-and-practice exercises improve programming
skill?

47 students participated in all parts of the syntax drill-and-practice
exercises. One participant submitted partially blank responses, so we assume
his responses are not genuine, so we remove the data as an outlier. Thus, our
sample consists of 46 students. Overall, the students received an average
score of 95.67 %. Students’ performance improved from exercise to exercise,
that is, from 92.16 % to 98.17 %.

(a)

(b)

Figure 8. (a) Percentage of correct submissions per exercise type. (b) Points
in the final exam, divided into students who participated in the syntax drill-
and-practice exercises (orange) and students who did not participate (purple).

Of these 46, 33 students participated in the final exam. The average score in
the syntax drill-and-practice exercises of these participants was 96.47 % (sd
= 2.69), so comparable to the entire sample (95.67 %, sd = 2.89). Broken down
per programming construct, the number of points also increases per construct
(93.41 % to 98.79 %), with a still stand from for loops (96.97 %) to do-while
loops (96.73 %).

Regarding error category, there are some interesting observations: First,
there were many mistakes students made with errors of the bracket category at
the first exercise (33), which drastically was reduced for the last exercise
(2). Second, for errors in the loop counter category, the only mistakes
students made was for the for loops, which was the first construct in which
these errors occurred. Last, for the errors that cannot be assigned to a
specific category, in the last exercises (while loop), students made most of
the mistakes (8 of 10). This specific error was that the loop counter (a.k.a.,
stepper) was missing, making the loop an infinite loop. Thus, this may
actually not be a syntactical error, but more a semantic error, or a
combination of both.

To evaluate whether the syntax drill-and-practice exercises have an effect on
the acquired programming skills, we compare the points in the exam for
students who participated in the syntax drill-and-practice exercises with
students who did not participate. Students who completed the syntax drill-and-
practice exercises received on average 74.61 points (sd = 15.04; the maximum
is 100 points). By contrast, the 69 students who did not participate in the
syntax drill-and-practice exercises scored an average of 62.96 points (sd =
18.5) in the exam, so considerably lower. This difference is significant
(Mann-Whitney-U test; U = 1555.5, p = 0.003) and indicates a medium effect
size of 0.37 (Cliff’s Delta).

Syntax drill-and-practice exercises can help students improve their
programming skill.

##  5\. Discussion

###  5.1. RQ1: Can the development of an early mental model predict
programming skill by the end of a programming course?

We found that students’ ability to develop a suitable mental model at the
beginning of a programming course can predict the performance of acquired
programming skill to a certain degree. Given that it is also easy to apply, it
is a suitable test for our early warning system. In contrast to the results by
Ahadi and others, we found that most students had reached the concrete
operational state and could implement a swap of values between two variables.
One reason might be that many of our students had some previous programming
experience: On average, our students have been programming for 1.8 years,
while Ahadi and others assume that their students do not have any programming
experience Ahadi et al. (2014). Thus, the students of our sample had
considerably more time to develop a suitable mental model of programming. This
might also explain why we observe a lower correlation, that is, in our sample,
the Early-MM test cannot differentiate sufficiently among our more experienced
students. In the second instance of the programming course, we have extended
the task to cover more advanced aspects of programming, such as logical
expressions and arrays. Data collection is currently ongoing, so we cannot
give an impression of how the revised version of the Early-MM works.

Interestingly, we observed that two of the questions are especially difficult
for our students. In these questions, students had to give a natural-language
description of what a piece of source code was doing. Especially describing
the swap proved more difficult than implementing one. This could hint to a
problem that students have difficulties in comprehending code, rather than
producing it. Another explanation could be that, since the swap is such a
basic algorithm, students might be used to implementing it without actually
having a natural-language representation. This result was also observed by
Ahadi and others for one of their sample of students, who also scored better
at implementing the swap than describing it, although the difference was less
pronounced (i.e., 50 %, 44 %, and 53 % for Tasks 6, 7, and 8, respectively; in
the other sample, the implementation of the swap was the most difficult, i.e.,
42 %, 33 %, and 31 % for Tasks 6, 7, and 8, respectively). Diving deeper, we
found that common mistakes were that participants (i) assumed that all
variables would get the same values, (ii) named the temporary variable to be
swapped, instead of the target variables, or (iii) described the swap on a too
high abstraction level. Thus, it might be helpful for students to focus more
attention on the purpose of code, so to also include program-comprehension
task in programming courses Izu et al. (2019). Although students were able to
read source code and trace variable values, they could not describe the exact
purpose of the code, so failed to integrate the statements into semantic
chunks, an important part of bottom-up program comprehension Pennington
(1987). Hence, students have not yet reached the concrete operational stage
and their mental model is not sufficiently developed to understand more
complex algorithms Lister (2020).

###  5.2. RQ2: Can language skills predict programming skill?

All the language tests showed a certain relationship with the performance in
the final exam. Thus, we could strengthen the result of previous studies, even
though the strength of the relationship is weaker in our data than, for
example, compared to Prat and others Prat et al. (2020). One reason could be
that most participants received an above average score, so the language tests
were too easy for our intended population of computer science majors.
Additionally, Prat and others used a different operationalization of language
skills that predicts whether someone is able to learn a new language, whereas
we tested whether someone already is proficient in a foreign language.

In the future, we intend to look for tests that are more difficult, but still
easy to apply. Currently, there is a C-Test being constructed that is more
difficult, and we plan to integrate this test in our ongoing data collection
among students of the two-semester programming course.

The advantage of our tests is that they are very quick to apply and easy to
access. That is a problem for the test used by Prat and others Prat et al.
(2020), which is not available for research, so it is not useful for our early
warning system.

###  5.3. RQ3: Can selective and sustained attention predict programming
skill?

It is unclear how selective and sustained attention is related to programming
skill. Considering the relationship between students who completed the d2-R
test and the exam, the correlation is not different from zero, but considering
the correlation of students who completed all tests and the exam, the
correlation increases and could almost be considered as medium (0.29, cf.
Table 1). One possible interpretation could be that the d2-R test can only
differentiate among the students who are highly motivated to perform well in a
course and take part in all learning opportunities. However, since this test
cannot differentiate between the students that we want to identify with it, it
is not suitable for our early warning system. This is in line with the
interpretation by Ambrosio and others, who concluded that simple attention
tasks might be too easy to really differentiate between programming students
Ambrósio et al. (2011). Even though they used a different operationalization
of attention, we observed a similar result.

###  5.4. RQ4: Can fluid intelligence predict programming skill?

We found weak correlations between the fluid intelligence and programming
skill, strengthening previous findings that higher fluid intelligence is
useful when acquiring programming skills. As with the other cognitive skills,
this is less pronounced compared to other studies, which often reported medium
to high correlations between fluid intelligence and programming skill Ambrósio
et al. (2011); Prat et al. (2020); Román-González et al. (2017). The disparity
between these and our results might be due to the difference in
operationalization of fluid intelligence, since the DESIGMA-A uses a solution-
composition approach to figural matrices (cf. Section 2.2). The
operationalization of programming skill also differs between studies, so it is
difficult to truly compare the results.

Additionally, we found that fluid intelligence and the ability to build a
correct early mental model of programming have a medium correlation. Thus, in
the beginning of a programming course, a higher fluid intelligence appears to
be helpful, but this advantage may vanish during the programming course. This
is also in line with neuro-imaging studies, in which researchers observed that
experience in programming leads to a more specific brain activation during
programming tasks, compared to novices, who typically show a pattern of
individual cognitive processes and less efficient use of neuronal resources
Helmlinger et al. (2020); Siegmund et al. (2017); Crk et al. (2015). Thus, the
DESIGMA-A might be a good predictor for how students start a programming
course, but might be less relevant when students have acquired a certain skill
level in programming.

Hence, the DESIGMA-A could be a useful predictor for our case, but is rather
tedious to apply, as it takes over one hour to administer. Recently, a short
version of the DESIGMA has been released, which only takes up to 23 minutes
Krieger et al. (2022), so less than half the time of the long version. With
this reduced effort, the short version of DESIGMA may be of help for us, so we
have integrated this version in our ongoing study.

###  5.5. RQ5: What skills are suitable for an early warning system?

Unfortunately, the small sample size for students who participated in all
tests and the final exam did not allow us to compute a statistical model, such
as a stepwise regression model, to build an early warning system.
Nevertheless, we could get a first impression of possible candidates by
manually analyzing the correlations among all tests and exam scores.

Especially language skills and a correct early mental model appear to be able
to predict whether students have developed programming skills at the end of a
two-semester programming course. In addition, fluid intelligence may also be a
suitable predictor. The advantage of assessing cognitive skills, rather than
programming skills, is that this can happen directly at the beginning of a
programming course or even before that, so that students who have a higher
risk of failing a course can be made aware of that risk early on. This way,
they know that they should be careful not to fall behind, because it is
difficult to catch up. Furthermore, these students can receive special course
offers, for example, in terms of tailored assignments or especially trained
teaching assistants, which can help students not to fall behind.

In a sample where students have some previous exposure to programming, the
Early-MM test can also be applied directly at the beginning or before the
actual course, with the same goal of identifying students who are at high risk
of struggling. When applied a few weeks into a programming course, it may need
some adjustments to fit a more experienced demographic. With the integration
of programming education in more and more middle and high schools, it is
highly likely that more and more students start a major and already have some
programming skills, so an adjusted test is definitely useful in the future.

###  5.6. RQ6: Can syntax drill-and-practice exercises improve programming
skill?

We found that syntax drill-and-practice exercises can improve students’
programming skill at the end of a two-year programming course. Unfortunately,
our data does not allow us to draw conclusions on whether these exercises are
especially useful for students who have a low score in the early warning
system, as we only have 12 students who participated in all tests, the exam,
and the syntax drill-and-practice exercises. Nevertheless, since these
exercises do not require much effort, neither for students nor for educators,
and since there is a measurable positive effect, we are currently integrating
these exercises into the course material. Additionally, we are increasing
their difficulty level, so that not only the inexperienced students profit
from them, but that also more skilled students can improve their programming
skills.

Interestingly, we found that, for later syntax drill-and-practice exercises,
the number of mistakes decreases across error categories, especially for
bracket errors (33 vs. 2) and loop counter errors (only for for loops). This
could mean that, when students have familiarized themselves with certain
errors, they can transfer that to different programming constructs. Thus, it
may be possible to create a more efficient version of syntax drill-and-
practice exercises, such that programming constructs and error categories can
be combined and the number of items can be decreased.

More efficient syntax drill-and-practice exercises also means that we can
include more programming exercises that build on the syntax drill-and-practice
exercises, such as fill-in-the-blank tasks (e.g., used by Kramer and others
Kramer et al. (2019) and Kyaw and others Kyaw et al. (2021)) or parsons puzzle
Denny et al. (2008); Ericson et al. (2017); Parsons and Haden (2006).
Additionally, errors that can be a combination of both, syntactical and
semantic errors, can be integrated, such as the missing loop counter that we
accidentally as syntax error, but might actually turn out to be a semantic
error, or a combination of both.

##  6\. Implications

The results of our study are promising for an early warning system. It can be
applied even before actual programming courses start to help students by
assigning them to specially trained tutors or provide them with tailored
training tasks. These could be similar to the tasks defined by Thurner and
others, who provide tailored tasks based on performance of preceding
programming tasks Thurner et al. (2017). However, this can only happen a few
weeks into a course, but with an early warning system, this process can be
started even before that, so that we do not lose valuable time at the
beginning of a course.

Syntax exercises help, even when they are applied later in the course. Of
course, earlier might even be more helpful, as found by the use of Phanon by
Edwards and others in several studies Edwards et al. (2018, 2020); Sullivan et
al. (2021); Ly et al. (2021). Additionally, special training in reading and
spatial skills can also help improve programming skill, and these can happen
even before a programming course Endres et al. (2021).

Despite having rather preliminary data, we are hopeful that our candidate
early warning system is a good starting point to identify struggling students
so early that we can still implement measures to support them in overcoming
their struggles. Noteworthy are the consistent ceiling effects for all tests,
meaning that, for all tests, our sample performed well above the mean. This
can be expected, considering that our sample typically has higher cognitive
skills than the average population (i.e., including persons not pursuing a
degree in higher education). For building an early warning system, it may
increase predictive power if we include tests for cognitive skills that are
designed for a population of higher cognitive skills and can differentiate
better among students. This is what we currently do in our ongoing studies.

Another interesting trend is that students who persevered in the course until
the final exam tend to score better in all the tests that we applied.
Additionally, students who took part in the study for at least one test or
even all of the tests have a higher mean in the final exam. This may be
because these students actually have higher cognitive skills or acquired
better programming skills, but might also hint at a selection bias, such that
only the highly motivated students participated (participation in the tests
was voluntary). There might also be a moderator effect, such that higher
cognitive skills lead to higher motivation in the course, because students
have an easier time to understand the course material. In future studies, we
hope that we can nudge more students to complete these tests, so that we can
differentiate the true effect of cognitive skills and motivation.

To conclude, even though the early warning system does not solve the
difficulties of teaching programming, it is still an important piece in the
puzzle to do so, because when detected early, it is possible to provide
support for struggling students to overcome their struggles and successfully
acquire programming skills.

Many thanks to our students who participated in our studies.

## References

  * (1)
  * Ahadi et al. (2014) Alireza Ahadi, Raymond Lister, and Donna Teague. 2014.  Falling Behind Early and Staying Behind When Learning to Program. In _Proceedings of the 25th Psychology of Programming Conference_ , Vol. 14. 77 – 88.
  * Ajideh and Mozaffarzadeh (2012) Parviz Ajideh and Sorayya Mozaffarzadeh. 2012.  A Comparative Study on C-Test vs. Cloze Test as Tests of Reading Comprehension.  _Journal of Basic and Applied Scientific Research_ 2, 11 (2012), 1159–11163.
  * Ambrósio et al. (2011) Ana Paula Ambrósio, Fábio Moreira Costa, Leandro Almeida, Amanda Franco, and Joaquim Macedo. 2011.  Identifying Cognitive Abilities to Improve CS1 Outcome. In _2011 Frontiers in Education Conference (FIE)_. IEEE, F3G–1–F3G–7.  https://doi.org/10.1109/FIE.2011.6142824
  * Becker et al. (2014) Nicolas Becker, Franzis Preckel, Julia Karbach, Nathalie Raffel, and Frank M Spinath. 2014.  Die Matrizenkonstruktionsaufgabe.  _Diagnostica_ (2014).  https://doi.org/10.1026/0012-1924/a000111
  * Bennedsen and Caspersen (2007) Jens Bennedsen and Michael E. Caspersen. 2007.  Failure Rates in Introductory Programming.  _SIGCSE Bull._ 39, 2 (2007), 32–36.  https://doi.org/10.1145/1272848.1272879
  * Brickenkamp et al. (2016) R Brickenkamp, L Schmidt-Atzert, and D Liepmann. 2016.  The d2 Test of Attention–Revised. A Test of Attention and Concentration.
  * Council of Europe. (2001) Council of Europe. 2001.  _Common European Framework of Reference for Languages (CEFR)_.  https://www.coe.int/en/web/common-european-framework-reference-languages/home
  * Crk et al. (2015) Igor Crk, Timothy Kluthe, and Andreas Stefik. 2015.  Understanding Programming Expertise: An Empirical Study of Phasic Brain Wave Changes.  _ACM Transactions on Computer–Human Interaction_ 23, 1, Article 2 (2015), 2:1–2:29 pages.
  * Denny et al. (2008) Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008.  Evaluating a New Exam Question: Parsons Problems. In _Proceedings of the Fourth International Workshop on Computing Education Research_ (Sydney, Australia) _(ICER ’08)_. Association for Computing Machinery, New York, NY, USA, 113–124.  https://doi.org/10.1145/1404520.1404532
  * Denny et al. (2011) Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011.  Understanding the Syntax Barrier for Novices. In _Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education_ (Darmstadt, Germany) _(ITiCSE ’11)_. Association for Computing Machinery, New York, NY, USA, 208–212.  https://doi.org/10.1145/1999747.1999807
  * Edwards et al. (2020) John Edwards, Joseph Ditton, Dragan Trninic, Hillary Swanson, Shelsey Sullivan, and Chad Mano. 2020.  Syntax Exercises in CS1. In _Proceedings of the 2020 ACM Conference on International Computing Education Research_ (Virtual Event, New Zealand) _(ICER ’20)_. Association for Computing Machinery, New York, NY, USA, 216–226.  https://doi.org/10.1145/3372782.3406259
  * Edwards et al. (2018) J. M. Edwards, E. K. Fulton, J. D. Holmes, J. L. Valentin, D. V. Beard, and K. R. Parker. 2018.  Separation of Syntax and Problem Solving in Introductory Computer Programming. In _2018 IEEE Frontiers in Education Conference (FIE)_. IEEE Computer Society, Los Alamitos, CA, USA, 1–5.  https://doi.org/10.1109/FIE.2018.8658852
  * Endres et al. (2021) Madeline Endres, Madison Fansher, Priti Shah, and Westley Weimer. 2021.  To Read or to Rotate? Comparing the Effects of Technical Reading Training and Spatial Skills Training on Novice Programming Ability. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 754–766.  https://doi.org/10.1145/3468264.3468583
  * Ericson et al. (2017) Barbara J. Ericson, Lauren E. Margulieux, and Jochen Rick. 2017.  Solving Parsons Problems Versus Fixing and Writing Code. In _Proceedings of the 17th Koli Calling International Conference on Computing Education Research_ (Koli, Finland) _(Koli Calling ’17)_. Association for Computing Machinery, New York, NY, USA, 20–29.  https://doi.org/10.1145/3141880.3141895
  * Floyd et al. (2017) Benjamin Floyd, Tyler Santander, and Westley Weimer. 2017.  Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise. In _2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)_. IEEE, Buenos Aires, 175–186.  https://doi.org/10.1109/ICSE.2017.24
  * Helmlinger et al. (2020) Birgit Helmlinger, Markus Sommer, Martina Feldhammer-Kahr, Guilherme Wood, Martin E Arendasy, and Silvia E Kober. 2020.  Programming Experience Associated With Neural Efficiency During Figural Reasoning.  _Scientific reports_ 10, 1 (2020), 1–14.  https://doi.org/10.1038/s41598-020-70360-z
  * Huang et al. (2019) Yu Huang, Xinyu Liu, Ryan Krueger, Tyler Santander, Xiaosu Hu, Kevin Leach, and Westley Weimer. 2019.  Distilling Neural Representations of Data Structure Manipulation Using FMRI and FNIRS. In _Proceedings of the 41st International Conference on Software Engineering_ (Montreal, Quebec, Canada) _(ICSE ’19)_. IEEE Press, 396–407.  https://doi.org/10.1109/ICSE.2019.00053
  * Institute for Test Research and Test Development Leipzig (2019) Institute for Test Research and Test Development Leipzig 2019.  _Vocabulary Tests_.  https://itt-leipzig.de/about-the-vocabulary-tests-2/?lang=en
  * Izu et al. (2019) Cruz Izu, Carsten Schulte, Ashish Aggarwal, Quintin Cutts, Rodrigo Duran, Mirela Gutica, Birte Heinemann, Eileen Kraemer, Violetta Lonati, Claudio Mirolo, and Renske Weeda. 2019.  Fostering Program Comprehension in Novice Programmers - Learning Activities and Learning Trajectories. In _Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education_ _(ITiCSE-WGR ’19)_. Association for Computing Machinery, New York, NY, USA, 27–52.  https://doi.org/10.1145/3344429.3372501
  * Keller et al. (2020) Stefan D. Keller, Johanna Fleckenstein, Maleika Krüger, Olaf Köller, and André A. Rupp. 2020.  English Writing Skills of Students in Upper Secondary Education: Results From an Empirical Study in Switzerland and Germany.  _Journal of Second Language Writing_ 48 (2020), 100700.  https://doi.org/10.1016/j.jslw.2019.100700
  * Kramer et al. (2019) Matthias Kramer, Mike Barkmin, and Torsten Brinda. 2019.  Identifying Predictors for Code Highlighting Skills: A Regressional Analysis of Knowledge, Syntax Abilities and Highlighting Skills. In _Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education_ _(ITiCSE ’19)_. Association for Computing Machinery, New York, NY, USA, 367–373.  https://doi.org/10.1145/3304221.3319745
  * Krieger et al. (2022) Florian Krieger, Nicolas Becker, Samuel Greiff, and Frank M Spinath. 2022.  _Design a Matrix - Standard_.  https://www.testzentrale.de/shop/design-a-matrix-standard.html
  * Kyaw et al. (2021) Htoo Htoo Sandi Kyaw, Nobuo Funabiki, Shune Lae Aung, Nem Khan Dim, and Wen Chung Kao. 2021.  A Study of Element Fill-In-Blank Problems for C Programming Learning Assistant System.  _International Journal of Information and Education Technology_ 11, 6 (June 2021), 255–261.  https://doi.org/10.18178/ijiet.2021.11.6.1520 Publisher: International Journal of Information and Education Technology.
  * Lee et al. (2016) SeolHwa Lee, Andrew Matteson, Danial Hooshyar, SongHyun Kim, JaeBum Jung, GiChun Nam, and HeuiSeok Lim. 2016.  Comparing Programming Language Comprehension Between Novice and Expert Programmers Using EEG Analysis. In _2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE)_. IEEE, 350–355.  https://doi.org/10.1109/BIBE.2016.30 ISSN: 2471-7819.
  * Lister (2011) Raymond Lister. 2011.  Concrete and Other Neo-Piagetian Forms of Reasoning in the Novice Programmer. In _Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114_ (Perth, Australia) _(ACE ’11)_. Australian Computer Society, Inc., AUS, 9–18.
  * Lister (2020) Raymond Lister. 2020.  On the Cognitive Development of the Novice Programmer: And the Development of a Computing Education Researcher. In _Proceedings of the 9th Computer Science Education Research Conference_ (Virtual Event, Netherlands) _(CSERC ’20)_. Association for Computing Machinery, New York, NY, USA, Article 2, 15 pages.  https://doi.org/10.1145/3442481.3442498
  * Luxton-Reilly (2016) Andrew Luxton-Reilly. 2016.  Learning to Program is Easy. In _Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education_. ACM, Arequipa Peru, 284–289.  https://doi.org/10.1145/2899415.2899432
  * Ly et al. (2021) Anna Ly, John Edwards, Michael Liut, and Andrew Petersen. 2021.  Revisiting Syntax Exercises in CS1. In _Proceedings of the 22st Annual Conference on Information Technology Education_ _(SIGITE ’21)_. Association for Computing Machinery, New York, NY, USA, 9–14.  https://doi.org/10.1145/3450329.3476855
  * Parsons and Haden (2006) Dale Parsons and Patricia Haden. 2006.  Parson’s Programming Puzzles: A Fun and Effective Learning Tool for First Programming Courses. In _Proceedings of the 8th Australasian Conference on Computing Education - Volume 52_ (Hobart, Australia) _(ACE ’06)_. Australian Computer Society, Inc., AUS, 157–163.
  * Pennington (1987) Nancy Pennington. 1987.  Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs.  _Cognitive Psychology_ 19, 3 (1987), 295–341.
  * Prat et al. (2020) Chantel S Prat, Tara M Madhyastha, Malayka J Mottarella, and Chu-Hsuan Kuo. 2020.  Relating Natural Language Aptitude to Individual Differences in Learning Programming Languages.  _Scientific reports_ 10, 1 (2020), 1–10.  https://doi.org/10.1038/s41598-020-60661-8
  * Román-González et al. (2017) Marcos Román-González, Juan-Carlos Pérez-González, and Carmen Jiménez-Fernández. 2017.  Which Cognitive Abilities Underlie Computational Thinking? Criterion Validity of the Computational Thinking Test.  _Computers in human behavior_ 72 (2017), 678–691.  https://doi.org/10.1016/j.chb.2016.08.047
  * Román-González et al. (2018) Marcos Román-González, Juan-Carlos Pérez-González, Jesús Moreno-León, and Gregorio Robles. 2018.  Can Computational Talent Be Detected? Predictive Validity of the Computational Thinking Test.  _International Journal of Child-Computer Interaction_ 18 (2018), 47–58.  https://doi.org/10.1016/j.ijcci.2018.06.004
  * Siegmund et al. (2014) Janet Siegmund, Christian Kästner, Sven Apel, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2014.  Understanding Understanding Source Code with Functional Magnetic Resonance Imaging. In _Proceedings of the 36th International Conference on Software Engineering_ (Hyderabad, India) _(ICSE 2014)_. Association for Computing Machinery, New York, NY, USA, 378–389.  https://doi.org/10.1145/2568225.2568252
  * Siegmund et al. (2017) Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister, Christian Kästner, Andrew Begel, Anja Bethmann, and André Brechmann. 2017.  Measuring Neural Efficiency of Program Comprehension. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ (Paderborn, Germany) _(ESEC/FSE 2017)_. Association for Computing Machinery, New York, NY, USA, 140–150.  https://doi.org/10.1145/3106237.3106268
  * Soloway et al. (1983) Elliot Soloway, Jeffrey Bonar, and Kate Ehrlich. 1983.  Cognitive Strategies and Looping Constructs: An Empirical Study.  _Commun. ACM_ 26, 11 (nov 1983), 853–860.  https://doi.org/10.1145/182.358436
  * Stewart et al. (2021) William H Stewart, Youngkyun Baek, Gina Kwid, and Kellie Taylor. 2021.  Exploring Factors That Influence Computational Thinking Skills in Elementary Students’ Collaborative Robotics.  _Journal of Educational Computing Research_ 59, 6 (2021), 1208–1239.
  * Sullivan et al. (2021) Shelsey Sullivan, Hillary Swanson, and John Edwards. 2021.  Student Attitudes Toward Syntax Exercises in CS1.  In _Proceedings of the 52nd ACM Technical Symposium on Computer Science Education_. Association for Computing Machinery, New York, NY, USA, 782–788.  https://doi.org/10.1145/3408877.3432399
  * Thurner et al. (2017) Veronika Thurner, Philipp Chavaroche, Axel Böttcher, and Daniela Zehetmeier. 2017.  A Concept for an Intelligent Tutoring System to Support Individual Learning Paths in Software Development Courses. In _International Conference on Interactive Collaborative Learning_. Springer, 774–783.
  * Vee (2013) Annette Vee. 2013.  Understanding Computer Programming as a Literacy.  _Literacy in Composition Studies_ 1, 2 (Oct. 2013), 42–64.  https://doi.org/10.21623/1.1.2.4