# A Systematic Literature Review of Empiricism and Norms of Reporting in
Computing Education Research Literature

Sarah Heckman  0000-0003-4351-8611 North Carolina State University
<EMAIL_ADDRESS>,  Jeffrey C. Carver  0000-0002-7824-9151 University of
Alabama<EMAIL_ADDRESS>,  Mark Sherriff  0000-0002-1745-205X University of
Virginia<EMAIL_ADDRESS>and  Ahmed Al-Zubidy  0000-0003-4305-2153
University of Alabama<EMAIL_ADDRESS>
(2021)

###### Abstract.

Context. Computing Education Research (CER) is critical to help the computing
education community and policy makers support the increasing population of
students who need to learn computing skills for future careers. For a
community to systematically advance knowledge about a topic, the members must
be able to understand published work thoroughly enough to perform
replications, conduct meta-analyses, and build theories. There is a need to
understand whether published research allows the CER community to
systematically advance knowledge and build theories.

Objectives. The goal of this study is to characterize the reporting of
empiricism in Computing Education Research literature by identifying whether
publications include content necessary for researchers to perform
replications, meta-analyses, and theory building. We answer three research
questions related to this goal: RQ1) What percentage of papers in CER venues
have some form of empirical evaluation? RQ2) Of the papers that have empirical
evaluation, what are the characteristics of the empirical evaluation? RQ3) Of
the papers that have empirical evaluation, do they follow norms (both for
inclusion and for labeling of information needed for replication, meta-
analysis, and, eventually, theory-building) for reporting empirical work?

Methods. We conducted a systematic literature review of the 2014 and 2015
proceedings or issues of five CER venues: Technical Symposium on Computer
Science Education (SIGCSE TS), International Symposium on Computing Education
Research (ICER), Conference on Innovation and Technology in Computer Science
Education (ITiCSE), ACM Transactions on Computing Education (TOCE), and
Computer Science Education (CSE). We developed and applied the CER Empiricism
Assessment Rubric to the 427 papers accepted and published at these venues
over 2014 and 2015. Two people evaluated each paper using the Base Rubric for
characterizing the paper. An individual person applied the other rubrics to
characterize the norms of reporting, as appropriate for the paper type. Any
discrepancies or questions were discussed between multiple reviewers to
resolve.

Results. We found that over 80% of papers accepted across all five venues had
some form of empirical evaluation. Quantitative evaluation methods were the
most frequently reported. Papers most frequently reported results on
interventions around pedagogical techniques, curriculum, community, or tools.
There was a split in papers that had some type of comparison between an
intervention and some other data set or baseline. Most papers reported related
work, following the expectations for doing so in the SIGCSE and CER community.
However, many papers were lacking properly reported research objectives,
goals, research questions, or hypotheses; description of participants; study
design; data collection; and threats to validity. These results align with
prior surveys of the CER literature.

Conclusions. CER authors are contributing empirical results to the literature;
however, not all norms for reporting are met. We encourage authors to provide
clear, labeled details about their work so that readers can use the study
methodologies and results for replications and meta-analyses. As our community
grows, our reporting of CER should mature to help establish computing
education theory to support the next generation of computing learners.

systematic literature review, empiricism, computing education research

††journal: TOCE††copyright: acmcopyright††journalyear: 2021††ccs: General and
reference Surveys and overviews††ccs: Social and professional topics Computer
science education

##  1\. Introduction

From 2009 to 2015, the number of bachelor’s degrees in Computer Science
increased by 74 percent, while overall growth across all fields of study only
rose by 16 percent (NAP24926). Taulbee’s 2019 (taulbee2019) report shows an
increase from a recent minimum of under 10,000 bachelor’s degrees conferred in
2009 to over 30,000 degrees in 2019 (see Figure 1B (taulbee2019)). It is
undeniable that there has been a rapid increase in interest in computer
science courses at higher-education institutions. We see similar demand in
K-12 with CSforAll111See https://www.csforall.org/ for more details.
initiatives. Educators have struggled with how to cope with new and
exacerbated challenges in computing education - ranging from how to scale up
to handle new enrollments to which teaching techniques are best during this
time of growth. The COVID-19 pandemic has added additional investigation into
the challenges of online instruction.

At the same time, attendance at the ACM SIGCSE Technical Symposium has
continued to increase year over year, coinciding with a similar increase in
the number of submissions (Hawthorne:2019SIGCSEReport;
Zhang:2020SIGCSETSRecap). Computing faculty are eagerly trying new teaching
methodologies to address the challenges they face with growth in enrollment
and with ever-emerging technologies. Other computing education research (CER)
venues are also seeing increases. The SIGCSE Board 222We use SIGCSE to
represent the ACM Special Interest Group in Computer Science Education as a
community of computing education researchers and practitioners. We use SIGCSE
Technical Symposium (SIGCSE TS) to represent the Technical Symposium on
Computer Science Education, which is commonly called SIGCSE. This
differentiates the community from the conference. added a fourth conference to
its yearly schedule to address the growing need for CS education in parts of
the world not served by existing conferences
(Impagliazzo:2018:SLN:3287087.3287088). There are many open questions in
computing education that the CER community must answer
(blikstein_moghadam_2019; kinnunen2010have).

While this growth in the CER community is exciting, the real benefit comes
when community members report innovations in a manner that allows other
researchers to build on them and educators to appropriately apply them in
their own contexts. For a community to systematically advance knowledge, its
members must be able to understand published work thoroughly enough to perform
replications, conduct meta-analyses, and build theories (bd12; fp04;
Ahadi2016Koli; mcgill-koli2019-replication; margulieux2019measurements;
Hao:2019TOCE:Replication; WWC2020; schmidt2009replication; NSF19002; gomez-
reser2010-replication; ihantola2015educational). In education literature, this
concept is referred to as the Scholarship of Teaching and Learning or SoTL.
SoTL is broadly defined as “posing [a problem] about an issue of teaching or
learning, study of the problem through methods appropriate to the disciplinary
epistemologies, application of results to practice, communication of results,
self-reflection, and peer review” (bd12).

In 2004, Fincher and Petre posited that most CER publications lack the type of
evidence or replications required for meta-analysis or theory building (fp04).
Clear also observed that many of the standard research practices used to
demonstrate rigor in “traditional”, non-human, computing topic areas are
ignored or even actively avoided when performing CER
(Clear:2006:VCS:1315803.1315806). These observations occurred just prior to
the transition to what Guzdial and du Boulay (guzdial_duboulay_2019) call the
“modern era” of CER in 2005 with the creation of the International Computing
Education Research Conference (ICER). Because this work was more than ten
years ago, there is a need to understand how well more recently published
research provides the type of information necessary for the computing
education community to systematically advance knowledge. Our work complements
recent work in understanding the quality of reporting in CER literature
(Hao:2019TOCE:Replication; ihantola2015educational; mcgill-
koli2019-replication; mcgill2018improving; margulieux2019measurements;
Ahadi2016Koli) and more broadly (WWC2020; NSF19002; Schulzc-
BMJ2010-ConsortStatement; apa-jars) to support replication.

Therefore, to provide insight into the current state of CER publications, the
goal of this study is:

To characterize the reporting of empiricism in Computing Education Research
literature by identifying whether publications include content necessary for
researchers to perform replications, meta-analyses, and theory building.

To accomplish this goal, we first defined the CER Empiricism Assessment Rubric
for analyzing CER papers. The rubric identifies the information necessary for
replication, meta-analysis, and theory building. We then applied the rubric to
427 papers published in the Technical Symposium on Computer Science Education
(SIGCSE TS), International Symposium on Computing Education Research (ICER),
Conference on Innovation and Technology in Computer Science Education
(ITiCSE), ACM Transactions on Computing Education (TOCE) and Computer Science
Education (CSE) during 2014 and 2015 to categorize published work and identify
whether information needed for replication is present and clearly labeled. We
chose these years because they were the most recent editions of the conference
when we began our research. In addition, these years coincide with ten years
after the first ICER (ICER14-Proceedings; ICER15-Proceedings), a conference
focused on empirical CER and just as the number of empirical research papers
published at the SIGCSE TS began to increase. Therefore, the results of this
analysis will serve as a baseline against which we can compare the results of
a similar analysis in subsequent years.

The contributions of this paper are:

  * •

The CER Empiricism Assessment Rubric for evaluating the completeness of the
empirical content of CER papers;

  * •

An analysis of the empiricism present in CER papers published during 2014 and
2015 in five CER venues;

  * •

A baseline for future analyses; and

  * •

Overall observations that serve as suggestions for how the CER community can
advance scientific reporting standards.

##  2\. Related Work

We explore related work in three ways: 1) CER literature reviews and community
reflection; 2) guidelines for reporting empirical work, particularly
educational work, outside of computing; and 3) an overview of recommendations
for replications. These views of the related literature show the current
efforts in transforming and increasing the impact of CER results and transfer
into practice that support learners at a variety of levels and needs. We close
the section with a discussion on where our work fits into the broader
literature.

###  2.1. CER Literature Reviews and Community Reflection

There have been many reviews of computing education literature, especially
over the last 20 years. holmboe2001research and Clear:2006:VCS:1315803.1315806
provide early reflections on CER and they note the biases in the broader
research community, including computing, against (computing) education
research as a distinct and robust research area (holmboe2001research;
Clear:2006:VCS:1315803.1315806). Supporting the growth and recognition of CER
and researchers through literature reviews and community reflection continues
the progress made by the CER community in the last 15-20 years.

For this section, our focus is on reviews that categorize or evaluate CER
papers and reporting quality, including the use of theories and measurements.
We do not include systematic literature reviews on specific topics in
computing education if they do not also include some discussion on the quality
of or challenges in aggregation due to reporting (e.g., reviews of K-12
(garneli2015computing; szabo2019fifteen); introductory programming
(pears2007survey; medeiros2018systematic; vihavainen2014systematic); teaching
assistants (mirza2019undergraduate); tools, languages, environments in K-12
(mcgill2020construction); meta-cognition and self-regulated learning
(prather2020we); and mapping of theories in CER (szabo2019review)). Table 1
summarizes the reviews that include categorization or evaluation of CER papers
from a quality and reporting perspective. For each review, we provide the
reference information, area of focus, the years and venues considered, the
number of papers reviewed, and a summary of inclusion/exclusion criteria
beyond the area of focus. The text below also includes non-systematic
literature review papers; however, Table 1 does not list these papers. We
organize the discussion in each subsection in chronological order by
publication date to highlight the changes in reporting since the early 2000s.

Table 1. Summary of CER Reviews Reference |  Area of Focus |  Years |  Venues |  # Papers |  Inclusion Criteria
---|---|---|---|---|---
valentine2004cs (valentine2004cs) |  First-year CS courses including CS1 and CS2 |  1984-2003 |  SIGCSE TS |  444 |  CS1, CS2, and CS1/CS2 topics
simon2007koli (simon2007koli) |  Computing education paper classification |  2001-2006 |  Koli |  102 |  full papers
randolph2008methodological (randolph2008methodological) |  methodological properties of reported research |  2000-2005 |  SIGCSE Bulletin, CSE, JCSE, Koli, SIGCSE TS, ITiCSE, ICER, ACE |  352 |  stratified random sample from all articles from listed venues between given years
carbone2008classifying (carbone2008classifying) |  computing education paper classification |  2005-2007 |  ICER |  43 |  full papers
simon2008eight (simon2008eight) |  computing education paper classification |  2000-2007 |  NACCQ |  157 |  full papers accepted to CompEd track
simon2009ace (simon2009ace) |  computing education paper classification |  1996-2008 |  ACE |  328 |  full papers
joy2009categorising (joy2009categorising) |  categorizing types of CER literature |  2004 and/or 2005 |  Too many to list |  3598 |  one proceeding/volume from conference/journal
sheard2009analysis (sheard2009analysis) |  teaching and learning of programming |  2005-2008 |  ICER, ITiCSE, SIGCSE TS, ACE, Koli, NACCQ |  164 |  programming-focused paper classified as an experiment, study, or analysis
kinnunen2010have (kinnunen2010have) |  didactic-focus-based categorization |  2005-2009 |  ICER |  67 |  focus on instructional process
malmi2010characterizing (malmi2010characterizing) |  CER research processes |  2005-2009 |  ICER |  72 |  full papers
malmi2014theoretical (malmi2014theoretical) |  theories, conceptual models, frameworks of CER literature |  2005-2011 |  ERIC/TOCE, CSE, ICER |  308 |  peer-reviewed full papers
lishinski2016methodological (lishinski2016methodological) |  use of educational theory empirical results & methodological rigor |  2012-2015 |  CSE, ICER |  136 |  full papers
ihantola2015educational (ihantola2015educational) |  educational data mining and learning analytics |  2005-2015 |  SIGCSE TS, ICER, ITiCSE, TOCE, CSE, ACE, EDM, JEDM, Koli, PPIG, TLT |  76 |  open-ended programming problems, programming process, automated data collection and analysis, length >3 pages
al2016updated (al2016updated) |  empircism in CS education |  2014-2015 |  SIGCSE TS |  162 |  full papers with empirical results
luxton2018introductory (luxton2018introductory) |  introductory programming |  2003-2017 |  ACM full-text collection, IEEE Explore, Science Direct (Elsevier), SpringerLink, Scopus |  1666 |  introductory programming courses student in computing degrees
mcgill2018improving (mcgill2018improving) |  status of reporting of K-12 program elements to support replication |  2012-2016 |  SIGCSE TS, ICER, TOCE |  92 |  primary and secondary education, research report or experience report, pre-college computing activities
margulieux2019measurements (margulieux2019measurements) |  standardization of measurement in CER |  2013-2017 |  CSE, TOCE, ICER |  197 |  human-subjects research that measured variables excluded reviews, evaluation papers, case studies, measurement validation studies
sanders2019icer (sanders2019icer) |  inferential statistics |  2005-2018 |  ICER |  270 |  full papers
decker-sigcsets2019-instruments (decker-sigcsets2019-instruments) |  research-based evaluation instruments |  2012-2016 |  csedreesarch.org, ICER, TOCE, CSE, Americal Evaluation Association, STELAR The Pear Institute Institute for the Integration of Technology into Teaching and Learning, MSPNet, Engineering is Elementary |  47 |  research-based evaluation instruments available in publications or public websites and databases
Hao:2019TOCE:Replication (Hao:2019TOCE:Replication) |  replications in CER literature |  2009-2018 |  SIGCSE TS, ICER, ITiCSE, TOCE, CSEJTLT |  54 |  articles including replicat[a-z]* manual review for replications
simon2020ace (simon2020ace) |  computing education paper classification |  1996-2020 |  ACE |  541 |  full papers
malmi2020theories (malmi2020theories) |  theoretical constructs and instruments for emotion, attitude, belief, and self-efficacy in computing learners |  2010-2019 |  ICER, TOCE, CSE, LAK, Scopus search |  50 |  computing domain containing theoretical constructs, including statistical models
papamitsiou2020computing (papamitsiou2020computing) |  keyword analysis of CER literature |  2005-2019 |  ITiCSE + WGR, ICER |  1274 |  full papers and working group reports
sheard2020twenty (sheard2020twenty) |  computing education paper classification |  1996-2019 |  ITiCSE + WGR |  1295 |  full papers and working group reports
This paper [2021] |  quality of empirical reporting in CER |  2014-2015 |  SIGCSE TS, ICER, ITiCSE, TOCE, CSE |  427 |  full papers

####  2.1.1. CER Topics in Literature

We start with an overview of how authors categorized topics in CER
publications.

The seminal book by fp04 (2004) identified 10 core areas for CER and provided
an initial definition of the practices and methods of the field (fp04). Their
work laid the foundation for CER in the next 15 years. pears2005constructing
(2005) created a taxonomy of four key areas in CER that builds on and groups
Fincher’s and Petre’s (fp04) initial categorization. The categories included
studies in teaching and learning; institutions and educational settings;
problems and solutions; and CER as a discipline (pears2005constructing). From
these categories, pears2005constructing created a core CER literature
including influential, seminal, and synthesis work.

joy2009categorising (2009) created a taxonomy to categorize the types of CER
publications in 21 journals and 21 conference proceedings from either 2004 or
2005. Their findings demonstrate that conference venues tend to have a
technical focus to their proceedings while journals have a pedagogical focus.

Some survey papers identified gaps in the topics covered by CER literature.
kinnunen2010have (2010) reviewed 67 ICER papers between 2005 and 2009. Their
categorization considered eight categories based on a three-layered didactic
structure including students, teachers, and goals/context at a classroom,
organization, and societal level. Expanding their corpus of reviewed papers by
13 and considering venues beyond ICER (e.g., ACE, SIGCSE TS, PPIG, ITiCSE,
ITiCSE WGR, CSE, Comput. Small Coll.), they found papers reported results at
the course-level and focused on student characteristics and process. Eight
papers included additional focus areas of students’ conceptions on and actions
to achieve course goals and content. The literature at that time did not
provide adequate coverage of categories around content/goals and teachers.

Simon and others (2007-2020) have conducted extensive classification studies
of computing education literature for ACE (simon2009ace; simon2020ace), ICER
(carbone2008classifying), ITiCSE (sheard2020twenty), Koli (simon2007koli), and
NACCQ (simon2008eight). The classification scheme considers the context (e.g.,
subject matter or course); theme (e.g., what the paper is about like teaching
technique); scope (e.g., extent of collaboration and work); and nature (e.g.,
distinction between research and practice) (simon2007classification). A recent
classification of ITiCSE papers in 2020 found an increase in reported research
over experience reports and that teaching and learning techniques remain a
common theme in published work (sheard2020twenty).

papamitsiou2020computing (2020) categorized papers from 15 years of ITiCSE,
ITiCSE Working Group Reports (WGR), and ICER using keywords and abstracts. The
main themes identified are around introductory programming, assessments, and
student performance.

The categorization of topics covered in the literature provides guidance to
researchers and practitioners about areas that are well studied and ripe for
meta-review and gaps where additional work is needed for a better
understanding of computing education. For example, “curriculum” is a common
categorization in many of the above surveys, however, they each have their own
nuance (fp04; joy2009categorising; kinnunen2010have; simon2007classification;
papamitsiou2020computing). While our work did not identify the context of the
reviewed study (e.g., CS1, databases, etc.), we do consider the subject of the
evaluation (e.g., curriculum or tool) which has overlap with categorizations
from related work.

####  2.1.2. Classification by Study Type

Several studies have classified CER literature by study type, which could
include high-level categories of quantitative and qualitative or more granular
categories like experiments, quasi-experimental studies, and experience
reports.

In one of the oldest CER surveys, valentine2004cs (2004) examined 444 CS1/CS2
papers from SIGCSE TS between 1984 and 2003 and found that only 21% included
experimental evaluation or were experience reports. Other categories
considered were Marco Polo (“I went there and I saw this”), Philosophy, Tools,
and John Henry (“outrageously difficult”) papers. There was an increase in the
number of experimental papers presented in the last 10 years of the study
period.

randolph2008methodological (2008) found that of 352 papers published in
various CER venues between 2000 and 2005, 40% contained only anecdotal
evidence and of the less than one-third of papers that did have experimental
or quasi-experimental designs, 54.8% of those papers used a weaker post-test
only design. randolph2008methodological categorized their sampled papers
against valentine2004cs’s categorization and found that 40.9% of sampled
papers were experimental or experience reports. Further,
randolph2008methodological used their own categorization of research
methodologies and found that of the papers that reported human subject
research (n = 144), 64.6% were experimental or quasi-experimental, 26.4% were
qualitative, 18.1% were causal comparative, 10.4% were co-relational, and 7.6%
were survey research.

malmi2010characterizing (2010) identified two dimensions that described the
type of research: purpose and framework. They found that 86% of papers had an
evaluative purpose and that 79% of papers reported a research framework. The
most common research frameworks were survey (39%), experimental (15%),
constructive (14%), and grounded theory (13%).

ihantola2015educational (2015) conducted a survey on educational data mining
research published at various venues between 2005 and 2015. Of the 76 papers
that met their inclusion criteria, 78% described studies in a natural setting.
Only 14% reported on formal experimental research.

al2016updated (2016) found that 162 (70%) of the papers reviewed had some form
of empirical evaluation. Their definition for empirical studies was broader
than the definitions used by valentine2004cs and randolph2008methodological.
However, the detailed evaluation type numbers show that 28% of the papers were
experimental, suggesting some increase over earlier surveys.

lishinski2016methodological (2016) found that 71% of CSE and 87% of ICER
papers between 2012 and 2015 reported empirical results and that 26% from CSE
and 19% from ICER were experimental as defined by randolph2008methodological.
The percentages of experimental work were much lower when considering the more
specific definition of single group post-test only
(lishinski2016methodological).

Early surveys of CER literature (valentine2004cs; randolph2008methodological)
reported low rates of empirical work in the surveyed papers. More recent
surveys (e.g., the last 10 years) of CER literature have shown an increase of
empirical work published at a variety of CER venues. While two of the surveys
reviewed literature from the CER-focused venues of ICER and CSE
(malmi2010characterizing; lishinski2016methodological), others surveys did
consider other venues (al2016updated; ihantola2015educational) suggesting the
increase in empiricism is happening more broadly in the computing community.
However, with differences in the definitions of empirical work between
surveys, direct comparisons cannot be made.

####  2.1.3. Theory

Many CER surveys discussed the use and creation of theory.

malmi2010characterizing (2010) found that 60% of the surveyed papers reported
explicit use of theories, models, frameworks, and instruments (TMFI) in their
studies. Additionally, they found a great diversity of TMFI used during that
time frame – 68 distinct resources out of 78 instances.

Tenenberg’s (Tenenberg2014TOCE) (2014) position paper argues for the
importance of recognizing the theories and theoretical frameworks that
underlie CER research. His argument is that authors should recognize the
theoretical frameworks utilized when creating research questions because
frameworks may introduce limitations on the types of questions asked and the
methods used to answer research questions. Including the theoretical stance of
the authors is an important aspect of reported works and serves as a
foundation for future CER.

malmi2014theoretical (2014) conducted a deeper review of CER literature that
considered additional years and venues to identify the use of theories,
models, and frameworks in the literature. They found that 51% of papers
described at least one of 216 distinct theories, models, or frameworks. The
interdisciplinary nature of CER benefits from the use of prior theoretical
work, especially from outside fields. However, the disparate usage of theory,
models, and frameworks is a challenge for creating a “stable theoretical base”
for CER.

lishinski2016methodological (2016) reviewed CSE and ICER literature between
2012 and 2015 to identify the use of outside educational and learning theory
and the methodological quality of the research using indicators that build on
prior work (e.g., (randolph2008methodological; malmi2014theoretical)). They
identified an increase in the use of theory to support research and in the
reporting of empirical CER work. However they found many studies utilized less
rigorous methods.

nelson2018use (2018) summarize the use of theory in CER and identify three
areas of concern in the community including tensions between explanation goals
and design goals, lack of work on domain-specific theories, and publication
bias due to theoretical lens of the work. They also provide concrete
suggestions to move the community forward by focusing on design and using
theory as a guide, investing in the creation of CER-specific theories, and
reducing reviewer bias in regards to manuscripts with novel designs where
theory may not be appropriate.

malmi2020theories (2020) reviewed 50 papers with theoretical constructs around
the topics of emotions, attitudes, beliefs, and self-efficacy of computing
learners from a variety of venues between 2010 and 2019. They found that
three-quarters of the papers were published between 2014 and 2019 suggesting a
maturation of the CER field in using theory and validated instruments. More
recent papers on theoretical constructs relied on quantitative methods rather
than qualitative.

There are theoretical underpinnings to CER literature and authors utilize work
from outside of computing to support their research questions. More recent
years have shown an increase in the use of theory to support the increased
empirical work in CER. However, the development of CER theories is an open
area of future work as the field matures. While we do not consider theory in
our review of CER literature, there are enough surveys on the topic to include
a discussion here for completeness.

####  2.1.4. Methods & Analysis

Other reviews examined the published literature to identify the type of
research methods and analysis techniques used in CER studies.

randolph2008methodological (2008) found that of the 144 studies that utilized
human participants, 107 (74.3%) used quantitative methods. Only 15.3% of the
studies used qualitative methods and 10.4% of studies used mixed methods. The
authors found that 44 of the 352 papers reviewed used some form of inferential
statistics. Additionally, 120 of the 123 of the behavioral, quantitative, and
empirical studies reported effect size.

sheard2009analysis (2009) conducted a review of papers on the teaching and
learning of programming, published in various CER venues between 2005 and
2008, which they classified as empirical. Of the 164 papers reviewed, 37%
reported quantitative results, 21% reported qualitative results, and 45%
utilized mixed methods.

malmi2010characterizing studied ICER papers published between 2005 and 2009
found a variety of analysis methods used. These methods included statistical
analysis (42%), exploratory statistical analysis (17%), descriptive statistics
(11%), interpretive qualitative analysis (35%), and interpretative
classification or content analysis (26%).

al2016updated (2016) reported that 54% the papers with empirical results in
SIGCSE TS 2014 and 2015 proceedings utilized surveys and 37% utilized
experimental methods, which were typically quantitative. Very few papers
utilized qualitative methods like observations. margulieux2019measurements
(2019) found 32% of papers published in TOCE, CSE, and ICER between 2013 and
2017 collected both qualitative and quantitative data and over half used
multiple measures.

sanders2019icer reviewed the use of inferential statistics in ICER papers
published between 2005 and 2018. They found that 51% of papers used
inferential statistics. However, they noted that the reporting associated with
inferential statistics tended to lack precise test names, confidence levels,
and p-values for results. Other noted concerns with statistical test reporting
included a lack of detail about data preparation, missing discussion about the
assumptions for statistical tests, lack of explanation for “obscure” tests,
lack of corrections for multiple statistical tests, and lack of statistical
significance and effect size discussions.

The literature suggests that published CER work tends to use quantitative
methods, but there is movement towards using multiple measures and appropriate
statistical tests when answering research questions.

####  2.1.5. Participants & Context

Several reviews categorized how CER literature reports on the participants of
the research study and the context of the intervention(s).

ihantola2015educational (2015) reported that 34% of the educational data
mining studies in their survey did not report any details about course
context, which could include the course level, programming language, and
topics. Additionally, 17% of the studies did not report the number of students
in the study, details about student level, or demographics.

al2016updated (2016) found that 11% of papers did not report the number of
participants in their study. Additionally, when trying to identify the number
of participants from the papers, there were discrepancies between reviewers
due to inconsistent reporting.

mcgill2018improving (2018) reviewed 92 papers from SIGCSE TS, ICER, and TOCE
between 2012 and 2016 related to pre-college computing activities and how well
papers reported key information to support future replication. One area
studied was the activity component data or information about the activities
and context of the intervention. They found that papers infrequently reported
learning outcomes (25%), curriculum (33%), number of students in the activity
(41.7%), and details about the duration of the activity within the larger
context (78.6%), but many did not list the number of contact hours, which is
important in K-12 work. They additionally found the reporting of both
instructor and student demographic data lacking. The authors provided a
checklist of recommendations to support better, and more consistent, reporting
of subject and context data to support replication.

luxton2018introductory (2018) found that a “large proportion” of publications
related to introductory programming did not provide sufficient contextual
details about the reported study, which increases the difficulty for readers
to determine transferability of the results. They recommend that authors
report details about the population of students and teaching context.

margulieux2019measurements (2019) identified several empirical reporting best
practices that are lacking in CER, particularly related to the context of the
study. While 85% of papers reviewed reported the number of participants only
49% reported additional participant characteristics (e.g., prior knowledge and
basic demographics). Other contextual information that would be beneficial to
include are details about the task and learning environment.

Literature surveys that considered the context and participants in reviewed
research found gaps or inconsistencies in reporting related to these items.
Several reviewed papers excluded contextual information about the classroom or
environment of the study (mcgill2018improving; ihantola2015educational;
luxton2018introductory; margulieux2019measurements). Several studies found
that between 10-20% of papers did not report the number of participants
(ihantola2015educational; al2016updated; margulieux2019measurements).

####  2.1.6. Data Sources

Several studies reported on the source of the data collected, including the
use of validated and non-validated instruments.

randolph2008methodological (2008) reported on the independent, dependent, and
mediating/moderating variables in 123 behavioral, quantitative, and empirical
articles in their sample of CER literature. The most common independent
variable was student instruction (98.9%). The most common dependent variables
were attitudes (60.2%) and achievement in computer science (56.1%). Only 29%
of papers used mediating or moderating variables, including gender, grade
level, and learning styles. The identification of measures showed that
questionnaires were the most commonly used (52.5%) followed by grades (29.3%),
teacher- or researcher-made tests (22.0%), and student work (17.9%).

sheard2009analysis (2009) found that empirical papers from a variety of CER
venues utilized formal course assessments (42%), tasks students complete
(38%), and questionnaires (33%) as the top three data gathering techniques.
The authors note that the use of “established measurement instruments” was low
in reviewed studies and that publications lacked details about how the
instruments were used.

malmi2010characterizing (2010) classified the data sources from ICER papers as
(1) naturally occurring, (2) research specific data, (3) reflection, or (4)
software. The results showed 79% of those papers used research specific data,
i.e. data collected specifically for the needs of the research through
interviews, questionnaires, observational data, assignments, or tasks.

al2016updated (2016) found that most SIGCSE TS papers that used empirical
evaluation reported on pedagogical techniques followed by courses and
curriculum. Additionally, they found that over 75% of authors reported on new
subjects specific to the paper. There was little replication or even reuse of
data, even from the authors’ previous studies.

ihantola2015educational (2016) found in their survey of educational data
mining literature that 81% of studies reported work from a single institution,
80% of studies did not consider longitudinal data, and 66% of studies only
considered a single course. Additionally, they considered methods and analysis
from several perspectives including details about the tasks students performed
as part of the study, the type of data collected (e.g., logging, key-stroke),
and the analysis methods. They found several gaps in reporting for each of
these.

decker-sigcsets2019-instruments (2019) consider evaluation instruments as a
key data source in CER literature. They found 47 evaluation instruments that
measured cognitive, non-cognitive, and program evaluation constructs that are
useful for CER researchers and would complement other quantitative and
qualitative measures. They categorized the instruments based on number of
items, type of items, target demographic, reliability, and validity as
reported in the literature.

The sources of data for CER work covers a variety of measures from student
grades, attitudinal surveys, key-logging, and other automated data collection.
A variety of evaluation instruments are available to measure various
constructs (decker-sigcsets2019-instruments). Several of the literature
surveys found that data was created specifically for the research study
(malmi2010characterizing; al2016updated) and focused on a single class or
institution (ihantola2015educational). Many surveys noted that reviewed
literature lacked details in reporting on the data collected and that reuse of
data was low (ihantola2015educational; al2016updated).

####  2.1.7. Comparisons & Replications

Several recent papers consider the impact of reporting quality on the
possibility for comparisons between and replications of CER work.

ihantola2015educational (2015) identified 5 studies (7% of their selected
papers) as replication studies in the domain of educational data mining.
al2016updated (2016) found that 46% of papers reviewed had no comparison point
as part of the study. The remaining papers either had a comparison to
historical data or a comparison to a data set created specifically for the
study. Additionally, fewer than six papers each year of the review (2014 and
2015) reported replications of prior work (al2016updated).

Ahadi2016Koli (2016) surveyed 73 CER researchers about their perspectives on
the value of replication work in the community. They found challenges related
to the language used to describe replication/reproducibility, bias and
incentives towards original work, and a lack of community value on replication
work. An example of a language challenge is related to the definitions we use
for replication. A survey quote suggests that replications are challenging due
to the highly-contextualized nature of the research environment. The authors
further suggest that terminology is used inconsistently in the field
(Ahadi2016Koli). However, a replication in a new environment is a conceptual
replication and can help generalize the work (NSF19002;
schmidt2009replication).

mcgill-koli2019-replication (2019) summarizes current work on replication,
reproducibility, and meta-analysis in CER and provides a call to action on how
to improve the field. The first suggestion is to improve individual studies
because replication, reproducibility, and meta-analysis rely on high quality
reporting. Other suggested actions are to pre-register studies, support open
science, invest in large-scale collaborative research, report power analysis
and effect size, create systems to store data and research tools, and
incentivize replication through community support.

margulieux2019measurements (2019) found that while CER literature has adopted
measurement instruments outside of computing and has created computing-
specific instruments, most papers do not use standardized instruments to
measure variables of interest. Use of standardized and validated instruments
increases the reliability and validity of study results and will support meta-
analysis by providing a common measure to compare across studies.

Hao:2019TOCE:Replication (2019) completed a systematic literature review on
replications in the CER community (including SIGCSE TS, ICER, ITiCSE, TOCE,
CSEJ) between 2009 and 2018. Of the 2,269 articles, only 54 (2.38%) were
considered a replication (as defined by schmidt2009replication). Of the 54
replications, 63% were successful replications. The others reported failures
or mixed results. Three-quarters of the replications were conceptual (e.g.,
methods varied from the original study), while the remaining replications were
direct (e.g., methods were as similar as possible to the original study).

CER literature does contain replications and comparisons, but replication
studies are a small percentage of published CER work
(Hao:2019TOCE:Replication). The CER community views direct replications as
challenging, with most replication studies being conceptual replications
(Ahadi2016Koli; Hao:2019TOCE:Replication; schmidt2009replication; NSF19002).
Comparisons and replications can be supported with the use of validated
instruments (margulieux2019measurements; decker-sigcsets2019-instruments) and
other open science practices can support replications (mcgill-
koli2019-replication).

####  2.1.8. Gaps in Reporting

Many literature reviews included a discussion about gaps in reporting of the
literature surveyed. These gaps are places where the literature review was
more challenging because papers were missing key pieces of information.

randolph2008methodological (2008) reviewed 352 papers randomly selected from
all papers published in several computing education venues between 2000 and
2005. As part of their methodological review, they identified whether papers
contained elements considered important by the American Psychological
Association (apa-jars), which include items like an abstract, research
questions, and a description of participants. They found that less than 50% of
the sampled papers reported purpose/rationale (36.6%), research
questions/hypotheses (22.0%), participants description (45.5%), procedures
(37.4%), and separate results and discussion (29.3%).

sheard2009analysis (2009) report on a lack of connection between findings
related to the process of learning programming and relevant theories and
models of learning. malmi2010characterizing (2010) found it challenging to
identify the theories, models, frameworks, and instruments utilized in the
papers reviewed with similar challenges in other dimensions considered during
the review.

ihantola2015educational demonstrated the challenges to re-analysis,
replication, and reproduction resulting from a lack of context and details in
existing literature through three case-studies. Additionally, their analysis
of quality measures on reviewed papers found that reporting related to
confounding factors, threats to validity, and ethical issues were lacking.
Some survey responses from Ahadi2016Koli’s (2016) work suggests that
replications can be difficult due to gaps in the reporting of a research
study.

al2016updated (2016) identified several areas where gaps in reporting lead to
challenges when reviewing the literature particularly with regard to lack of
replication and inconsistent paper organization. In particular, they found
that between 40% of 70% of papers in the years evaluated from the SIGCSE TS
were lacking threats to validity.

lishinski2016methodological (2016) found that 47% of CSE and 56% of ICER
papers during their study timeframe reported explicit research questions.
While this level of reporting is an improvement over the use of research
questions as reported by randolph2008methodological, the lack of explicit
research questions in some empirical studies is a concern.
lishinski2016methodological’s finding about research questions are lower than
the 79% reported by ihantola2015educational possibly due to the focus on
educational data mining and learning analytics.

mcgill2018improving (2018) found many gaps in reporting of pre-college
activities related to the study context and the demographics of instructors
and students. They provide a checklist with recommendations that would also be
useful for education researcher more broadly.

luxton2018introductory (2018) found in their review of introductory
programming that many papers lacked details about the construct/intervention
and its operationalization and effect sizes, which limits replication. They
suggest archiving course information including “syllabus, learning outcomes,
infrastructure, teaching approaches, and population demographics…”.

Prior work found several gaps in reporting quality
(randolph2008methodological; ihantola2015educational; al2016updated;
lishinski2016methodological), lack of connection between results and theories
(sheard2009analysis; malmi2010characterizing), and lack of contextual
information (ihantola2015educational; mcgill2018improving;
luxton2018introductory). Open science practices, like archiving research study
information, could supplement the details in the literature
(luxton2018introductory; mcgill-koli2019-replication).

###  2.2. Guidelines for Reporting and Quality

Several organizations provide guidelines for reporting and assessing study
quality. We encourage interested readers to review these resources to consider
additional quality expectations when designing and reporting on empirical CER
studies.

The What Works Clearinghouse (WWC) has developed a Standards Handbook
(WWC2020) that provides resources for a systematic review process, including
the development of a quality rubric, to provide summary reports about
empirical educational research that can be of use to policymakers. The
structured review is intended to assess the internal validity of the study.
WWC focuses on four types of research: randomized controlled trials, quasi-
experimental design, regression discontinuity design, and single-case design.
Studies related to a particular area of interest and quality rubric provide
one of three labels: meets WWC Design Standards without reservations, meets
WWC Design Standards with reservations, does not meet WWC Design Standards.
For example, quasi-experimental designs which lack randomization can only
achieve the category “meets WWC Design Standards with reservations” due to the
lack of randomization in the process. The WWC Design Standards provide a
deeper quality assessment, specifically around the methods, results, and
internal validity of educational research studies than our rubric on reporting
norms.

The CONSORT 2010 Statement (Schulzc-BMJ2010-ConsortStatement) provides
guidelines and a checklist for reporting parallel group randomized trials.
While the authors of CONSORT do not prescribe a specific article structure,
they do suggest using subheadings to help readers find information in the
manuscript. Appropriate subheadings provide readers with guideposts on where
to find important information and can be especially useful for supporting
possible replications.

The American Psychological Association (APA) provides Journal Article
Reporting Standards (JARS) for researchers, reviewers, and editors (apa-jars).
The website provides guidelines for reporting standards on quantitative,
qualitative, and mixed research publications and study designs. It also
provides guidance on meta-analysis reporting standards.

The American Education Research Association (AERA) Standards for Reporting on
Empirical Social Science Research in AERA Publications provides guidance for
reporting on empirical educational research (area-empirical-social-
science-2006). Guidance includes a clear statement of the problem, description
of the study design, overview of the data collection and sources of evidence,
a description of key measurements and how they are operationalized by the
study, details of analysis procedures, scope, and ethics.

Within the computing education community, several articles provide guidance on
high quality reporting of CER work. daniels2012models provide a framework for
designing action research to support answering research questions about
“concrete teaching and learning challenges” in computer science classrooms.
Their framework provides guidance on reporting the researcher’s epistemology
and theoretical perspective(s), describing the context of the research study,
and describing the methodology and methods associated with the research
question. mcgill-fie2018-cer-repository describe the challenges in creating a
repository for K-12 CER research due to inconsistencies in reporting standards
and questions about study quality identified by a focus group gathered to
create repository requirements. Some of the findings from the focus group
suggest that reporting on basic technical components would support
replication. The www.csedresearch.org website serves as a repository for
instruments and guidelines for reporting on K-12 studies, which builds on
other work by McGill, Decker, and others (mcgill2018improving). As part of
their systematic literature review, ihantola2015educational created a quality
assessment rubric for selected studies that reported on educational data
mining and learning analytics.

Researchers in related fields which also consider human subjects, like
software engineering, have proposed reporting guidelines for empirical work.
runeson2009guidelines describes guidelines for reporting case studies,
including suggested headings and subsections. The work (runeson2009guidelines)
compares case study reporting guidelines to experimental reporting guidelines
created by jedlitschka2005reporting and refined by kitchenham2008evaluating.
Additionally, kitchenham2008evaluating provides checklists that researchers,
practitioners, meta-reviewers, replicators, and reviewers should consider when
reading experimental work. Carver-RESER2010-ReportingGuidelines provides
guidelines for reporting experimental replications in software engineering
that can also be useful as general reporting guidelines for empirical work. As
part of reporting on a replication, the original study should be discussed,
including the research question(s), participants and their characteristics and
context, experimental design, artifacts or resources used in the study,
context variables, and summary of the results as general reporting guidelines
for empirical work.

###  2.3. Guidelines for Replication

Our focus on reporting quality is in the goal of supporting replications,
meta-analysis, and theory-building in the CER community. This section
describes key references on replications in CER and more broadly.

schmidt2009replication provides a pragmatic and operational overview of
replication, particularly in the social sciences. Their definitions of direct
replication and conceptual replication form the foundation of the discussion
of replication and capture our views of replication. Replications provide
additional confirmatory power about a research question. Schmidt suggests five
functions of replication: “to control for sampling error, to control for
artifacts, to control for fraud, to generalize results to a larger or to a
different population, and to verify the underlying hypothesis of the earlier
experiment.” While schmidt2009replication provides a framework for
replications and reproductions, he additionally provides a pragmatic
reflection on practical publications of reflections given community biases
towards original work by using follow-up studies and systematic replications.
Follow-up studies may include direct replication of earlier work and
additional experimental conditions that either provide generalizability or
novelty beyond the replicative piece. Systematic replications provide a way of
exploring the variation around a research replication modeled as a data
matrix. Replications consider different cases within the matrix to
systematically explore the research space.

The National Science Foundation (NSF) and the Institute of Education Sciences
(IES) issued joint Companion Guidelines on Replication & Reproducibility in
Education Research (NSF19002) as a supplement to the Common Guidelines for
Education Research and Development (NSF13126). The Companion Guidelines define
reproducibility and replication and describe the importance of both in
furthering theory in education. The guidelines for designing reproducible
studies suggest that “Analyses should be described in sufficient detail as to
allow other researchers to reproduce the results using the same dataset.” When
reporting results, researchers should share data and analysis details, how
results compare to replicated or reproduced studies, and specific details
about data that was excluded or omitted (NSF19002). The definitions from the
Companion Guidelines, which are similar to those provided by
schmidt2009replication, are:

  * •

reproducibility: “the ability to achieve the same findings as another
investigator using extant data from a prior study” (NSF19002).

  * •

replication: “involved collecting and analyzing data to determine if the new
studies (in whole or in part) yield the same findings as a previous study”
(NSF19002). Replication studies are further broken down into two categories:

    * –

direct replication: “seek to replicate findings from a previous study using
the same, or as similar as possible, research methods and procedures as a
previous study” (NSF19002; schmidt2009replication).

    * –

conceptual replication: “seek to determine whether similar results are found
when certain aspects of a previous study’s method and/or procedures are
systematically varied” (NSF19002; schmidt2009replication).

nosek2014method describe the benefits of direct replication as increasing the
size of the data that can help identify false positive results, help establish
generalizability of results, and identify the boundaries of results.

gomez-reser2010-replication explore the language and definitions of how other
disciplines verify findings of experimental research. A review of literature
found 18 sources with classifications of replications. Their synthesis
identified three key groupings of definitions based on whether the authors use
the same methods for either operational replication of the methods or
empirical generalization of the results, different methods for a conceptual
replication to determine if the results around a given hypothesis are
reproducible, or existing data sets where the analysis is replicated for
internal replication or for a reanalysis of the data using different methods.
gomez-reser2010-replication provide the following terms and definitions:

  * •

Re-analysis: uses the same or different analysis on “the data of a previously
run experiment are used to verify the results rather than re-running the
experiment.” (gomez-reser2010-replication)

  * •

Replication: uses the same methods with different data that “verifies that the
observed findings are stable enough to be discovered more than once.” (gomez-
reser2010-replication)

  * •

Reproduction: uses the same hypothesis, but different methods and data to
verify “that the findings are not to be attributed to the experimental
method.” (gomez-reser2010-replication)

ihantola2015educational build on Gómez et al.’s (gomez-reser2010-replication)
work to create a novel classification of reproduction studies in the R.A.P.
Taxonomy. They consider three key criteria: researchers, data analysis, and
data production to identify seven classifications where one or more of the
criteria are changed for study reproduction. By considering the taxonomy,
researchers can examine the state space of possible study reproductions
related to a specific hypothesis. The seven categories and their definitions
are:

  * •

Re-Analysis: “a different different experimenter is following the same
analysis done before with the original data set for review purposes”
(ihantola2015educational)

  * •

Extended Analysis: “an experimenter is extending the baseline study by looking
at a previously analyzed data set, but using new analysis methods
(ihantola2015educational)

  * •

Repetition: “an experimenter is repeating the same analysis with new data”
(ihantola2015educational)

  * •

Verification: “the same data set is looked into again by a different
experimenter and using a different analysis method to verify the conclusions”
(ihantola2015educational)

  * •

Replication Study: “a different experimenter is following the same analysis
method as in the baseline study, but using their own different data set”
(ihantola2015educational)

  * •

Triangulation: “an experimenter is collecting a new data set to be analyzed
with a new method” (ihantola2015educational)

  * •

Reproduction: “a different experimenter is analyzing their own new data set
and following a new analysis method designed for the study in order to test
the hypothesis in the baseline study.” (ihantola2015educational)

ACM updated their Artifact Review and Badging definitions (acm-artifact-
review) in 2020 to correspond to definitions used by the National Information
Standards Organization (NISO). ACM uses the following definitions:

  * •

Repeatability: “Same team, same experimental setup” (acm-artifact-review)

  * •

Reproducibility: “Different team, same experimental setup” (acm-artifact-
review)

  * •

Replicability: “Different team, different experimental setup” (acm-artifact-
review)

The terminology utilized by NSF and ISE Companion Guidelines (NSF19002),
gomez-reser2010-replication, ihantola2015educational, and ACM (acm-artifact-
review) is inconsistent . Table 2 maps the definitions to each other. We
utilize the definitions from the Companion Guidelines (NSF19002) in our
discussion.

Table 2. Alignment of Replication Definitions | NSF/ISE
---
Companion Guidelines
(NSF19002) Gómez, et al. (gomez-reser2010-replication) |  Ihantola, et al. (ihantola2015educational) |  ACM (acm-artifact-review)
Reproducibility | Re-analysis |  | Re-analysis
---
Verification
Repeatability
Direct Replication | Replication |  | Repetition
---
Replication Study
Reproducibility
Conceptual Replication | Reproduction |  | Extended Analysis
---
Triangulation
Reproduction
Replicability

###  2.4. Contributions

The recent increase in characterizing the reporting standards in CER
literature through various reporting lenses demonstrates the need for the
community to identify and use common reporting standards. We see our
contribution as complementing the recent work of others
(Hao:2019TOCE:Replication; sanders2019icer; margulieux2019measurements;
ihantola2015educational; mcgill2018improving; lishinski2016methodological;
Ahadi2016Koli; al2016updated; mcgill-koli2019-replication;
mcgill2020construction; malmi2020theories; nelson2018use; Tenenberg2014TOCE)
to improve reporting standards in the CER community. Specifically, the current
paper provides an advance over prior work in three dimensions:

  * •

Scale \- we considered the proceedings of five CER venues for a full two
years, including the SIGCSE TS;

  * •

Scope \- our study provides a broad snapshot that covers the full
proceedings/issues of CER venues compared with the more focused snapshots of
prior work.

  * •

Focus \- our study investigates the quality of empirical reporting in CER
rather than classification.

##  3\. Methodology

Our goal is to describe the reporting of empiricism across the most common
venues for computing education research and practice. We have identified
elements of empiricism that should be included in reports of research results,
which parallel the key items needed during design of an educational research
study (fp04; bd12; creswell_creswell_2018). Our systematic literature review
(kitchenham2004slr) will reveal the state of the practice for reporting
empirical research results and provide guidance to the CER community for how
to improve reporting to provide the basis the community needs to advance in a
scientifically rigorous manner.

Our study investigates the following research questions (RQs):

  * •

RQ1: What percentage of papers in CER venues have some form of empirical
evaluation?

  * •

RQ2: Of the papers that have empirical evaluation, what are the
characteristics of the empirical evaluation?

  * •

RQ3: Of the papers that have empirical evaluation, do they follow norms (both
for inclusion and for labeling of information needed for replication, meta-
analysis, and, eventually, theory-building) for reporting empirical work?

###  3.1. Selection Criteria

Previous work done through 2008 (randolph2008methodological; valentine2004cs)
showed a small increase in the presence of empirical evaluation in papers in
CER conferences in general. Several more recent papers show an increase in
empirical work across a variety of venues (malmi2010characterizing;
al2016updated; lishinski2016methodological; ihantola2015educational). Our
evaluation includes two years of papers from five venues (SIGCSE TS, ICER,
ITiCSE, TOCE, and CSE). We included all papers from the TOCE and CSE editions
and two iterations of each conference during 2014 and 2015.

###  3.2. CER Empiricism Assessment Rubric

We developed and piloted the first version of the CER Empiricism Assessment
Rubric in 2015 (al2016updated). This initial version of the rubric was based
on the evaluation rubrics used in previous reviews of CER literature
(randolph2008methodological; valentine2004cs) and on items considered when
designing empirical studies (fp04; bd12). Specifically, the rubric was focused
on aspects of a research paper that are essential for others to understand and
potentially replicate the experiment, as evidenced by prior work
(randolph2008methodological; valentine2004cs) and supported by guidelines in
educational research (WWC2020; Schulzc-BMJ2010-ConsortStatement; NSF19002;
schmidt2009replication). Randolph et al.’s rubric, however, incorporated more
granular details in its categories. For example, Randolph et al.’s rubric took
into account the computing domain from which the research intervention
originated, including categories such as “Visualization” and “Simulation” as
discrete options (randolph2008methodological). Our rubric takes a more high-
level approach, focusing more on the general role of the intervention in how
it relates to computing education. In this example, instead of noting whether
the intervention incorporated “Visualization” or “Simulation,” we focused on
whether the intervention was an assignment, a tool, a curricular innovation,
etc.

After we applied the initial version of the rubric in our study of the SIGCSE
TS proceedings, we evolved both the rubric and the methodology for applying it
to a large number of papers to arrive at the current version of the rubric
found in Appendix A. In updating our rubric, we referenced the evolving
reviewing guidelines for ACM SIGCSE conferences and work in the scholarship of
teaching and learning (SoTL) space to ensure that our core rubric items were
capturing current best practices in CER work (bd12). Further, we streamlined
our evaluation methodology to make the application of the rubric to a large
set of papers more feasible, while still maintaining the quality of our
evaluation by examining the inter-rater reliability. Finally, we discovered
some papers that exposed corner cases that we needed to address, such as
papers that purported to be empirical studies, but reported no number of
actual participants in the study.

The rubric is comprised of a Base Rubric that is applicable to all papers with
some level of empirical results and three additional rubrics for specific
categories of research projects.

The Base Rubric contains a set of characteristics that captures the overall
nature of the empirical work including information such as the type of
evaluation method, the evaluation subject, whether there is any comparison,
and the number of participants. These items provide a high-level
characterization of the empirical work presented, answering questions such as:

  * •

What is the balance between quantitative and qualitative work?

  * •

At what rate do researchers publish studies on curricula?

  * •

Are researchers performing replication studies or creating their own
interventions?

  * •

How many participants are in the typical CER study?

Depending upon the type of empirical study included in the paper, i.e.
quantitative, qualitative, survey, and/or descriptive, we then employed one or
more additional rubrics. Each of these rubrics allowed us to characterize the
presence and clarity of important information that CER papers should include
relative to each type of study. The items on these additional rubrics all use
the same categorization scale that identifies whether a piece of information
is present and how easy it is for the reader to identify that piece of
information. The scale captures both types of information as follows:

  * •

Completeness - the level of completeness of presented information

    * –

Complete - Answers/addresses all questions for a rubric category. There is no
assessment on the quality of the answer.

    * –

Partial - Answers/addresses some of the questions for a rubric category. There
is no assessment on the quality of the answer.

    * –

Not Present - Answers/addresses none of the questions for a rubric category.
The questions should be addressed.

    * –

Not Applicable - The rubric item is not applicable to the paper.

  * •

Labeled - whether the presented information is clearly labeled

    * –

Labeled - There is a heading appropriate for the rubric item or there is
emphasis (bold/italics) for the rubric item.

    * –

Not Labeled - There is no heading or emphasis for the rubric item to easily
find the item in the paper.

Therefore each rubric item can take one of six labels:

  * •

Complete and Labeled

  * •

Complete and Not Labeled

  * •

Partial and Labeled

  * •

Partial and Not Labeled

  * •

Not Present

  * •

Not Applicable

The items included on the rubric represents the type of information necessary
to support reproducibility and replication of research studies. We use the
definitions from the Companion Guidelines on Replication & Reproducibility in
Education Research published jointly by the National Science Foundation (NSF)
and the Institute of Education Sciences (IES) (NSF19002).

  * •

reproducibility: “the ability to achieve the same findings as another
investigator using extant data from a prior study” (NSF19002).

  * •

replication: “involved collecting and analyzing data to determine if the new
studies (in whole or in part) yield the same findings as a previous study”
(NSF19002). Replication studies are further broken down into two categories:

    * –

direct replication: “seek to replicate findings from a previous study using
the same, or as similar as possible, research methods and procedures as a
previous study” (NSF19002; schmidt2009replication).

    * –

conceptual replication: “seek to determine whether similar results are found
when certain aspects of a previous study’s method and/or procedures are
systematically varied” (NSF19002; schmidt2009replication).

  * •

meta-analysis \- is a quantitative, formal, systematic analysis of a number of
related study results in order to identify general patterns and draw
overarching conclusions about the entire body of research (Haidich:2010).

  * •

theory-building \- is the process of using replication, repetition, and meta-
analysis to systematize knowledge. Theories can drive the generation of
hypotheses and produce predictive theory through scientific enquiry (fp04).
Additionally, conceptual replications can produce understanding and confirm
underlying theory (schmidt2009replication).

For the remainder of this paper, we use the term replication to cover any type
of study design that builds on prior study designs through replication in the
same setting (i.e., direct replication) or through systematic variance (i.e.,
conceptual replication).

Reporting common information and using open science supports replications
(NSF19002). Meta-analysis can then utilize these common results, aggregate
them, and increase overall generalizability of findings to “gain a better
understanding of what interventions improve (or do not improve) educational
outcomes, for whom, and under what conditions” (NSF19002). Because individual
studies are more prone to error or bias, meta-analysis or meta-studies provide
the opportunity to synthesize results around a research question to understand
impacts on a broader set of learners (mcgill-koli2019-replication;
al2016updated; schmidt2009replication; NSF19002). This process provides for a
mechanism for theory, either confirmatory of educational theory in a computing
context or the emergence of theory to support new phenomenon (fp04;
schmidt2009replication; mcgill-koli2019-replication).

For example, good practice is for a paper to provide an overview of the
participants, including demographics and the sampling or recruitment method
(mcgill2018improving). If a paper met all the criteria specified in the
rubric, the paper receives a label of “Complete” for that rubric item. If the
paper meets at least one, but not all, of the criteria, the paper receives a
label of “Partial” for that rubric item. If the paper meets none of the
criteria in the rubric, the paper receives a label of “Not Present” for the
rubric item. Finally, if this particular rubric item does not apply to the
paper, the paper receives the label of “Not Applicable” for that rubric item.

To ease the replication, meta-analysis, or theory-building process, it is
important for a paper to not only contain the appropriate information but also
to present that information in a way that readers can easily locate it,
similar to the checklists in CONSORT (Schulzc-BMJ2010-ConsortStatement) and
suggested information for reporting in APA JARS (apa-jars). We determined a
rubric item to be easy to find if there was a relevant section heading in the
paper for the item. We also considered text that emphasized a specific item
through italics, bolding, or a bulleted list. These are common ways of noting
research questions rather than using a section header. We assigned a value of
either “Labeled” or “Not Labeled” for each rubric item in each paper.

To reiterate, the purpose of the CER Empiricism Assessment Rubric is to
characterize the study and the rigor in which a paper reports an empirical
study. The goal is to comment on the efficacy of the reporting from the
perspective of a reader who wishes to replicate, perform meta-analysis, or
build theories. The rubric does not comment in any way on the quality of the
study or reporting itself, beyond the presence of the expected information.

We completed the Experimental Rubric for each paper that contained some form
of empirical evaluation. This rubric characterizes the rigor in which the
paper reports and labels the key aspects of an empirical study (Schulzc-
BMJ2010-ConsortStatement; fp04; bd12; creswell_creswell_2018). These aspects
include concepts and questions such as:

  * •

Are the research objectives obvious and easy to find?

  * •

Do the authors present related work?

  * •

Is the study design properly presented?

  * •

How was the data gathered and analyzed for this work?

  * •

Are the results presented in a succinct, direct way?

  * •

Are threats to validity discussed?

In order for an empirical study to be replicated, a paper needs to thoroughly
discuss and clearly label these items so readers can easily find them.

We completed the Survey Rubric for each paper that uses a survey as either the
primary research methodology or one of the data sources. Within those topics,
the rubric items examine whether the paper describes the survey creation
process, the rationale behind the questions, and the execution of the survey,
including its administration and the medium used.

We completed the Descriptive/Persuasive Rubric for papers that do not claim
any cause/effect relationship. These papers are often presented as position
papers. This rubric contains items focused on the goal of the paper’s
argument, the presence of related work, the soundness of the argument, and
whether supporting evidence is present.

###  3.3. Gathering Data and Applying the Rubric

Because we included all papers from each venue in the chosen years, we
manually extracted the DOI entries for each paper from the digital library.
Then we downloaded the PDFs of the papers to analyze locally.

In an initial phase, we examined the first 50 papers to refine our
methodology. We randomly assigned each paper to two researchers for initial
categorization. Each researcher independently evaluated the paper using the
rubric. Then we met to discuss any discrepancies in the analysis. If needed,
we asked a third researcher to review a paper to resolve any discrepancies. At
the end of this process, all researchers agreed on the final categorization of
the paper.

Based on the experience with the first 50 papers, we modified our methodology
for the remaining papers. Overall, we found very few discrepancies in this
first set of papers. We computed kappa to measure the inter-rater reliability
for each of the rubric items across the 50 papers. Overall, our level of
agreement was very high, with the kappa values for each rubric item above 0.8
(p <0.01). Based on this high level of agreement, we decided it was not
necessary for two reviewers to complete the full rubric on each paper.
However, we also did not want to fully rely on only one reviewer for each
paper. As a compromise, because the Base Rubric contains the information that
determines the type of research present in the paper and determines which
other rubric(s) apply, we had two researchers complete the Base Rubric for
each paper. After resolving any discrepancies, this process resulted in a
consensus on the type of research present in each paper. Based on the type of
research present, one researcher then reviewed the paper using one or more of
the remaining rubrics (Experimental, Survey, and Descriptive/Persuasive
Rubrics). Finally, we merged all results into a single spreadsheet for
analysis and calculations (dataset_2021).

##  4\. Results

This section describes the results of our analysis of our dataset
(dataset_2021), organized around the three research questions.

###  4.1. RQ1: Empirical Evaluation

RQ1: What percentage of papers in CER venues have some form of empirical
evaluation?

We classify a paper as empirical if its Evaluation Method is Descriptive,
Survey, Qualitative, or Quantitative such that it contains empirical evidence
addressing a research goal, question, or hypothesis. As shown in Table 3, 351
of the 427 papers (82%) contained some type of empirical evaluation. While
this number is quite high, note that we did not assess the quality or
correctness of the empirical evaluation, just the presence of it. When
considering specific venues, we found that TOCE had a lower percentage of
empirical papers. This result is likely due to the fact that a portion of the
TOCE papers were editorials and opinion papers, which we would not expect to
contain empirical evaluation. On the other end of the spectrum, 24 out of 25
papers in ICER 2015 had some sort of empirical evaluation.

Table 3. Empirical Evaluation by Venue and Year Venue | Year | # Papers |  | Acceptance
---
Rate
| # Empirical
---
Papers
| % Empirical
---
Papers
| # Empirical
---
Papers
by Venue
SIGCSE TS | 2014 | 110 | 39.4% | 91 | 82.7% | 176
SIGCSE TS | 2015 | 103 | 36.0% | 85 | 82.5%
ICER | 2014 | 17 | 25.0% | 14 | 82.4% | 38
ICER | 2015 | 25 | 26.0% | 24 | 96.0%
ITiCSE | 2014 | 51 | 35.0% | 43 | 84.3% | 85
ITiCSE | 2015 | 52 | 43.5% | 42 | 80.8%
TOCE | 2014 | 21 | n/a | 15 | 71.4% | 29
TOCE | 2015 | 21 | n/a | 14 | 66.7%
CSE | 2014 | 11 | n/a | 10 | 90.9% | 23
CSE | 2015 | 16 | n/a | 13 | 81.3%
Totals |  | 427 |  | 351 | 82.2% |

When comparing these results with those from earlier literature reviews, we
see an increase in the percentage of papers containing empirical evaluation
over time. valentine2004cs found 21% of papers discussing CS1 and CS2 topics
had “experimental” evaluation and randolph2008methodological found 35% of
papers with behavioral, quantitative, or empirical research from a broader set
of CS education venues from 2000-2005 contained empirical evaluation. Early
surveys of ICER papers between 2005-2009 found 86% of papers had an evaluative
purpose, appropriate for a conference focused on CER
(malmi2010characterizing). lishinski2016methodological’s analysis considered
ICER and CSE papers between 2012 and 2015 and found that 71% of CSE papers and
87% of ICER papers reported empirical results. These numbers are lower than
some of our results but consider a larger time frame. A review of SIGCSE TS
papers from 2014 and 2015 found that 70% of papers had some form of empirical
evaluation (al2016updated). Our definition of empirical evaluation is more
broad resulting in higher numbers during the same time period. When looking at
data mining and automated learning research across venues,
ihantola2015educational found that 78% of papers reported on a study in a
natural (e.g., empirical) setting. Our findings, along with complementary
work, suggest an increase in the amount of empirical work since the early
surveys by valentine2004cs and randolph2008methodological, which in turn
should lead to an increase of CER in the SIGCSE community. The remaining
research questions investigate the state-of-the-practice for reporting
empirical studies to further understand the concerns raised in our previous
work (al2016updated).

###  4.2. RQ2: Characteristics of Empirical Evaluation

RQ2: Of the papers that have empirical evaluation, what are the
characteristics of empirical evaluation?

To answer this question, we characterize the papers using the CER Empiricism
Assessment Rubric – Base Rubric. For each characteristic, we provide a table
that reports the raw number of papers with that characteristic type followed
by the percentage of the total empirical papers for the venue (based on the
last column of Table 3). We gave some papers multiple values for a
characteristic. In those cases, the percentages may total more than 100%.
Additionally, we summarize the characteristic items in the total column as the
percentage of all 351 empirical papers.

Table 4. Evaluation Methods by Venue | Evaluation
---
Method
| SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
| # | % | # | % | # | % | # | % | # | % | # | %
| Descriptive/
---
Persuasive
7 | 4% | 0 | 0% | 1 | 1% | 2 | 7% | 3 | 13% | 13 | 4%
Survey | 32 | 18% | 4 | 11% | 12 | 14% | 7 | 24% | 4 | 17% | 59 | 17%
Qualitative | 15 | 9% | 12 | 32% | 13 | 15% | 4 | 14% | 5 | 22% | 49 | 14%
Quantitative | 126 | 72% | 22 | 58% | 59 | 69% | 17 | 59% | 12 | 52% | 236 | 67%

Evaluation Method (Table 4). The CER Empiricism Assessment Rubric describes
each evaluation method in detail. However, for clarity we highlight the Survey
and Descriptive/Persuasive evaluation methods more carefully here to describe
their specific usage as an evaluation method. We coded papers as Survey when
they described a community survey with the goal of describing the current
state of that community (i.e. the paper does not involve any interventions).
Qualitative and Quantitative evaluation methods may have surveys as one of
their Data Sources in the study. We coded papers as Descriptive/Persuasive if
its focus was to describe a current situation or to persuade the reader about
a position. These papers do not test relationships among variables
(statistically or otherwise).

Examining the venues in more detail reveals that the papers published at
SIGCSE TS are overwhelmingly Quantitative followed by Survey. SIGCSE TS has
very few Qualitative papers in the years studied. The ITiCSE papers are also
mostly Quantitative, but do have a higher percentage of Qualitative papers.
ICER and CSE had the most balanced split between Quantitative and Qualitative
studies, but a smaller percentage of Survey papers.

Over two-thirds of empirical studies in CER literature are Quantitative (67%).
We coded only five papers with multiple evaluation methods. These papers were
either Survey – Qualitative or Survey – Quantitative. While the large emphasis
that these CER papers place on Quantitative methods provides interesting
numerical results, it does suggest the lack of nuance that Qualitative methods
bring to help understand the why and how of a study, which can expand and
deepen our understanding of computing phenomena (bd12; hazzan2006qualitative).

Table 5. Evaluation Subject by Venue Evaluation Subject |  | SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
| # | % | # | % | # | % | # | % | # | % | # | %
Assignment | 13 | 7% | 4 | 11% | 2 | 2% | 2 | 7% | 0 | 0% | 21 | 6%
Community | 24 | 14% | 7 | 18% | 15 | 18% | 8 | 28% | 2 | 9% | 56 | 16%
Curriculum | 50 | 28% | 3 | 8% | 13 | 15% | 7 | 24% | 8 | 35% | 81 | 23%
Pedagogical Technique | 59 | 34% | 17 | 45% | 34 | 40% | 9 | 31% | 7 | 30% | 126 | 36%
Tool | 24 | 14% | 3 | 8% | 17 | 20% | 2 | 7% | 2 | 9% | 48 | 14%
Other | 6 | 3% | 4 | 11% | 4 | 5% | 2 | 7% | 4 | 17% | 20 | 6%

Evaluation Subjects (Table 5). Pedagogical Techniques, or teaching methods,
were the most common evaluation subject in all venues. Tools papers were more
common at the SIGCSE TS and ITiCSE. All venues had papers evaluating the
broader Community. Curriculum papers were more common at the SIGCSE TS, TOCE,
and CSE. Readers may find the distribution of paper types at these venues for
2014 and 2015 interesting as they consider future studies that either fit with
or fill gaps with existing literature.

Table 6. Evaluation Subject Source by Venue | Evaluation Subject
---
Source
| SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
| # | % | # | % | # | % | # | % | # | % | # | %
Authors Here | 101 | 57% | 21 | 55% | 51 | 60% | 17 | 59% | 18 | 82% | 208 | 59%
Authors Elsewhere | 11 | 6% | 4 | 11% | 9 | 11% | 3 | 10% | 1 | 5% | 28 | 8%
Other Modified | 32 | 18% | 6 | 16% | 8 | 9% | 1 | 3% | 2 | 9% | 49 | 14%
Other Not Modified | 10 | 6% | 7 | 18% | 6 | 7% | 4 | 14% | 0 | 0% | 27 | 8%
Community | 22 | 13% | 1 | 3% | 11 | 13% | 4 | 14% | 1 | 5% | 39 | 11%

Evaluation Subject Source (Table 6). This rubric item is a measure of
replication and reproducibility in the community. This item could take one of
five values: (1) Authors Here – the authors created the evaluation subject for
use in the current study; (2) Authors Elsewhere – the authors created and
published the evaluation subject in a previous publication; (3) Other Modified
– someone other than the authors created the evaluation subject, but the
authors modified it; (4) Other Not Modified – someone other than the authors
created the evaluation subject, and the authors did not modify it; or (5)
Community – self-identifying group of people related to an area of interest
(if the evaluation subject is Community, then the source is Community as
well).

More than half of all empirical papers used evaluation subjects developed for
the specific study (i.e. Authors Here), indicating that researchers may be
working in isolation and not using existing data sets or possible comparisons
to similar participants in a different context. Papers that reuse an
evaluation subject from a previous study by another author demonstrate
replication and sharing within the broader community (NSF19002;
schmidt2009replication). In some cases, the authors conducted a study on an
evaluation source that they utilized in earlier work, which we coded as
Authors Elsewhere. Overall, ICER had the highest percentage of papers in which
authors utilized evaluation subjects from other researchers (e.g. Other
Modified and Other Not Modified).

Table 7. Comparison in Evaluation by Venue | Comparison in
---
Evaluation
| SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
| # | % | # | % | # | % | # | % | # | % | # | %
Comparison | 87 | 49% | 20 | 53% | 46 | 54% | 15 | 52% | 8 | 35% | 176 | 50%
None | 89 | 51% | 18 | 47% | 39 | 46% | 14 | 48% | 15 | 65% | 175 | 50%

Comparison (Table 7). By comparing the results from an intervention to the
results from a baseline approach, a paper can provide additional support to
strengthen a conclusion that the intervention caused the observed effect. The
lack of Comparison leaves open the possibility of confounding factors
influencing the observed result. However, when papers report a case study
(i.e. an in-depth analysis of a single case), the lack of comparison is
expected (e.g., None). Case studies can provide valuable findings that other
researchers can attempt to replicate in their own environments, using more
experimental approaches. Approximately half of the papers in each venue did
provide some type of Comparison between two or more groups of participants. To
promote comparisons between researchers, margulieux2019measurements suggest
utilizing common, standardized, measurements.

Table 8. Number of Participants by Venue | Number of
---
Participants
| SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
| # | % | # | % | # | % | # | % | # | % | # | %
1 | 6 | 3% | 0 | 0% | 2 | 2% | 1 | 3% | 0 | 0% | 9 | 3%
2 | 1 | 1% | 2 | 5% | 2 | 2% | 1 | 3% | 0 | 0% | 6 | 2%
3-10 | 10 | 6% | 2 | 5% | 6 | 7% | 3 | 10% | 0 | 0% | 20 | 6%
11-30 | 38 | 22% | 7 | 18% | 21 | 25% | 3 | 10% | 2 | 10% | 71 | 20%
31-75 | 38 | 22% | 10 | 26% | 15 | 18% | 6 | 21% | 6 | 29% | 75 | 21%
76-150 | 25 | 14% | 6 | 16% | 13 | 15% | 3 | 10% | 1 | 5% | 48 | 14%
151-300 | 20 | 11% | 5 | 13% | 7 | 8% | 3 | 10% | 5 | 24% | 40 | 11%
301-999 | 19 | 11% | 3 | 8% | 12 | 14% | 4 | 14% | 5 | 24% | 43 | 12%
1000+ | 13 | 7% | 3 | 8% | 7 | 8% | 4 | 14% | 1 | 5% | 21 | 8%
Missing | 6 | 3% | 0 | 0% | 0 | 0% | 1 | 3% | 1 | 5% | 8 | 2%
Table 9. Summary Statistics of Participants by Venue Summary Statistics | SIGCSE TS | ICER | ITiCSE | TOCE | CSE | TOTAL
---|---|---|---|---|---|---
Average | 1937.6 | 913.1 | 921.2 | 583.8 | 314.5 | 1363.7
Median | 63.5 | 75 | 60 | 95.5 | 183 | 72
Mode | 1 | 75 | 23 | 5 | 183 | 12

Participants (Table 8 & Table 9). The number of Participants in published
studies can provide some insight into the strength of the conclusions drawn.
Of concern are the papers that did not provide any numbers when discussing the
participants in the study. Of the 349 empirical papers that should report
subject data333Two CSE papers categorized as descriptive had N/A recorded for
subjects., 98% of papers reported subjects in some form. This percentage is
higher than the 83.7% of articles reporting subjects in mcgill2018improving.
We additionally report summary statistics for the number of participants by
venue and across all venues. Since we are considering all publications in the
venues rather than a subset based on an inclusion criteria, the mean and
median number of participants is higher than the mean of 328 and median of 45
reported by McGill et al., likely due to the K-12 focus (mcgill2018improving).

Table 10. Type of Study for Non-Descriptive Papers by Venue and for Quantitative Only Papers by Venue |  | SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
| Type of Study
---
(Non-Descriptive)
# | % | # | % | # | % | # | % | # | % | # | %
Observational | 86 | 51% | 20 | 53% | 37 | 44% | 16 | 59% | 12 | 60% | 171 | 51%
Interventional | 83 | 49% | 18 | 47% | 47 | 56% | 11 | 41% | 8 | 40% | 167 | 49%
| Type of Study
---
(Quantitative Only)
# | % | # | % | # | % | # | % | # | % | # | %
Observational | 63 | 50% | 9 | 41% | 20 | 34% | 8 | 47% | 5 | 42% | 105 | 44%
Interventional | 63 | 50% | 13 | 59% | 39 | 66% | 9 | 53% | 7 | 58% | 131 | 56%

Type of Study (Table 10). An Observational study is performed in a natural
setting in which the researcher collects data via observation without
manipulation of the situation. Conversely, an Interventional study is
performed by assigning participants into groups (e.g., control and
experimental) and applying the treatment to the experimental group to measure
its effect. Our use of observational and interventional are not intended as a
direct relationship to qualitative and quantitative methods, but are instead a
way of describing the setting. Observational studies are similar to Fincher’s
and Petre’s (fp04) definition on in situ – the normal or natural setting.
Interventional studies correspond to Fincher’s and Petre’s (fp04) settings of
under constraints and in a laboratory where there is some direct intervention
to the environment.

There is a fairly even split between the two types of studies across all
venues. This result indicates a nice balance in CER work. However, when
looking at Quantitative work only, we do see an increase in the proportion of
Interventional studies. These results suggest that there is a gap in using
Qualitative methods to assess CER interventions.

Table 11. Data Source of Papers by Venue | Data
---
Source
| Assessment
---
Data

Automated |  | Course
---
Evaluations

Interviews |  | Focus
---
Group

Observations |  Survey |  Other |  | More than One
---

| Survey ONLY
---

SIGCSE | Quant | 46% | 13% | 7% | 6% | 1% | 8% | 35% | 5% | 19% | 24%
Qual | 40% | 0% | 0% | 47% | 13% | 7% | 27% | 0% | 33% | 7%
Sur | 6% | 0% | 0% | 3% | 3% | 0% | 100% | 0% | 13% | 88%
ICER | Quant | 45% | 27% | 0% | 0% | 0% | 5% | 32% | 14% | 23% | 14%
Qual | 8% | 8% | 0% | 67% | 0% | 33% | 0% | 17% | 25% | 0%
Sur | 0% | 0% | 0% | 0% | 25% | 0% | 100% | 0% | 25% | 75%
ITiCSE | Quant | 53% | 17% | 2% | 5% | 0% | 12% | 47% | 10% | 41% | 15%
Qual | 23% | 15% | 0% | 15% | 8% | 15% | 15% | 15% | 8% | 8%
Sur | 0% | 0% | 0% | 0% | 0% | 0% | 100% | 0% | 0% | 100%
TOCE | Quant | 47% | 12% | 0% | 12% | 6% | 12% | 29% | 6% | 24% | 18%
Qual | 0% | 0% | 0% | 50% | 0% | 25% | 0% | 25% | 0% | 0%
Sur | 14% | 14% | 0% | 0% | 0% | 0% | 100% | 0% | 29% | 71%
CSE | Quant | 67% | 0% | 0% | 25% | 0% | 8% | 58% | 8% | 42% | 25%
Qual | 20% | 0% | 0% | 0% | 20% | 40% | 40% | 20% | 40% | 40%
Sur | 0% | 0% | 0% | 0% | 25% | 0% | 100% | 0% | 25% | 25%
Totals | 38% | 11% | 3% | 11% | 3% | 9% | 47% | 7% | 24% | 30%

Data Source (Table 11). We identified seven key data sources and included an
option for data sources that do not fit any of the existing categories.
Overall, the most common data sources utilized in Quantitative, Qualitative,
and Survey work were Surveys, Assessment Data, and Automated Data. In
addition, the Qualitative studies frequently featured Interviews, Focus
Groups, and Observations. Our categorization of data sources has some overlap
with related work, but our focus was identifying high-level data sources. That
means that categories in other studies, like attitudinal data described in
randolph2008methodological was typically grouped in a larger category of
“Survey”.

The analysis showed that 76% of the papers utilized only one type of data
source, even if they collected multiple samples. By only utilizing a single
data source, authors limit their understanding of the phenomena under study
because they cannot triangulate the results to clarify or understand (fp04;
margulieux2019measurements). The analysis also showed that 30% of all papers
used Surveys as the sole data source. While surveys are an excellent tool for
collecting attitudinal and self-report data, as the only source of data, they
may not be robust enough to provide strong evidence to answer a research
question. We did not collect data on the use of validated survey instruments
as described in decker-sigcsets2019-instruments.

###  4.3. RQ3: Norms of Empirical Evaluation

RQ3: Of the papers that have empirical evaluation, do they follow norms (both
inclusion and labeling of information needed for replication, meta-analysis,
and, eventually, theory-building) for reporting empirical work?

For empirical papers, we expect a paper should contain a research objective,
related work, an overview of the research participants, an overview of the
study design or methods, a description of the data collection process, a
description of the analysis procedures, a report of the results, and a listing
of the threats to validity for the research (fp04; bd12;
creswell_creswell_2018; Schulzc-BMJ2010-ConsortStatement) as defined in the
CER Empiricism Assessment – Experimental Rubric. Any papers reporting the
results of a Survey as a data source should have information about how the
survey was conducted and the survey questions asked as defined in the CER
Empiricism Assessment – Survey Rubric. Descriptive/Persuasive work should
include the goal of the argument, related work, at least three premises and a
conclusion, all with supporting evidence as defined in the CER Empiricism
Assessment – Descriptive/Persuasive Rubric. However, due to the small numbers
of Descriptive/Persuasive papers in our analysis (e.g., 10), we do not provide
an analysis of that category in this section.

To provide an overall characterization of the papers, we classified each paper
into one of three categories for each of the key elements listed above:

  * •

Strongly Supports Replication \- items that were “Complete and Labeled”
provide all the details necessary to support replication;

  * •

Weakly Supports Replication \- items that were “Complete and Not Labeled” or
“Partial and Labeled” provide some, but not all of the details required to
support replication;

  * •

Not Present \- items that were “Not Present” are missing information necessary
to support replication.

Table 12 summarizes the results of our analysis of reporting norms. Overall,
the only two elements where more than 50% of papers reach the Strongly
Supports Replication level are Related Work and Results. Looking across all
elements, the papers published in ICER do a better job by having more then 50%
of papers reach the Strongly Supports Replication level for all elements
except for Threats to Validity. However, as the numbers in Table 12 show,
there is still room for improvement as a community to reach the Strongly
Supports Replication level for all attributes most (or all) of the time.

####  4.3.1. Experimental Rubric

Table 12. Norms of Reporting Empirical Evaluation |  | Replication
---
Support
| SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
Research Objective | Strong | 20.1% | 55.3% | 17.9% | 55.6% | 55.0% | 28.4%
Weak | 58.0% | 31.6% | 61.9% | 37.0% | 45.0% | 53.6%
Not Present | 21.9% | 13.2% | 20.2% | 7.4% | 0.0% | 18.0%
Related Work | Strong | 62.1% | 81.6% | 70.2% | 81.5% | 65.0% | 68.0%
Weak | 27.8% | 15.8% | 19.0% | 18.5% | 30.0% | 23.7%
Not Present | 10.1% | 2.6% | 10.7% | 0.0% | 5.0% | 8.3%
Subjects | Strong | 32.0% | 50.0% | 27.4% | 44.4% | 44.4% | 34.5%
Weak | 60.4% | 47.4% | 61.9% | 51.9% | 50.0% | 58.0%
Not Present | 7.7% | 2.6% | 10.7% | 3.7% | 5.6% | 7.4%
Study Design | Strong | 33.7% | 60.5% | 56.0% | 48.1% | 68.8% | 45.2%
Weak | 59.2% | 36.8% | 39.3% | 51.9% | 31.3% | 49.7%
Not Present | 7.1% | 2.6% | 4.8% | 0.0% | 0.0% | 5.1%
Data Collection | Strong | 27.8% | 52.6% | 23.8% | 25.9% | 60.0% | 30.9%
Weak | 65.7% | 44.7% | 70.2% | 66.7% | 40.0% | 63.4%
Not Present | 6.5% | 2.6% | 6.0% | 7.4% | 0.0% | 5.7%
Analysis Procedures | Strong | 28.6% | 57.9% | 23.8% | 48.1% | 71.4% | 34.1%
Weak | 60.7% | 39.5% | 65.5% | 51.9% | 28.6% | 57.4%
Not Present | 10.7% | 2.6% | 10.7% | 0.0% | 0.0% | 8.5%
Results | Strong | 61.5% | 97.4% | 78.6% | 81.5% | 100.0% | 73.5%
Weak | 37.3% | 2.6% | 21.4% | 18.5% | 0.0% | 25.9%
Not Present | 1.2% | 0.0% | 0.0% | 0.0% | 0.0% | 0.6%
Threats | Strong | 7.7% | 31.6% | 4.8% | 25.9% | 20.0% | 11.8%
Weak | 13.0% | 23.7% | 13.1% | 7.4% | 20.0% | 14.2%
Not Present | 79.3% | 44.7% | 82.1% | 66.7% | 60.0% | 74.0%

The Experimental Rubric describes the standard information we would expect to
see in a paper reporting on empirical work. Table 12 summarizes the results
for the items on the Experimental Rubric.

Research Question. One of the most critical pieces of information in an
empirical paper is the research objective, goals, hypotheses, or other
summative statement that provides the context of the work reported in the
paper. It is concerning that, across all venues, 18% of the papers lacked a
summative statement about the work (i.e. scored as Not Present).
randolph2008methodological reported only 22% of papers between 2000 and 2005
had research questions or hypotheses and lishinski2016methodological found
that only 47% of CSE and 56% of ICER papers between 2012 and 2015 had explicit
research questions. Our numbers for CSE (55%) and ICER (55.3%) for complete
and labeled are similar to lishinski2016methodological and higher than
randolph2008methodological suggesting some improvement in the reporting of
research questions.

A summative statement guides the reader to the main ideas of the study and is
critical for determining if the methods and analysis are appropriate for the
statement and if the results answer or address the statement. One or more
research statements, objectives, or hypotheses are recommended in standards
like CONSORT (Schulzc-BMJ2010-ConsortStatement), APA JARS (apa-jars), and AERA
(area-empirical-social-science-2006). In addition, we found over half of the
papers reached only the Weakly Supports Replication level because they lacked
important information. These papers either did not highlight the summative
statement in a meaningful way, through text attributes (e.g., italics or
bold), callouts (e.g., boxes or bullets), or clear labeling (e.g., RQ1, Goal,
etc.), or did not clearly label the goal of the paper, thereby requiring the
reader to infer the context of the paper through other statements in the
paper.

Related Work. Most papers reported and labeled related work demonstrating
Strongly Supports Replication for this category. The SIGCSE Board policy444See
SIGCSE Board Program Chair Responsibilities policy at
https://sigcse.org/policies/pcr.html. states that Program Chairs should
“request that all paper submissions include a review of previous, related
work.” This guidance demonstrates the importance of establishing reporting
norms for the community to ensure high quality dissemination of empirical
results. The papers we rated as Weakly Supports Replication typically did not
link the related work to the research objective or did not contain an
explicitly labeled related work or background section. For those papers that
lacked a related work section, the authors typically discussed the related
work or background information in the introduction.

Participants. Most papers did include information about the study
participants. However, over half of the papers fell into the Weakly Supports
Replication category because they lacked either full details about the
participants or a clear label for the section of the paper describing the
participants. A description of participants should contain demographic
information to provide context for the results (mcgill2018improving; NSF19002;
apa-jars; Schulzc-BMJ2010-ConsortStatement; area-empirical-social-
science-2006). For example, when papers include student populations, the
demographics should include: the number of participants, age groups, education
levels, gender, race/ethnicity, region or area of the world, and prior
computing experience (mcgill2018improving). Other student demographics that
may be of interest depending on the study include student disabilities,
socioeconomic status, family history (e.g., first generation college student),
and veteran status. Demographics may vary based on the populations under study
and the research questions.

In addition to lacking demographics, papers often lacked a description of the
recruitment process for the participants. Classroom research may lack the
formal control and treatments present in laboratory or controlled studies. By
clarifying the inclusion and exclusion criteria for participants, an author
clarifies the participant pool and research context (NSF19002; Schulzc-
BMJ2010-ConsortStatement; apa-jars; area-empirical-social-science-2006).
Papers should also include a statement about how the authors recruited
participants and obtained consent to participate in the study. This statement
demonstrates ethical treatment of research participants and assure readers
that authors are following standards for human subject research in their
context.

Study Design. Most papers contained a discussion about study design. We found
a split between Strongly Supports Replication and Weakly Supports Replication.
There were a number of papers that lacked a clear study design or methods
section. In many of these cases, the study design information may have been
integrated with the results, rather than as a separate independent section. By
providing the study design information in a separate section, the authors help
readers more easily extract important information about the design to support
replication. In addition to not being labeled, some study designs sections
omitted key information. One key piece of information is identification of the
dependent and independent variables (Schulzc-BMJ2010-ConsortStatement; apa-
jars), which could be as simple as identifying the key item under observation
or providing overview of an intervention.

Data Collection. Most papers contained a discussion about data collection.
However, 63.4% of papers fell into the Weakly Supports Replication category.
Some of those papers did not include a clearly defined data collection
section. Sometimes, the papers integrated the discussion about data collection
into the discussion about the study design or methods. Other times, papers
integrated the discussion about data collection into the results section. By
providing a clear section or subsection around data collection (apa-jars;
area-empirical-social-science-2006), authors can provide details to readers
who may be interested in using similar techniques in their own work.

Other Weakly Supports Replication papers omitted important information
describing the how, where, and who of the data collection. For example, papers
frequently lacked details about survey administration, which is discussed
further in Section 4.3.2.

Analysis Procedures. Around 34% of papers demonstrated Strongly Supports
Replication related to the reporting of analysis procedures. Around half of
the Weakly Supports Replication papers were both incomplete and not properly
labeled. These papers were commonly missing a full description of how the
authors worked with or processed that data after collection. Additionally, the
papers often did not provide a justification for why the particular
statistical tests were appropriate for the analysis. A discussion on the use
of inferential statistics for ICER’s proceedings history provides a
justification for full reporting of statistical tests and examples of papers
with Strongly Supports Replication in this space (sanders2019icer). Standards
support clear reporting on analysis procedures and statistical tests used
(apa-jars; area-empirical-social-science-2006)

Results. Most papers reported results and tied the results back to the
research question, goal, or hypothesis (which may have been inferred if not
explicitly stated). A large percentage of ICER and TOCE reached the level of
Strongly Supports Replication. Papers with Weakly Supports Replication tended
to have partial reporting of results, usually by not tying the results back to
the research questions, goals, or hypotheses as suggested by APA JARS (apa-
jars). Some papers also omitted a clearly labeled results section. Standards
provide guidance on the details that are included when reporting results
(Schulzc-BMJ2010-ConsortStatement; apa-jars; area-empirical-social-
science-2006).

Threats. Most papers (almost 75%) did not report any threats to validity or
limitations to work. Our results are similar to the upper range reported by
al2016updated. ihantola2015educational found only 22% of papers they reviewed
on educational data mining reported threats to validity. If papers did discuss
threats or limitations, the threats/limitations were typically unlabeled and
appeared in either a results or discussion section. Papers with a Weakly
Supports Replication may have listed the threats, but typically did not
discuss how the authors addressed the threats. A threats to validity section
provides context to the study design and provides details on how readers
should interpret the work in a broader context and potential biases that might
impact the result (al2016updated; Schulzc-BMJ2010-ConsortStatement; apa-jars).
Because CER is complex work, threats to validity describe how the authors
controlled that complexity (or not) (ihantola2015educational; al2016updated).
Replications help control for sampling error and artifacts which identify
potential weaknesses with internal validity of the original study. Clearly
articulated threats to validity can help identify key variables to vary in
replications (schmidt2009replication). Replications also support increasing
external validity through generalizability (schmidt2009replication).

####  4.3.2. Survey Rubric

We completed the Survey Rubric for any paper that contained a survey as a data
source. To provide replicable survey research, authors must provide details on
the survey design and survey execution. The reuse and standardization of
surveys would be a good starting point for building a culture of replication
(margulieux2019measurements; area-empirical-social-science-2006). If authors
provide details about their surveys, even as an un-reviewed supplement to the
the paper, it would help build this culture. Table 13 summarizes the results
for the items on the Survey Rubric.

Table 13. Norms of Reporting Surveys |  |  | SIGCSE
---
TS
ICER | ITiCSE | TOCE | CSE | TOTAL
Conducting the Survey | Strong Replication | 28% | 27% | 14% | 18% | 36% | 24%
Weak Replication | 58% | 73% | 79% | 82% | 64% | 67%
Not Present | 14% | 0% | 7% | 0% | 0% | 9%
Survey Design | Strong Replication | 30% | 0% | 24% | 36% | 20% | 26%
Weak Replication | 50% | 91% | 50% | 45% | 20% | 51%
Not Present | 20% | 9% | 26% | 18% | 60% | 23%

Conducting the Survey. We found that most papers demonstrated Weak Replication
because they did not fully discuss survey execution in their context. The most
frequently missing information was details about survey administration and
survey medium. Additionally, many authors did not clearly label the
information about conducting the survey.

Survey Design. Most papers did not provide a justification for the selection
of survey questions based upon how those questions measure items of interest
to answer the research questions, which is a category of reporting in APA JARS
(apa-jars; area-empirical-social-science-2006). Additionally, most papers did
not include the survey questions, which makes it difficult for others to adopt
or reuse the survey in their context. In particular, we found that the
majority of papers published in CSE during 2014 and 2015 did not discuss the
design of the surveys or how the questions were derived, but instead focused
more on the distribution and execution of the survey. Validation and adoption
of standardized, citable survey instruments will also support replication
(margulieux2019measurements).

##  5\. Discussion

The CER community has seen a growth in submissions and participation. As our
community grows we need to mature the norms of reporting empirical research so
the broader community can benefit from the dissemination of high-quality
results, that answer well-stated research questions, situated in the
appropriate literature, with a clear discussion of limitations and threats to
validity. Papers should clearly document methods and analysis procedures to
allow others to replicate studies in their own contexts. Reviewers should
begin to expect paper authors to follow reporting norms and use those norms as
guidance when performing paper reviews.

In the following sections, we provide observations and recommendations from
our review of reporting norms. Where appropriate, we connect our
recommendations to the broader literature and guidelines, particularly APA
JARS (apa-jars), CONSORT, (Schulzc-BMJ2010-ConsortStatement), What Works
Clearinghouse (WWC2020), AERA (area-empirical-social-science-2006), and the
IES/NSF Guidelines (NSF13126; NSF19002).

###  5.1. Observations and Recommendations from the Results

Based on the results described in the previous section, we make a number of
observations and recommendations for authors. We note that these
recommendations may not apply in all situations and that the individual
research questions and researcher context will impact the decisions that are
made during a study.

Evaluation Methods. The results showed that only five papers had used multiple
evaluation methods. We encourage researchers to consider using mixed methods
approaches more often in study designs to utilize the complementary strengths
of both Qualitative and Quantitative methods (see fp04, bd12, APA JARS (apa-
jars) for details on mixed method studies). However, we acknowledge that it is
not always possible or advisable to use mixed methods approaches and that page
limits often constrain the level of detail researchers can include in their
publications.

Evaluation Subject Source. The results showed that more than half of the
papers used evaluation subjects developed for the specific study. This result
suggests that replication in CER is weak. Replication provides a mechanism for
generalizing findings about a research question or set of research questions
that can lead to the development of theory to support the computing education
community (schmidt2009replication; ihantola2015educational). We challenge the
computing education community to consider using common methods, evaluation
metrics, and data collection and reporting procedures to support the
comparison and aggregation of data to generalize CER findings. Utilizing
standards can support this goal (Schulzc-BMJ2010-ConsortStatement; WWC2020;
NSF19002; apa-jars; area-empirical-social-science-2006).

Data Source. The results showed that a large number of papers utilized only
one source of data. We encourage researchers to utilize multiple data sources
to provide more robust insight to the phenomena under study and stronger
answers to research questions. In addition, 29% of the papers used surveys as
the only data source. There are many situations where surveys are an excellent
form of data collection (i.e., understanding a community); however, in
classroom studies, researchers should supplement surveys with additional
information to provide stronger evidence to answer a research question. For
example, a research question about how an intervention impacts student
learning should not rely solely on student’s self-report of their learning.
margulieux2019measurements argues that collecting both process (e.g., progress
or experience) and product (e.g., performance or outcome) data can increase
the applicability of the results to a larger group of educators.

Study Design. We observed a number of papers that lacked a clear study design
or methods section. A study design section should tie the observation or
intervention to the research question(s) and provide justification that the
observation or intervention will support answering the research question(s).
The study design section should clearly describe the steps of the study so
others could replicate the study in their own environments. For subjective
measures, the paper should contain a discussion of the coding process and the
method for obtaining agreement or consensus among multiple raters. These items
are important for determining how to properly interpret the results. In
addition, citing seminal work and standards about the study methods
strengthens the methods discussion and can help build the community’s
knowledge about CER methods (fp04; bd12; Schulzc-BMJ2010-ConsortStatement;
apa-jars; WWC2020; area-empirical-social-science-2006).

Survey Design. We observed that most authors did not provide a justification
for the specific survey questions included. While page restrictions can make
it difficult to provide the full survey, authors can include the full survey
as an un-reviewed supplemental content or on a website. In cases where an
author does not want to make the survey publicly available because it may bias
future results, the author can include a statement to that effect in the paper
and provide contact information for interested researchers to obtain a copy of
the survey. The use of standardized survey instruments can support replication
(margulieux2019measurements).

###  5.2. Author Guidelines for Empirical Papers

Many CER researchers may not have any formal training in conducting human
subject research in educational environments. Examples of high-quality
literature and study frameworks for replication can help new researchers get
started investigating interesting challenges in their own classrooms,
departments, and communities. We therefore propose the following guidelines
for CER authors when preparing their studies and writing up their work. These
guidelines are similar to expectations when designing CER studies (fp04; bd12)
and have elements in common with standards (Schulzc-BMJ2010-ConsortStatement;
WWC2020; NSF19002; apa-jars; area-empirical-social-science-2006). CER Venues,
like TOCE, are now recommending that authors utilize APA JARS for future
submissions.

General Reporting Guidelines

  * •

Clearly identify the evaluation method(s) and subject(s) under evaluation in
the study (Schulzc-BMJ2010-ConsortStatement; apa-jars; area-empirical-social-
science-2006).

  * •

Clearly identify what work is new to the study and where the study builds on
prior work or data (apa-jars; NSF19002; area-empirical-social-science-2006).

  * •

Clearly identify any comparisons with prior work or with a control group, if
appropriate (apa-jars; NSF19002; area-empirical-social-science-2006).

  * •

Provide information about participants, including demographics
(mcgill2018improving; Schulzc-BMJ2010-ConsortStatement; apa-jars; area-
empirical-social-science-2006).

  * •

Identify the study as observational or interventional (apa-jars).

  * •

Identify data sources (apa-jars; area-empirical-social-science-2006).

  * •

For surveys, describe the administration process and provide the survey
questions.

  * •

Where possible, provide supplemental resources that would help others
replicate the work, e.g. code used to analyze data on a public version control
system or anonymized, aggregate data on a website (Schulzc-
BMJ2010-ConsortStatement; NSF19002).

  * •

Utilize pre-registration and open science to strengthen research integrity and
transparency (NSF19002; Schulzc-BMJ2010-ConsortStatement; mcgill-
koli2019-replication; nosek2014method).

Research Question (Schulzc-BMJ2010-ConsortStatement; apa-jars; area-empirical-
social-science-2006)

  * •

The research goal or question is of upmost importance because it drives the
rest of the work. Utilize resources on writing high quality and actionable
research questions.

  * •

Highlight the research goal and questions in the text. For example, italicize
the goal statements in the paper’s introduction. List and name more specific
research questions (e.g. RQ1, RQ2) in the introduction and revisit them in the
results.

Related Work

  * •

Report related work in its own section. It should situate the research
question or goal in the context of the broader literature. Because most SIGCSE
conferences now provide extra pages for references, authors should be able to
provide a robust discussion of related work with sufficient citations of
relevant literature.

  * •

Provide a discussion that links the related work to the research goal,
objectives, or questions for the paper. The section should also discuss how
the related work informs the study and how the study builds on previous work.
The information about previous work is especially important for replication
papers (apa-jars; area-empirical-social-science-2006).

  * •

Synthesize the key themes in the prior work to provide context and motivation
for the current study, which should not simply be an annotated bibliography.

Participants

  * •

Report demographic information about the participant groups in the study. For
students, minimally report numbers, ages, education levels, gender,
race/ethnicity, prior computing experience, and regional location for the
students (mcgill2018improving; apa-jars; area-empirical-social-science-2006).
mcgill2018improving provide other suggestions for reporting demographic
information for other participant populations.

  * •

If number of participants is too small that reporting demographics would
become identifying for those participants, clearly state this information in
lieu of reporting specific demographics.

  * •

Describe the process for selecting participants. If the study uses multiple
groups, the paper should discuss the process for allocating participants to
each group. If the authors exclude participants from the study (e.g., minors
in a university-level study), the paper should explain the exclusion criteria
(Schulzc-BMJ2010-ConsortStatement; apa-jars; area-empirical-social-
science-2006).

  * •

Explain the process for obtaining consent from participants and any ethical
considerations associated with human subjects research. Since ethical
standards vary by country and institution, authors should clarify expectations
in their context and assure reviewers and readers that the human subjects
study was handled in an appropriate manner for the author’s context (apa-jars;
area-empirical-social-science-2006).

Study Design

  * •

Separate the study design or methods from the discussion of results. This
separation will allow readers to assess the quality of the study and to more
easily see the steps needed for replication.

  * •

Identify independent and dependent variables (Schulzc-
BMJ2010-ConsortStatement; apa-jars).

  * •

Describe each step in conducting the study (Schulzc-BMJ2010-ConsortStatement;
apa-jars; area-empirical-social-science-2006).

Data Collection

  * •

Identify data collection procedures including mediums for data collection, who
collected the data, and how the data addresses or answers the research
questions (NSF19002; apa-jars; area-empirical-social-science-2006).

  * •

Provide a rationale for why the data collected will help address the research
question(s) (apa-jars; area-empirical-social-science-2006).

Analysis Procedures

  * •

Describe the process for working with the collected data, including the
process for cleaning data, if necessary (NSF19002; area-empirical-social-
science-2006).

  * •

For qualitative analysis, describe the coding process and the process for
evaluating the correctness of the coding (e.g., multiple raters, inter-rater
reliability, etc.) (apa-jars; area-empirical-social-science-2006).

  * •

For quantitative analysis, describe, and justify, the statistical tests, if
appropriate. For other types of analysis, provide a justification on how the
analysis helps answer the research question(s) (Schulzc-
BMJ2010-ConsortStatement; apa-jars; area-empirical-social-science-2006).

Results

  * •

Ensure that results directly address the research question(s) (Schulzc-
BMJ2010-ConsortStatement; apa-jars; area-empirical-social-science-2006).

  * •

For qualitative results, provide tables, descriptions, quotes, and arguments
to answer the research questions (apa-jars; area-empirical-social-
science-2006).

  * •

For quantitative results, provide summary or descriptive statistics to answer
the research questions (apa-jars; area-empirical-social-science-2006).

Threats to Validity

  * •

Include a dedicated threats to validity section that lists internal, external,
construct threats, and biases as appropriate for the research question(s) and
study design (Schulzc-BMJ2010-ConsortStatement; apa-jars; area-empirical-
social-science-2006).

  * •

Discuss how the study design addresses the threats.

  * •

Discuss any threats not addressed by the study design.

  * •

Explain and justify the unaddressed threats.

  * •

Explain the potential impact on the interpretation of the study results due to
the unaddressed threats (Schulzc-BMJ2010-ConsortStatement; apa-jars; area-
empirical-social-science-2006).

###  5.3. Reviewer Guidelines for Reviewing Empirical Papers

The responsibility for high-quality publications about CER work does not rest
solely on authors. Review guidelines over the past several years have
clarified expectations for reviewing empirical work, both as CER and
experience reports. Reviewers have a responsibility to hold authors
accountable for following norms for reporting CER. We recommend that reviewers
use the author guidelines and standards documents above as a checklist for
things to provide feedback on during peer review. These guidelines and
standards echo guidelines and standards from education and other fields about
presentation of empirical work (Schulzc-BMJ2010-ConsortStatement; apa-jars;
NSF19002; area-empirical-social-science-2006; WWC2020).

##  6\. Threats to Validity

In developing our rubric and applying it to the work described in this paper,
we identified some limitations and threats to validity we must address. We
have backgrounds in conducting human-based empirical research in computing
education, software engineering, and security, but do not explicitly have
research degrees in education. However, two of authors have been members of
the program committee of SIGCSE TS for multiple years, including years
volunteering as program and symposium chairs, so we believe that we do have
the needed background.

During the creation of the CER Empiricism Assessment Rubric, we did not
explicitly review or reference other validated rubrics. Therefore, it is
possible that we omitted some aspects of educational research project design.
To address this threat to validity, after we completed our work, we compared
our rubric to other guidelines for reporting and assessing empirical research
study quality, such as APA JARS (apa-jars), the _What Works Clearinghouse
Standards Handbook_ (WWC2020) and CONSORT (Schulzc-BMJ2010-ConsortStatement),
as discussed in Section 2.2. We found that there was significant overlap in
the core concepts of these other guidelines and rubrics with our own,
indicating that our rubric captures many of the same reporting values.

In choosing the years and venues for inclusion in this review, we selected the
conferences and journals that are widely considered to be the top tier for CER
and only analyzed the proceedings or issues from two years, 2014 and 2015. We
did not include more general education conferences that have a computing
track, such as ASEE or FIE, and we excluded some other venues (e.g., Koli
Calling) to make the review more feasible. While this choice means we could
have missed a portion of the community, we believe that the venues we chose
cover the majority of the computing education literature. Further, some other
venues accepted papers based solely on the abstract, rather than the entire
paper, which we believe does not accurately represent the current state of
empiricism. We chose the particular set of years to create a baseline on
empirical work before more recent changes to paper tracks and reviewing
guidelines at SIGCSE TS and other venues. Also, we recognize that the findings
from 2014 and 2015 may not represent the current state of empiricism in 2021.
Future work will consider a review of more recent publications.

Furthermore, we do not have the perspective of manuscripts that were not
accepted into these venues. As Fincher, et al., discussed in the conclusion of
their book chapter (fincher_tenenberg_dorn_hundhausen_mccartney_murphy_2019),
a weakness of systematic literature reviews is that they focus on a
quantitative analysis to characterize a body of work through an author defined
lens and may miss the broader context of the timeframe in which the work was
published. We do not consider the broader context of computing education
during 2014 and 2015 and how that may impact the types of papers accepted.
However, we minimize this limitation by focusing on characterizing empirical
elements independent of the specific topic of the work (e.g., K-12 or CS1).
Our work provides value by suggesting how the community can improve reporting
standards independent of the broader community context for the acceptance and
publication of the reviewed literature.

In our application of the rubric, we settled on a two-pass approach. Two
researchers evaluated each paper using the Base Rubric \- evaluation method,
evaluation subject, evaluation subject source, comparison, and number of
participants. We used this approach to better ensure proper overall
categorization of papers. We calculated the inter-rater reliability among the
research team working on the categorization and determined only minor
differences in the evaluations. After this initial evaluation, one researcher
then completed the remaining items in the CER Empiricism Assessment Rubric
sub-rubrics as appropriate for each paper. We chose this approach because it
gave us the best balance between efficiency and accuracy.

However, we recognize the possibility that individual researchers
mischaracterized aspects of individual papers. For example, one researcher
might rate General Rubric: Research Objectives for a paper as Partial and Not
Labeled because she found a discussion of research objectives, but not a
dedicated section, while another reader could have missed the discussion
altogether because he read too quickly. We have provided our dataset
(dataset_2021) for other researchers to consider when utilizing our rubric.

##  7\. Conclusion

Our research goal was to characterize the reporting of empiricism in Computing
Education Research literature by identifying whether publications include
content necessary for researchers to perform replications, meta-analyses, and
theory building. This systematic literature review summarizes the type of
papers and studies included during 2014 and 2015 in the SIGCSE TS, ICER,
ITiCSE, TOCE, and CSE venues. A majority of the accepted papers report
empirical work. However those papers do not consistently follow reporting
norms. We have provided suggestions to authors and reviewers to move the
community forward in publishing high-quality empirical work that can lead to
meta-analysis and theory building.

We did observe progress in reporting empirical work in recent years. With the
creation of the CS education research track at the SIGCSE TS for SIGCSE 2018
(Perez-Quinones:2017:MPT:3129166.3129172), the organizers updated the review
criteria to specifically request reviewers evaluate the items we include in
our General Rubric. Additionally, TOCE now recommends that authors utilize APA
JARS when organizing their submissions.

In future work, we will conduct a similar literature review on more recent
publications to gauge whether the community has made any progress. With
improved reporting, our next review will attempt to consider categorizing