# Expanding the scope of statistical computing: Training statisticians to be
software engineers

Alex Reinhart and Christopher R. Genovese

###### Abstract

Traditionally, statistical computing courses have taught the syntax of a
particular programming language or specific statistical computation methods.
Since the publication of Nolan and Temple Lang [2010], we have seen a greater
emphasis on data wrangling, reproducible research, and visualization. This
shift better prepares students for careers working with complex datasets and
producing analyses for multiple audiences. But, we argue, statisticians are
now often called upon to develop statistical _software_ , not just analyses,
such as R packages implementing new analysis methods or machine learning
systems integrated into commercial products. This demands different skills.

We describe a graduate course that we developed to meet this need by focusing
on four themes: programming practices; software design; important algorithms
and data structures; and essential tools and methods. Through code review and
revision, and a semester-long software project, students practice all the
skills of software engineering. The course allows students to expand their
understanding of computing _as applied to statistical problems_ while building
expertise in the kind of software development that is increasingly the
province of the working statistician. We see this as a model for the future
evolution of the computing curriculum in statistics and data science.

##  1 Introduction

When Nolan and Temple Lang [2010] wrote their seminal paper on the role of
computing in statistics and statistics curricula, they noted the rapid change
in the skills needed by practicing statisticians. It would no longer be
sufficient for statisticians to learn computing only as a collection of
numerical methods or specialized statistical algorithms, such as Markov chain
Monte Carlo or generating pseudo-random numbers. Statisticians now face large
quantities of data, often in new forms like text or networks, and this data
must be obtained—such as from Web services or databases—then managed, wrangled
in complex ways, and visualized. They argued that arming students with a solid
computational base will prepare them to adapt to the wide range of problems
they will see on the job—and that these computational skills will also give
them new ways to explore and understand the statistical concepts. They
suggested syllabi and curricula that would advance understanding of these
skills in both undergraduate and graduate programs [Nolan and Temple Lang,
2009].

This premise has only become more true in the intervening years. As the
conversation shifts to “data science” and organizations apply statistical
thinking to an ever wider range of problems, statisticians must use their
computational skills to acquire data from disparate sources, integrate it into
a useful form, conduct exploratory analyses and visualizations to understand
the data’s full complexity, and only then use statistical procedures to draw
conclusions. To ensure these conclusions are reproducible, statisticians must
also use computational tools like knitr [Xie, 2015] and the command line to
automate a pipeline of scripts, analyses and results.

In this paper, we argue that though these computational skills are important,
for some statisticians they are only a fraction of what is now needed. Many
statisticians now find themselves delivering not _analyses_ —in the form of
reports or presentations on some statistical analysis—but _products_ that are
used continually. In academia, these products might be R packages implementing
a newly developed statistical method, so that others can apply the same method
to their own problems. In industry, these products could be new methods to
detect fraud or improve advertising in a large online service, used
continuously as new data arrives and new decisions have to be made. In either
case, the product is often a large and complex piece of _software_ with a
codebase developed by a team over many months, and it’s never truly “done”: it
must be maintained and updated as conditions change and new requirements are
placed on it.

To build and maintain these products, statisticians need additional skills.
Structuring a large and complex codebase so it can be easily understood
requires principles from software engineering; writing code with a team
requires version control systems and collaboration skills; applying new
statistical methods to large and complicated data requires a firm
understanding of algorithms and data structures so the resulting code will be
efficient. And everything must be well-tested and debugged so colleagues,
bosses, and users can have confidence in the results. These skills are less
important for a one-off data analysis, but they are crucial for the tasks
statisticians face as they put their expertise into practice as part of long-
running systems and widely used products.

Beginning in 2015, we have developed a graduate-level course in statistical
computing intended to teach software engineering skills. The course is now
part of the required curriculum for both the Master’s in Statistical Practice
and the Ph.D. in Statistics & Data Science at Carnegie Mellon University,
serving roughly 40–50 students per year. Most of these students have prior
statistical programming experience from undergraduate courses, and our course
is their only required graduate-level statistical computing course. The
students have widely varied backgrounds, and though the master’s program
emphasizes professional skills [see Greenhouse and Seltman, 2018] while the
Ph.D. program emphasizes theoretical and applied research, the course goals
are shared: to prepare students to build complex statistical software.

In this paper, we set out the skills we aim to teach and the strategies we use
to teach them. Our pedagogy has evolved every year as we have discovered that
the course pedagogy is inextricable from its content. For students to learn
complex computational skills, we must give them regular practice with these
skills and rapid, targeted feedback on their performance. We argue that these
skills are becoming increasingly important for graduate-level statisticians
and cannot be left to others to fill in. Computation has only grown in
importance in the ten years since Nolan and Temple Lang issued their call to
action, and we expect it will only become _more_ important in the ten years to
come.

##  2 Role of Computing in Statistics and Data Science

Before we discuss the statistical computing course we developed, it will be
useful to briefly trace the evolution of computing’s role in statistics and
data science.

Since roughly 2000, a major focus of work and teaching in statistical
computing has been “Literate Statistical Practice”, which “encourages the
construction of documentation for data management and statistical analysis as
the code for it is produced” [Rossini, 2001]. Tools such as Sweave [Leisch,
2002] began to allow statisticians to embed the code producing their analysis
_inside_ the text of the analysis report, so that a single command would run
the analysis, produce the results, format tables and figures, and typeset the
report for distribution. This had many practical advantages, making it easy to
make small changes and then re-run analyses and reports from scratch, and has
become even more important as reproducibility has become a major concern.
Tools like knitr and R Markdown [Xie, 2015, Xie et al., 2018] have made
reproducible reports easier to write and easier to use, contributing to their
rapid spread.

These tools are now widely used in statistics education and in practice.
Baumer et al. [2014], for example, use knitr in introductory statistics
courses to “develop the basic capacity to undertake modern data analysis and
communicate their results,” and Çetinkaya-Rundel and Rundel [2018] described
its coordinated use in an undergraduate course sequence designed to develop
statistical computing skills in students from the introductory level onward.
In industry, Bion et al. [2018] describe the widespread adoption of knitr
among the data science team at Airbnb, who use it routinely to share their
analyses and business experiments with each other, with management, and even
publicly as blog posts and academic publications.

The other main emphasis of statistical computing work has been on tools to
make wrangling, restructuring, summarizing, and aggregating data much easier.
There is a growing emphasis on “tidy data” [Wickham, 2014], and the R
community has developed many new packages [e.g. the Tidyverse, Wickham et al.,
2019] that make it easy for statisticians to express the operations they need
to wrangle their data into the most convenient form. Other packages facilitate
interactive visualizations or make it easy to present statistical results in
written reports. Statistical computing curricula have adapted to include these
tools and to give students authentic practice wrangling messy data.

But we should not narrow our focus too quickly. Not every statistical task
fits into the framework of “receive a question, wrangle the data, conduct an
analysis, and write a report on the results.” No longer relegated to roles as
consultants or analysts brought in to answer specific questions, statisticians
and data scientists increasingly hold roles as integral parts of teams
developing products and delivering services. They need “data acumen,” which
includes facility with a much wider range of tools and the ability to
collaborate with software engineering teams and other disciplines [National
Academies of Sciences, Engineering, and Medicine, 2018, Chapter 2].

Bion et al. [2018] provided an insightful example of this shift. At Airbnb, a
service allowing property owners to list spaces for short-term rentals, the
data science team might build “a machine learning algorithm that takes into
account a variety of points of information” to suggest a fair price for a host
to charge for guests. But the outcome of this work is not a report to be
submitted to management describing the results of their modeling efforts—after
developing a prototype model, they “worked with engineers to bring the
prototype into production,” where hosts now use its recommendations every day.
That is, the final outcome was the _deployment_ of a piece of statistical
software, which now continually operates as a part of Airbnb’s core business.

We can also look beyond industry to see statistical software used for purposes
other than writing analytical reports. Consider a Ph.D. student conducting
theoretical work on new models for some complex type of data. This work may
involve thousands of lines of code: code to simulate data with known
parameters, code to estimate the model from data, code to run simulation
studies that verify theoretical results, code to calculate diagnostics or
measure goodness of fit, code to fit benchmark models and run comparisons,
etc. Much of the code forms a _product_ that an ambitious graduate student
might release as an R package submitted to CRAN or a Python package on PyPI,
allowing other researchers to benefit from their theoretical labor and use the
newly developed methods for their own practical purposes. And the wide
availability of these statistical products has been a boon for the field,
allowing new statistical methods to be quickly adopted in industry [Bion et
al., 2018]. The broader impact of this statistical software ecosystem is hard
to overstate.

The shift in statistical computing is noticeable in the work done by
statisticians in academia, but also by the jobs they take in industry. For
example, of 91 total graduates of Carnegie Mellon University’s Bachelor’s in
Statistics & Data Science program in 2018, 56 reported being employed at the
time of a survey of their career outcomes, and of these, 14 (25%) reported a
job title implying a software development role, such as “Software Engineer”
[CMU Career & Professional Development Center, 2018]. It has been our
experience that many industry roles titled “Data Scientist” or “Data Analyst”
also heavily involve software development.

##  3 Course Content

In the next sections, we discuss what it would mean for a statistical
computing curriculum to prepare students for these roles, and discuss a course
we developed to do so. We focus on four themes—four sets of skills students
must learn to effectively develop statistical products and not just
statistical reports. These themes are covered in lectures, but it is also
vital that the course give students repeated practice with all these skills,
and the necessary assignments and pedagogy will be discussed in Section 4.

###  3.1 Four Themes for Statistical Product Development

####  3.1.1 Effective Programming Practices

Students must learn practices that make software more reliable, more usable,
and easier to maintain. Such practices include testing, code review, clear
naming, and effective documentation.

Unit testing, for example, is often adopted in professional software
development to ensure code is correct and defects are not accidentally
introduced. A unit test isolates a specific “unit” of code, such as a function
or class, and runs that unit with specific inputs, then verifies that the
unit’s output matches the expected output. Unit tests are written using a
package designed to organize test cases, run all tests automatically with a
single command, and report summary results indicating which test cases failed
and giving descriptions of the failure. The testthat package is widely used
for R [Wickham, 2011], and similar packages are available for almost every
common programming language.

Unit testing is an essential part of software engineering for several reasons.
Most obviously, it helps ensure correctness of software. If each function or
method has detailed test cases, and these test cases can easily be checked
every time the code is changed, mistakes can be detected immediately. Software
engineering research shows that while writing unit tests takes extra time,
this time can in some settings be made up in the time saved fixing problems
and debugging errors [Williams et al., 2003, Bissi et al., 2016]. There have
been notable cases of errors in statistical and scientific software going
undetected for years, even as the software was used routinely for scientific
research, underscoring the importance of effective testing [Eklund et al.,
2016]. Less obviously, dividing up complex tasks into simple pieces—so they
can be easily tested—also encourages software to be composed of small, easily
understood pieces, which is a key software design recommendation (see Section
3.1.2).

Code review is another essential programming practice. In collaborative
software projects, such as software developed by a team in a large company or
an open-source package developed by a group of volunteers, collaborators often
practice _peer code review_ [Rigby and Bird, 2013, Sadowski et al., 2018].
Each proposed change to the software, such as a new feature or fix for a
problem, is submitted for review by a coworker or collaborator. The peer gives
line-by-line feedback on the code, enforcing project style guidelines, looking
for flawed reasoning and bugs, and giving feedback so the code can be
improved. Only after the proposed change has passed peer review is it merged
into the product or package.

Experiments have shown that code review detects bugs and improves software
quality, often by encouraging code to be clearer and easier to maintain
[Mäntylä and Lassenius, 2009, Beller et al., 2014]. Popular software
collaboration platforms like GitHub and GitLab support code review through
“pull requests” or “merge requests.” We give students experience with code
review in two ways. We first host an in-class activity in which students
reviewed real code written (by a course instructor) to solve a specific
problem. Students are given a code review checklist to follow, encouraging
them to look at specific features of the code and comment on them as part of
their review. Later in the semester, students conduct in-class peer code
review of their Challenge projects (see Section 4.3) using GitHub’s code
review features.

####  3.1.2 Fundamental Principles of Software Design

Throughout the semester, we emphasize a few key principles of _design_. This
includes modularity and code organization, the way that the many features
required of software are organized into files, functions, classes, scripts,
and so on. Effective software design is a powerful means to manage complexity.
In a poor design, functions may become large and complicated, and interact
with each other in complicated ways, so that changing one small part of the
code’s behavior requires intricate surgery on many separate functions. In an
effective design, functions are small and modular, and features are clearly
separated so that changing behavior only requires changing a few specific
functions that are clearly responsible for that behavior. Good design also
facilitates code reuse and refinement.

This kind of design is not a major concern when writing a literate statistical
report, which is mostly linear with a few helper functions. But when
developing a software package that’s intended to be reusable, careful design
is essential—a good design makes it easy to modify and extend the package, for
example as a Ph.D. student explores new methods in a thesis, while a bad
design can make changes excruciatingly difficult.

The semester-long Challenge project, described below in Section 4.3, is
designed to give students practical design experience. Since the project
requires students to build a complicated product over the entire semester, and
later portions of the project require students to build on or modify earlier
portions, they either experience the benefits of well-designed code or suffer
the pain of modifying poorly designed code. The teaching assistants also
provide extensive feedback on design, starting before students implement any
features.

####  3.1.3 Important Algorithms, Data Structures, and Representations

In recent years, a large amount of statistical research has been focused on
_scaling_ statistical methods to enormous datasets without an extravagantly
large computational budget. Commonly, statistical computing courses prepare
students to work with large datasets by teaching them different tools. SQL
database systems, for example, are designed to efficiently query massive
datasets that do not fit in memory, while software like Hadoop and Apache
Spark are designed to distribute calculations across multiple servers that
each have their own chunk of data. Students might also learn to use tools like
Rcpp, which allows users to write the most computationally intensive parts of
their R packages in efficient C++ code that can be easily called from within R
[Eddelbuettel and Francois, 2011]. (Cython [Behnel et al., 2011] serves a
similar role in the Python world.) And students are often exposed to R
programming folk wisdom: use built-in functions whenever possible, avoid for
loops in favor of vectorization, and perhaps use packages like data.table
instead of native data frames.

But this misses the ways that careful software design can make code efficient
and scalable. First, the designer must select an _algorithm_ appropriate to
the task at hand, meaning the designer must be familiar with general
strategies for designing algorithms. For example, the divide-and-conquer
strategy is to reduce a large problem into several smaller problems whose
solutions can be combined to yield the overall solution; by doing this
recursively, a complex problem can be reduced to many small and trivial
problems. The divide-and-conquer strategy is widely used in computer science
to produce algorithms that scale well to large datasets (for example,
mergesort is a divide-and-conquer sorting algorithm), and it has been recently
explored as a tool for implementing statistical methods on large datasets
[Jordan, 2013]. Dynamic programming is another widely used strategy to break
problems into smaller problems whose solutions can be combined efficiently;
for example, the fused lasso can be expressed as a dynamic programming
problem, leading to a linear-time algorithm [Johnson, 2013].

Along with the appropriate algorithm, the designer must also select
appropriate _data structures_ to store the data needed for an algorithm in an
efficient way. Students used to working in R for data analysis tend to think
of data frames, lists, matrices, and vectors as the only available data
structures, and often write algorithms that require repeatedly scanning
through an entire dataset to find relevant elements—which scales poorly to
large datasets. But data structures like hash tables (dictionaries), binary
trees, stacks, and queues all have their uses in statistical algorithms.

In statistics, for example, the $k$-d tree can store $n$ data points, each in
$k$ dimensions, and can find all data points in specific intervals or ranges
in $O(\log n)$ time, rather than requiring a loop through all $n$ data points
[Bentley, 1975]. This can also be used to solve $k$-nearest-neighbor problems
efficiently, and has been adapted to perform fast approximate kernel density
estimation [Gray and Moore, 2003]. Other tree data structures are widely used
by SQL databases to efficiently process queries with complex joins and WHERE
clauses.

Our Statistical Computing course covers basic algorithmic strategies such as
divide-and-conquer and dynamic programming, as well as basic data structures.
We emphasize to students that selecting the appropriate algorithm and data
structure can be much more important than the ordinary R performance tips. An
algorithm that uses repeated (but vectorized) scans through an entire data
frame or vector is intrinsically less efficient than one that uses a tree to
do the same operation in $O(\log n)$ time, for example.

In the course, various homework assignments pose simple problems that can be
solved in an obvious but tremendously inefficient way as well as a less-
obvious but efficient way using an appropriate algorithm and data structure.
(These can be challenging in R, which does not provide efficient data
structures by default; for example, looking up an item by name in an R list
requires an $O(n)$ scan through all entries, and base R does not provide
flexible collection data structures [Barry, 2018].) Along with the Challenge
projects, these assignments teach students that fast code often requires
careful thinking about the organization and use of data.

####  3.1.4 Essential Tools and Methods

There are many ways to produce good statistical software, but there are
several core tools that are almost universally useful. Such tools include
editors, integrated development environments, version control systems,
debuggers and profilers, databases (relational and otherwise), and the command
line. This theme focuses on giving students substantive experience with those
tools to give them a foundation for building good practices and habits going
forward.

We build experience with such tools into the structure of the course,
providing support for a range of quality tools while giving students as much
flexibility as possible. For instance, we cover using SQL in class and let
students interact with SQL databases through their favorite programming
language in assignments and class activities. Similarly, while it is possible
to work completely through graphical user interfaces (GUIs), we believe that
command line tools can add value for practioners and be a powerful tool in
many circumstances. We show students how to use these tools and build a set of
practices for the effective design and use of such tools.

Version control is a more challenging example. It is a critical tool for
successfully developing large-scale software in collaboration with others,
allowing team members to track the history of every source file. It allows
changes to be systematically recorded and reverted if necessary, and allows
collaborators working independently to make changes to code without
interfering with each other’s work. Version control software is now widely
used by companies and by collaborative open-source software projects. The R
system itself, for example, is developed using the Subversion version control
system.

There are many version control systems available, with non-trivial differences
in use and details. We focus particularly on Git, which is perhaps the most
widely used modern version control system, particularly with the growth of
web-based collaboration services such as GitHub, GitLab, and Bitbucket that
enhance Git with online tools for filing bug reports, reviewing proposed
changes to code, and tracking project timelines and milestones. Students who
are familiar with Git will be prepared to work at organizations that use Git
or similar systems, or to collaborate on any of the thousands of open source
data science packages that organize their development with Git. Bryan [2018]
has also persuasively argued that Git is valuable for managing the data, code,
and figures involved in a literate statistical analysis, such as data analysis
reports, further enhancing their reproducibility by making the history of
changes visible.

Unfortunately, Git is not known for being user-friendly. Its primary interface
is through the command line shell, and its documentation can be almost
impenetrable to new users. We have found that simply teaching the concepts to
students in a lecture is not sufficient; students need extensive practice
_using_ Git throughout the semester to begin to grasp its concepts. Hence
students use Git and GitHub to submit all homework assignments and course
projects, starting with an in-class tutorial during the first week.

Readers interested in using Git in their own courses may benefit from the
experiences of Çetinkaya-Rundel and Rundel [2018] and Fiksel et al. [2019],
who discuss how to use Git in statistics courses and describe common student
experiences, many of which match what we have seen in our course.

###  3.2 Anti-Themes

It would perhaps be most accurate to say that our course teaches a problem-
solving _philosophy_ , encompassed in the four themes presented above, rather
than simply a collection of tools suited for specific problems. This is
reflected by several topics we choose _not_ to cover in the course.

For example, our course _does not teach a programming language_. We assume
that our students have already had some exposure to programming, such as in an
undergraduate statistical computing course or through practical experience
conducting data analyses, and so we do not spend class time covering syntax or
programming constructs. We do not require students to use any specific
programming language for their work, and examples in our lectures are often
given in R, Python, C++, Clojure, Racket, and other languages.

As the concepts, practices, and skills covered in the course are widely
relevant, this design decision makes the course accessible to students from a
range of programming backgrounds. We believe an even bigger benefit of this
approach is the perspective it offers. A focus on a single language tends to
conflate approaches to problems with the way their solutions can be expressed
in that language. Instead, we often show examples in multiple languages so
that students can see both the commonalities that are conserved across most
languages as well as some contrast across other possible design choices and
idioms. Students quickly find that, even excepting a few syntactic details,
they can understand the approach taken in a wide range of languages and that
this affects how they approach problems even in their chosen language.

We encourage students to get some experience in a new language, even if only
on simple problems. Some students use this opportunity to explore languages
they expect to use in practice (such as Python, or C++ for use with Rcpp),
while others explore more widely and pick up functional or strongly typed
languages.

Similarly, the Statistical Computing course does not cover specific packages,
such as the Tidyverse [Wickham et al., 2019] or tools to obtain and wrangle
data (such as Web scraping systems). Such tools are important in practice, but
we feel it is more important for the course to cover fundamental computing
concepts that will enable students to effectively use whatever tools may
appear.

##  4 Pedagogy

One might suspect that a computing course emphasizing concepts without
teaching any specific programming languages or tools—as our does—can’t teach
the practical skills they need. But in a course intended to teach students a
complex skill, such as engineering statistical software, the only way for
students to learn the _skill_ —and not just the prerequisite knowledge for
that skill—is regular practice with targeted feedback. Regular practice gives
students the opportunity to practice the skills we teach, while targeted
feedback ensures they learn those skills and learn from their mistakes.

Hence the content of the Statistical Computing course cannot be separated from
its pedagogy. In this section, we describe a course design that ensures
students gain regular hands-on practice and detailed feedback, and the
practical considerations that went into it. Many of these pedagogical features
were developed through experience with each iteration of the course, and so
the course has changed significantly over time; these changes are summarized
in Table 1 and discussed in the subsections that follow.

Semester | Changes
---|---
Spring 2015 | Pilot version (half-semester)
Fall 2015 | Challenge project introduced (one short project)
Fall 2016 | Pull request & revision system; Master’s students join; 2 Challenge projects
Fall 2017 | Various small content & activity improvements
Fall 2018 | Switch to one four-part challenge
Fall 2019 | Recommended homework schedule provided
Fall 2020 | Problem bank rotation; Master’s students in separate course
Table 1: A summary of revisions and changes made to the Statistical Computing
course during each iteration, as discussed in Section 4. The course structure
has undergone many adjustments in response to experience.

###  4.1 Active Learning

In-class active learning has been repeatedly shown to improve student learning
in a variety of STEM fields [Freeman et al., 2014]. Most of our course
lectures incorporate active learning activities in various forms. For example,
early in the semester we cover unit testing; we have found that students often
struggle to think of test cases for code they write, so a large portion of the
unit-testing class is spent having the students work in small groups to think
of test cases for a few example functions.

Our experience has been that much of the student learning in a lecture seems
to come from these activities. We frequently discover that after spending 30
minutes lecturing on a particular topic and feeling that the lecture is going
well, an in-class activity reveals that some students are still deeply
confused and have misinterpreted much of the lecture. Without these
activities, the confusion could only be detected (and corrected) much later.

For key concepts that all students must master to succeed in the course, we
have gradually shifted from long lectures to in-class group activities that
students can later turn in individually for homework credit, ensuring that all
students practice the necessary skills before completing other assignments or
the Challenge project.

###  4.2 Homework Problem Bank

Because our students have varied levels of programming experience and have
varied goals for the course, we felt that an ordinary homework assignment
structure, where each student completes the same assignments, would be
inadequate. Some students may already be familiar with certain topics and
require little practice, while others may be most interested in a specific
topics they expect to use in their research or future job and desire specific
practice for that topic.

To allow students to choose assignments that suit their needs, we developed a
problem bank containing over 70 programming problems, categorized by topic and
difficulty level. Some problems have direct connections to statistics, while
others simply illustrate programming principles. Example assignments include:

  * •

Implement a kernel smoothing procedure, but allow the user to pass in any
kernel function and metric of their choice, not limited by any pre-specified
list of kernels and metrics built in to the code.

  * •

Implement graph search algorithms to solve mazes.

  * •

Implement simple text tokenization to produce bag-of-words vectors for
documents, then explore different distance metrics between documents of
different types.

To give students constant practice the themes discussed in Section 3.1,
students are required to write unit tests for every homework assignment, and
must submit them for review using Git. An automated system runs the unit tests
submitted with each homework assignment and verifies that all tests pass.

Throughout the semester, assignments from the problem bank are posted as the
relevant topics are covered. Students can select from all the posted
assignments those they believe are most interesting or relevant to their
needs, and complete the assignments at their own pace. They may use any
programming language that at least one of the course instructors or TAs is
able to read, though most students choose Python or R. Our grading system (see
Section 4.4) simply requires students to satisfactorily complete a certain
number of assignments by the end of the semester.

To encourage effective time management by students, who may be tempted to
abuse the flexibility of the homework system to put off submitting assignments
until near the end of the semester, we have used two different strategies. Our
first approach set a schedule by which students are expected to complete
certain numbers of homework points (see Section 4.4); our second approach
provides a rotating selection of assignments and retires assignments from the
problem bank after 2–3 weeks, so students must complete an assignment quickly
before it becomes unavailable. This also ensures that in a given week, the TAs
must only grade a few different types of assignment, allowing them to more
efficiently grade.

###  4.3 Challenge Project

The homework problem bank allowed students to gain practice in many topics
covered in the course, but small homework assignments do not cover a key
learning goal of the course: learning effective strategies to design and
develop large-scale software—with all the complexity it entails—over the
course of months or years. To achieve this, the course includes a semester-
long Challenge Project. Early in the semester, students choose between several
Challenges on varying topics, and work on their chosen project for the rest of
the semester.

For example, one Challenge asks students to implement an algorithm to build
and prune classification trees, then use this code to build random forest
classifiers [Breiman, 2001]. They then extend this code to build
classification trees using data stored in a SQL database without loading this
data into memory, in principle allowing the construction of trees for very
large datasets. Finally, they scrape abstracts from the arXiv preprint server
and use their random forest code to try to classify abstracts by subject
category using features extracted from the abstracts.

Initially the Challenge projects were designed to take several weeks and were
due in one unit. But since Fall 2018, the Challenge projects are broken into
four parts, due regularly throughout the semester. The entire project takes
roughly three months. This allows the projects to be more detailed and
ambitious, but also allows crucial scaffolding. In the first part, students
consider the design of their code but do not actually implement it. Instead,
they write function signatures but leave the bodies of the functions empty.
The only code required to be submitted is extensive unit tests demonstrating
what the code _should_ do, encouraging students to think more deeply about
their design before plunging in to implementation. The subsequent Challenge
parts ask students to successively add features, following the requirements
given in the assignment.

Besides the classification tree Challenge, other topics include applying the
isolation forest method for anomaly detection [Liu et al., 2012] to videos,
implementing fast data structures and algorithms for autocompletion [Wayne,
2016], and using audio fingerprinting to match short snippets of audios to a
database of recorded music [Li-chun Wang, 2003]. All the Challenges are
designed to produce a working piece of software that is usable in a relevant
context, e.g., a package or library, a web or mobile app, or a command-line
tool. All the Challenges are also designed to integrate multiple skills:
students must select appropriate data structures and algorithms for their code
to work efficiently, while using software design principles to keep it simple
and easy to maintain.

###  4.4 Specification-Based Grading and Revision

During the initial iteration of the course, homework grading was fairly
conventional: the teaching assistant graded code submissions using a simple
rubric, assigning a point value to each rubric category. However, we quickly
found this system to be ineffective, as students were not reviewing the
feedback or using it to improve their future submissions.

Beginning in Fall 2016, we switched to a new system. Students select
assignments from the problem bank and submit their code through GitHub as pull
requests. The teaching assistants then give detailed line-by-line feedback on
the pull requests using GitHub’s code review features. The reviews point out
bugs, critique difficult-to-read or poorly formatted code, suggest more
appropriate algorithms or data structures, request additional unit test cases,
and note design choices that make the code difficult to reuse or modify.

Crucially, there are only two possible outcomes of review: the assignment can
be marked Mastered, indicating the student has successfully solved the
problem, has used appropriate algorithms and data structures, and has written
the code with good style and with unit tests that verified its correctness;
or, if those criteria are not met, the assignment is marked “Changes
requested” and the student is asked to revise it according to the feedback.
Once revisions are complete, the student submits them for another review.

This system allows us to hold assignments to a very high standard. We expect
that a large fraction of submissions will be revised. (We cut the required
number of homework assignments in half at the same time as introducing the
revision system; student workload has not appreciably dropped, showing that
students are spending much more time on each assignment.) The revision process
gives students practice with a constellation of skills that are often
neglected in instruction and ensures they master the practical details of the
concepts covered in the course. One could even consider the code reviews to be
personalized tutoring provided by the teaching assistants, complementing the
lectures and activities led by the instructors. This tutoring is what allows
the course to cover topics at a high level and expect students to learn to
implement them in their chosen programming languages.

A similar revision system is used for the Challenge projects. Each part of the
Challenge is submitted to the teaching assistants for review as it is
completed, and the student must make satisfactory revisions before they can
submit the next part of the Challenge. Once all parts of the Challenge are
complete and meet the requirements, it can be graded either Mastered or
Sophisticated. Sophisticated submissions are those that demonstrate
exceptional software engineering skill, by being well-designed, clearly
written, thoroughly tested, unusually flexible and modular, and incorporating
apt choices of methods/algorithms and data organization. Earning a
Sophisticated grade on the Challenge increases the student’s course grade, as
discussed below in Section 4.5.

Prior research on mastery learning systems suggests strategies like this can
improve student learning [Kulik et al., 1990], though the additional
flexibility in our system distinguishes it from the more widely used mastery
grading systems.

###  4.5 Grading System

The course structure poses challenges for assigning final course grades. As
described in Section 4.2, students select homework assignments from a bank of
possible problems. Students can complete assignments in any order, and there
are no fixed deadlines for submitting individual assignments, nor are there
points to be averaged to give a final grade.

We base grades on the number of Mastered assignments. A simple table in the
course syllabus specifies how many assignments must be Mastered to achieve a
certain grade. The Challenge project is required, but achieving a
Sophisticated on the Challenge can also move the final course grade up one
grade level.

To account for the fact that individual homework assignments may involve
different degrees of difficulty, we assigned each homework assignment a
certain number of “points.” Typical assignments were 2 points, but difficult
assignments were 3 points and trivial assignments 1 point. The only possible
outcome is still either Mastered, meaning the student receives all the points,
or revision, meaning the student does not yet receive any points for the
assignment. There is no partial credit for submissions. The grade table is
then based on the number of points Mastered, rather than the number of
assignments, and accounts for assignment difficulty, preventing students from
simply choosing the easiest assignments to complete.

We found that this grading system has several advantages. It is noticeably
simpler than a normal points-based system to grade and administer, reducing
workload on the TAs. It reduces uncertainty for students, who know that if
they revise their submissions as instructed, they will obtain a certain number
of points, and these points translate into grades. There is no concern over a
final exam that heavily affects final grades—there is no final exam—and
students know exactly how much work they must do for a certain grade. It also
gives the students the flexibility to explore the problem bank to improve
their skills and gives them incentive to tackle some of the more challenging
problems.

##  5 Conclusion

The trends that motivated Nolan and Temple Lang’s call for a new focus on
computing in statistics curricula have only accelerated. The scope and
complexity of computing tasks expected of statisticians and data scientists
require not only a detailed knowledge of statistical methods and numerical
approaches but also skills related to data management, collaboration, and
software engineering. Our Statistical Computing course is designed to give
students a firm foundation—and authentic practice—in these skills. It is
intended to serve as a base on which their programming experience can be built
throughout their graduate career, and beyond. Novel features of our course
include emphasis on the practice of software design, our multi-path problem
bank, our grading system, integrated code review, regular revision, and a
language-agnostic approach. The course has been successful in both our Ph.D.
and Master’s program. Implementation, particularly at scale, is a continuing
challenge, and we will continue to develop and refine the course.

Statistical computing is a broad topic, and students come with varied
backgrounds and downstream needs. There are many reasonable approaches to
teaching students the computing skills they will need in their careers. We
believe, however, that working statisticians in industry and academia face
increasingly stringent demands on the capabilities, usability, and maintenance
of the software they produce, and that literate report-writing is only one
component of the many computational skills a successful statistician will
need. As the field’s computing curricula continue to evolve, we believe that
this reality needs to be faced head on.

###  5.1 Student Feedback

Our university conducts anonymous course evaluation surveys at the end of each
semester; student participation is voluntary and response rates can vary
widely. Nonetheless, the quantitative data and comments from students can
sometimes provide useful information about how a course is being received.

According to the survey results, students in our statistical computing course
report working roughly 11 hours per week on the course, which is above the
intended average of 9 for a course worth its number of credits. Student
comments attributed this to the fast pace of the course: for example, one
student wrote that “As the class was designed covers a lot of topics that
would take a couple semesters in normal C.S. courses, this course is
definitely conceptually difficult and has a quite heavy workload.” We do not
think this is an unfair characterization, particularly for students with less
prior programming experience, and continue to adjust the curriculum and pace
based on student feedback.

Nonetheless, students enjoyed the pedagogy of the course and its flexibility.
One student noted that “My favorite parts were the interactive parts;” the
interactive activities discussed in Section 4.1 “helped me feel more engaged
and helped me understand the problems better.” Similarly, one student noted
that “I could pick and choose easier and harder assignments, and get to
explore new areas that interested me without being overwhelmed and stressed
out constantly.” Though this flexibility was appreciated, it also has its
drawbacks, as noted by the student who complained that “The homework system
also really opened my eyes to how bad I am with time management.” With no
homework deadlines during the semester, some students experienced a mad rush
to get the required number of assignments completed in the last few weeks.

###  5.2 Future Improvements

The topics we emphasize have changed from year to year as our understanding of
student needs have changed, and the prior skills of each student cohort have
varied significantly from year to year. Also, the unconventional course
structure, while giving students significant freedom to explore their
interests, has required a great deal of experimentation to improve, and likely
will continue to change each year as we learn what structure best teaches our
intended skills.

Several challenges remain to be solved. Git has proven to be a major obstacle
to students; they must use it to submit each homework assignment on GitHub for
review by the TAs, but students who make mistakes often attempt to fix them
with ad-hoc solutions found online, leading to tangled Git histories that must
be carefully un-tangled by the instructors or TAs before assignments can be
graded. The flexible homework system can sometimes be too flexible, and
without formal deadlines, students can procrastinate and get into difficult
situations, or skip assignments selected as in-class activities and miss
important skills needed for the Challenge project. (This has been a bigger
issue for Master’s students than for Ph.D. students.) The demands on course
TAs to give high-quality feedback on assignments while also holding regular
office hours can be stressful, requiring skilled TAs and large time
commitments. We are working to streamline the course, automate some aspects of
homework submission and review, and improve the student experience.

###  5.3 Implementing a Similar Course

For those interested in teaching the core themes we describe in Section 3.1,
comprehensive lecture notes are available at our course website,
https://36-750.github.io/. (The website includes more than one semester worth
of material: it includes notes on every topic that has been taught in the
course, even as the selection of topics has changed from year to year.) The
notes include in-class active learning activities, example programs, and notes
that were used during lectures. The homework problem bank (Section 4.2),
including solutions to some problems, is kept privately by the authors and is
available to instructors on request.

But so far, we have left one key question open: Who can effectively teach a
statistical computing course on the topics we describe? The four core themes
require faculty with experience designing and implementing complex software;
while an introductory R programming class simply requires knowledge of syntax
and some basic principles, our themes include principles of algorithms, data
structures, and software design. To give effective feedback, the course
teaching assistants must also be experienced programmers who can recognize
inefficient algorithms or unnecessarily complex designs.

These constraints limit who can teach a course covering the skills we feel are
most important, at least until such teaching becomes more widespread and
faculty can be expected to have these skills. It may be practical, however, to
co-teach the class. An instructor experienced in statistics and data science
could cover those topics, while an instructor experienced in software
engineering, perhaps from another department, provides the core computing
content.

This raises a question: Why not have students take a computer science or
software engineering course from another department? While a fair portion of
the material we emphasize (including algorithms, data structures, testing,
software design, and wide-ranging assignments) might seem more naturally
obtained from a Computer Science department, we have found many reasons to
prefer that material within a Statistics curriculum. First, we _explicitly_
address these themes, with significant class time; these skills tend to be
threaded more implicitly throughout a typical computer science curriculum.
Second, we can focus the practice of our target skills with context and
examples that are meaningful to Statistics and Data Science students. Third,
having a single foundation course early in the Statistics graduate curriculum
has been a significant downstream productivity enhancer for our students, who
soon use the skills in their other courses and projects. Finally, we know of
no other course, in Computer Science or elsewhere, that achieves our target
balance on our themes and skill development.

## Acknowledgments

We thank Peter Freeman for contributions to the course design and content
during its first iteration. We are indebted to our excellent teaching
assistants for their outstanding work in the class: Philipp Burckhardt,
Niccolò Dalmasso, Sangwon Hyun, Nicolás Kim, Francis Kovacs, Taylor Pospisil,
and Shamindra Shrotriya. Jerzy Wieczorek provided helpful insight on our
mastery grading system. We thank the peer reviewers and guest editor for many
suggestions that improved the manuscript.

## References

  * Barry [2018] Timothy Barry.  Collections in R: Review and proposal.  _The R Journal_ , 10(1):455–471, 2018.  doi: 10.32614/RJ-2018-037.
  * Baumer et al. [2014] Benjamin S. Baumer, Mine Çetinkaya-Rundel, Andrew Bray, Linda Loi, and Nicholas J. Horton.  R Markdown: Integrating a reproducible analysis tool into introductory statistics.  _Technology Innovations in Statistics Education_ , 8(1), 2014.  URL https://escholarship.org/uc/item/90b2f5xh.
  * Behnel et al. [2011] S. Behnel, R. Bradshaw, C. Citro, L. Dalcin, D.S. Seljebotn, and K. Smith.  Cython: The best of both worlds.  _Computing in Science Engineering_ , 13(2):31 –39, 2011.  ISSN 1521-9615.  doi: 10.1109/MCSE.2010.118.
  * Beller et al. [2014] Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens.  Modern code reviews in open-source projects: Which problems do they fix?  In _Proceedings of the 11th Working Conference on Mining Software Repositories_ , pages 202–211, 2014.  doi: 10.1145/2597073.2597082.
  * Bentley [1975] Jon Louis Bentley.  Multidimensional binary search trees used for associative searching.  _Communications of the ACM_ , 18(9):509–517, 1975\.  doi: 10.1145/361002.361007.
  * Bion et al. [2018] Ricardo Bion, Robert Chang, and Jason Goodman.  How R helps Airbnb make the most of its data.  _The American Statistician_ , 72(1):46–52, 2018\.  doi: 10.1080/00031305.2017.1392362.
  * Bissi et al. [2016] Wilson Bissi, Adolfo Gustavo Serra Seca Neto, and Maria Claudia Figueiredo Pereira Emer.  The effects of test driven development on internal quality, external quality and productivity: A systematic review.  _Information and Software Technology_ , 74:45–54, 2016\.  doi: 10.1016/j.infsof.2016.02.004.
  * Breiman [2001] Leo Breiman.  Random forests.  _Machine Learning_ , 45(1):5–32, 2001.  doi: 10.1023/A:1010933404324.
  * Bryan [2018] Jennifer Bryan.  Excuse me, do you have a moment to talk about version control?  _The American Statistician_ , 72(1):20–27, 2018\.  doi: 10.1080/00031305.2017.1399928.
  * Çetinkaya-Rundel and Rundel [2018] Mine Çetinkaya-Rundel and Colin Rundel.  Infrastructure and tools for teaching computing throughout the statistical curriculum.  _The American Statistician_ , 72(1):58–65, 2018\.  doi: 10.1080/00031305.2017.1397549.
  * CMU Career & Professional Development Center [2018] CMU Career & Professional Development Center.  First destination outcomes: Dietrich College Statistics & Data Science, bachelor’s, 2018.  URL https://www.cmu.edu/career/documents/2018_one_pagers/dc/Bachelors%20Stats.pdf.
  * Eddelbuettel and Francois [2011] Dirk Eddelbuettel and Romain Francois.  Rcpp: Seamless R and C++ integration.  _Journal of Statistical Software_ , 40(8), 2011.  doi: 10.18637/jss.v040.i08.
  * Eklund et al. [2016] Anders Eklund, Thomas E Nichols, and Hans Knutsson.  Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates.  _Proceedings of the National Academy of Sciences_ , 113(28):7900–7905, 2016.  doi: 10.1073/pnas.1602413113.
  * Fiksel et al. [2019] Jacob Fiksel, Leah R Jager, Johanna S Hardin, and Margaret A Taub.  Using GitHub Classroom to teach statistics.  _Journal of Statistics Education_ , 27(2):110–119, 2019.  doi: 10.1080/10691898.2019.1617089.
  * Freeman et al. [2014] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. P. Wenderoth.  Active learning increases student performance in science, engineering, and mathematics.  _Proceedings of the National Academy of Sciences_ , 111(23):8410–8415, May 2014.  doi: 10.1073/pnas.1319030111.
  * Gray and Moore [2003] Alexander G. Gray and Andrew W. Moore.  Nonparametric density estimation: Toward computational tractability.  In _Proceedings of the 2003 SIAM International Conference on Data Mining_ , pages 203–211, 2003.  doi: 10.1137/1.9781611972733.19.
  * Greenhouse and Seltman [2018] Joel B Greenhouse and Howard J Seltman.  On teaching statistical practice: From novice to expert.  _The American Statistician_ , 72(2):147–154, 2018\.  doi: 10.1080/00031305.2016.1270230.
  * Johnson [2013] Nicholas A Johnson.  A dynamic programming algorithm for the fused lasso and $L_{0}$-segmentation.  _Journal of Computational and Graphical Statistics_ , 22(2):246–260, 2013.  doi: 10.1080/10618600.2012.681238.
  * Jordan [2013] Michael I Jordan.  On statistics, computation and scalability.  _Bernoulli_ , 19(4):1378–1390, 2013.  doi: 10.3150/12-BEJSP17.
  * Kulik et al. [1990] Chen-Lin C Kulik, James A Kulik, and Robert L Bangert-Drowns.  Effectiveness of mastery learning programs: A meta-analysis.  _Review of Educational Research_ , 60(2):265–299, 1990.  doi: 10.3102/00346543060002265.
  * Leisch [2002] Friedrich Leisch.  Sweave: Dynamic generation of statistical reports using literate data analysis.  In Wolfgang Härdle and Bernd Rönz, editors, _Compstat 2002 — Proceedings in Computational Statistics_ , pages 575–580. Physica Verlag, Heidelberg, 2002.  ISBN 3-7908-1517-9.
  * Li-chun Wang [2003] Avery Li-chun Wang.  An industrial-strength audio search algorithm.  In _Proceedings of the 4th International Conference on Music Information Retrieval_ , 2003.
  * Liu et al. [2012] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou.  Isolation-based anomaly detection.  _ACM Transactions on Knowledge Discovery from Data_ , 6(1):3:1–3:39, March 2012.  doi: 10.1145/2133360.2133363.
  * Mäntylä and Lassenius [2009] M. V. Mäntylä and C. Lassenius.  What types of defects are really discovered in code reviews?  _IEEE Transactions on Software Engineering_ , 35(3):430–448, May 2009.  ISSN 2326-3881.  doi: 10.1109/TSE.2008.71.
  * National Academies of Sciences, Engineering, and Medicine [2018] National Academies of Sciences, Engineering, and Medicine.  _Data Science for Undergraduates: Opportunities and Options_.  The National Academies Press, Washington, DC, 2018.  doi: 10.17226/25104.
  * Nolan and Temple Lang [2009] Deborah Nolan and Duncan Temple Lang.  Integrating computing into the statistics curricula, 2009.  URL https://www.stat.berkeley.edu/~statcur/.
  * Nolan and Temple Lang [2010] Deborah Nolan and Duncan Temple Lang.  Computing in the statistics curricula.  _The American Statistician_ , 64(2):97–107, 2010\.  doi: 10.1198/tast.2010.09132.
  * Rigby and Bird [2013] Peter C Rigby and Christian Bird.  Convergent contemporary software peer review practices.  In _Proceedings of the 9th Joint Meeting on Foundations of Software Engineering_ , pages 202–212, 2013.  doi: 10.1145/2491411.2491444.
  * Rossini [2001] A J Rossini.  Literate statistical practice.  In K Hornik and F Leisch, editors, _Proceedings of the 2nd InternationalWorkshop on Distributed Statistical Computing_ , 2001.
  * Sadowski et al. [2018] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli.  Modern code review: A case study at Google.  In _Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice_ , pages 181–190, 2018.  doi: 10.1145/3183519.3183525.
  * Wayne [2016] Kevin Wayne.  Autocomplete-me.  In _SIGCSE Nifty Assignments_ , 2016.  URL http://nifty.stanford.edu/2016/wayne-autocomplete-me/.
  * Wickham [2011] Hadley Wickham.  testthat: Get started with testing.  _The R Journal_ , 3(1):5–10, 2011.  URL https://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf.
  * Wickham [2014] Hadley Wickham.  Tidy data.  _Journal of Statistical Software_ , 59(10), 2014.  doi: 10.18637/jss.v059.i10.
  * Wickham et al. [2019] Hadley Wickham, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, Alex Hayes, Lionel Henry, Jim Hester, Max Kuhn, Thomas Lin Pedersen, Evan Miller, Stephan Milton Bache, Kirill Müller, Jeroen Ooms, David Robinson, Dana Paige Seidel, Vitalie Spinu, Kohske Takahashi, Davis Vaughan, Claus Wilke, Kara Woo, and Hiroaki Yutani.  Welcome to the tidyverse.  _Journal of Open Source Software_ , 4(43):1686, Nov 2019.  doi: 10.21105/joss.01686.
  * Williams et al. [2003] L. Williams, E. M. Maximilien, and M. Vouk.  Test-driven development as a defect-reduction practice.  In _14th International Symposium on Software Reliability Engineering_ , pages 34–45, Nov 2003.  doi: 10.1109/ISSRE.2003.1251029.
  * Xie [2015] Yihui Xie.  _Dynamic Documents with R and knitr_.  Chapman and Hall/CRC, Boca Raton, Florida, 2nd edition, 2015.  URL https://yihui.name/knitr/.  ISBN 978-1498716963.
  * Xie et al. [2018] Yihui Xie, J.J. Allaire, and Garrett Grolemund.  _R Markdown: The Definitive Guide_.  Chapman and Hall/CRC, Boca Raton, Florida, 2018.  URL https://bookdown.org/yihui/rmarkdown.  ISBN 9781138359338.


11institutetext: Metadata Research Center, Drexel University, Philadelphia,
PA, USA
11email<EMAIL_ADDRESS>22institutetext:
Department of Chemistry, University of Central Florida, Orlando, FL, USA
22email: kylerlanglois@knights.ucf.edu,jfurst<EMAIL_ADDRESS>33institutetext: Chemical and Biological Engineering, Colorado School of
Mines, Golden, CO, USA
33email<EMAIL_ADDRESS>
# Metadata for Scientific Experiment Reporting: A Case Study in Metal-Organic
Frameworks

Xintong Zhao 11 Kyle Langlois 22 Jacob Furst 22 Scott McClellan 11 Xiaohua Hu
11 Yuan An 11 Diego A. Gómez-Gualdrón Fernando J. Uribe-Romo Jane Greenberg
332211

###### Abstract

Research methods and procedures are core aspects of the research process.
Metadata focused on these components is critical to supporting the FAIR
principles, particularly reproducibility. The research reported on in this
paper presents a methodological framework for metadata documentation
supporting the reproducibility of research producing Metal Organic Frameworks
(MOFs). The MOF case study involved natural language processing to extract key
synthesis experiment information from a corpus of research literature.
Following, a classification activity was performed by domain experts to
identify entity-relation pairs. Results include: 1) a research framework for
metadata design, 2) a metadata schema that includes nine entities and two
relationships for reporting MOF synthesis experiments, and 3) a growing
database of MOF synthesis reports structured by our metadata scheme. The
metadata schema is intended to support discovery and reproducibility of metal-
organic framework research and the FAIR principles. The paper provides
background information, identifies the research goals and objectives, research
design, results, a discussion, and the conclusion.

###### Keywords:

FAIR,Research Reproducibility, Experiment Metadata, Metal-Organic Frameworks
(MOFs), Synthesis Reporting

##  1 Introduction

Metadata has been a key component of scientific research and data management
even in the analog world. The significance of metadata has expanded with the
growth in digital and networked technologies, data-driven science, and the
evolution of data science and AI. This importance is central to the
development and implementation of the FAIR principles, which are reviewed
below. FAIR principles, together with open science, open data, and data
sharing have also helped to highlight new metadata needs. One area in
particular, is the need for metadata that specifically records research
methods and procedures.

Metadata developments have emphasized resource discovery, access, and use, and
aspects of provenance tracking and linking, interconnected with the linked
data and the Semantic Web. Research methods follow general procedural
frameworks, with some more specific than others. In the sciences research
experiments, particularly those in the laboratory, are generally recorded in
analog or digital lab notebooks. While an individual lab may have a
recommended protocol or examples for reporting the experiment procedure,
individual guidelines are over generalized or absent when researchers publish
their results in scientific articles. Standardization in reporting of
experimental research procedures in scientific publications is necessary to
support reproducibility. The research presented in this paper considers this
need with a specific materials science research case for the production of
Metal Organic Framework (MOFs), which are produced through synthesis
experiments.

The paper sections that follow provide background information on FAIR metadata
for scientific research, metadata for research methods and procedures, and the
MOF case. Next, the research goals, objectives, and research design and
framework for identifying key MOF metadata entities is presented, followed by
the results, a discussion, and, the conclusion.

##  2 FAIR Metadata for Scientific Research

Metadata is essential for supporting the FAIR principles [32], and making
research data Findable, Accessible, Interoperable and Reusable. The goals
embodied in the FAIR principles have been key to the open science [17, 27] and
open data/data sharing movements [30], even prior to the publication wide-
spread global adoption of these principles. The FAIR principles along with
good data management practices explain, in part, the wide availability of
metadata standards for scientific research data. Key examples of globally used
metadata standards for research data include the Data Document Initiative
(DDI) [25, 4] for social science research data, the Darwin Core (DwC) [31, 3]
for museum specimens, and the Ecological Metadata Language (EML) [9, 7] for
ecological research data. The Digital Curation Centre’s (DCC) Disciplinary
Metadata directory [6], and the Metadata Standards Catalog [19] provide
information on and links to these metadata standards (DDI, DwC, and EML) along
with many other data-focused metadata standards. In addition to these
disciplinary-wide directories, there are agency/domain focused initiatives,
such as the United States Geological Survey (USGS) Metadata Creation
guidelines [8, 33] and the United Nations Food and Agriculture (UNFAO)
vocabulary services [28, 10]. These domain focused examples provide access to
a suite of resources and tools to guide metadata generation and verification
for research data and other digital resources.

The spectrum of metadata standards for scientific research further includes an
array of value standards (e.g., ontologies, taxonomies, vocabularies), and
metadata applied to other aspects of a research undertaking, such as metadata
reflecting access/usage rights [12], metadata for physical samples [5], or
even research instrumentation and software [20, 26, 15]. And, finally, we
cannot overlook the core component of scientific research metadata,
specifically the well-known metadata that represents the entities associated
with a publication citation, such as author name(s), title, publication date,
persistent identifier (e.g., DOI) and metadata representing other publication
properties. This associated metadata is critical for research databases,
citation indexes, and supporting FAIR. Overall, while the availability of
metadata standards and metadata generation approaches for documenting
scientific research has rapidly advanced over the last two decades, it’s clear
that less attention has been given to metadata specifically documenting the
research methods and underlying procedures. This limitation motivates the work
presented in this paper.

##  3 Metadata for Research Methods and Procedures: A Case for MOFs

Research is the drive for knowledge discovery. Research is shaped by the
scientific method, with the goal that research should be verifiable [2].
Researchers pursue new knowledge by asking questions, testing hypotheses,
measurement, experimentation, and other mechanisms. The field of research is
shaped by research methods and procedures. Among commonly taught textbook
methods are surveys, interviews, observational studies, anthropological
approaches, and experimental design. Each of these methods involve a set of
prescribed steps, and researchers frequently pursue mixed or multi-method
approaches. For example, some researchers will provide details, indicating the
steps, whereas others may tend to report the procedures at a more general
level.

Research has emphasized the need for more explicit metadata standards
targeting research methods. For example, Chao [1] sought to identify metadata
properties used to record methods across a set of eight schemes frequently
used for reporting scientific research. Her results show explicit and implicit
aspects, with four schemes having a mechanism for recording methods. Among
this set, only DDI and ThermoML (TML), indicated that the methods-related
metadata properties were mandatory. Researchers focusing on paradata [14],
provenance and pipelines [13], and reproducible computational research [16]
underscore the importance of metadata in this area. Additionally, some
research journals provide more concrete guidelines on reporting research
procedures, although such guidelines are not an official standard, they
provide a framework. To this end, some researchers also publish supplemental
material that provides detail on the procedures.

The developments highlighted here and Chao’s work, in particular, point to an
interest in standardization of research reporting. This need has is an
important goal for materials scientists working in the area of Metal Organic
Frameworks (MOFs), the research area that presents the case for the work
reported on in this research.

Metal Organic Frameworks (MOFs) materials are a type of crystalline materials
with open porous architectures with a predetermined structure [34]. MOFs
present an important class of materials as they have shown significant
potential in various applications such as CO2 emission, catalysis and drug
delivery. The large number of publications in the MOF area shows the diversity
of research approaches. In addition, given the complexity of design space of
MOF materials, a significant difference appears among synthesis experiment
reports from researchers. Since no specific standards for experiment reports
exist, the details from MOF synthesis experiments are not consistently
reported. For example, some synthesis reports can be more detail-driven
whereas some are more generalized; also, experimentalists could include
different types of details in their reports. The current state of reporting
severely limits researchers’ abilities to achieve FAIR, particularly
reproducibility, and motivates research to our goal to identify key metadata
entities for standardizing MOF synthesis experiment reporting.

##  4 Research Goals

The goal of this study is to identify key metadata components to facilitate
experiment reporting, using Metal-Organic Framework (MOFs) as a case study. We
present this work as a case study in that the results can contribute to larger
arena of reporting scientific research. In this paper, we report our framework
to design a metadata standard that structures synthesis experiments reported
in research articles. As a result, this “codification” of synthesis recipes is
beneficial to support discovery and reproducibility of metal-organic framework
materials.

To achieve this goal, we identified the following objectives:

  * •

What are the key components in a synthesis experiment?

  * •

How to quantify the key components by pairs of entities and relations?

##  5 Research Design and Framework

We pursued the MOF case study method using natural language processing and
interviews to gather domain expert feedback to develop the framework for
metadata standard for MOFs synthesis experiment. Overall, our framework
contains four main steps:

  * •

Identifying key components in MOF synthesis experiments and initialize
metadata design

  * •

Gather corpus of research articles and filter paragraphs related to synthesis
procedure

  * •

Apply the initial design of metadata to filtered synthesis paragraphs and send
to domain experts for assessment

  * •

Modification of metadata structure based on the feedbacks from domain experts
Detailed description of each step is shown below.

###  5.1 Identifying key components in MOF synthesis experiments and
initialize metadata design

(a) An example of MOF synthesis procedure

(b) Metadata Design to Describe Synthesis Experiment for MOF materials

Figure 1: Identifying key components in MOF synthesis experiments and
initialize metadata design

We conducted qualitative analysis to gain better understanding of domain-
specific needs in order to design the metadata. Namely, we performed the
following two steps: first, we reviewed previously established research about
MOF article text mining [11, 24]; second, we arranged interviews with
materials scientists to identify the critical components in synthesis
experiments. Based on the results of two steps above, we quantified the
synthesis procedure using the metadata structure demonstrated in figure 1.
Figure 1(a) is an example of a real synthesis procedure to create a MOF
material. Based on the findings from literature review and interviews with
domain scientists, we quantify such synthesis procedure as a sequence of
actions associated with specific attributes. The design of metadata is shown
in figure 1(b): the whole synthesis experiment is linked by synthesis actions
(e.g., ultrasonicate, cool, dry), and specific details (e.g., acids, chemical
composition/precursors, numerical values) are associated with actions. To this
end, we design our metadata for synthesis reports as a set of entities linked
by a set of relations. An initial list of key entity types is determined. We
identified entity types that appeared in previous studies and incorporated
them into our metadata design. The following entity types are included in the
initial list: chemical composition(i.e., precursors), synthesis action and
numeric descriptor.

In addition to the entity types, we also defined two relation types to link
pairs of entities: (1) has_value and (2) associated_with. The entity relation
pair plays an important role in the synthesis procedure of metal-organic
frameworks. It can indicate links between two different entities; for example,
when several synthesis actions (e.g., cooled, heated, ultrasonicated) and some
descriptors (e.g., room temperature, 2 hours) are identified, we can link
actions with their corresponding descriptor explicitly by assigning relations
between them (e.g., cooled under room temperature). In this step, we
initialized a list of entity and relation types to describe synthesis data.

###  5.2 Gather corpus of research articles and filter paragraphs related to
synthesis procedure

After initializing our metadata to describe synthesis experiments, the next
step will be to assess the initial design and make further modification. To
this end, we decided to apply our initial metadata design to synthesis
experiments reported in different research articles. To gather such a corpus
consisting of MOF synthesis experiments, we collected 215,866 full research
articles from the American Chemical Society (ACS) that are categorized under
materials science study. Then we filtered MOF-related articles by matching
their DOI numbers to DOIs listed in the Cambridge Structural Database (CSD),
which is a major database registering crustal structures like MOFs.

Figure 2: An Example of Used Prompt for Named Entity Recognition

After finding MOF articles, we used a few-shot learning method based on GPT
(Generative Pre-trained Transformers) models to identify specific paragraphs
that describe the synthesis procedure - we first manually identified a few
synthesis paragraphs as examples, then applied the fine-tuned GPT models to
all collected articles to identify synthesis paragraphs. We selected GPT model
Curie to perform the namend entity recognition. To fine-tune the Curie GPT
model, we applied prompt engineering based on the manually annotated examples.
The fine-tuning process was handled by OpenAI API[22]. An example of prompt
and completion structure for the few-shot learning is demonstrated in figure
2. Among all MOF-related articles, 2796 synthesis paragraphs were identified.

###  5.3 Send sample paragraphs to domain experts for initial assessment

Among synthesis paragraphs collected above, 15 paragraphs were randomly
selected and sent to domain scientists to perform a first round assessment of
the metadata schema. Two domain scientists were asked to annotate the 15
synthesis reports based on the metadata designed above. During the annotation
process, they need to provide answers to the following questions:

  * •

Is current metadata able to include all important information described in the
experiment report?

  * •

If the current metadata does not capture all critical information from the
synthesis steps, what should we add to the metadata?

  * •

Does current metadata design contain redundant components? If yes, then what
should we remove from current metadata design?

Figure 3: An Example of Annotated Paragraph about MOF Synthesis based on our
Metadata Schema

During the annotation process, domain scientists recorded entity types that
they believe worth incorporating into the schema. The annotation procedure was
performed on a software tool called Doccano. Figure 3 provides a visualized
example of synthesis report annotation based on our metadata design.

###  5.4 Further Modification based on Experts’ Feedback

We analyzed the 15 annotated paragraphs and arranged follow up discussion with
domain scientists. Based on this first-round annotation, our domain scientists
suggested that all components in the current metadata design are important and
hence should be kept. However, the current metadata design is not sufficient
to capture all critical information from a synthesis procedure. In this way,
we incorporated new entity types suggested by scientists to our metadata
schema. The following three entity types were added: 1) Vessel 2) MOF
structure and 3) Acid.

##  6 Results: Current Progress and Future Work

Having observed these disparities in MOF experiment reporting above, we
noticed an urge to standardize these unstructured yet very valuable synthesis
reports. To this end, we address this need by providing the following three
outputs: (1) a research framework for metadata design, (2) a metadata schema
that includes nine entities and two relationships for reporting MOF synthesis
experiments, and (3) a growing database of MOF synthesis reports structured by
our metadata scheme.

In order to better adapt our framework to broader scientific experiment areas,
we generalized our proposed research framework in section 5 and proposed a
general framework for experimental metadata design. As demonstrated in Figure
4, our research framework with human-in-the-loop can be generalized to four
main steps: (1) key component identification, (2) underlying data collection,
(3) applying designed metadata to the underlying data and (4) assessment on
designed metadata and further modification.

Figure 4: A Generalized Framework for Designing Metadata for Scientific
Experiments

While designing a metadata to describe experimental data, identifying key
components in the target underlying data is the first step. The initial design
of metadata for scientific experiments can be done by conducting literature
review of previous established literature and obtaining knowledge from human
experts. Then it is important to test our metadata design by mapping it to
collected target experimental data. Finally, domain experts will jump in again
at the final assessment of metadata design and make further modification on
the design if needed.

As a result of the Metal-Organic Framework case study, our final design of
metadata for MOF synthesis experiment is shown in figure 5 as follows.
Overall, we defined nine entities in hierarchical level: We have precursors,
synthesis actions, acid, vessel, descriptor and resulted mof chemical
composition as main entity types; the entity type “precursor” also has three
following sub entity types: metal precursor, organic precursor and solvent
precursor.

Figure 5: Overall Metadata Design for MOF Synthesis Experiments

From May through August 2023, domain experts engaged in manually annotating 80
synthesis paragraphs based on the metadata schema reported in this paper. The
annotated paragraphs are stored as a database of MOF materials synthesis
experiments and each synthesized MOF structure is linked to its corresponding
research article. In the near future, one work priority is to expand our
synthesis database tagged by the above metadata standard. To apply our
metadata design to a large amount of synthesis reports, we are conducting
experiments on using generative AI models and 80 annotated reports as training
data to perform automatic semantic labeling to the rest of 2000+ synthesis
reports.

##  7 Discussion

The need of structuring scientific experiment reports has gained increasing
attention across various research communities [29, 21, 18]. There’s even
interest amongst publishers. This especially holds true for the metal-organic
framework studies[23, 24]: due to the complex design space of metal-organic
framework materials, the way experimentalists describe their synthesis
procedures significantly varies from each other. Other than the difference in
writing style, they also tend to report different sets of aspects related to
the synthesis experiment. As reported in the results, not all experimentalists
report the type of vessels used in the synthesis procedure: some researchers
believe vessels used during the experiment could make a difference in the
final output, whereas some researchers do not think it would matter.
Similarly, there is also a difference in the scope of reports, where some
experiment reports are more detail-driven and some reports are more
generalized.

Overall, this study on metadata design for MOF experimental data is beneficial
to the broader research community – our framework for metadata design can be
adapted to different scientific research areas.The growing MOF synthesis
database structured by our metadata schema has high potential for multiple
aspects, such as synthesis information retrieval, synthesis design and it
ultimately helps the reproducibility of synthesis experiments.

This study is not without limitations. While initializing our metadata design
for MOF experimental data, we interviewed four material scientists studying
metal-organic framework structures. Three MOF experts were on the experimental
side and one MOF expert was on the computational side. However, there are many
more scientists conducting research in the MOF area and we plan for further
assessment of the initial entity set. In this way, this study can be limited
in terms of the scope of human expert interviews.

##  8 Conclusion

In this paper, we identified the need to structure scientific experiment
reports and conducted a case study on Metal-Organic Framework studies. We
address this need by providing a research framework to design experimental
metadata and a growing database for MOF synthesis reports structured by our
designed metadata schema. The resulting metadata schema for MOF synthesis
experiments can be beneficial to facilitate MOF material discovery. The
database will be further expanded by leveraging large pre-trained language
models for automatic semantic labeling in the short future.

## References

  * [1] Chao, T. C. (2014). Enhancing metadata for research methods in data curation. Proceedings of the American society for information science and technology, 51(1), 1-4.
  * [2] Cowles, H. M. (2020). The scientific method: An evolution of thinking from Darwin to Dewey. Harvard University Press.
  * [3] Darwin Core https://www.tdwg.org/standards/dwc/
  * [4] DDI Alliance https://ddialliance.org/Specification/
  * [5] Devaraju, A., Klump, J., Tey, V., Fraser, R., Cox, S., & Wyborn, L. (2017). A digital repository for physical samples: Concepts, solutions and management. In Research and Advanced Technology for Digital Libraries: 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings 21 (pp. 74-85). Springer International Publishing.
  * [6] Digital Curation Centre https://www.dcc.ac.uk/guidance/standards/metadata
  * [7] Ecological Metadata Language https://eml.ecoinformatics.org/
  * [8] Faundeen, J. L., Burley, T. E., Carlino, J., Govoni, D. L., Henkel, H. S., Holl, S., … & Zolly, L. S. (2013). The United States geological survey science data lifecycle model. Reston, VA, USA: US Department of the Interior, US Geological Survey.
  * [9] Fegraus, E. H., Andelman, S., Jones, M. B., & Schildhauer, M. (2005). Maximizing the value of ecological data with structured metadata: an introduction to ecological metadata language (EML) and principles for metadata creation. Bulletin of the Ecological Society of America, 86(3), 158-168.
  * [10] Food and Agricultural Organization of the United Nations https://www.fao.org/faoterm/language-resources/en/
  * [11] Glasby, L. T., Gubsch, K., Bence, R., Oktavian, R., Isoko, K., Moosavi, S. M., … & Moghadam, P. Z. (2023). DigiMOF: A Database of Metal–Organic Framework Synthesis Information Generated via Text Mining. Chemistry of Materials.
  * [12] Grabus, S., & Greenberg, J. (2019). The landscape of rights and licensing initiatives for data sharing. Data Science Journal, 18, 29-29.
  * [13] Huvila, I., Greenberg, J., Sköld, O., Thomer, A., Trace, C., & Zhao, X. (2021). Documenting Information Processes and Practices: Paradata, Provenance Metadata, Life‐Cycles and Pipelines. Proceedings of the Association for Information Science and Technology, 58(1), 604-609.
  * [14] Huvila, I., & Sinnamon, L. (2022). Sharing Research Design, Methods and Process Information in and out of Academia. Proceedings of the Association for Information Science and Technology, 59(1), 132-144.
  * [15] Katz, D. S., Gruenpeter, M., & Honeyman, T. (2021). Taking a fresh look at FAIR for research software. Patterns, 2(3).
  * [16] Leipzig, J., Nüst, D., Hoyt, C. T., Ram, K., & Greenberg, J. (2021). The role of metadata in reproducible computational research. Patterns, 2(9).
  * [17] McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., … & Yarkoni, T. (2016). How open science helps researchers succeed. elife, 5, e16800.
  * [18] McLeroy, K. R., Garney, W., Mayo-Wilson, E.,& Grant, S. (2016). Scientific reporting raising the standards. Health Education & Behavior, 43(5), 501-508.
  * [19] Metadata Standards Catalog https://rdamsc.bath.ac.uk/
  * [20] Michiorri, A., Sempreviva, A. M., Philipp, S., Perez-Lopez, P., Ferriere, A., & Moser, D. (2022). Topic Taxonomy and Metadata to Support Renewable Energy Digitalisation. Energies, 15(24), 9531.
  * [21] Nandy, A., Duan, C., & Kulik, H. J. (2021). Using machine learning and data mining to leverage community knowledge for the engineering of stable metal–organic frameworks. Journal of the American Chemical Society, 143(42), 17535-17547.
  * [22] https://pypi.org/project/openai/
  * [23] Park, S., Kim, B., Choi, S., Boyd, P. G., Smit, B., & Kim, J. (2018). Text mining metal–organic framework papers. Journal of chemical information and modeling, 58(2), 244-251.
  * [24] Park, H., Kang, Y., Choe, W., & Kim, J. (2022). Mining insights on metal–organic framework synthesis from scientific literature texts. Journal of Chemical Information and Modeling, 62(5), 1190-1198.
  * [25] Rasmussen, K. B. (2014). Social Science Metadata and the Foundations of the DDI. IASSIST Quarterly, 37(1-4), 28-28.
  * [26] Stocker, M., Darroch, L., Krahl, R., Habermann, T., Devaraju, A., Schwardmann, U., … & Häggström, I. (2020). Persistent identification of instruments. arXiv preprint arXiv:2003.12958.
  * [27] Stodden, V. (2010). Open science: policy implications for the evolving phenomenon of user-led scientific innovation. Journal of Science Communication, 9(01), A05.
  * [28] Subirats, I., Malapela, T., Dister, S., Zeng, M., Goovaerts, M., Pesce, V., … & Keizer, J. (2012). Reorienting open repositories to the challenges of the semantic web: experiences from FAO’s contribution to the resource processing and discovery cycle in repositories in the agricultural domain. In Metadata and Semantics Research: 6th Research Conference, MTSR 2012, Cádiz, Spain, November 28-30, 2012. Proceedings 6 (pp. 158-167). Springer Berlin Heidelberg.
  * [29] Trench, B. (2009). Science reporting in the electronic embrace of the Internet. na.
  * [30] Uhlir, P. F., & Schröder, P. (2007). Open data for global science. Data Science Journal, 6, OD36-OD53.
  * [31] Wieczorek, J., Bloom, D., Guralnick, R., Blum, S., Döring, M., Giovanni, R., … & Vieglais, D. (2012). Darwin Core: an evolving community-developed biodiversity data standard. PloS one, 7(1), e29715.
  * [32] Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., … & Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific data, 3(1), 1-9.
  * [33] U.S. Geological Survey
https://www.usgs.gov/data-management/metadata-creation

  * [34] Yaghi, O. M., O’Keeffe, M., Ockwig, N. W., Chae, H. K., Eddaoudi, M., & Kim, J. (2003). Reticular synthesis and the design of new materials. Nature, 423(6941), 705-714.