Terminologies for Reproducible Research

Lorena A. Barba, the George Washington University, Washington D.C.

January 2018

## Introduction

Reproducible research—by its many names—has come to be regarded as a key
concern across disciplines and stakeholder groups. Funding agencies and
journals, professional societies and even mass media are paying attention,
often focusing on the so-called “crisis” of reproducibility. One big problem
keeps coming up among those seeking to tackle the issue: different groups are
using terminologies in utter contradiction with each other. In July 2017, over
a dozen participants joined the Workshop on Reproducibility Taxonomies for
Computing and Computational Science at the National Science Foundation, to
appraise the variety of terminologies. My presentation at that event Barba
(2017) condensed a catalog of uses of the recurrent terms _reproduce_ and
_replicate_ , often meaning different things but sometimes interchangeable.

Looking at a broad sample of publications in different fields, we can classify
their terminology via decision tree: they either, _A_ —make no distinction
between the words _reproduce_ and _replicate_ , or _B_ —use them distinctly.
If _B_ , then they are commonly divided in two camps. In a spectrum of
concerns that starts at a minimum standard of “same data$+$same methods$=$same
results,” to “new data and/or new methods in an independent study$=$same
findings,” group $1$ calls the minimum standard _reproduce_ , while group $2$
calls it _replicate_. This direct swap of the two terms aggravates an already
weighty issue. By attempting to inventory the terminologies across
disciplines, I hope that some patterns will emerge to help us resolve the
contradictions.

## Pioneers of reproducible research

The first appearance of the phrase “reproducible research” in a scholarly
publication appears to be an invited paper presented at the 1992 meeting of
the Society of Exploration Geophysics (SEG), from the group of Jon Claerbout
at Stanford Claerbout and Karrenbach (1992). Claerbout pioneered the use of
computers in processing and filtering seismic exploration data Wikipedia
(2017). From at least 1990, he required his students’ PhD theses to conform to
a standard of reproducibility. His idea of reproducible research was to leave
finished work (an article or a thesis) in a state that allowed colleagues to
reproduce the calculation, analysis and final figures by executing a single
command. The goal was to merge a publication with its underlying computational
analysis. They used an automation tool called make, which builds software from
source code by reading through a makefile: a list of commands to be executed
in sequence. Their workflow combined a set of standardized commands (burn,
build, view, clean), and a filing system for the research compendium
associated with the paper (data sets, programs, scripts, parameter files,
makefiles). Explicitly, Claerbout and Karrenbach (1992) provide a “definition
of reproducibility in computationally oriented research.” The original SEG
paper is somewhat dated, but the group presented an updated overview in Schwab
et al. (2000). Here, the authors emphasize the limitations of the traditional
methods of scientific publication, especially for computational research. In
their vision of reproducible research, readers should be able to rebuild
published results “using the author’s underlying programs and raw data.”
Implicitly, they are advocating for open code and data.

At Stanford, statistics professor David Donoho learned of Claerbout’s methods
in the early 1990s, and began adopting (and later promoting) them. In Buckheit
and Donoho (1995), the authors state that “reproducibility of experiments in
seismic exploration requires having the complete software environment
available in other laboratories and the full source code available for
inspection, modification, and application under varied parameter settings.”
This is because the goal in this field is to produce an image of the earth’s
sub-surface, from the datasets of seismic exploration signals. The software
generating the image is key to the final results. And here is where the
authors write the often-quoted “slogan” of reproducible research (paraphrasing
Claerbout):

> _“An article about computational science in a scientific publication is not
> the scholarship itself, it is merely advertising of the scholarship. The
> actual scholarship is the complete software development environment and the
> complete set of instructions which generated the figures.”_

Buckheit and Donoho (1995) include many technical details that seem dated now,
so perhaps a better place to read about this group’s thinking and practice is
Donoho et al. (2009). This appears to be the first article to publicly state
that reproducibility depends on open code and data. The authors define
_reproducible computational research_ as that “in which all details of
computations—code and data—are made conveniently available to others.” They
used the now-recurrent charged language about a “credibility crisis” in
computational science, and worry that computation cannot claim to be the
“third branch” of science because most computational results cannot be
verified. In the two traditional branches, standards of practice already exist
for managing the ubiquity of error: deductive science uses formal logic and
the mathematical proof, while empirical science uses statistical hypothesis
testing and detailed methods reporting. The authors also counter the view that
reproducibility means re-implementing the research software from the ground
up. Because, in that case, if the new results did not match, “the only way
we’d ever get to the bottom of such a discrepancy is if we both worked
reproducibly and studied detailed differences between code and data.”

The pioneering efforts of Claerbout and Donoho influenced many others. But
their concerns centered on research involving computational analysis of unique
data (recordings of seismic waves). They did not deal with the situation where
another researcher might collect _new_ data, re-doing a study or an experiment
following an original design, then analyze this data to compare the final
findings with the original work. Situations like this arise in many empirical
fields, where scientific findings must be confirmed by independent studies.
Peng et al. (2006) distinguish the term _replication_ for this scenario. They
say: “Scientific evidence is strengthened when important findings are
_replicated_ by multiple independent investigators using independent data,
analytical methods, laboratories, and instruments.” But epidemiologic studies
are expensive, time-consuming, and often impossible to replicate (e.g., when
they rely on opportunistic data, say, from an infectious outbreak). “An
attainable minimum standard is _reproducibility_ , which calls for data sets
and software to be made available for verifying published findings and
conducting alternative analyses,” they say.

In the social sciences, Harvard professor Gary King, one of the most cited
political scientists of this generation, also pioneered the reproducibility
concerns. In King (1995), he writes: “The replication standard holds that
sufficient information exists with which to understand, evaluate, and build
upon a prior work if a third party could replicate the results without any
additional information from the author.” He uses the term _replication_
throughout, not making a distinction like we have in the Claerbout/Donoho/Peng
writings on reproducible research. In fact, his paper uses “replication” 96
times, “replicate” 21 times, and “reproduce” (as verb) only twice. E.g., “good
science requires that we be able to reproduce existing numerical results…” A
more recent work co-authored by him Lazer et al. (2014), still does not use
the words “reproduce/reproducibility/reproducible,” and only uses “replicate.”
This appears to be the common usage in his field, as a Google Scholar search
with “replication political science“ gives 339,000 results, and with
“reproducible political science” gives just 59,700 results (checked
01/14/2017).

## Conflicting terminologies

As mentioned above, Claerbout and Donoho’s pioneering work influenced many
others, who adopt their definition of reproducible research. For example,
Gentleman and Temple Lang (2007): “By reproducible research, we mean research
papers with accompanying software tools that allow the reader to directly
reproduce the results and employ the computational methods that are presented
in the research paper.” Vandewalle et al. (2009): “A research work is called
reproducible if all information relevant to the work, including, but not
limited to, text, data and code, is made available, such that an independent
researcher can reproduce the results.” LeVeque (2009): “The idea of
‘reproducible research’ in scientific computing is to archive and make
publicly available all the codes used to create a paper’s figures or tables,
preferably in such a manner that readers can download the codes and run them
to reproduce the results.” The distinction with the idea of replication,
introduced by Peng, also appears in other works. The _Annals of Internal
Medicine_ issued a statement saying: “Independent replication by independent
scientists in independent settings provides the best assurance that a
scientific finding is valid; however, the resources and time required for
high-quality clinical studies makes literal replication of published studies a
slow corrective to any errors in the original publication. However, scientists
and journal editors can promote ‘reproducible research,’ [which ensures] that
independent scientists can reproduce published results by using the same
procedures and data as the original investigators” Laine et al. (2007). In an
editorial, the _Int. Journal of Forecasting_ announces that it will publish
_replication studies_ : works attempting to independently verify research
findings “under the same or very similar conditions” Hyndman (2010). _Science_
published a special issue on “Data Replication & Reproducibility” in 2011, to
which the introduction reads: “Replication—the confirmation of results and
conclusions from one study obtained independently in another—is considered the
scientific gold standard” Jasny et al. (2011). Peng’s own (widely cited)
article in this special issue introduces the idea of a _reproducibility
spectrum_ , in which reproducible research is a “minimum standard for judging
scientific claims when full independent replication of a study is not
possible.” Peng (2011)

On the basis of the multiple references cited above, here are concise
definitions that convey the meanings of the Claerbout/Donoho/Peng convention:

Reproducible research:

Authors provide all the necessary data and the computer codes to run the
analysis again, re-creating the results.

Replication:

A study that arrives at the same scientific findings as another study,
collecting new data (possibly with different methods) and completing new
analyses.

Unfortunately, conflicting terminologies started to spread in recent years. It
appears that the first work to directly swap the usage of _reproducible_ and
_replicable_ is Drummond (2009). This is a workshop contribution to a
conference in the field of machine learning (meaning, it received light or no
peer review). The author seems to arbitrarily swap existing terminology; he
says: “I have used the term replicability for what others have called
reproducibility in our literature.” He provides no more justification than “I
think it reasonable.” Mark Liberman, Distinguished Professor of Linguistics at
the University of Pennsylvania, analyzed the usage of terms in the literature,
and referring to Drummond’s paper he concluded: “Since the technical term
‘reproducible research’ has been in use since 1990, and the technical
distinction between reproducible and replicable at least since 2006, we should
reject [the] attempt to re-coin technical terms reproducible and replicable in
senses that assign the terms to concepts nearly opposite to those used in the
definitions by Claerbout, Peng and others.” Liberman (2015)

Alas, before the scrutiny of the linguistics professor, others picked up on
the swapped terminology. Casadevall and Fang (2010), citing Drummond, adopt it
and discuss various scenarios in the fields of microbiology and immunology.
Other examples include Davison (2012), Loscalzo (2012), Crook et al. (2013),
Cooper et al. (2015), among many others. Drummond’s 2009 paper has 134
citations in Google Scholar (checked Jan. 15, 2018), although some are for
other reasons de Winter and Happee (2013) or to plainly point out the
discrepant terminology Boylan et al. (2015).

Some other works adopted the swapped terms without a clear link to Drummond
(2009). In LeVeque et al. (2012), referring to topics of discussion at a July
2011 workshop titled “Reproducible Research: Tools and Strategies for
Scientific Computing,” the authors write: “As an example of the lack of a
common nomenclature, two sequential speakers provided opposite definitions for
replicable and reproducible. (We believe the first refers to the ability to
run a code and produce exactly the same results as published, and the second
refers to the ability to create a code that independently verifies the
published results using the information provided.)” A citation follows to
Stodden (2011), where we read: “Replication, using author-provided code and
data, and independent reproduction work hand-in-hand. We can reserve the term
‘replicability’ for the regeneration of published results from author-provided
code and data.” Followed by an immediate citation to King (1995), maybe this
was prompted by the similarity between King’s conditions for a “replication
standard” and the Claerbout/Donoho concept of reproducible research.
(Although, as mentioned above, King simply uses “replication” for everything.)
Also in LeVeque et al. (2012), the practice of “private reproducibility” is
explained as being able to rebuild own past research results from the precise
version of the code that was used to create them. In this sense,
“reproducibility” is that minimum standard in the spectrum, in contrast to
“replicability,” as Drummond would have it. The opposing terminologies coexist
in this paper.

A more recent use of terminology in contradiction with the
Claerbout/Donoho/Peng convention appears in the effort of the Association of
Computing Machinery (ACM) on Result and Artifact Review and Badging.111
Approved June 2016, https://www.acm.org/publications/policies/artifact-review-
badging It deploys across ACM publications a badging system for articles
complying with various standards of code and data sharing. With that purpose,
it defines the following terminology:

Repeatability

(Same team, same experimental setup.) The measurement can be obtained with
stated precision by the same team using the same measurement procedure, the
same measuring system, under the same operating conditions, in the same
location on multiple trials. For computational experiments, this means that a
researcher can reliably repeat her own computation.

Replicability

(Different team, same experimental setup.) The measurement can be obtained
with stated precision by a different team using the same measurement
procedure, the same measuring system, under the same operating conditions, in
the same or a different location on multiple trials. For computational
experiments, this means that an independent group can obtain the same result
using the author?s own artifacts.

Reproducibility

(Different team, different experimental setup.) The measurement can be
obtained with stated precision by a different team, a different measuring
system, in a different location on multiple trials. For computational
experiments, this means that an independent group can obtain the same result
using artifacts which they develop completely independently.

The source cited for the definitions adopted by the ACM is the International
Vocabulary of Metrology JCGM (2008), which establishes terminology for
physical measurements. Relevant passages from this source are (abridged):

repeatability condition of measurement

…same measurement procedure, same operators, same measuring system, same
operating conditions and same location, and replicate measurements on the same
or similar objects over a short period of time (p.23);

reproducibility condition of measurement

…set of conditions that includes different locations, operators, measuring
systems, and replicate measurements on the same or similar objects (p.24).

The document contains no definition of “replicability” but both definitions
above use the phrase ‘replicate measurements.’ The scenario for these
definitions is a physical quantity being measured, and the precision of that
measurement. Repeatability is the precision over successive measurements of
the same quantity, with everything kept the same (even the operator), over a
short period of time. Because measuring tools and procedures have inherent
errors, it’s relevant to document repeatability in the form of standard
deviation or other dispersion characteristics (e.g., graphical).
Reproducibility of measurements involves changing at least one condition,
e.g., the instrument, the location, or the operator. Because the _same_
physical quantity is being measured (“on the same or similar object”),
reproducibility of measurement is also expressed by a dispersion statistic,
like standard deviation (Taylor and Kuyatt, 1994, p.14).

Along the way of creating the ACM terminology, based on the metrology
document, some leap occurred. Replicability is not defined in JCGM (2008), and
reproducibility could just as well be mapped to the minimum standard of the
Peng spectrum, where certainly the operator (researcher) changes, the
instrument (computer) does as well, and other conditions may also change, but
the object of study remains the same. Arguably, conditions for physical
measurements are a distant analogy for the complex processes of a full
scientific workflow.

## Cataloguing the reproducibility literature

The conflicting terminologies are at least an annoyance, and at worst an
impediment to the progress of science. Yet, no solution is at hand beyond a
general good practice of always defining the terms used in any particular
writing. Here is a decision tree to catalogue the terminologies in the
literature: authors either, $A$—make no distinction between the words
_reproduce_ and _replicate_ , or $B$—use them distinctly. If $B$, then they
are commonly divided in two camps. In a spectrum of concerns that starts at a
minimum standard of “same data$+$same methods$=$same results,” to “new data
and/or new methods in an independent study$=$same findings,” group $1$ calls
the minimum standard _reproduce_ , while group $2$ calls it _replicate_. $A$
includes King (1995). $B1$ corresponds to the Claerbout/Donoho/Peng
convention, while $B2$ agrees with Drummond (2009) and the ACM terminology.
Table 2 classifies into these groups all the references cited above, and more.

Table 1: Catalogue of terminologies in the literature, with Google Scholar citations (checked Jan. 20, 2018). $A$ | $B1$ | $B2$
---|---|---
King (1995), 527 |  Peng et al. (2006), 177 |  Drummond (2009), 135
JCGM (2008), 32 |  Gentleman and Temple Lang (2007), 216 |  Casadevall and Fang (2010), 58
|  Laine et al. (2007), 134 |  Stodden (2011), 30
Dewald et al. (1986), 506 |  Vandewalle et al. (2009), 266 |  Davison (2012), 80
Pesaran (2003), 12 |  LeVeque (2009), 32 |  Loscalzo (2012), 31
McCullough et al. (2008), 93 |  Hyndman (2010), 20 |  LeVeque et al. (2012), 74
Garijo et al. (2013), 52 |  Jasny et al. (2011), 180 |  Crook et al. (2013), 16
Open Science Collaboration (2012), 300 |  Peng (2011), 552 |  Cooper et al. (2015), 26
Open Science Collaboration (2015), 1573 |  |
Stodden (2015), 19 |  Koenker and Zeileis (2009), 58 |  Cartwright (1991), 81
Duvendack et al. (2017), 13 |  Delescluse et al. (2012), 22 | Pellizzari et al. (2017)
Lejaeghere et al. (2016),199 |  Sandve et al. (2013), 227 | FASEB (2016)
Coudert (2017), 3 |  Stodden et al. (2014), 119 |
|  Topalidou et al. (2015), 14 |
|  Iqbal et al. (2016), 67 |
|  Kafkafi et al. (2016), 2 |
|  Stevens (2017), 1 |
|  Kitzes et al. (2017), 10 |
|  Benureau and Rougier (2017), 1 |
|  Bollen et al. (2015), 12 |
|  Broman et al. (2017), 4 |

### Added references and commentary

Table 2 includes additional references to those cited in the narrative above.
Quotes from these and/or commentary to justify their grouping follow below,
starting by what seems to be the predominant terminology: that of group $B1$.

Koenker and Zeileis (2009) cite the Claerbout ‘slogan,’ from Buckheit and
Donoho (1995), and list various software tools and techniques to enable
reproducible research. They then describe two ‘replication case studies,’
i.e., efforts to obtain the same results that appeared in other publications.
In the conclusion, they say: “the real challenge of reproducible econometric
research lies in restructuring incentives to encourage better archiving and
distribution of the gory details of computationally oriented research.”
Despite absence of explicit definitions, the usage agrees with group $B1$.
Similarly, Delescluse et al. (2012) cite Claerbout, Donoho, and Peng (among
many others), and adopt the $B1$ terminology. The authors describe their
preferred tools for reproducible data analysis of neurophysiological data, and
illustrate the process with a detailed example of implementation.

The work by Sandve et al. (2013) is also in group $B1$. It says: “As full
replication of studies on independently collected data is often not feasible,
[…] reproducible research [is] an attainable minimum standard for assessing
the value of scientific claims.” The authors use the phrase ‘replication
studies’ several times, and refer to ‘inability to replicate findings’ and
‘replication in studies with different data.’ Thus, although they included no
explicit terminology definition, their usage agrees with
Claerbout/Donoho/Peng.

Garijo et al. (2013) lacks any definitions, and does not seem to make a
distinction between _replicate_ and _reproduce_. However, the first term is
used only 3 times, while terms with the root ‘reproduc-’ appear 161 times. The
paper deals with reproducing another group’s published work using both
artifacts provided by original authors and others. It’s listed in group $A$ as
a conservative choice, but it could be argued to belong in $B1$.

The first comprehensive review of the field in book form, edited by Stodden et
al. (2014), includes this passage in the preface: “Replication, the practice
of independently implementing scientific experiments to validate specific
findings, is the cornerstone of discovering scientific truth. Related to
replication is reproducibility, which is the calculation of quantitative
scientific results by independent scientists using the original datasets and
methods.” This usage agrees with group $B1$, though on inspection of the
various chapters, some authors don’t make a clear distinction between the
terms ‘reproduce/replicate,‘ and often use ‘reproducibility‘ as an umbrella
term. An example, from chapter 11 (by the Open Science Collaboration):
“…narrowly, reproducibility is the repetition of a simulation or data analysis
of existing data by re-executing a program. More broadly, reproducibility
refers to _direct replication_ , an attempt to replicate the original
observation using the same methods of a previous investigation but collecting
new [data].”

Topalidou et al. (2015) describe an undertaking to reproduce published results
that failed initially when the code provided by the original authors did not
compile. The effort resulted in a collaboration with the original authors, and
a reimplementation of the code from scratch, “following the principles of
reproducible computational science as proposed in Peng (2011).” The language
agrees with group $B1$.

In Iqbal et al. (2016), we read: “There are many different proposals on how
reproducible research can be guaranteed. These include approaches at
reproducible practices, i.e., making other investigators able to repeat the
process and calculations; re-analysis (as in the case of randomized trials);
and replication by independent investigators, as in genetics, psychology, and
cancer biology.” This usage agrees with group $B1$.

A report of the January 2015 workshop “Replicability and reproducibility of
discoveries in animal phenotyping” at Tel Aviv University Kafkafi et al.
(2016), which included researchers from many disciplines (genetics, behavior
genetics, behavioral neuroscience, ethology, statistics, bioinformatics and
database programming), also aligns with the $B1$ terminology. It says: “the
scientific community has become increasingly concerned with cases of published
‘discoveries’ that could not be replicated in subsequent studies, and
sometimes could not even be reproduced in reanalysis of the original data.”
The meeting was hosted by the Replicability Research Group, with funding from
the European Research Council. On their website,222
http://www.replicability.tau.ac.il/index.php/replicability-in-
science/replicability-vs-reproducibility.html this group notes their
terminology usage, citing the policy in the _Biostatistics_ journal Peng
(2009).

Stevens (2017) includes explicit definitions: “Replicability is ‘re-performing
the experiment and collecting new data,’ whereas reproducibility is ‘re-
performing the same analysis with the same code using a different analyst.’
Therefore, one can replicate a study or an effect (outcome of a study) but
reproduce results (data analyses).” This usage aligns with group $B1$.

The recent book edited by Kitzes et al. (2017) consists of 31 case studies
that present concrete practices for reproducibility in data-intensive
research. The book also includes synthesis chapters that aim to summarize the
lessons learned from the various contributions. In the first of these, Marwick
et al. (2017) address definitions, mentioning the confusion that has spread
over the recent years, and decidedly adopting the $B1$ terminology. They note
that this is “broadly consistent with usage of these terms throughout this
book” (more than 40 authors).

Benureau and Rougier (2017) use an example to discuss the necessary
characteristics of scientific codes: “The code should be executable (re-
runnable) and produce the same result more than once (repeatable); it should
allow an investigator to reobtain the published results (reproducible) while
being easy to use, understand and modify (reusable), and it should act as an
available reference for any ambiguity in the algorithmic descriptions of the
article (replicable).” Their meanings for ‘reproducible’ and ‘replicable’ are
aligned with group $B1$, but they also expand the spectrum of concerns, as it
specifically relates to code. Using a concrete example lends to clear and
illuminating explanations of the potential pitfalls with scientific use of
software.

Moving on to group $B2$, with the opposite terminology, we find in Cartwright
(1991): “I propose to distinguish replicability—doing the same experiment
again—from reproducibility—doing a new experiment.” This essay by a
philosopher of science is a commentary on the work of another, who had
proposed that “replication is the establishment of a new and contested result
by agreement over what counts as a correctly performed series of experiments”
Collins (1991). Both authors are concerned with replication in economics,
where usage generally makes no distinction with the term ‘reproduce,’ and most
often defaults to the term ‘replicate’ for everything. It does appear that
Cartwright’s terminology proposal remained obscure. In the book by Guala
(2005), we read: “Genuine _replications_ […] generally involve some (minor or
major) modifications of the original design.” And: “It is necessary to
distinguish, however, between the _replication_ and mere _repetition_ of an
experiment. Repetition is the business of doing an experiment again, trying to
keep exactly the same design as in the original.” In a footnote: “The
distinction introduced here is well known in the philosophical literature on
experiments, although other authors sometimes use another terminology (e.g.,
‘replication’ vs. ‘reproduction’).”—citing Cartwright (1991). Like in
political science, the economics literature uses ‘replication‘ much more
frequently, often as an umbrella term. (A Google Scholar search with
“replication economics” gives 298,000 results, and with “reproducibility
economics,” 53,700 results; checked Jan. 19, 2018). An even older example in
the field of economics is Dewald et al. (1986), while a more recent one is
McCullough et al. (2008). They don’t distinguish ‘reproduce‘ and ‘replicate.‘
The new-section announcement for replication studies in the _Journal of
Applied Econometrics_ Pesaran (2003), meanwhile, explains what they should
include: checking consistency and accuracy of data, checking validity of
computations either directly or using different software (preferred), and
checking “if the substantive empirical finding of the paper can be replicated
using data from other periods, countries, regions, or other entities as
appropriate.” Succinctly put by Duvendack et al. (2017): “we operationalize
‘replication’ as any study whose main purpose is to determine the validity of
one or more empirical results from a previously published study.”

The new monograph by Pellizzari et al. (2017), derived from a report of a 2015
workshop, aims to illuminate the semantics problems of the field, falling in
agreement with the ACM definitions (group $B1$). It cites the National
Institute of Standards and Technology (NIST) as “the most authoritative
source” for the adopted lexicon (though NIST has _not_ addressed lexicon for
reproducible research, specifically). Citing from page 8: “Of particular
importance is the crucial distinction between reproducibility and
replicability. The differences lie mainly in whether every element of a given
study must be done in exactly the same way in any effort either to do it again
or to confirm original results. That requirement is basically the
“restrictive” aspect of replicability. By contrast, reproducibility does not
turn so critically on such elements as having identical subjects, methods, and
other components; that is, at least one component differs from the original
research study.” Table 1 includes the “NIST definition” for reproducibility:
“closeness of the agreement between the results of measurements of the same
measurand carried out under changed conditions of measurement.” This language
appears verbatim on page 14 of NIST Technical Note 1297 Taylor and Kuyatt
(1994), where it says that it comes directly from the International Vocabulary
of Metrology (VIM). It is followed by a note explaining that reproducibility
can be expressed by dispersion statistics (e.g., standard deviation), because
it refers to measuring the _same_ physical quantity, with at least one
condition changed (e.g., the location, or the operator, or the method of
measurement). In this sense, I see no contradiction with the usage of
‘reproducible‘ by Claerbout and Donoho, where at least operator, location and
time are meant to change on reproducing some results. The key, though, is that
‘replication’ or ‘replicability’ are _not_ defined terms in any NIST document
or in the VIM. Pellizzari et al. (2017) claim that “NIST does, however, use
the term in specific applications without defining it or distinguishing it
from repeatability”—without a citation. They do reference the agreement with
Casadevall and Fang (2010), which in turn cite Drummond (2009), as we saw
previously. The phrase ‘replicable measurements’ appears in both the
definitions of repeatability and reproducibility conditions of measurement in
the International Vocabulary of Metrology JCGM (2008), as also mentioned.
Finally, we should note that NIST, despite its name, is not a standards
developing organization. So to speak of “NIST definitions” is itself
problematic, because they do not, in fact, write standards. They only endorse
the standards of other organizations like ANSI, ISO, IEEE, etc.

In group $A$ are the widely cited works of the Open Science Collaboration—two
large-scale efforts to replicate published empirical and correlational studies
in psychology. They use the terms ‘reproduce’ and ‘replicate’ interchangeably.
Explicitly, the first work says: “Some distinguish between ‘reproducibility’
and ‘replicability’ by treating the former as a narrower case of the latter
(e.g., computational sciences) or vice versa (e.g., biological sciences). We
ignore the distinction.” Open Science Collaboration (2012). In their later
work Open Science Collaboration (2015), the authors recurrently use
‘replication’ (208 times), and less frequently apply ‘reproducible,’ as an
umbrella term (60 times). However, the lead author of these studies, Brian
Nosek, adopted the distinction between terms according to
Claerbout/Donoho/Peng when speaking in a recent video interview National
Science Foundation (2015). He says: “[by] reproducibility…what we’re usually
referring to is: can you take the data and findings that I produced in some
research, and run the analysis again and get the result back that I got. [In]
replicability…I do a study, get a finding with some data, and you do your own
study…”

Stodden (2015) also uses the two recurrent terms indistinctly: “A published
finding may not reproduce in independent replications of the original
experimental design (i.e., reimplementing the experiment) for any of several
statistical reasons.” This work focuses on _statistical reproducibility_ ,
which “refers to the failure to replicate an experiment owing to flawed
experimental design or statistical analysis.” Like some others (see below),
this work sidesteps the definition tangle using _reproducibility_ as an
umbrella term, and proposing expanded terminology that distinguishes between
_empirical_ reproducibility, _computational_ reproducibility, and
_statistical_ reproducibility.

Lejaeghere et al. (2016) tackle for the first time the issue of
reproducibility in the context of computational chemistry and materials
science. This is an impressive work involving dozens of researchers, and
comparing 15 solid-state codes on 71 elemental crystals, over many thousands
of calculations. ‘Reproducibility’ here takes the meaning of a new analysis
with different methods (i.e., on the ‘replication’ side of Peng’s spectrum).
But their use of the term is generic: “Scientific results are expected to be
reproducible. When the same study is repeated independently, it should reach
the same conclusions.” I have grouped this work with $A$ since the terms
‘replication/replicate’ are absent, and the authors appear unaware of the
different terminologies. A recent editorial in one of the top chemistry
journals, on the other hand, plainly aligns with the Claerbout/Donoho sense
for ‘reproducible research‘ when giving a description of what this means in
computational chemistry of materials: open data, using standard machine-
readable formats, hosting in archival-quality repositories providing a
persistent URL or DOI, fully sharing input and output files, and open-source
software Coudert (2017).

Hundreds of works could of course be added to this discussion. I used no
special criteria to choose works to cite here, beyond having come across them
in the past, read them, and taken notes about their terminology (plus a random
walk through Google Scholar while writing).

## Expanded terminologies

Stodden (2013) made first a separation between _empirical_ reproducibility and
_computational_ reproducibility, then further unpacked _statistical_
reproducibility Stodden (2015). A few others have also invented expanded
terminologies. In the fields of economics and political science, the umbrella
term is _replication_. Hamermesh (2007) expands this into _pure_ replication,
_statistical_ replication, and _scientific_ replication. The first refers to
re-analyzing the same data with the same model and estimation parameters; the
second may use different but comparable data, but the same model. The last one
would use “different sample, different population, and perhaps similar but not
identical model […] and, indeed, comprises most of what economists view as
replication.”

In the context of neuroscience, Crook et al. (2013)—after citing Drummond
(2009) as their source for terminology—propose to make further distinctions
among: internal replicability, external replicability, cross replicability,
and reproducibility. In the first case, the original authors or someone else
in the same group can re-create the results, re-executing the same software.
In the second case, a reader of published results can re-create them using the
data and code supplied by the authors. And cross-replicability would mean
running the same model but with different software. But then Crook et al.
(2013) say that “the boundary line between cross-replicability and
reproducibility is not always clear.”

Goodman et al. (2016) recognize that “basic terms—reproducibility,
replicability, reliability, robustness, and generalizability—are not
standardized.” They propose new lexicon: _methods_ reproducibility (the
original meaning of reproducibility by Claerbout/Donoho), _results_
reproducibility (the meaning of replicability in Peng), and _inferential_
reproducibility.

## Beyond the academic literature

### Funding agencies

The National Science Foundation Subcommittee on Replicability in Science of
the Advisory Committee to the NSF Directorate for Social, Behavioral, and
Economic Sciences (SBE) published a report titled “Perspectives on Robust and
Reliable Science” in May 2015 Bollen et al. (2015). They define terminology as
follows: “reproducibility refers to the ability of a researcher to duplicate
results of a prior study using the same materials as were used by the original
investigator,” and “new evidence is provided by new experimentation, defined
in the NSF report as ‘replicability’.” The report recognizes a lack of
consensus on terminology, but adopts clear definitions for the two recurrent
terms. It also states that “reproducibility is a minimum necessary condition
for a finding to be believable and informative.” These definitions are
consistent with the Claerbout/Donoho/Peng convention (group $B1$)—despite the
fact that in economics and political science ‘replicability‘ had been widely
used as an umbrella term.

The NSF Directorate for Computer & Information Science & Engineering (CISE)
issued a “Dear Colleague Letter: Encouraging Reproducibility in Computing and
Communications Research” in October 21, 2016.333
https://www.nsf.gov/pubs/2017/nsf17022/nsf17022.jsp It encourages PIs to
“embrace completeness and transparency in developing rigorous protocols as
well as in making experimental parameters and collected data available to
other researchers’,’ but it makes no attempt to define ‘reproducibility.’ It
cites the December 2011 special issue of _Science_ containing the perspective
of Peng (2011), and a May 2016 editorial in _Nature_ introducing a reader
survey of concern with reproducibility Baker (2016); the latter work uses
reproduce/reproducibility as umbrella terms.

The National Institutes of Health (NIH) maintains a website titled “Rigor and
Reproducibility,” but does not provide terminology.444
https://grants.nih.gov/reproducibility/index.htm It set up requirements for
strengthening reproducibility of biomedical research in January 2016,
providing instructions for grant proposals. NIH focuses on _rigor_ (concerning
methods and materials) and _transparency_ (concerning publication). Their
definition of reproducibility is the straight combination of those two
notions, while making no particular distinction with the word ‘replication’
Collins and Tabak (2014).

### Professional societies

The 1999 Ethics and Values document 555
https://www.aps.org/policy/statements/99_6.cfm of the American Physical
Society (APS) reads: “The success and credibility of science are anchored in
the willingness of scientists to […] Expose their ideas and results to
independent testing and replication by others. This requires the open exchange
of data, procedures and materials.” The use of the word ‘replication’
coincides with the ideas of the Peng spectrum, but it could be a coincidental
choice of words.

The American Economic Association (AEA) has a policy666
https://www.aeaweb.org/journals/policies/data-availability-policy that applies
to all its journals: “to publish papers only if the data used in the analysis
are clearly and precisely documented and are readily available to any
researcher for purposes of replication.” (This includes all computer codes,
configuration files and scripts.) They use the term ‘replication‘ throughout,
but the common sense is the replication of a full study or published paper.
For example, Coffman et al. (2017) propose that journals publish replication
reports, which “summarize novel work replicating an existing high-impact
paper, or they could highlight a replication result embedded in a wider-scope
published paper.”

The Association for Psychological Science (APS) has been promoting open
science and replication efforts for some years. In 2012, their journal
_Perspectives on Psychological Science_ began publishing special sections on
replication Pashler and Wagenmakers (2012). The APS flagship journal,
_Psychological Science_ , started several initiatives to enhance replicability
Eich (2014), including: stronger methods reporting (not counted towards word
limits), expanded statistical requirements, and new badges for recognizing and
promoting open-science practices. A later editorial Lindsay (2015) reaffirmed
the commitment to replication, while discussing typical practices that weaken
it, like low statistical power and $p$-hacking. In all these contexts,
‘replication’ is used broadly, but often referring to independent studies
aimed and confirming the scientific findings of a previous publication. The
APS is also an original signatory of the Transparency and Openness Promotion
(TOP) Guidelines Nosek et al. (2015). These guidelines, signed so far by
nearly 3,000 journals and organizations, associate reproducibility to sharing
of data, open code, research design disclosure, pre-registration of analysis
plans and study details, and replication studies.

The American Psychological Association (APA) just released new journal article
reporting standards, with recurrent mention of the phrase ‘replication
studies’ to mean complete studies (including data collection) meant to confirm
the findings of another Appelbaum et al. (2018).

The Federation of American Societies for Experimental Biology published a
recommendations document FASEB (2016) targeted at biomedical and biological
research (particularly two key tools: mouse models and antibodies). It
provides definitions that fall squarely in group $B2$:

Replicability:

the ability to duplicate (i.e., repeat) a prior result using the same source
materials and methodologies. This term should only be used when referring to
repeating the results of a specific experiment rather than an entire study;

Reproducibility:

the ability to achieve similar or nearly identical results using comparable
materials and methodologies. This term may be used when specific findings from
a study are obtained by an independent group of researchers.

The American Statistical Association (ASA) released in January 2017 a document
with recommendations for funding agencies on supporting reproducible research
Broman et al. (2017). Its definitions align with group $B1$:

Reproducibility:

A study is reproducible if you can take the original data and the computer
code used to analyze the data and reproduce all of the numerical findings from
the study. This may initially sound like a trivial task but experience has
shown that it?s not always easy to achieve this seemingly minimal standard.

Replicability:

This is the act of repeating an entire study, independently of the original
investigator without the use of original data (but generally using the same
methods).

### Other journal policies and editorials

The _Journal of Applied Econometrics_ (JAE) has maintained a data archive777
http://qed.econ.queensu.ca/jae/ since the late 1980s, making it mandatory for
papers published after January 1994 to deposit their (non-confidential) data.
Depositing software, however, is voluntary. JAE announced in Pesaran (2003) a
new section dedicated to replications of previously published empirical
results. This journal is the leader in publishing such works among journals in
the field, followed by the _American Economic Review_ , according to the
survey of Duvendack et al. (2017).

The _American Journal of Political Science_ requires authors888
https://ajps.org/ajps-replication-policy/ to “provide replication materials
that are sufficient to enable interested researchers to reproduce all of the
analytic results that are reported in the text and supporting materials.”
Their terminology does not distinguish between the two recurrent terms (group
$A$), but their use of ‘replication files’—meaning, all data and code that
produce the results presented in an accepted paper—corresponds to the
yardstick of Claerbout and Donoho’s ‘reproducible research.’ What is
remarkable about this journal’s process is that they contract a third party999
The Odum Institute for Research in Social Science, in the case of quantitative
analyses, and the Qualitative Data Repository at Syracuse University, in the
case of qualitative analyses. to complete a full verification that the author-
provided data and analysis code do produce the reported results. They make
this a condition for publication. Author-provided files are deposited in the
journal’s section on the Dataverse repository,101010
https://dataverse.harvard.edu/dataverse/ajps where they are labeled
‘Replication Data for <paper title>.’

The statement of the _Annals of Internal Medicine_ Laine et al. (2007) talks
about ‘replication by independent scientists in independent settings’ and
‘reproducible research,’ when published results can be re-created using the
procedures and data used by the original authors. This language is consistent
with that of group $B1$.

The journal _Biostatistics_ announced its reproducible-research initiative in
Peng (2009), with a clear distinction between replication of scientific
findings and the minimum standards of reproducible research. It designates
articles as reproducible when three conditions are met: open data, open-source
code, and the associate editor for reproducibility was able to re-run the
analysis to recreate the reported results.

Adding to an existing firm open-data policy, the journal _Genome Biology_
recently announced an open-source code requirement. The editorial reads: “We
consider source code availability and accessibility to be important to ensure
that computational analyses can be easily repeated and research shown to be
reproducible.” Marszalek and Flintoft (2016). A more recent piece advocating
for structured supplemental materials submits that an entire workflow could be
included in supplements: “Workflows are especially relevant for in silico
analyses, as reproducibility can turn on the ability to recreate the exact
parameters employed.” Greenbaum et al. (2017). This sense is similar to
Claerbout and Donoho’s idea of reproducible research, while the term
‘replication‘ is hardly used in this field, where the word is more likely to
appear in phrases like ‘DNA replication’ and ‘genome replication.’

I also want to mention the new _ReScience_ journal:111111
https://rescience.github.io “a peer-reviewed journal that targets
computational research and encourages the explicit replication of already
published research.” As described in their FAQs, a publication in _ReScience_
achieves two goals: “by replicating the original work, it provides an
independent implementation of the original computational protocol, and by
making this new implementation public, it is reproducible and thus a safer
basis for future research to build on.” A recent paper reporting on the
initiative’s progress gives the adopted terminology Rougier et al. (2017):

Reproducing

the result of a computation means running the same software on the same input
data and obtaining the same results. The goal of a reproduction attempt is to
verify that the computational protocol leading to the results has been
recorded correctly.

Replicating

a published result means writing and then running new software based on the
description of a computational model or method provided in the original
publication, and obtaining results that are similar enough to be considered
equivalent.

## Conclusions

The Claerbout/Donoho/Peng terminology is broadly disseminated across
disciplines (see Table 2). But the recent adoption of an opposing terminology
by two large professional groups—ACM and FASEB—make standardization awkward.
The ACM publicizes its rationale for adoption as based on the International
Vocabulary of Metrology, but a close reading of the sources makes this
justification tenuous. The source of the FASEB adoption is unclear, but
there’s a chance that Casadevall and Fang (2010) had an influence there. They,
in turn, based their definitions on the emphatic but essentially flawed work
of Drummond (2009).

Table 2: Grouping of terminologies, as in Table 1, but by discipline. $A$ | $B1$ | $B2$
---|---|---
political science | signal processing | microbiology, immunology (FASEB)
economics | scientific computing | computer science (ACM)
| econometry |
| epidemiology |
| clinical studies |
| internal medicine |
| physiology (neuro) |
| computational biology |
| biomedical research |
| statistics |

It seems that any attempt to unify terminologies can only start by
conversations with ACM and FASEB to explore how slim the chance might be that
they backtrack their adoption to align with the predominant usage.
Conversations with leaders in the fields of political science and economics,
where ‘replication‘ is predominant, should follow. Use of the term here often
applies to a new study that aims to confirm findings reported in a publication
(and thus aligns with Peng’s), but it also spills over to calling the research
compendia ‘replication files.’ This custom in the _American Journal of
Political Science_ is unlikely to be reversed, but it can be absorbed into the
Claerbout/Donoho/Peng convention, as reproducible research is also necessary
for successful replication. Whereas economics and political science are
predominantly empirical disciplines, using software for statistical analysis
is ubiquitous. A feasible way out would be if these fields accept calling
‘reproducible research’ that which openly archives full sets of ‘replication
files.’

## References

  * Appelbaum et al. (2018) Appelbaum, M., H. Cooper, R. B. Kline, E. Mayo-Wilson, A. M. Nezu, and S. M. Rao, 2018: Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25, doi:10.1037/amp0000191.
  * Baker (2016) Baker, M., 2016: 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452–454.
  * Barba (2017) Barba, L. A., 2017: Science reproducibility taxonomy. Figshare, presentation slides under CC-BY, doi:10.6084/m9.figshare.5248273.v1.
  * Benureau and Rougier (2017) Benureau, F. C. and N. P. Rougier, 2017: Re-run, repeat, reproduce, reuse, replicate: Transforming code into scientific contributions. Frontiers in Neuroinformatics, 11, doi:10.3389/fninf.2017.00069.
  * Bollen et al. (2015) Bollen, K., J. Cacioppo, R. Kaplan, J. Krosnick, and J. Olds, 2015: Social, behavioral, and economic sciences: perspectives on robust and reliable science. Report of the Subcommittee on Replicability in Science, Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.
  * Boylan et al. (2015) Boylan, J. E., P. Goodwin, M. Mohammadipour, and A. A. Syntetos, 2015: Reproducibility in forecasting research. International Journal of Forecasting, 31(1), 79–90, doi:10.1016/j.ijforecast.2014.05.008.
  * Broman et al. (2017) Broman, K., M. Cetinkaya-Rundel, A. Nussbaum, C. Paciorek, R. Peng, D. Turek, and H. Wickham, 2017: Recommendations to funding agencies for supporting reproducible research. American Statistical Association (ASA), http://www.amstat.org/asa/files/pdfs/POL-ReproducibleResearchRecommendations.pdf.
  * Buckheit and Donoho (1995) Buckheit, J. B. and D. L. Donoho, 1995: Wavelab and reproducible research. In Antoniadis A., O. G., ed., Wavelets and statistics, Springer, New York, NY, vol. 103 of Lecture Notes in Statistics, pp. 55–81, doi:10.1007/978-1-4612-2544-7_5.
  * Cartwright (1991) Cartwright, N., 1991: Replicability, reproducibility, and robustness: comments on harry collins. History of Political Economy, 23(1), 143–155, doi:10.1215/00182702-23-1-143.
  * Casadevall and Fang (2010) Casadevall, A. and F. C. Fang, 2010: Reproducible science. Infection and Immunity, 78(12), 4972–4975, doi:10.1128/IAI.00908-10.
  * Claerbout and Karrenbach (1992) Claerbout, J. F. and M. Karrenbach, 1992: Electronic documents give reproducible research a new meaning. In SEG Technical Program Expanded Abstracts 1992, Society of Exploration Geophysicists, pp. 601–604, doi:10.1190/1.1822162.
  * Coffman et al. (2017) Coffman, L. C., M. Niederle, and A. J. Wilson, 2017: A proposal to organize and promote replications. American Economic Review, 107(5), 41–45, doi:10.1257/aer.p20171122.
  * Collins and Tabak (2014) Collins, F. S. and L. A. Tabak, 2014: Nih plans to enhance reproducibility. Nature, 505(7485), 612, doi:10.1038/505612a.
  * Collins (1991) Collins, H. M., 1991: The meaning of replication and the science of economics. History of Political Economy, 23(1), 123–142, doi:10.1215/00182702-23-1-123.
  * Cooper et al. (2015) Cooper, J., J. O. Vik, and D. Waltemath, 2015: A call for virtual experiments: accelerating the scientific process. Progress in Biophysics and Molecular Biology, 117(1), 99–106, doi:10.1016/j.pbiomolbio.2014.10.001.
  * Coudert (2017) Coudert, F.-X., 2017: Reproducible research in computational chemistry of materials. Chemistry of Materials, 29(7), 2615–2617, doi:10.1021/acs.chemmater.7b00799.
  * Crook et al. (2013) Crook, S. M., A. P. Davison, and H. E. Plesser, 2013: Learning from the past: approaches for reproducibility in computational neuroscience. In Bower, J., ed., 20 Years of Computational Neuroscience, Springer, vol. 9 of Springer Series in Computational Neuroscience, pp. 73–102, doi:10.1007/978-1-4614-1424-7_4.
  * Davison (2012) Davison, A., 2012: Automated capture of experiment context for easier reproducibility in computational research. Computing in Science & Engineering, 14(4), 48–56, doi:10.1109/MCSE.2012.41.
  * de Winter and Happee (2013) de Winter, J. and R. Happee, 2013: Why selective publication of statistically significant results can be effective. PloS one, 8(6), e66463, doi:10.1371/journal.pone.0066463.
  * Delescluse et al. (2012) Delescluse, M., R. Franconville, S. Joucla, T. Lieury, and C. Pouzat, 2012: Making neurophysiological data analysis reproducible: Why and how? Journal of Physiology–Paris, 106(3), 159–170, doi:10.1016/j.jphysparis.2011.09.011.
  * Dewald et al. (1986) Dewald, W. G., J. G. Thursby, and R. G. Anderson, 1986: Replication in empirical economics: The journal of money, credit and banking project. The American Economic Review, 587–603, www.jstor.org/stable/1806061.
  * Donoho et al. (2009) Donoho, D. L., A. Maleki, I. U. Rahman, M. Shahram, and V. Stodden, 2009: Reproducible research in computational harmonic analysis. Computing in Science & Engineering, 11(1), 8–18, doi:10.1109/MCSE.2009.15.
  * Drummond (2009) Drummond, C., 2009: Replicability is not reproducibility: nor is it good science. [Conference Paper] Cogprints ID 7691, http://cogprints.org/7691/.
  * Duvendack et al. (2017) Duvendack, M., R. Palmer-Jones, and W. R. Reed, 2017: What is meant by “replication” and why does it encounter resistance in economics? American Economic Review, 107(5), 46–51, doi:10.1257/aer.p20171031.
  * Eich (2014) Eich, E., 2014: Business not as usual. Psychological Science, 25(1), 3–6, doi:10.1177/0956797613512465.
  * FASEB (2016) FASEB, 2016: Enhancing research reproducibility. Federation of American Societies for Experimental Biology, https://www.faseb.org/Portals/2/PDFs/opa/2016/FASEB_Enhancing%20Research%20Reproducibility.pdf.
  * Garijo et al. (2013) Garijo, D., S. Kinnings, L. Xie, L. Xie, Y. Zhang, P. E. Bourne, and Y. Gil, 2013: Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PloS one, 8(11), e80278, doi:10.1371/journal.pone.0080278.
  * Gentleman and Temple Lang (2007) Gentleman, R. and D. Temple Lang, 2007: Statistical analyses and reproducible research. Journal of Computational and Graphical Statistics, 16(1), 1–23, doi:10.1198/106186007X178663.
  * Goodman et al. (2016) Goodman, S. N., D. Fanelli, and J. P. A. Ioannidis, 2016: What does research reproducibility mean? Science translational medicine, 8(341), 341ps12, doi:10.1126/scitranslmed.aaf5027.
  * Greenbaum et al. (2017) Greenbaum, D., J. Rozowsky, V. Stodden, and M. Gerstein, 2017: Structuring supplemental materials in support of reproducibility. Genome biology, 18(1), 64, doi:10.1186/s13059-017-1205-3.
  * Guala (2005) Guala, F., 2005: The Methodology of Experimental Economics. Cambridge University Press.
  * Hamermesh (2007) Hamermesh, D. S., 2007: Replication in economics. Canadian Journal of Economics/Revue canadienne d’économique, 40(3), 715–733, doi:10.1111/j.1365-2966.2007.00428.x.
  * Hyndman (2010) Hyndman, R. J., 2010: Encouraging replication and reproducible research. Int. J. Forecasting, 26, 2–3, doi:10.1016/j.ijforecast.2009.12.003.
  * Iqbal et al. (2016) Iqbal, S. A., J. D. Wallach, M. J. Khoury, S. D. Schully, and J. P. Ioannidis, 2016: Reproducible research practices and transparency across the biomedical literature. PLoS Biology, 14(1), e1002333, doi:10.1371/journal.pbio.1002333.
  * Jasny et al. (2011) Jasny, B. R., G. Chin, L. Chong, and S. Vignieri, 2011: Again, and again, and again…. Science, 334(6060), 1225, doi:10.1126/science.334.6060.1225.
  * JCGM (2008) JCGM, 2008: The international vocabulary of metrology—basic and general concepts and associated terms (VIM). Joint Committee for Guides in Metrology, 3rd edn. JCGM 200: 2012, https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf.
  * Kafkafi et al. (2016) Kafkafi, N., J. Agassi, E. J. Chesler, J. C. Crabbe, W. E. Crusio, D. Eilam, R. Gerlai, I. Golani, A. Gomez-Marin, R. Heller, et al., 2016: Reproducibility and replicability of rodent phenotyping in preclinical studies. bioRxiv, 079350, doi:10.1101/079350.
  * King (1995) King, G., 1995: Replication, replication. PS: Political Science & Politics, 28(3), 444–452, doi:10.2307/420301.
  * Kitzes et al. (2017) Kitzes, J., D. Turek, and F. Deniz, 2017: The practice of reproducible research: case studies and lessons from the data-intensive sciences. University of California Press.
  * Koenker and Zeileis (2009) Koenker, R. and A. Zeileis, 2009: On reproducible econometric research. Journal of Applied Econometrics, 24(5), 833–847, doi:10.1002/jae.1083.
  * Laine et al. (2007) Laine, C., S. N. Goodman, M. E. Griswold, and H. C. Sox, 2007: Reproducible research: Moving toward research the public can really trust. Annals of Internal Medicine, 146(6), 450–453, doi:10.7326/0003-4819-146-6-200703200-00154.
  * Lazer et al. (2014) Lazer, D., R. Kennedy, G. King, and A. Vespignani, 2014: The parable of google flu: traps in big data analysis. Science, 343(6176), 1203–1205, doi:10.1126/science.1248506.
  * Lejaeghere et al. (2016) Lejaeghere, K., G. Bihlmayer, T. Björkman, P. Blaha, S. Blügel, V. Blum, D. Caliste, I. E. Castelli, S. J. Clark, A. Dal Corso, et al., 2016: Reproducibility in density functional theory calculations of solids. Science, 351(6280), aad3000, doi:10.1126/science.aad3000.
  * LeVeque (2009) LeVeque, R. J., 2009: Python tools for reproducible research on hyperbolic problems. Computing in Science & Engineering, 11(1), 19–27, doi:10.1109/MCSE.2009.13.
  * LeVeque et al. (2012) LeVeque, R. J., I. M. Mitchell, and V. Stodden, 2012: Reproducible research for scientific computing: Tools and strategies for changing the culture. Computing in Science & Engineering, 14(4), 13–17, doi:10.1109/MCSE.2012.38.
  * Liberman (2015) Liberman, M., 2015: Replicability vs. reproducibility — or is it the other way around? The Language Log, http://languagelog.ldc.upenn.edu/nll/?p=21956.
  * Lindsay (2015) Lindsay, D. S., 2015: Replication in psychological science. Psychological Science, 26(12), 1827–1832.
  * Loscalzo (2012) Loscalzo, J., 2012: Irreproducible experimental results. Circulation, 125(10), 1211–1214, doi:10.1161/circulationaha.112.098244.
  * Marszalek and Flintoft (2016) Marszalek, R. T. and L. Flintoft, 2016: Being open: our policy on source code. Genome Biology, 17, 172, doi:10.1186/s13059-016-1040-y.
  * Marwick et al. (2017) Marwick, B., A. Rokem, and V. Staneva, 2017: Assessing Reproducibility. _In The Practice of Reproducible Research Case Studies and Lessons from the Data-Intensive Sciences, Kitzes et al., eds._ , University of California Press. pp. 3–18.
  * McCullough et al. (2008) McCullough, B. D., K. A. McGeary, and T. D. Harrison, 2008: Do economics journal archives promote replicable research? Canadian Journal of Economics/Revue canadienne d’économique, 41(4), 1406–1420, doi:10.1111/j.1540-5982.2008.00509.x.
  * National Science Foundation (2015) National Science Foundation, 2015: The basics of reproducibility. Video on the NSF website, https://www.nsf.gov/discoveries/disc_summ.jsp?cntn_id=136139&org=NSF.
  * Nosek et al. (2015) Nosek, B. A., G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, C. D. Chambers, G. Chin, G. Christensen, et al., 2015: Promoting an open research culture. Science, 348(6242), 1422–1425, doi:10.1126/science.aab2374.
  * Open Science Collaboration (2012) Open Science Collaboration, 2012: An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660, doi:10.1177/1745691612462588.
  * Open Science Collaboration (2015) —, 2015: Estimating the reproducibility of psychological science. Science, 349(6251), aac4716, doi:10.1126/science.aac4716.
  * Pashler and Wagenmakers (2012) Pashler, H. and E.-J. Wagenmakers, 2012: Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530, doi:10.1177/1745691612465253.
  * Pellizzari et al. (2017) Pellizzari, E., K. Lohr, A. Blatecky, and D. Creel, 2017: Reproducibility: A Primer on Semantics and Implications for Research. RTI Press.
  * Peng (2009) Peng, R. D., 2009: Reproducible research and biostatistics. Biostatistics, 10(3), 405–408, doi:10.1093/biostatistics/kxp014.
  * Peng (2011) —, 2011: Reproducible research in computational science. Science, 334(6060), 1226–1227, doi:10.1126/science.1213847.
  * Peng et al. (2006) Peng, R. D., F. Dominici, and S. L. Zeger, 2006: Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789, doi:10.1093/aje/kwj093.
  * Pesaran (2003) Pesaran, H., 2003: Introducing a replication section. Journal of Applied Econometrics, 18(1), 111–111, http://www.jstor.org/stable/30035191.
  * Rougier et al. (2017) Rougier, N. P., K. Hinsen, F. Alexandre, T. Arildsen, L. A. Barba, F. C. Benureau, C. T. Brown, P. De Buyl, O. Caglayan, A. P. Davison, et al., 2017: Sustainable computational science: the rescience initiative. PeerJ Computer Science, 3, e142, doi:10.7717/peerj-cs.142.
  * Sandve et al. (2013) Sandve, G. K., A. Nekrutenko, J. Taylor, and E. Hovig, 2013: Ten simple rules for reproducible computational research. PLoS Computational Biology, 9(10), e1003285, doi:10.1371/journal.pcbi.1003285.
  * Schwab et al. (2000) Schwab, M., M. Karrenbach, and J. Claerbout, 2000: Making scientific computations reproducible. Computing in Science & Engineering, 2(6), 61–67, doi:10.1109/5992.881708.
  * Stevens (2017) Stevens, J. R., 2017: Replicability and reproducibility in comparative psychology. Frontiers in Psychology, 8, 862, doi:10.3389/fpsyg.2017.00862.
  * Stodden (2011) Stodden, V., 2011: Trust your science? open your data and code. Amstat News, 21–22, http://magazine.amstat.org/blog/2011/07/01/trust-your-science/.
  * Stodden (2013) —, 2013: Resolving irreproducibility in empirical and computational research. IMS Bulletin, http://bulletin.imstat.org/2013/11/resolving-irreproducibility-in-empirical-and-computational-research/.
  * Stodden (2015) —, 2015: Reproducing statistical results. Annual Review of Statistics and Its Application, 2, 1–19, doi:10.1146/annurev-statistics-010814-020127.
  * Stodden et al. (2014) Stodden, V., F. Leisch, and R. D. Peng, eds., 2014: Implementing Reproducible Research. CRC Press, iSBN 9781466561595, preprint at osf.io/s9tya/.
  * Taylor and Kuyatt (1994) Taylor, B. N. and C. E. Kuyatt, 1994: Guidelines for evaluating and expressing the uncertainty of NIST measurement results. US Department of Commerce, Technology Administration, National Institute of Standards and Technology, https://www.nist.gov/sites/default/files/documents/2017/05/09/tn1297s.pdf.
  * Topalidou et al. (2015) Topalidou, M., A. Leblois, T. Boraud, and N. P. Rougier, 2015: A long journey into reproducible computational neuroscience. Frontiers in Computational Neuroscience, 9, 30, doi:10.3389/fncom.2015.00030.
  * Vandewalle et al. (2009) Vandewalle, P., J. Kovacevic, and M. Vetterli, 2009: Reproducible research in signal processing. IEEE Signal Processing Magazine, 26(3), 37–47, doi:10.1109/MSP.2009.932122.
  * Wikipedia (2017) Wikipedia contributors, 2017: Jon Claerbout. Wikipedia, The Free Encyclopedia, https://en.wikipedia.org/w/index.php?title=Jon_Claerbout&oldid=777337051 [Online; accessed 14-January-2018].


# FORCASTing the spectroscopic dust properties of the WC+O binary WR137 with
SOFIA

Megan J. Peatt  Department of Physics and Astronomy, Embry-Riddle Aeronautical
University, 3700 Willow Creek Rd., Prescott, AZ 86301, USA  Noel D. Richardson
Department of Physics and Astronomy, Embry-Riddle Aeronautical University,
3700 Willow Creek Rd., Prescott, AZ 86301, USA  Peredur M. Williams Institute
for Astronomy, University of Edinburgh, Royal Observatory, Edinburgh EH9 3HJ,
United Kingdom  Nicole Karnath Space Science Institute, 4765 Walnut St, Suite
B Boulder, CO 80301, USA  Center for Astrophysics Harvard & Smithsonian,
Cambridge, MA 02138, USA  Victor I. Shenavrin Sternberg Astronomical
Institute, Moscow State University, Universitetskij pr., 13, Moscow, 119991,
Russia  Ryan M. Lau NSF’s NOIRLab, 950 N. Cherry Avenue, Tucson, Arizona
85719, USA  Anthony F. J. Moffat Département de physique, Université de
Montréal, Complexe des Sciences, 1375 Avenue Thérèse-Lavoie-Roux, Montréal,
Quebéc, H2V 0B3, Canada  Gerd Weigelt Max Planck Institute for Radio
Astronomy, Auf dem Hügel 69, 53121 Bonn, Germany

###### Abstract

WR 137 (HD 192641) is a binary system consisting of a carbon-rich Wolf-Rayet
star and an Oe companion star in a 13-year orbit. Near periastron, the winds
of the two stars collide and form carbonaceous dust. We obtained three mid-
infrared grism spectra of the system with SOFIA and FORCAST during the last
year of SOFIA’s operations in July 2021, February 2021, and May 2022 (Cycle
9). Within these spectra, we have identified several wind lines from He I, He
II, C III, and C IV that are emitted from the Wolf-Rayet wind as well as a
weak emission feature around 6.3–6.4 $\mu$m that may have shifted its peak
flux from 6.29 to 6.41$\mu$m through this time period. The weak feature grew
as the continuum dust emission grew while the WR emission appeared to decline
due to lower contrast with the continuum. Furthermore, we observe that the
peak of the feature shifts to redder wavelengths during the observations. We
compare this feature to the UIR feature and other emission lines identified in
dusty WC binaries. For WR 137, we speculate that mixing of the winds in the
system with the Oe star’s disk is important for starting the dust formation
and that it is less important as dust formation continues. Previous infrared
photometry shows “mini-eruptions” of dust production which could then be
explained with variations of the Oe star disk.

Wolf-Rayet stars (1806), Early-type stars (430), Oe stars (1153), Dust
formation (2269), Binary stars (154), Stellar winds (1636)

††journal: ApJ††facilities: SOFIA(FORCAST)††software: astropy (Astropy
Collaboration et al., 2013, 2018)

##  1 Introduction

Massive stars have large impacts on their environments through strong stellar
winds and terminal supernova explosions. The exact nature of the impacts can
be changed through binary companions and their orbital architecture. These
stars are usually formed in binary systems, and a majority of the systems are
expected to interact during their lifetimes (Sana et al., 2012, 2013). Amongst
the massive stars, Wolf-Rayet (WR) stars represent highly evolved massive
stars which have lost their outer hydrogen envelope from either strong stellar
winds or binary interactions. They are helium burning and have strong
hydrogen-free stellar winds ($\dot{M}\sim 10^{-5}M_{\odot}\ {\rm yr}^{-1}$;
$v_{\infty}\gtrsim 2000\ {\rm km\ s}^{-1}$) and are assumed to have gone
through rapid mass-loss before becoming a WR star. These stars have highly
structured, strong stellar winds with terminal speeds of several thousand km
s-1. From their spectra, the WR stars are classified into either nitrogen-rich
(WN) or carbon-rich (WC) depending on if the wind emission is dominated by
nitrogen and ionized helium or carbon.

WR 137 was one of the first WR stars discovered by Wolf & Rayet (1867) at the
Paris Observatory along with WR 134 and WR 135 (where they were labeled by
their 1850 coordinates, the present names coming from the ‘Sixth Catalogue’
(van der Hucht et al., 1981). The latter authors classified its spectrum as
WC7+abs, reflecting its uncertain status as a binary. From moderate-resolution
spectra, Underhill (1962) deduced that it was a WC7+Be shell star binary but
Massey et al. (1981) and Moffat et al. (1986) did not find variations in its
radial velocities, deducing that WR 137 was not a (close) binary system.

Multi-wavelength infrared (IR) photometry of late sub-type WC stars found a
few to show emission from carbonaceous dust (Allen et al., 1972). Two WR
stars, WR 140 (HD 193793, Williams et al. (1978), WC7+abs) and WR 48a (Danks
et al. (1983), WC9) were observed to show rapid increases in the IR attributed
to dust formation. Infrared photometry of WR 137 in 1978–84 (Williams et al.,
1985) also showed a dust-formation episode, with maximum near 1984.5. Re-
examination of IR photometry of WR 137 in 1970–77 when the emission was fading
(Hackwell et al., 1976) suggested that WR 137 was at that time recovering from
an earlier, unobserved, dust formation event, leading Williams et al. (1985)
to suggest that such eruptions recurred at intervals of about 15 years.

The episodic dust formation resembling that by WR 140 revived the search for
binary motion. Annuk (1991) demonstrated that the WR 137 system was a binary
by presenting a first RV orbit, adopting a period of 4400 days from the IR
data but Underhill (1992) did not accept this and advocated a single star
model for WR 137.

Continued IR photometry (Williams et al., 2001) revealed another dust
formation episode peaking in 1997, suggesting a period near 4765$\pm$50 days.
Lefèvre et al. (2005) derived a spectroscopic period of 4766$\pm$66 days from
their RV orbit, confirming the binary status of WR 137. Richardson et al.
(2016) collected $H$-band interferometry with the CHARA Array, which allowed
them to separate the two components and support a nearly edge-on inclination
despite only one epoch of observations. The two stars contribute nearly
equally to the $H$-band flux. They also found a spectral type of WC7pd+O9V
using spectroscopic modeling.

The WR 137 dust cloud was imaged in the IR by Marchenko et al. (1999) in 1997
and 1998, shortly after IR maximum, using the Hubble Space Telescope’s NICMOS
instrument in the $H^{\prime}-$ and $K^{\prime}-$bands. They found IR-emitting
clumps near the central binary, and evidence for a stream of dust emanating
from the central source with an extent of $\sim 0.25\arcsec$ ($\sim 4$ pc).
This type of structure was confirmed by Lau et al. (2023) who presented JWST
aperture masking interferometric observations with the NIRISS instrument.

The dust may form in the shocked gas between a WC star and companion O star
where their winds collide, such as inferred from the rotating ‘pinwheel’ of
heated dust made in the WC9 system WR 104 (which still requires determination
of its orbit) and the canonical episodic dust-maker WR 140. This system was
recently imaged by JWST+MIRI in the mid-infrared with three filters spanning
7.7–21$\mu$m, showing more than 150 years of dust created and sent into
interstellar space in concentric arcs (Lau et al., 2022). Spatially-resolved
spectroscopy of the dust indicated two unidentified infrared (UIR) features
around 6.4 and 7.8 $\mu$m, as well as a strong [S IV] feature at 10.5$\mu$m.
This greatly extended the ground-based IR imaging of WR 140’s dust by Monnier
et al. (2002) and Williams et al. (2009).

A decretion disk around the companion star in WR 137 was discovered from the
observation of double-peaked hydrogen and helium emission lines, making it an
O9e star. This was first noted by Underhill (1962), and recently investigated
in much more detail by St-Louis et al. (2020). This may explain the constant
component of the continuum polarization (Harries et al., 2000) and the thinner
spiral of dust around WR137 (St-Louis et al., 2020, Lau et al. submitted).

Figure 1: The long-term infrared light curve of WR 137. We show the archival
photometry (references in Section 1) of WR 137 as well as our new
measurements. When the binary nears periastron, the infrared brightness
increases due to increased dust production with the next periastron happening
soon.

In this paper, we report on new mid-infrared spectroscopy of WR 137 taken with
the Stratospheric Observatory for Infrared Astronomy (SOFIA) and the FORCAST
instrument as the recent dust formation began. We describe these observations
in Section 2. Section 3 describes measurements we made of spectral features in
our observations, including a measurement of a line that appears to be
associated with the dust being created. We discuss these results with a
comparison to other WR systems and their dust features in Section 4,
concluding this study in Section 5.

##  2 Observations

###  2.1 Infrared photometry

Near- and mid-infrared photometry of WR 137 has been collected at the
Sternberg Astronomical Institute’s (SAI) Crimean Laboratory of Moscow State
University using a $JHKLM$ photometer with an InSb photovoltaic detector
cooled with liquid nitrogen (Shenavrin et al., 2011). Observations have been
taken regularly since 2020 August, allowing us to compare the outburst with
the previously observed outburst. Each data point consists of multiple sub-
exposures of 30–60 s with a total integration time of 5–10 minutes in each of
the $JHKL$ filters and 20–25 minutes in the $M$ filter. The observations are
compared to standard star observations taken before or after each observation
of the target. The light curve is tabulated as an online machine readable
table.

###  2.2 SOFIA Observations

We obtained mid-IR grism spectroscopy using the FORCAST instrument (Herter et
al., 2013, 2018) onboard SOFIA (Temi et al., 2018). The first observation,
taken in 2021 July, used both the G063 and G111 grating setups, with two
subsequent observations in 2022 February and May111We note that the final
flight where our data were collected was nicknamed “Perry”, which is extremely
appropriate given Peredur Williams’ expertise and involvement in the long-term
monitoring and discovery of dust around WC stars. with only observations using
the G063 grism. This setup covers the wavelength range of 4.9–-8.0 $\mu$m, but
we found the spectra near the edge of the chip to be too noisy for analysis,
effectively reducing the useful range to 5.1–7.9 $\mu$m. These spectra were
taken with the 2.4-$\arcsec$ slit, yielding a spectral resolving power of
$\sim 180$. The FORCAST grism data are known to have variable slit-losses,
adding uncertainty to the flux calibration, but these data should be accurate
to a few percent (e.g., Gehrz et al., 2021). For this analysis, we focus on
the variability of line flux and equivalent widths, so we concentrate on only
the G063 spectra.

Figure 2: In the top panel, we show the spectrum from July 2021 with line
identifications for He I, He II, C III, and C IV. We also show the region of
the weak UIR feature at 6.3–6.4$\mu$m. In the lower panel, we show the three
spectra taken with SOFIA/FORCAST compared to a smoothed model of the binary
without dust emission (Richardson et al., 2016). Note that the model only
includes the stellar and wind flux, neglecting all dust emission. Some of
these emission lines from the wind were not included in the model.  Figure 3:
Here we show the three spectra with the corresponding times in the infrared
evolution. Unfortunately, no IR photometry was able to be collected at times
near the February 2022 observation, so the interpolated flux on a line plot is
likely overestimated compared with the actual flux at that time.

##  3 Measurements

Richardson et al. (2016) presented models of the combined spectrum of WR 137
based on optical and ultraviolet spectroscopy. The model, along with the three
flux-calibrated spectra from SOFIA, are shown in Fig. 2. While the SOFIA data
are noisy, a few things are readily apparent. The blended lines at 5.4 and 5.9
$\mu$m are not present in the models presented by Richardson et al. (2016).
The He I line at $\sim$7.5 $\mu$m appears anomalously strong compared with the
model or other helium transitions in our spectra and it is possible that
hydrogen Pfund-$\alpha$ (7.46 $\mu$m) from the decretion disk around the
companion star reported by St-Louis et al. (2020) could be a significant
contributor. We were able to measure the 6.4 $\mu$m feature equivalent width
and then compare it to known WR lines of reasonable strength at 5.6, 5.8, 6.9,
and 7.45 $\mu$m. Our errors are statistical in nature as we measured the
strength with several assumptions of where the continuum was present, allowing
us a reasonable confidence on the strengths of the lines measured. In Fig. 2,
we also show line identifications for the spectrum obtained in 2021 July. We
identify lines of He I, He II, C III, and C IV, including several that were
not included in the PoWR model reported by Richardson et al. (2016).

The spectra are compared with the IR light curve in Fig. 3, showing that the
changing flux observed by SOFIA was seen in the NIR photometry. We show our
equivalent width measurements for the 6.4$\mu$m feature as well as other lines
in Fig. 4. To the first order, we see that the equivalent width of the lines
from the wind appear to decrease as the continuum increased through these
epochs. This is typical behavior for the wind lines given the additional
continuum flux from dust being formed in the wind, as demonstrated in the
infrared photometry (Fig. 1). We also show in Table 1 the net flux for the
blended lines at 6.9 $\mu$m (the closest line to the UIR band) that shows a
constant flux (within uncertainties) across the three observations.

While the equivalent width of the WR wind lines decreased during our three
observations, the opposite is true for the 6.4 $\mu$m feature which grew
stronger with time. This shows that this feature is inconsistent with the WR
wind lines and is therefore associated with the dust production near
periastron. We tested this by taking ratios of the 6.4$\mu$m line to each of
the wind lines. These ratios, shown in Fig. 5, show that this line indeed
appears stronger with time, consistent with this feature being formed in the
newly formed dust around the system.

Line ($\mu$m) | July 2021 | February 2022 | May 2022
---|---|---|---
JD: | 2,459,396.5 | 2,459,626.5 | 2,459,719.5
|  $W_{\lambda}$($\mu$m) |  $W_{\lambda}$($\mu$m) |  $W_{\lambda}$($\mu$m)
6.3–6.4$\mu$m feature | -6.47 $\pm$ 0.44 [6.29$\mu$m] | -9.53 $\pm$ 0.44 [6.36$\mu$m] | -13.81 $\pm$ 0.44 [6.41$\mu$m]
5.6 C IV+He II | -16.47 $\pm$ 0.71 | -16.48 $\pm$ 0.71 | -11.42 $\pm$ 0.71
5.8 C III+C IV | -18.47 $\pm$ 0.41 | -10.38 $\pm$ 0.41 | -5.42 $\pm$ 0.41
6.9 C IV+He II | -32.26 $\pm$ 0.47 | -19.95 $\pm$ 0.47 | -11.28 $\pm$ 0.47
7.5 He I+He II | -20.12 $\pm$ 1.00 | -16.50 $\pm$ 1.00 | -11.21 $\pm$ 1.00
Flux | (Jy) | (Jy) | (Jy)
6.9 C IV+He II | (4.44$\pm$0.16)$\times 10^{-2}$ | (4.68$\pm$0.41)$\times 10^{-2}$ | (4.77$\pm$0.17)$\times 10^{-2}$
Table 1: The averages of each equivalent width (in $\mu$m) for the three
SOFIA/FORCAST spectra with uncertainties calculated from standard deviation.
We note the JD for each spectrum and the central wavelength of the 6.4 $\mu$m
is given in square brackets. In the bottom row, we show the integrated net
flux from the 6.9 $\mu$m line in Jy.  Figure 4: The WR lines can be seen
decreasing in strength as the continuum increases, while the 6.4$\mu$m
feature’s strength grows, demonstrating a relation to the dust being produced
during periastron. The error bars are the standard deviation from the mean of
the measurements. 6.4$\mu$m is shown with a black dot and the others are
colored symbols, which match the symbols used in Fig 5.  Figure 5: To compare
the strengths of the lines, a ratio was taken of the 6.4$\mu$m feature over
each WR line. Ratios are represented by the corresponding symbol from our
previous plot. A clear trend is shown of the ratios steadily increasing over
time, though the WR line at 5.8$\mu$m does so at a quicker rate, potentially
related to colliding winds or the dust.  Figure 6: The portion of the SOFIA
grism spectroscopy around the UIR feature at 6.2–6.5 $\mu$m. The data from May
2022 were shifted downward by 2 Jy for clarity. The average peak position from
our measurements is highlighted and the data are smoothed for clarity with the
unsmoothed spectra shown in grey. We have overplotted a dotted line to guide
the eye in the shift of the peak wavelength.

##  4 Discussion

We have found that a seemingly weak emission line at $\sim$6.2-6.5 $\mu$m (see
Fig. 6) in the infrared spectra of WR 137 taken with SOFIA during its last
year of operation. The emission line increased in strength as the dust
formation began leading up to the periastron passage that will occur in
$\sim$2024\. Chiar et al. (2002) observed a similar feature in the dusty WR
binary WR 48a. In this discussion, we will compare this feature with that seen
in other WCd binaries and how its form changed throughout these three
observations and what that could imply for dust creation in WC binaries, and
in particular for the case of WR 137.

Endo et al. (2022) observed a broad 8 $\mu$m feature in the spectrum of the
28.1-year period system WR 125 when the system was near its infrared maximum.
This feature was seen in ISO spectra of several WCd stars and was compared to
these other systems by Endo et al. The broad 8-$\mu$m feature is often
associated with the unidentified infrared (UIR) features, and thus it probably
represents emission from the building blocks of the dust particles in the
colliding wind region. Our SOFIA spectra of WR 137 are cut off by the detector
and do not cover the region of the 8-$\mu$m feature.

However, the 8 $\mu$m feature is usually associated with a feature at 6.2
$\mu$m in infrared dust spectra. Duley & Williams (1981) found that the UIR
features are often inherent in compounds made of aromatic C$-$H and C$=$C
chemical bonds, with some suggesting that this arises from PAH emission (e.g.,
Allamandola et al., 1985). Since the UIR feature is usually seen at 6.2$\mu$m,
the observed wavelength of the feature would imply that the feature is shifted
to a longer wavelength in the spectrum of WR 137. However, it is also seen at
a longer wavelength in the spectrum of WR 48a presented by Chiar et al.
(2002), which is classified as “class C” according to the procedures outlined
by Peeters et al. (2002).

The longer wavelength of the feature can be caused by hydrogen being absent in
the PAH molecules that form this line (e.g., Pauzat et al., 1997).
Furthermore, we saw some evidence that the central wavelength of the feature
shifted to longer wavelengths during the year of our observations. In Fig. 6,
we show the spectra in the region of this line with the average peak
wavelength shown. For clarity, we also smooth these spectra and shift the last
spectrum to a similar flux level as the other observations for clarity. We
caution that the signal-to-noise of these spectra is low as evidenced in the
unsmoothed spectra in Fig. 6, but the overall shift of the line peak went from
$\sim$6.29$\mu$m in July 2021 to $\sim$6.35$\mu$m in February 2022, and then
to $\sim$6.41$\mu$m in May 2022. We speculate that this shift implies that the
dust begins forming in the WR 137 system in a mixing environment of the WC
wind and the Oe hydrogen disk. As periastron nears, the dust is sent along the
collision region which becomes more intense. These initial grains can then
grow with additional carbon from the WC wind but with less hydrogen available
from the Oe wind.

Lau et al. (submitted) presented aperture masking interferometry of the WR 137
system at 3.8 $\mu$m and 4.8 $\mu$m taken with the NIRISS instrument on JWST.
The resulting images of the dust around WR 137 show a very narrow extension of
the dust compared to the models that are normally able to reproduce the
geometry of the WCd binary dust (e.g., WR 112, Lau et al., 2020b). These
models have been widely used both for systems with well-established orbits
like WR 140 (Lau et al., 2022) as well as for inferring orbital information
from long-period systems like Apep (WC8+WN; Callingham et al., 2020), WR 48a
(WC6; Lau et al., 2020a), or WR 112 (WC8; Lau et al., 2020b). The narrow
signature of dust formation in the WR 137 system could imply that the mixing
with the Oe star’s disk material is important and crucial for the dust
formation in this system, which allows for periodic dust formation as the disk
is remarkably stable (St-Louis et al., 2020).

We suggest that the wind of the WC star collides with the equatorial disk
surrounding the Oe star. As this begins, the disk provides a large reservoir
of hydrogen that mixes with the carbon-rich WC wind. This mixing begins at
times where the PAH-like formation can create molecules that are not very
hydrogen-deficient. However, as the collisions in the system build and the
binary approaches periastron, we see the feature move to longer wavelengths
indicating a potential for fewer hydrogen atoms to be bound to the PAHs
emitting from the feature. As a result of this mixing, the dust is built from
the disk, but gradually grows with more carbon-rich material than with the
mixed materials.

The infrared light curve of WR 137 (Fig. 1) has been seen in the past to be
somewhat more irregular than other episodic dust-producing WC binaries. In the
WR 140 system, the light curve is almost completely repeatable with very few
or no excursions. However, in the light curve of WR 137, Williams et al.
(2001) show evidence of “mini-eruptions” happening in 1987, 1988, 1990 on top
of the regular activity. The current outburst shows some deviations from these
previous outbursts. These differences from the repeatable behavior of WR 140
shows that variable disk activity from the Oe star could be a protagonist in
our understanding of dust production for WR 137.

The observations presented here, especially when coupled to the stellar
parameters presented by St-Louis et al. (2020) and the JWST observations
presented by Lau et al. (2023), provide some speculative evidence for mixing
being an important aspect for dust creation in the WCd binaries with episodic
dust creation, or at least for the case of WR 137. Lau et al. (2023) also
presented evidence that the collision strength between the two winds for WR
137 was not strong enough for regular dust formation, stating that the disk
was required for the dust formation in WR 137 in order to provide an
environment with high enough density to begin the dust formation process.

The disk seems to be an essential ingredient in this system’s dust creation,
so we also want to consider how such a star evolved in the system. Oe stars,
like the Be stars (Rivinius et al., 2013), are rapid rotators. This is also
observed in WR 137 where Richardson et al. (2016) measured a $v\sin i$ of 220
km s-1 for the O star, although this is close to an average rotational
velocity for a Be star (e.g., Martayan et al., 2007). The results of Marchenko
et al. (1999?) and Richardson et al. (2016) seem to point towards an edge-on
orbital geometry for WR 137, a result that is being confirmed with follow-up
interferometry (Richardson et al., in prep), which could mean the Oe star is
not aligned with the orbit.

##  5 Conclusions

We have found that the mid-infrared emission line at 6.3–6.4 $\mu$m in the
spectrum of WR 137 is associated with the growth of dust during the approach
towards periastron passage in 2021–2022. The feature grew in strength as the
continuum emission from the dust also grew while the relative strength of the
emission lines from the Wolf-Rayet wind appeared to shrink due to the
increasing continuum. The peak wavelength of the feature appeared to shift to
redder wavelengths as the feature grew, likely associated with hydrogen being
less prevalent in the grains as the grains grew. We do caution that this is a
marginal detection with the signal-to-noise of the data, but the long dust
plume, as recently imaged with aperture-masking interferometry with JWST, is
incredibly narrow. We speculate that these results show that the dust grain
growth begins in a mixing of material from the WC wind and the Oe disk. Such a
mixing is less important as the dust continues to build, leading to a shift in
the peak wavelength of the 6.2 $\mu$m feature that is typically considered a
UIR feature.

SOFIA observed both WR 137 and WR 125 (Daly et al., in prep) during recent
dust-producing episodes. These data along with some infrared data taken from
the ground (e.g., Endo et al., 2022) provide a look into the dust formation at
critical epochs in the binary clocks for these systems. Such observations will
provide a pivotal context for interpretation of higher-resolution and higher
signal-to-noise spectroscopy of spatially resolved dust around these massive
binaries as has already been accomplished with JWST with the prototype of
these systems, WR 140 (Lau et al., 2022).

Based on observations made with the NASA/DLR Stratospheric Observatory for
Infrared Astronomy (SOFIA). SOFIA was jointly operated by the Universities
Space Research Association, Inc. (USRA), under NASA contract NNA17BF53C, and
the Deutsches SOFIA Institut (DSI) under DLR contract 50 OK 2002 to the
University of Stuttgart. Financial support for this work was provided in part
by NASA through award #09-0163 issued by USRA. MJP is grateful to Embry-Riddle
Aeronautical University’s Undergraduate Research Institute for financial
support through their IGNITE program. NDR is grateful for support from the
Cottrell Scholar Award #CS-CSA-2023-143 sponsored by the Research Corporation
for Science Advancement.

## References

  * Allamandola et al. (1985) Allamandola, L. J., Tielens, A. G. G. M., & Barker, J. R. 1985, ApJ, 290, L25, doi: 10.1086/184435
  * Allen et al. (1972) Allen, D. A., Swings, J. P., & Harvey, P. M. 1972, A&A, 20, 333
  * Annuk (1991) Annuk, K. 1991, in Wolf-Rayet Stars and Interrelations with Other Massive Stars in Galaxies, ed. K. A. van der Hucht & B. Hidayat, Vol. 143, 245
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Callingham et al. (2020) Callingham, J. R., Crowther, P. A., Williams, P. M., et al. 2020, MNRAS, 495, 3323, doi: 10.1093/mnras/staa1244
  * Chiar et al. (2002) Chiar, J. E., Peeters, E., & Tielens, A. G. G. M. 2002, ApJ, 579, L91, doi: 10.1086/345316
  * Danks et al. (1983) Danks, A. C., Dennefeld, M., Wamsteker, W., & Shaver, P. A. 1983, A&A, 118, 301
  * Duley & Williams (1981) Duley, W. W., & Williams, D. A. 1981, MNRAS, 196, 269, doi: 10.1093/mnras/196.2.269
  * Endo et al. (2022) Endo, I., Lau, R. M., Sakon, I., et al. 2022, ApJ, 930, 116, doi: 10.3847/1538-4357/ac63bd
  * Gehrz et al. (2021) Gehrz, R. D., Banerjee, D. P. K., Evans, A., et al. 2021, The Astronomer’s Telegram, 14794, 1
  * Hackwell et al. (1976) Hackwell, J. A., Gehrz, R. D., Smith, J. R., & Strecker, D. W. 1976, ApJ, 210, 137, doi: 10.1086/154811
  * Harries et al. (2000) Harries, T. J., Babler, B. L., & Fox, G. K. 2000, A&A, 361, 273
  * Herter et al. (2013) Herter, T. L., Vacca, W. D., Adams, J. D., et al. 2013, PASP, 125, 1393, doi: 10.1086/674144
  * Herter et al. (2018) Herter, T. L., Adams, J. D., Gull, G. E., et al. 2018, Journal of Astronomical Instrumentation, 7, 1840005, doi: 10.1142/S2251171718400056
  * Lau et al. (2020a) Lau, R. M., Eldridge, J. J., Hankins, M. J., et al. 2020a, ApJ, 898, 74, doi: 10.3847/1538-4357/ab9cb5
  * Lau et al. (2020b) Lau, R. M., Hankins, M. J., Han, Y., et al. 2020b, ApJ, 900, 190, doi: 10.3847/1538-4357/abaab8
  * Lau et al. (2022) —. 2022, Nature Astronomy, 6, 1308, doi: 10.1038/s41550-022-01812-x
  * Lefèvre et al. (2005) Lefèvre, L., Marchenko, S. V., Lépine, S., et al. 2005, MNRAS, 360, 141, doi: 10.1111/j.1365-2966.2005.09017.x
  * Marchenko et al. (1999) Marchenko, S. V., Moffat, A. F. J., & Grosdidier, Y. 1999, ApJ, 522, 433, doi: 10.1086/307630
  * Martayan et al. (2007) Martayan, C., Floquet, M., Hubert, A. M., et al. 2007, A&A, 472, 577, doi: 10.1051/0004-6361:20077390
  * Massey et al. (1981) Massey, P., Conti, P. S., & Niemela, V. S. 1981, ApJ, 246, 145, doi: 10.1086/158907
  * Moffat et al. (1986) Moffat, A. F. J., Lamontagne, R., Shara, M. M., & McAlister, H. A. 1986, AJ, 91, 1392, doi: 10.1086/114116
  * Monnier et al. (2002) Monnier, J. D., Tuthill, P. G., & Danchi, W. C. 2002, ApJ, 567, L137, doi: 10.1086/340005
  * Pauzat et al. (1997) Pauzat, F., Talbi, D., & Ellinger, Y. 1997, A&A, 319, 318
  * Peeters et al. (2002) Peeters, E., Hony, S., Van Kerckhoven, C., et al. 2002, A&A, 390, 1089, doi: 10.1051/0004-6361:20020773
  * Richardson et al. (2016) Richardson, N. D., Shenar, T., Roy-Loubier, O., et al. 2016, MNRAS, 461, 4115, doi: 10.1093/mnras/stw1585
  * Rivinius et al. (2013) Rivinius, T., Carciofi, A. C., & Martayan, C. 2013, A&A Rev., 21, 69, doi: 10.1007/s00159-013-0069-0
  * Sana et al. (2012) Sana, H., de Mink, S. E., de Koter, A., et al. 2012, Science, 337, 444, doi: 10.1126/science.1223344
  * Sana et al. (2013) Sana, H., de Koter, A., de Mink, S. E., et al. 2013, A&A, 550, A107, doi: 10.1051/0004-6361/201219621
  * Shenavrin et al. (2011) Shenavrin, V. I., Taranova, O. G., & Nadzhip, A. E. 2011, Astronomy Reports, 55, 31, doi: 10.1134/S1063772911010070
  * St-Louis et al. (2020) St-Louis, N., Piaulet, C., Richardson, N. D., et al. 2020, MNRAS, 497, 4448, doi: 10.1093/mnras/staa2214
  * Temi et al. (2018) Temi, P., Hoffman, D., Ennico, K., & Le, J. 2018, Journal of Astronomical Instrumentation, 7, 1840011, doi: 10.1142/S2251171718400111
  * Underhill (1962) Underhill, A. B. 1962, ApJ, 136, 14, doi: 10.1086/147347
  * Underhill (1992) —. 1992, ApJ, 398, 636, doi: 10.1086/171889
  * van der Hucht et al. (1981) van der Hucht, K. A., Conti, P. S., Lundstrom, I., & Stenholm, B. 1981, Space Sci. Rev., 28, 227, doi: 10.1007/BF00173260
  * Williams et al. (1978) Williams, P. M., Beattie, D. H., Lee, T. J., Stewart, J. M., & Antonopoulou, E. 1978, MNRAS, 185, 467, doi: 10.1093/mnras/185.3.467
  * Williams et al. (1985) Williams, P. M., Longmore, A. J., van der Hucht, K. A., et al. 1985, MNRAS, 215, 23P, doi: 10.1093/mnras/215.1.23P
  * Williams et al. (2001) Williams, P. M., Kidger, M. R., van der Hucht, K. A., et al. 2001, MNRAS, 324, 156, doi: 10.1046/j.1365-8711.2001.04284.x
  * Williams et al. (2009) Williams, P. M., Marchenko, S. V., Marston, A. P., et al. 2009, MNRAS, 395, 1749, doi: 10.1111/j.1365-2966.2009.14664.x
  * Wolf & Rayet (1867) Wolf, C. J. E., & Rayet, G. 1867, Academie des Sciences Paris Comptes Rendus, 65, 292