# Detecting Stance in Scientific Papers: Did we get more Negative Recently?

Dominik Beese1, Begüm Altunbaş2, Görkem Güzeler2, Steffen Eger1
1 Technische Universität Darmstadt, 2 Sabancı University

###### Abstract

In this paper, we classify scientific articles in the domain of natural
language processing (NLP) and machine learning (ML) into whether (i) they
extend the current state-of-the-art by introduction of novel techniques which
beat existing models or whether (ii) they mainly criticize the existing state-
of-the-art, i.e., that it is deficient with respect to some property (e.g.,
wrong evaluation, wrong datasets, misleading task specification). We refer to
contributions under (i) as having a ?positive stance? and contributions under
(ii) as having a ?negative stance? to related work. We annotate over 2k papers
from NLP and ML to train a SciBERT based model to automatically predict the
stance of a paper based on its title and abstract. We then analyze large-scale
trends on over 41k papers from the last $\sim$35 years in NLP and ML, finding
that papers have gotten substantially more positive over time, but negative
papers also got more negative and we observe considerably more negative papers
in recent years. Negative papers are also more influential in terms of
citations they receive.

##  1 Introduction

Deep learning has revolutionized machine learning (ML) and natural language
processing (NLP) in the last decade. In particular, deep learning has led to
unprecedented performance gains on a large number of NLP and ML tasks,
including machine translation (Lample et al., 2018), image classification
(Krizhevsky et al., 2012), natural language understanding (Devlin et al.,
2018), and text generation (Radford et al., 2019).

Abstract (excerpt)
---
?We are surprised to find that BERT’s peak performance of 77% on the Argument
Reasoning Comprehension Task reaches just three points below the average
untrained human baseline. However, we show that this result is entirely
accounted for by exploitation of spurious statistical cues in the dataset. We
analyze the nature of these cues and demonstrate that a range of models all
exploit them. This analysis informs the construction of an adversarial dataset
on which all models achieve random accuracy. […]?
Table 1: Example of a paper with negative stance. The underlying paper is
Niven and Kao (2019). Negative elements are highlighted by us in red.

On the other hand, there has seemingly also been a recent flood of papers
highlighting limitations of (deep learning) approaches, including claims about
models exploiting dataset biases (Niven and Kao, 2019), flawed evaluation
(Marie et al., 2021), and general “troubling trends” in machine learning
practice (Lipton and Steinhardt, 2019).

Indeed, from a historic perspective, deep learning – formerly known under the
name of ‘artificial neural networks’ – is a prime exemplar of a technology
that has received very mixed assessments over time, ranging from initial hype
to negative and positive evaluation in repeating cycles (Hendler, 2008; Guo et
al., 2020).

Motivated especially by (recent) observations of negative assessment of
individual papers regarding the existing literature and its claims, we define
a new NLP task of determining the _stance_ of a paper with respect to its
related work.111This concept is related to positive/negative citations within
a paper, which has been annotated in a few works, e.g., Teufel et al. (2006).
Our work goes beyond individual citations and assesses the stance of the
authors’ main message. We take a prototypical paper of positive stance to be
one that generally accepts the premises of related work (even though it may
identify specific – minor – issues and weaknesses), extends it and sets a new
state-of-the-art.222This definition of positiveness is not uncritical; under
our definition, a paper that does not discuss related work but merely
proclaims a new state-of-the-art would be judged as positive towards related
work. Our intuition is that if a paper declares a new state-of-the-art, then
it extends related work and would thus typically (at least implicitly) need to
accept its premises. One could also discuss our concept of stance under the
framing of ‘critique’ vs. ‘non-critique’ papers. In contrast, a prototypical
paper of negative stance concludes that related work is basically flawed,
mistaken and potentially based on false assumptions (cf. 1). (Any particular
paper may mix positive and negative elements, so we treat the task as a
continuous regression problem.)333The authors of this paper have no ‘stance’
on ‘stance’: we believe that both positive and negative papers can be valuable
contributions to the scientific community.

We hold this task important in order to be able to (i) analyze trends in
science, (ii) which could potentially anticipate pessimistic future
developments (“the end of the party”). (iii) By comparing trends across two
disciplines – ML and NLP – we can also contrast the evolution and current
state of each. The task is also timely, as interest in the analysis of
scientific literature in the NLP community (cf. 2) has been steadily on the
rise recently.

Our contributions are:

  * •

we define the new task of stance detection for scientific literature;

  * •

we provide a human annotated dataset of over 2k scientific papers, labeled for
their stance;

  * •

we provide a large-scale trend analysis on over 41k papers from the NLP and ML
community in the past $\sim$35 years;

  * •

we address various trend questions including (a) whether negativity has
recently increased; (b) whether positive/negative papers are more influential
and (c) more likely to be accepted.

We point out that ‘negativity’, as is a focus of our work, plays a central
role in various contexts: in the social sciences, signed social networks are
networks in which agents have positive and negative relations to each other,
potentially explaining phenomena such as long-term disagreement (Altafini,
2012; Eger, 2016); in science, negative citations may be a form of self-
correction (Bordignon, 2020) and publishing negative results may reduce waste
on resources for disputed approaches (Mlinarić et al., 2017); in economics,
the principle of _creative destruction_ (Aghion and Howitt, 1990) may explain
the evolution and progress of capitalism.

##  2 Related work

Historically, analysis of scientific literature is the scope of the fields of
_scientometrics_ and _science-of-science_. Classical results include the
relation between title length and the number of citations a paper receives
(Letchford et al., 2015) as well as quantitative laws underlying citation
patterns or the number of co-authors of papers over time (Fortunato et al.,
2018). In recent years, with the rise in quality of models and approaches,
more and more NLP approaches are also devoted to the analysis of scientific
literature.

Gao et al. (2019) ask how much the author response in the ?rebuttal phase? of
the peer review process influences the final scores of a reviewer, finding its
impact to be marginal, especially compared to the scores of other reviewers.

Prabhakaran et al. (2016) predict whether a scientific topic will rise or fall
in popularity based on how authors frame the topics in their work. They use a
subset of the Web of
Science444https://clarivate.com/webofsciencegroup/solutions/web-of-science/
Core Collection with papers from 1991 to 2010 and analyze abstracts by
assigning scientific topics (e.g., stem cell research) as well as rhetorical
roles (i.e., scientific background, methods used, results, etc.) to phrases.
Prabhakaran et al. (2016) find that topics that are currently discussed as
results and background are at their peak and tend to fall in popularity in the
future, whereas topics that are mentioned as methods or conclusions tend to
start to rise in popularity.

Arguably the paper most similar to ours is that of Jurgens et al. (2018). They
study the entire content of a scientific publication in order to predict its
future impact, based on how citations are framed. They differentiate between
different _functions_ of citations: Background (the other work provides
relevant information), Uses (usage of data, methods, etc. from the other
work), and Comparison or Contrast (express similarities/differences to the
other work). Using human annotation and trained classifiers on top of them,
they annotate 21k papers from the ACL Anthology555https://aclanthology.org/
and analyze the evolution of the functions over (a) the course of a paper, (b)
different venues, and (c) time. For (c), papers citing many other technologies
with Uses citations tend to have large future impact. They further show that
the field of NLP has shown a significant increase in consensus when authors
started to use less Comparison or Contrast citations and simply acknowledged
previous work as Background. The amount of Uses citations also increased when
researchers began to agree on the same methods and technologies. The authors
argue that these trends imply that NLP has become a rapid discovery science
(Collins, 1994), i.e., a particular shift a scientific field can undergo when
it reaches a high level of consensus on its research topics, methods, and
technologies, and then starts to continually improve on each other’s methods.
A number of recent papers also leverages or studies citation context,
including Cohan et al. (2019); Jebari et al. (2021); Wright and Augenstein
(2021); Lauscher et al. (2021). Our approach differs from Jurgens et al.
(2018) in several ways: for example, we do not analyze individual citations,
but directly evaluate the stance of a complete paper (as measured by its
framing in the paper’s abstract); most importantly, we are particularly
interested in _negative_ stances, which as relation is absent in the
classification scheme of Jurgens et al. (2018).

Beyond classification, Yuan et al. (2021) use NLP models to automatically
generate reviews for scientific papers. To do so, they compile a dataset of
scientific articles and reviews from OpenReview666https://openreview.net/ and
NeurIPS Proceedings777https://papers.neurips.cc/. They conclude that their
review generation model is not good enough to fully automate the reviewing
process, but could still make the reviewer’s job more effective. Wang et al.
(2020) create automatic reviews for papers from OpenReview, NeurIPS, and the
PeerRead dataset (Kang et al., 2018a) by defining multiple knowledge graphs,
one extracted from the paper, one from the papers the paper cites, and one for
background knowledge. Beltagy et al. (2019) train a language model (extending
BERT) on a large multi-domain corpus of scientific publications.

##  3 Data

We extract our data from two sources: the ACL Anthology which contains papers
and meta-data for all major NLP events. Our second data source are machine
learning conferences.

####  NLP dataset

From the ACL Anthology, we extract papers from eight different NLP venues
between 1984 and 2021.888We find by introspection that papers before 1984
often have different formatting and may even lack specifically delineated
abstracts. For the same reason, we exclude papers from CL before 1986. For all
venues, we only include papers from the main conference and exclude papers
from workshops (by manually selecting the volumes) and contributions like book
reviews and title indices (by filtering the titles). To extract the data, we
download the provided metadata from the ACL Anthology website in the BibTeX
format that contains information of authors, title, venue and year. We then
use Allen AI’s Science Parse999https://github.com/allenai/science-parse/ to
extract abstract information and collect author and citation information from
Semantic Scholar.101010https://www.semanticscholar.org/ We refer to this
dataset as NLP in the following. NLP contains more than 23k papers in total.
The distribution over the venues is shown in 3.

Venue | # papers
---|---
ACL | 6,819
EMNLP | 5,013
COLING | 4,653
NAACL | 2,636
SemEval | 2,384
CoNLL | 1,033
CL | 952
TACL | 351
Overall | 23,841
Table 2: NLP dataset.

Venue | # papers
---|---
NeurIPS | 7,537
AAAI | 4,384
ICML | 3,667
ICLR | 2,720
Overall | 18,308
Table 3: ML dataset.

####  ML dataset

We download papers from the respective websites of
NeurIPS111111https://proceedings.neurips.cc/, AAAI121212https://www.aaai.org/,
ICML131313https://icml.cc/, and ICLR141414https://iclr.cc/, and then use
Science Parse to extract abstracts and Semantic Scholar to collect citation
information as above. ML contains over 18k papers between 1989 and 2021. The
distribution over the venues is given in 3.

##  4 Data annotation

We annotate the data from NLP and ML for each paper’s stance towards related
work (as given in the authors’ _framing_ in a paper’s abstract). In contrast
to some related work, we do not annotate stance towards individual citations,
but infer the authors’ stance from the paper abstracts and titles as we are
interested in the stance of the authors’ overall core message. Indeed, in
abstracts, authors typically condense the most important information they
intend to convey. This agrees with the insight that abstracts and titles are
typically the only piece of the paper that the majority of readers consumes
(Andrade, 2011). Our focus on title and abstract is therefore a deliberate
choice.

On a coarse-grained level, we consider three possible stances: (1) a paper may
express a _positive stance_ towards related work, e.g., extend the current
approaches and reach a new state-of-the-art; (2) a paper may express a
_negative stance_ towards related work, e.g., state that datasets, evaluation
protocols or techniques are basically flawed; (3) if a paper does not
particularly fit into this categorization, e.g., because it discusses or
summarizes previous work without criticizing it, then we consider a paper as
expressing a _neutral stance_. In the annotation, we relax this coarse-grained
scheme and instead give continuous numbers for annotation, ranging from -1
(severe negative stance) towards +1 (very positive stance), with 0 as neutral
stance. We provide guidelines for the annotation in the Appendix.

In the following, we describe the annotation process and provide statistics.

###  4.1 Annotation statistics and procedure

In total, we manually annotated 2,170 papers from NLP and ML. The distribution
of paper over venues is given in 4. In this human annotated dataset, the ACL
conference, which takes place annually since 1979, has most papers (353),
followed by NeurIPS (246) and AAAI (218). We also included some workshop
papers (WS) in our human annotated, but remove them in the automatic analysis
later on, except for SemEval.

Venue | # papers
---|---
ACL | 353
NeurIPS | 246
AAAI | 218
NAACL | 207
ICLR | 200
ICML | 200
CL | 190

Venue | # papers
---|---
EMNLP | 180
COLING | 151
WS | 87
CoNLL | 67
SemEval | 50
Other | 21
Overall | 2,170

Table 4: Human annotated dataset.

In our human annotated data, 1,338 papers (61.7%) exhibit _positive stance_
($\geq 0.1$), 389 papers (17.9%) exhibit _negative stance_ ($\leq-0.1$), and
443 papers (20.4%) are _neutral_ ($>-0.1\>\land\><0.1$). We show the more
detailed distribution of papers in terms of stance in 1.

Figure 1: Stance distribution of the human annotated dataset.

This statistic does not reflect the true distribution of stance in our data
(which is dominated by positive papers, as we will show below), as we
oversampled negative papers using heuristics (e.g., looking for particular
keywords in abstracts and titles such as _fail, limitation_). We did this in
order to ensure classifiers trained on datasets that are not too class
imbalanced.

#### Inter-annotator agreement

We had up to 5 annotators annotate abstracts for stances. The annotators were
computer science undergraduate students and one computer science faculty
member from NLP. 71% of the human annotated data set is annotated by one, 22%
by two, 5% by three, and 2% by four annotators. We label each abstract’s
stance as the average over all the annotators.

We measure agreement on stance annotation using Pearson’s Correlation
Coefficient as well as Cohen’s Kappa Coefficient. The resulting correlations
among all pairs of annotators (on a common set of instances) range from 0.67
to 0.95. On a coarse level with three stances, we have an agreement between
0.56 to 0.86 kappa. Overall, we thus have good agreement.

#### Historic vs. modern NLP data

We refer to NLP papers published before the year 2000 as _historic_ papers
(Hist) and papers published since 2000 as _modern_ papers (NLP). The historic
dataset consists of 350 papers, the modern dataset of 1,020 papers.

##  5 Model implementation

For all our experiments, we use SciBERT (Beltagy et al., 2019). We feed each
paper as concatenation of title and abstract separated by special tokens to
SciBERT: [CLS] <title> [SEP] <abstract>. We set the maximum token length to
300, which is sufficient for most papers and a good compromise for efficient
memory usage. We add a fully connected layer with one output neuron and linear
activation on top of the pooled output to obtain a single prediction for the
stance of a paper. Since we define stance as a value between -1 and +1, we
clip the prediction to the desired range.

The model is trained with the following hyperparameters, which we found to
work best in a preliminary experiment by performing a full grid search over
the hyperparameters given in parenthesis and evaluating on the dev set: a
batch size of 16 (8, 16, 32), a _slanted triangular learning rate_ (Howard and
Ruder, 2018) with a maximum learning rate of 5e-5 (5e-6, 1e-5, 2e-5, 5e-5,
8e-5, 1e-4), a warmup ratio of 0.06, and linear decay. We train for 3 epochs
(2, 3, 4, 5) and optimize using Adam (Kingma and Ba, 2015) with an $\epsilon$
of 1e-6, $\beta_{1}$ of 0.9, $\beta_{2}$ of 0.999, and the MSE as the loss
function.

For our experiments in 6, we train 10 models with those hyperparameters and
keep the best model based on the MSE on the dev set. We repeat this 3 times
and calculate performance metrics (cf. 6.1) based on those models.

##  6 Experiments

In the following, we first verify the reliability of our stance detection
model described in 5. To do so, we assess its cross-domain and in-domain
performance and compare it to several baselines. Once the quality of the model
is assured, we apply it large-scale to determine trends over time and venues
in 7.

###  6.1 Experimental setup

#### Metrics

We use two different metrics to evaluate our models. The Coefficient of
Determination (R²) is similar to the Mean Squared Error (MSE) but also takes
the distribution of the data into account which makes it more informative and
truthful than the MSE (Chicco et al., 2021). A model that always predicts the
expected value has an $R^{2}$ score of 0. The range of the metric is
$(-\infty,1]$. The Macro F1 Score (M. F1) is a standard metric to assess the
quality of multi-class classification which can take class-imbalance into
account. We compute the macro F1 score on coarse-grained stance labels
(positive, negative, neutral), see above.

#### Baselines

We compare our models to simple baselines. POS: Always predict a positive
stance +1; ZERO: Always predict a neutral stance 0; NEG: Always predict a
negative stance -1; AVG: Always predict the average of manual annotations.

#### Cross-domain experiments

Due to exponential growth of science, our model is mostly trained on more
recent data. However, we also want to make sure that we obtain reliable
predictions, e.g., for past papers. As a consequence, we first evaluate our
model in a cross-domain setting. In this, we train our model on papers from
different time stamps or domains and evaluate on a respective out-of-domain
test set. For the source data, we set the train-dev split ratio to 0.7 and
0.3, and we use the whole annotated data for the target data.

#### In-domain experiments

We also perform in-domain tests as an upper bound for our cross-domain
evaluation. We set the train-dev-test ratio to 0.6/0.1/0.3.

#### Combined experiments

We create a combined test set which consists of NLP, ML and Hist papers. We
set the train-dev-test ratio to 0.6/0.1/0.3.

#### Data

As mentioned in 4.1, we divided our human-annotated dataset into 3 groups. The
Hist dataset consists of a total of 350 papers, of which 14% negative, 44% are
neutral, and 42% positive. The NLP dataset consists of 1,020 annotated
abstracts and has 21% negative, 17% neutral, 62% positive papers. The ML
dataset consists of 800 annotated samples and has 15% negative, 14% neutral,
71% positive papers.

Figure 2: Different metric scores for combined, in-domain, and cross-domain
evaluation. The x-axis represents the test data and the colored bars represent
different train sets. Cross-domain is the train set which consists of the two
remaining datasets. We choose the best baselines for comparison, which is AVG,
except for M. F1 on Hist where it is ZERO.

###  6.2 Results

Results are shown in 2. We observe a very clear trend: the in-domain
performance is typically better than the cross-domain performance which is
better than the (best) baselines. The model trained on combined data even
outperforms the in-domain results. In-domain and cross-domain performance on
Hist is lowest, which is not surprising as this dataset is smallest in size
and, due to temporal divergence (Lazaridou et al., 2021), assumedly has
largest divergence to the modern datasets. In general, however, cross-domain
performance does not lag too much behind the other experimental conditions,
suggesting that our model is reliable across (our) domains. It is interesting
to note that a model trained on a combination of all time periods performs
best. It even leads to good results on the Hist portion of our data, which is
why we use it for the analysis below.

In 5 (in the appendix), we illustrate sample predictions of our best
performing model, randomly sampled from a predicted stance of 1.0 (very
positive papers) or below -0.8 (very negative papers). We note that the
predicted values look very plausible, except possibly for the very positive
paper from ML, which also contains some negative elements and should probably
be labeled as slightly less positive.

##  7 Analysis

We analyze large-scale trends from the model’s predictions and smooth graphs
with Gaussian blurs.

####  ?Are there more positive or more negative papers??

The histogram of stance values predicted by the model, aggregated over all
venues and years, is visualized in 3a. It shows that most papers have a
positive stance and that the more negative the stance gets, the fewer papers
there are. Less than 4% of all papers have a negative stance and more than 80%
of all papers have a stance of $\geq$0.6.

####  ?Are NLP papers more positive/negative than ML papers??

3b shows the histogram of the predicted stance values, aggregated over all
years, for both datasets. The distribution is similar, but ML has more papers
with stance values between 0.5 and 0.8, whereas NLP has more papers with a
stance of 1.0. Overall, 4.3% of all NLP papers and 2.1% of all ML papers
exhibit a negative stance, which makes NLP more negative than ML.

(a) Overall distribution of stance values.

(b) Distribution of stance values for each domain.

Figure 3: Distribution of stance values. The line plot on the left shows how
the stance values are distributed; the bar plot on the right shows the
percentage of papers with positive and negative stance.

####  ?Did we get more positive/negative recently??

We analyze the development of the _average stance value_ over time in 4. This
shows that the average stance is always positive with a minimum value of 0.29
for NLP in the 1980s. When our ML dataset started in 1989 it was more positive
than NLP. In the late 1990s, the stance of papers in NLP and ML came closer
together, but around 2000 (when NAACL and SemEval were first held) NLP took
over and became slightly more positive than ML. The positiveness reached its
peak around 2015 with an average stance above 0.80 for NLP. It then started to
get more negative for NLP which means that the field of NLP got more negative
recently. The ML domain, however, got more positive with a maximum stance of
0.80 in 2021.

Figure 4: Development of the average stance value over the years for both
domains, average stance and 95% confidence interval.

We further analyze whether the increase in positiveness from 1990 to 2010 and
the decrease in positiveness in the most recent years for NLP comes from more
positive/negative papers overall or from positive/negative papers getting
less/more positive/negative. 5 shows how many papers have a negative stance in
each year. We observe that negativity has peaked in the late 1980s and late
1990s for NLP and ML, respectively. There was then a continuous downward trend
in negativity until the early (ML) and mid (NLP) 2010s. In the recent years,
negativity has increased for both domains, but considerably more sharply so
for NLP, from below 4% of all papers in 2015 to over 6%.

Figure 5: Percentage of papers with a negative stance for both domains over
the years.

6 shows the _average stance value of all positive papers_ and the _average
stance value of all negative papers_ over time. The development of the average
positive stance value is very similar to the development of the average stance
value overall. The average negative stance value has a decreasing trend for
both datasets which means that negative papers have gotten more negative over
time.

Figure 6: Distribution of the average stance value of papers with a positive
(left) and a negative (right) stance over time for both domains, average
stance and 95% confidence interval. The values indicate the average
positiveness/negativeness of positive/negative papers.

Finally, we analyze trend curves for individual venues, visualized in 7. The
trend towards more negative papers in the most recent years is visible for
most venues, especially ACL, EMNLP, COLING, NAACL, CoNLL, ICML and ICLR. TACL
has the sharpest increase. NeurIPS, AAAI, CL, and SemEval do not follow this
trend, however. Many venues were more negative before the 2000s and least
negative in the 2000s. The CL journal is a noticeable outlier: it is the most
negative venue with up to 40% negative papers (we note that it is also the
only journal in our dataset).

Figure 7: Percentage of negative papers for each venue over the years on a
logarithmic scale.

####  ?Do positive/negative papers receive more/fewer citations??

8 shows how many citations a paper with a certain stance value has received in
comparison to papers published in the same year. Positive values indicate more
citations than the average, negative values indicate fewer citations. The
graph shows that papers with a negative stance receive more citations than the
average paper in the same year; very negative papers receive even more
citations; except for the most negative papers with stance values of -1.0. In
contrast, a paper with a positive stance receives less citations on average,
but very positive papers with a stance value of 1.0 receive slightly more
citations.

Figure 8: Normalized number of citations a paper with a certain stance value
has received, average number of normalized citations and 95% confidence
interval. Normalized values indicate how many citations more or less a paper
has received in comparison to the average number of citations a paper
published in the same year has received, measured in multiples of the standard
deviation of citation counts in that year.

Similar results can be found for NLP by analyzing the individual domains
separately, which is shown in 9. The domain of ML is more extreme in that
negative papers receive even more citations and positive papers with stance
values from 0.1 to 0.8 even less.

Figure 9: Normalized number of citations a paper with a certain stance and
domain has received.

####  ?Do positive/negative papers have lower/higher acceptance chances??

We use 2,606 accepted and 6,100 rejected papers from the ICLR conferences in
2013 and 2017–2021, collected from OpenReview, and 3,063 accepted and 9,142
rejected papers from ACL, EMNLP, NAACL, NeurIPS, AAAI, ICML, and ICLR over the
years 2007–2017 as collected by Kang et al. (2018b). 10 shows how many papers
with a certain stance value were accepted. Although the variance is higher,
the trend shows that papers with a negative stance have higher acceptance
rates. The acceptance rate for papers with stance values between -0.6 and 0.8
is lower than the overall acceptance rate, but for very positive papers the
acceptance rate is slightly higher again.

Figure 10: Acceptance rate of a submitted paper with a certain stance value,
average acceptance rate and 95% confidence interval. The dotted line indicates
the overall acceptance rate of 26.7%.

We also calculate the acceptance rates for two separate time spans, 2007–2014
and 2015–2021, as shown in 11. For the most recent years 2015–2021, the trend
is similar to the overall trend with the exception that very positive papers
are not more likely to be accepted than the average paper. However, the
acceptance rates in earlier years 2007–2014 were different. They show that
papers with a negative stance used to be much less likely to be accepted and
that papers with a positive stance used to have better acceptance chances.
This is consistent with the trend over time, which shows that fewer papers
were negative in the years 2007–2014 than in 2015–2021, implying a bias:
positive papers were more popular back then and therefore more positive papers
got accepted.

Figure 11: Normalized acceptance rate of a submitted paper with a certain
stance value for two time spans, average acceptance rate and 95% confidence
interval. Normalized values indicate how many percentage points more or less a
paper with a certain stance value is likely to be accepted in comparison to
the average acceptance rate in each time span. The dotted line indicates the
average acceptance rate.

##  8 Concluding remarks

We analyzed stance in abstracts of scientific publications, where authors
position themselves positively or negatively with respect to related work. We
annotated over 2k abstracts from ML and NLP venues and trained a SciBERT model
on a subset of the annotated abstracts, verifying that the model is of
sufficiently high quality for the task. We then used this model to
automatically predict the stance of a paper based on its title and abstract.
We applied the model large-scale to a collection of 41k scientific
publications in the domain of NLP and ML from the years 1984 to 2021 to enable
large-scale analysis.

The analysis revealed that the majority of papers in the past and today have a
positive stance, that the average stance has substantially increased over
time, yielding support for the hypothesis that ML and NLP have become “rapid
discovery sciences”, and that the ML domain is more positive than the NLP
domain. Scientific publications used to be more negative in the early days,
then became very positive until they started to get more negative again
recently. Overall, publications got also more extreme over time, which means
that papers with a positive stance became more positive and papers with a
negative stance more negative. We found negative papers to be more influential
than positive ones in terms of citations they receive and, recently, more
likely to be accepted to NLP/ML venues.

Future work should address other scientific disciplines beyond NLP and ML for
a broader scientific trend analysis, examine the correlation between overall
stance of a paper and individual (negative) citations in its related work
sections, annotate word-level rationales for our sentence-level scores, assess
the correlation between stance and socio-demographic factors (gender,
nationality, affiliation, h-index, etc.) and analyze how negative papers may
potentially transform a field.

## References

  * Aghion and Howitt (1990) Philippe Aghion and Peter Howitt. 1990.  A model of growth through creative destruction.
  * Altafini (2012) Claudio Altafini. 2012.  Consensus problems on networks with antagonistic interactions.  _IEEE transactions on automatic control_ , 58(4):935–946.
  * Andrade (2011) Chittaranjan Andrade. 2011.  How to write a good abstract for a scientific paper or conference presentation.  _Indian Journal of Psychiatry_ , 53:172 – 175.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Bordignon (2020) Frederique Bordignon. 2020.  Self-correction of science: a comparative study of negative citations and post-publication peer review.  _Scientometrics_ , 124:1225–1239.
  * Cai et al. (2016) Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016.  Bidirectional recurrent convolutional neural network for relation classification.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 756–765, Berlin, Germany. Association for Computational Linguistics.
  * Chicco et al. (2021) Davide Chicco, Matthijs J Warrens, and Giuseppe Jurman. 2021.  The coefficient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation.  _PeerJ Computer Science_ , 7:e623.
  * Cohan et al. (2019) Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019.  Structural scaffolds for citation intent classification in scientific publications.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Collins (1994) Randall Collins. 1994.  Why the social sciences won’t become high-consensus, rapid-discovery science.  _Sociological Forum_ , 9(2):155–177.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Eger (2016) Steffen Eger. 2016.  Opinion dynamics and wisdom under out-group discrimination.  _Mathematical Social Sciences_ , 80:97–107.
  * Fortunato et al. (2018) Santo Fortunato, Carl T Bergstrom, Katy Börner, James A Evans, Dirk Helbing, Staša Milojević, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. 2018.  Science of science.  _Science_ , 359(6379).
  * Gao et al. (2019) Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019.  Does my rebuttal matter? insights from a major NLP conference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1274–1290, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Guo et al. (2020) Qinghua Guo, Shichao Jin, Min Li, Qiuli Yang, Kexin Xu, Yuanzhen Ju, Jing Zhang, Jing Xuan, Jin Liu, Yanjun Su, et al. 2020.  Application of deep learning in ecological resource research: Theories, methods, and challenges.  _Science China Earth Sciences_ , pages 1–18.
  * Hendler (2008) James Hendler. 2008.  Avoiding another ai winter.  _IEEE Intelligent Systems_ , 23(2):2–4.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Jebari et al. (2021) Chaker Jebari, Enrique Herrera-Viedma, and Manuel Jesus Cobo. 2021.  The use of citation context to detect the evolution of research topics: a large-scale analysis.  _Scientometrics_ , 126(4):2971–2989.
  * Jurgens et al. (2018) David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018\.  Measuring the evolution of a scientific field through citation frames.  _Transactions of the Association for Computational Linguistics_ , 6:391–406.
  * Kang et al. (2018a) Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018a.  A dataset of peer reviews (PeerRead): Collection, insights and NLP applications.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kang et al. (2018b) Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018b.  A dataset of peer reviews (PeerRead): Collection, insights and NLP applications.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25:1097–1105.
  * Lample et al. (2018) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018\.  Unsupervised machine translation using monolingual corpora only.  In _International Conference on Learning Representations_.
  * Lauscher et al. (2021) Anne Lauscher, Brandon Ko, Bailey Kuehl, Sophie Johnson, David Jurgens, Arman Cohan, and Kyle Lo. 2021.  Multicite: Modeling realistic citations requires moving beyond the single-sentence single-label setting.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Letchford et al. (2015) Adrian Letchford, Helen Susannah Moat, and Tobias Preis. 2015.  The advantage of short paper titles.  _Royal Society open science_ , 2(8):150266.
  * Levy et al. (2015) Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015.  Do supervised distributional methods really learn lexical inference relations? In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 970–976, Denver, Colorado. Association for Computational Linguistics.
  * Lipton and Steinhardt (2019) Zachary C. Lipton and Jacob Steinhardt. 2019.  Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research. _Queue_ , 17(1):45–77.
  * Liu et al. (2015) Q. Liu, Shu Wu, and Liang Wang. 2015.  Cot: Contextual operating tensor for context-aware recommender systems.  In _AAAI_.
  * Marie et al. (2021) Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2021.  Scientific credibility of machine translation research: A meta-evaluation of 769 papers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7297–7306, Online. Association for Computational Linguistics.
  * Mlinarić et al. (2017) Ana Mlinarić, Martina Horvat, and Vesna Šupak Smolčić. 2017\.  Dealing with the positive publication bias: Why you should really publish your negative results.  _Biochemia medica_ , 27(3):447–452.
  * Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4658–4664, Florence, Italy. Association for Computational Linguistics.
  * Prabhakaran et al. (2016) Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and Dan Jurafsky. 2016\.  Predicting the rise and fall of scientific topics from trends in their rhetorical framing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1170–1180, Berlin, Germany. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Schlegel et al. (2021) Viktor Schlegel, Goran Nenadic, and Riza Batista-Navarro. 2021.  Semantics altering modifications for evaluating comprehension in machine reading.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(15):13762–13770.
  * Teufel et al. (2006) Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006.  Automatic classification of citation function.  In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_ , EMNLP ’06, page 103–110, USA. Association for Computational Linguistics.
  * Wang et al. (2020) Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020.  ReviewRobot: Explainable paper review generation based on knowledge synthesis.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 384–397, Dublin, Ireland. Association for Computational Linguistics.
  * Wright and Augenstein (2021) Dustin Wright and Isabelle Augenstein. 2021.  CiteWorth: Cite-worthiness detection for improved scientific document understanding.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1796–1807, Online. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2021.  Can we automate scientific reviewing? _CoRR_ , abs/2102.00176.

## Appendix

### Annotation guidelines

We issued the following guidelines to annotators: (1) The stance is a value in
the range from -1 (very negative) to +1 (very positive). (2) Only the title
and abstract of a paper is taken into account.

A contribution has a _positive stance_ and is annotated with a positive number
up to +1 when: (a) it clearly indicates to improve the state-of-the-art by
beating existing standards; (b) it presents novel techniques; (c) it proposes
solutions to problems of previous work; (d) it gives insights to existing
models or methods and explains why they work.

A contribution has a _negative stance_ and is annotated with a negative number
up to -1 when: (a) it clearly criticizes previous work of being wrong; (b) it
presents flaws of existing work, i.e. that an approach is deficient with
respect to some property; (c) it analyzes errors of other methods and explains
why they do not work as expected.

Contributions that have positive and negative parts are annotated with a value
between -1 and +1, taking into account the following: (a) The importance of
individual parts matters, i.e., ‘some problems’ is less negative than ‘fails
to work’. (b) The amount of positive and negative parts matters. (c) The last
sentence of an abstract is usually the most important sentence. If the last
sentence of a contribution is positive, it is more positive than a
contribution with a negative last sentence.

Contributions that fall outside this labeling scheme are _neutral_ and
annotated with a 0. Those include: (a) contributions that explore existing
work without beating other systems or explaining why it works or doesn’t work;
(b) contributions that compare, discuss, study, or summarize existing work
without criticizing it.

Stance |  Abstract (excerpt)
---|---
NLP (1.0) |  ?[…] In this paper, we present a novel model BRCNN to classify the relation of two entities in a sentence. […] We propose a bidirectional architecture to learn relation representations with directional information along the SDP forwards and backwards at the same time […]. Experimental results show that our method outperforms the state-of-the-art approaches […].? (Cai et al., 2016)
ML (1.0) |  ?[…] However, the state-of-the-art context modeling methods treat contexts as other dimensions similar to the dimensions of users and items, and cannot capture the special semantic operation of contexts. […] In this work, we propose Contextual Operating Tensor (COT) model, which represents the common semantic effects of contexts as a contextual operating tensor and represents a context as a latent vector. […] Experimental results show that the proposed COT model yields significant improvements over the competitive compared methods on three typical datasets […].? (Liu et al., 2015)
NLP (-0.83) |  ?Distributional representations of words have been recently used in supervised settings for recognizing lexical inference relations between word pairs, such as hypernymy and entailment. We investigate a collection of these state-of-the-art methods, and show that they do not actually learn a relation between two words. Instead, they learn an independent property of a single word in the pair: whether that word is a ?prototypical hypernym?.? (Levy et al., 2015)
ML (-0.72) |  ?[…] In this paper, we investigate whether state-of-the-art MRC models are able to correctly process Semantics Altering Modifications (SAM): […] We present a method to automatically generate and align challenge sets featuring original and altered examples. We further propose a novel evaluation methodology to correctly assess the capability of MRC systems […]. […] We […] find that […] optimised models consistently struggle to correctly process semantically altered data.? (Schlegel et al., 2021)
Table 5: Predicted highly positive and highly negative papers from our
datasets. Blue/red text represents positive/negative rationales; these are
added by us.


which is a relatively new open source Python library for GP modelling. While
this library has not yet been widely used in the astrophysics literature, we
believe111111Note that the development of the tinygp library is led by D.
Foreman-Mackey, an author of this review. that it is a good choice for many
applications. tinygp uses the JAX121212https://jax.readthedocs.io library as
its computational backend, giving it good computational performance (including
support for GPU hardware) with a pure-Python user interface. JAX also provides
support for automatic differentiation (e.g., Margossian, 2019), which can be
used to accelerate all the inference schemes described in Sections 5.1 and
5.2.

Perhaps the most widely used Python library for GP inference in astrophysics
is george131313https://george.readthedocs.io (Ambikasaran et al., 2015), which
is a relatively simple library that implements most of the standard kernel
functions described in Section 3.1, and an approximate solver that can be used
to scale to moderately large ($N\sim 1000$) datasets. More recently, the
scalable methods implemented by celerite141414https://github.com/dfm/celerite
(Foreman-Mackey et al., 2017) and
S+LEAF151515https://obswww.unige.ch/~delisle/spleaf/doc (Delisle et al., 2020,
2022) have become popular for time domain GP models in astrophysics. These
libraries require one-dimensional—or appropriately structured (Gordon et al.,
2020; Delisle et al., 2022)—input data, and a specific set of kernel
functions, although these kernels form a complete (Loper et al., 2021) and
physically interpretable (Foreman-Mackey et al., 2017) basis, and it has
recently been demonstrated that the Matérn class of kernels can be exactly
represented (Jordán et al., 2021).

Outside of astrophysics, the GPy161616https://github.com/SheffieldML/GPy,
GPyTorch171717https://gpytorch.ai (Gardner et al., 2018), and
GPFlow181818https://www.gpflow.org (Matthews et al., 2017; van der Wilk et
al., 2020) projects provide mature and widely used GP implementations.
GPyTorch, in particular, is designed for scaling GP modelling to large
datasets, using approximate methods (e.g., Wilson & Nickisch, 2015; Wilson et
al., 2015) that do not have the same restrictions as celerite and S+LEAF.
GPyTorch has been used for a variety of applications in astrophysics (e.g.,
Dharmawardena et al., 2022; Li et al., 2022), but not yet as widely as the
other libraries discussed so far, perhaps because it does not integrate into
the astronomical workflow as easily as the projects that were designed with
astrophysicists in mind.

##  10 CONCLUSIONS

Gaussian Process Regression has become a key element in the toolkit of modern
astronomers, particularly those working with time-domain datasets. It offers a
principled approach to modelling stochastic signals and nuisances,
particularly correlated noise, whose importance has grown significantly in the
past decade with the development of major precision time-domain monitoring
surveys (e.g. planetary transit surveys). GPR is explicitly rooted in
probability theory, making it readily interpretable and enabling astronomers
to quantify their confidence (or lack thereof) in the results of their
analysis. Despite the simplicity of the underlying mathematical framework,
even basic GP models are extremely flexible, and a small handful of commonly
used “building blocks” have been used successfully to explain the observed
variability of astronomical objects on all scales. On the other hand, this
flexibility comes at a price: a “good fit” with a particular model is no
longer a fail-safe indication that this particular model is “the correct one”,
so users must proceed with extra caution.

###  10.1 Summary of the review

After a very brief historical introduction, we started this review by
introducing two simulated examples, which demonstrate the two main classes of
use for GPR in astronomy, namely to account for a nuisance signal (in the case
of a planetary transit) or to model a stochastic signal of interest (in the
quasar time-delay example). The source code for these and other worked
examples in the review is available alongside this review in the form of
Jupyter notebooks that can be accessed by clicking on the icon next to each
figure caption, and interested users are encouraged to use these to further
their understanding of the methodology and potentially adapt the code to their
own datasets. Next, we gave a succinct introduction to the key equations
underlying GPR regression, and sketched out a typical workflow for applying
GPR to a new dataset. Like any data analysis method, GPR requires the user to
make a number of choices, the most important of which is the choice of
covariance function. We listed some of the most widely used covariance
functions, outlined how to construct more complex ones, and gave practical
advice on how to assess the validity of one’s modelling choices. These
strategies include standard Bayesian model comparison but also cross-
validation, which can be more appropriate in cases where predictive power is
important.

We then reviewed the growing literature on applications of GPR to time-domain
datasets, from exoplanet detection and characterisation, where GPs are mostly
used to account for instrumental or stellar nuisance signals, to the
characterisation of stellar variability itself, where the GP is now used to
model a stochastic signal for its own sake. These examples are drawn from the
exoplanet and stellar astrophysics literature, which we are most familiar
with, but we also attempted to give an overview of recent applications of GPR
to two other classes of astronomical objects where it is increasingly
important, namely AGN and compact objects. Altogether, these applications
include a very diverse array of GP models with varying levels of complexity,
including multidimensional inputs and outputs, derivative and integral
observations, and applications where the GP model is physically motivated
versus cases where it is merely an effective model. We noted how GPR can be
used to tease apart the deterministic and stochastic components of a model,
but also different components of a stochastic process.

We discussed the challenges and pitfalls of GPR, noting recent algorithmic and
technical developments that have brought down their computational cost, once
prohibitive for large datasets, and outlining some strategies to limit the
risk of overfitting. Nonetheless, the size of the dataset remains an important
consideration when deciding whether or not to use GPR for a particular
problem. If too small, learning the kernel hyperparameters and flagging
outliers can be difficult, while computing time can still be problematic for
certain types of kernels and/or computations on very large datasets. Finally,
we reviewed the ecosystem of free software resources for GPR, which make it
easy to test a GP model for a new problem in just a few lines of code, and to
implement a computationally efficient version suitable for a large dataset
with minimal additional effort.

###  10.2 Future perspectives

In certain application domains, such as the analysis of photometric and radial
velocity time-series from exoplanet surveys, GPR is already used fairly
widely. Its usefulness as an effective model for astrophysical or instrumental
nuisance signals is well established, and it is seen as a fairly standard
element of the astronomer’s data analysis toolkit. In these existing
applications, we foresee three main routes for further progress in the near
future. One is the development of more physically motivated (as opposed to
merely effective) GP models for stellar variability, so that GPR can be used
to characterise and understand these stellar signals for their own sake. The
other is a more systematic use of GPR as part of the de-trending and detection
pipelines for exoplanet transit and RV surveys. Computational cost
considerations would have made this impractical just a few years ago, but this
obstacle has been lifted thanks to the recent development of scalable GP
models. Finally, as JWST begins to deliver much exoplanet atmosphere
observations with much higher signal-to-noise (including, importantly, more
precise and relevant housekeeping parameters), a new generation of systematics
models will no doubt be developed to get the most out of these observations,
and it seems likely that GPR would have a role to play in that.

In other applications, such as pulsars, QPOs and astrophysical transients, GPR
remains more of a niche pursuit so far, but promising early results should
motivate further work in the next few years. The next decade will see a vast
increase in the scale of the relevant datasets, with major surveys such as the
Vera Rubin Observatory’s Legacy Survey of Space and Time (LSST) or the Square
Kilometre Array (SKA) routinely delivering tens of TB per night. Even with the
recent development of scalable GP models, it is unclear whether GPR cam be
useful as a “first contact” analysis tool for these very large surveys, but it
has clear potential for the analysis of specific subsets of data.

One area of active development concerns the problem of non-Gaussianity. While
the assumption of Gaussianity implicit to GPR is often good enough in
practice, it is almost never strictly correct, and this can become problematic
when extreme precision is required on the model hyperparameters, or when the
data contain (a relatively small proportion of) significant outliers.
Stochastic process models based on distributions with higher kurtosis than the
Gaussian, for example Student’s-$t$ processes are an interesting alternative,
though they have not been extensively used in astronomy.

Another area where GPR could become more widely used in future is for
intelligent observation planning: if a set of variable objects is being
monitored as part of a survey, using a GP to model existing observations would
enable the observer to make predictions (including uncertainties) for future
observations, and therefore to prioritise among the target list in real time
according to some metric of choice.

#### Parting remarks

GPs are an extremely powerful and general data analysis method, which explains
their rapidly growing popularity in a wide range of application domains,
including time-domain astronomy. However, their very flexibility, together
with their built-in property of automatic marginalisation over random
functions, can make them appear arcane, occasionally leading to lukewarm
reception from the wider community. We hope this review will have de-mystified
the subject somewhat, and provided our readers with enough information to make
informed decisions about when and how to use GPs for their datasets.

## DISCLOSURE STATEMENT

The authors are not aware of any affiliations, memberships, funding, or
financial holdings that might be perceived as affecting the objectivity of
this review.

## ACKNOWLEDGMENTS

S. A. wishes to acknowledge support from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(Grant agreement No. 865624), and useful discussions with Steve Roberts, Mike
Osborne, Oscar Barragán, Vinesh Rajpaul, and Aris Karastergiou.

D. F. M. acknowledges useful conversations with Eric Agol, Will Farr, Alex
Gagliano, Tyler Gordon, and Maximiliano Isi.

The authors would like to thank the community members who sent feedback on the
public draft of this review: Eric Agol, Nestor Espinoza, Christopher Kochanek,
Konstantin Malanchev, Maria Pruzhinskaya, Steve Roberts, Fergus Simpson,
Joshua Winn, Dahai Yan, and Ying Zu.

This research has made use of NASA’s Astrophysics Data System Bibliographic
Services.

## References

  * Abbott et al. (2020) Abbott BP, Abbott R, Abbott TD, Abraham S, Acernese F, et al. 2020. Classical and Quantum Gravity 37(5):055002
  * Agol et al. (2020) Agol E, Luger R, Foreman-Mackey D. 2020. AJ 159(3):123
  * Aigrain et al. (2015) Aigrain S, Hodgkin ST, Irwin MJ, Lewis JR, Roberts SJ. 2015. MNRAS 447(3):2880–2893
  * Aigrain et al. (2016) Aigrain S, Parviainen H, Pope BJS. 2016. MNRAS 459(3):2408–2419
  * Aigrain et al. (2012) Aigrain S, Pont F, Zucker S. 2012. MNRAS 419(4):3147–3158
  * Albert (2020) Albert JG. 2020. arXiv e-prints :arXiv:2012.15286
  * Almosallam et al. (2016) Almosallam IA, Jarvis MJ, Roberts SJ. 2016. MNRAS 462(1):726–739
  * Ambikasaran (2015) Ambikasaran S. 2015. Numerical Linear Algebra with Applications 22(6):1102–1114
  * Ambikasaran et al. (2015) Ambikasaran S, Foreman-Mackey D, Greengard L, Hogg DW, O’Neil M. 2015\. IEEE Transactions on Pattern Analysis and Machine Intelligence 38:252
  * Angus et al. (2018) Angus R, Morton T, Aigrain S, Foreman-Mackey D, Rajpaul V. 2018. MNRAS 474(2):2094–2108
  * Antoniadis et al. (2022) Antoniadis J, Arzoumanian Z, Babak S, Bailes M, Bak Nielsen AS, et al. 2022. MNRAS 510(4):4873–4887
  * Bailer-Jones (2012) Bailer-Jones CAL. 2012. A&A 546:A89
  * Barclay et al. (2015) Barclay T, Endl M, Huber D, Foreman-Mackey D, Cochran WD, et al. 2015\. ApJ 800(1):46
  * Barnes et al. (1980) Barnes JA, Tryon PV, Sargent H. H. I. 1980. Sunspot cycle simulation using random noise. In The Ancient Sun: Fossil Record in the Earth, Moon and Meteorites, eds. RO Pepin, JA Eddy, RB Merrill
  * Barragán et al. (2019) Barragán O, Aigrain S, Kubyshkina D, Gandolfi D, Livingston J, et al. 2019. MNRAS 490(1):698–708
  * Barragán et al. (2022) Barragán O, Aigrain S, Rajpaul VM, Zicher N. 2022. MNRAS 509(1):866–883
  * Barros et al. (2020) Barros SCC, Demangeon O, Díaz RF, Cabrera J, Santos NC, et al. 2020\. A&A 634:A75
  * Blandford & Narayan (1992) Blandford RD, Narayan R. 1992. Annual Review of Astronomy and Astrophysics 30(1):311–358
  * Boone (2019) Boone K. 2019. AJ 158(6):257
  * Bradbury et al. (2018) Bradbury J, Frostig R, Hawkins P, Johnson MJ, Leary C, et al. 2018. JAX: composable transformations of Python+NumPy programs
  * Brook et al. (2019) Brook PR, Karastergiou A, Johnston S. 2019. MNRAS 488(4):5702–5712
  * Brook et al. (2016) Brook PR, Karastergiou A, Johnston S, Kerr M, Shannon RM, Roberts SJ. 2016. MNRAS 456(2):1374–1393
  * Camacho et al. (2022) Camacho JD, Faria JP, Viana PTP. 2022. arXiv e-prints :arXiv:2205.06627
  * Carter & Winn (2009) Carter JA, Winn JN. 2009. ApJ 704(1):51–67
  * Champion et al. (2020) Champion D, Cognard I, Cruces M, Desvignes G, Jankowski F, et al. 2020\. MNRAS 498(4):6044–6056
  * Charbonneau et al. (2000) Charbonneau D, Brown TM, Latham DW, Mayor M. 2000. ApJ 529(1):L45–L48
  * Conroy et al. (2020) Conroy KE, Kochoska A, Hey D, Pablo H, Hambleton KM, et al. 2020. ApJS 250(2):34
  * Constable & Parker (1988) Constable CG, Parker RL. 1988. J. Geophys. Res. 93(B10):11569–11581
  * Crossfield et al. (2016) Crossfield IJM, Ciardi DR, Petigura EA, Sinukoff E, Schlieder JE, et al. 2016. ApJS 226(1):7
  * Cunningham et al. (2008) Cunningham JP, Shenoy KV, Sahani M. 2008. Fast Gaussian process methods for point process intensity estimation. In Proceedings of the 25th international conference on Machine learning
  * Czekala et al. (2015) Czekala I, Andrews SM, Mandel KS, Hogg DW, Green GM. 2015. ApJ 812(2):128
  * David et al. (2019) David TJ, Petigura EA, Luger R, Foreman-Mackey D, Livingston JH, et al. 2019. ApJ 885(1):L12
  * Delisle et al. (2020) Delisle JB, Hara N, Ségransan D. 2020. A&A 638:A95
  * Delisle et al. (2022) Delisle JB, Unger N, Hara NC, Ségransan D. 2022. A&A 659:A182
  * Deming et al. (2015) Deming D, Knutson H, Kammer J, Fulton BJ, Ingalls J, et al. 2015. ApJ 805(2):132
  * Dharmawardena et al. (2022) Dharmawardena TE, Bailer-Jones CAL, Fouesneau M, Foreman-Mackey D. 2022\. A&A 658:A166
  * Dumusque et al. (2012) Dumusque X, Pepe F, Lovis C, Ségransan D, Sahlmann J, et al. 2012\. Nature 491(7423):207–211
  * Dvorak & Edelman (1976) Dvorak R, Edelman C. 1976. Mitteilungen der Astronomischen Gesellschaft Hamburg 38:192
  * Evans et al. (2015) Evans TM, Aigrain S, Gibson N, Barstow JK, Amundsen DS, et al. 2015. MNRAS 451(1):680–694
  * Evans et al. (2013) Evans TM, Pont F, Sing DK, Aigrain S, Barstow JK, et al. 2013. ApJ 772(2):L16
  * Evans et al. (2018) Evans TM, Sing DK, Goyal JM, Nikolov N, Marley MS, et al. 2018. AJ 156(6):283
  * Evans et al. (2017) Evans TM, Sing DK, Kataria T, Goyal J, Nikolov N, et al. 2017. Nature 548(7665):58–61
  * Farr et al. (2018) Farr WM, Pope BJS, Davies GR, North TSH, White TR, et al. 2018. ApJ 865(2):L20
  * Foreman-Mackey et al. (2017) Foreman-Mackey D, Agol E, Ambikasaran S, Angus R. 2017. AJ 154(6):220
  * Foreman-Mackey et al. (2015) Foreman-Mackey D, Montet BT, Hogg DW, Morton TD, Wang D, Schölkopf B. 2015. ApJ 806(2):215
  * Foreman-Mackey et al. (2022) Foreman-Mackey D, Yadav S, Tronsgaard R, Schmerler S, Rashid T. 2022. dfm/tinygp: tinygp v0.2.2
  * Gabry et al. (2019) Gabry J, Simpson D, Vehtari A, Betancourt M, Gelman A. 2019. Journal of the Royal Statistical Society: Series A (Statistics in Society) 182(2):389–402
  * Gardner et al. (2018) Gardner J, Pleiss G, Weinberger KQ, Bindel D, Wilson AG. 2018. Advances in Neural Information Processing Systems 31
  * Gelfand et al. (1992) Gelfand AE, Dey DK, Chang H. 1992. Model determination using predictive distributions with implementation via sampling-based methods. Tech. rep., Stanford Univ CA Dept of Statistics
  * Gelman et al. (1995) Gelman A, Carlin JB, Stern HS, Rubin DB. 1995. Bayesian data analysis. Chapman and Hall/CRC
  * Gelman et al. (1996) Gelman A, Meng XL, Stern H. 1996. Statistica sinica :733–760
  * Gibson (2014) Gibson NP. 2014. MNRAS 445(4):3401–3414
  * Gibson et al. (2012) Gibson NP, Aigrain S, Roberts S, Evans TM, Osborne M, Pont F. 2012. MNRAS 419(3):2683–2694
  * Gibson et al. (2011) Gibson NP, Pont F, Aigrain S. 2011. MNRAS 411(4):2199–2213
  * Gillen et al. (2020) Gillen E, Briegal JT, Hodgkin ST, Foreman-Mackey D, Van Leeuwen F, et al. 2020. MNRAS 492(1):1008–1024
  * Gillon et al. (2007) Gillon M, Pont F, Demory BO, Mallmann F, Mayor M, et al. 2007. A&A 472(2):L13–L16
  * Gordon et al. (2020) Gordon TA, Agol E, Foreman-Mackey D. 2020. AJ 160(5):240
  * Gordon et al. (2021) Gordon TA, Davenport JRA, Angus R, Foreman-Mackey D, Agol E, et al. 2021\. ApJ 913(1):70
  * Grunblatt et al. (2017) Grunblatt SK, Huber D, Gaidos E, Lopez ED, Howard AW, et al. 2017. AJ 154(6):254
  * Haywood et al. (2014) Haywood RD, Collier Cameron A, Queloz D, Barros SCC, Deleuil M, et al. 2014. MNRAS 443(3):2517–2531
  * Henry et al. (2000) Henry GW, Marcy GW, Butler RP, Vogt SS. 2000. ApJ 529(1):L41–L44
  * Hippke et al. (2019) Hippke M, David TJ, Mulders GD, Heller R. 2019. AJ 158(4):143
  * Hoffman & Gelman (2014) Hoffman MD, Gelman A. 2014. Journal of Machine Learning Research 15(47):1593–1623
  * Hogg et al. (2010) Hogg DW, Bovy J, Lang D. 2010. arXiv e-prints :arXiv:1008.4686
  * Hogg & Foreman-Mackey (2018) Hogg DW, Foreman-Mackey D. 2018. ApJS 236(1):11
  * Hogg & Villar (2021) Hogg DW, Villar S. 2021. PASP 133(1027):093001
  * Holman & Murray (2005) Holman MJ, Murray NW. 2005. Science 307(5713):1288–1291
  * Horne (2003) Horne K. 2003. Status aand Prospects of Planetary Transit Searches: Hot Jupiters Galore. In Scientific Frontiers in Research on Extrasolar Planets, eds. D Deming, S Seager, vol. 294 of Astronomical Society of the Pacific Conference Series
  * Hu & Tak (2020) Hu Z, Tak H. 2020. AJ 160(6):265
  * Hübner et al. (2022) Hübner M, Huppenkothen D, Lasky PD, Inglis AR, Ick C, Hogg DW. 2022\. ApJ 936(1):17
  * Ingram & Motta (2019) Ingram A, Motta S. 2019. New Astronomy Reviews 85:101524
  * Isi & Farr (2021) Isi M, Farr WM. 2021. arXiv e-prints :arXiv:2107.05609
  * Jekeli (1991) Jekeli C. 1991. Manuscr. Geod. 16(5):313–325
  * Jordán et al. (2021) Jordán A, Eyheramendy S, Buchner J. 2021. Research Notes of the American Astronomical Society 5(5):107
  * Kasliwal et al. (2017) Kasliwal VP, Vogeley MS, Richards GT. 2017. MNRAS 470(3):3027–3048
  * Keeling & Whorf (2004) Keeling CD, Whorf TP. 2004. Atmospheric CO2 Records from Sites in the SIO Air Sampling Network. In Trends: A Compendium of Data on Global Change. Carbon Dioxide Information Analysis Cente, Oak Ridge National Laboratory, US Department of Energy, Oak Ridge
  * Kelly et al. (2009) Kelly BC, Bechtold J, Siemiginowska A. 2009. ApJ 698(1):895–910
  * Kelly et al. (2014) Kelly BC, Becker AC, Sobolewska M, Siemiginowska A, Uttley P. 2014. ApJ 788(1):33
  * Kostov et al. (2019) Kostov VB, Schlieder JE, Barclay T, Quintana EV, Colón KD, et al. 2019\. AJ 158(1):32
  * Kozłowski et al. (2010) Kozłowski S, Kochanek CS, Udalski A, Wyrzykowski Ł, Soszyński I, et al. 2010. ApJ 708(2):927–945
  * Kreidberg (2018) Kreidberg L. 2018. Exoplanet Atmosphere Measurements from Transmission Spectroscopy and Other Planet Star Combined Light Observations. In Handbook of Exoplanets, eds. HJ Deeg, JA Belmonte. SpringerLink, 100
  * Krige (1951) Krige DG. 1951. A statistical approach to some mine valuations and allied problems at the Witwatersrand. Master’s thesis, University of Witwatersrand
  * Kundić et al. (1995) Kundić T, Colley WN, Gott J. Richard I, Malhotra S, Pen UL, et al. 1995. ApJ 455:L5
  * Kundić et al. (1997) Kundić T, Turner EL, Colley WN, Gott J. Richard I, Rhoads JE, et al. 1997. ApJ 482(1):75–82
  * Li et al. (2022) Li T, Davies GR, Lyttle AJ, Ball WH, Carboneau LM, García RA. 2022\. MNRAS 511(4):5597–5610
  * Lochner et al. (2016) Lochner M, McEwen JD, Peiris HV, Lahav O, Winter MK. 2016. ApJS 225(2):31
  * Loper et al. (2021) Loper J, Blei D, Cunningham JP, Paninski L. 2021. Journal of Machine Learning Research 22(234):1–36
  * Luger et al. (2016) Luger R, Agol E, Kruse E, Barnes R, Becker A, et al. 2016. AJ 152(4):100
  * Luger et al. (2021a) Luger R, Foreman-Mackey D, Hedges C. 2021a. AJ 162(3):124
  * Luger et al. (2021b) Luger R, Foreman-Mackey D, Hedges C, Hogg DW. 2021b. AJ 162(3):123
  * Luger et al. (2018) Luger R, Kruse E, Foreman-Mackey D, Agol E, Saunders N. 2018. AJ 156(3):99
  * MacLeod et al. (2010) MacLeod CL, Ivezić Ž, Kochanek CS, Kozłowski S, Kelly B, et al. 2010. ApJ 721(2):1014–1033
  * MacLeod et al. (2012) MacLeod CL, Ivezić Ž, Sesar B, de Vries W, Kochanek CS, et al. 2012. ApJ 753(2):106
  * Mandel & Agol (2002) Mandel K, Agol E. 2002. ApJ 580(2):L171–L175
  * Margossian (2019) Margossian CC. 2019. Wiley interdisciplinary reviews: data mining and knowledge discovery 9(4):e1305
  * Matthews et al. (2017) Matthews AGdG, van der Wilk M, Nickson T, Fujii K, Boukouvalas A, et al. 2017\. Journal of Machine Learning Research 18(40):1–6
  * Meunier et al. (2010) Meunier N, Desort M, Lagrange AM. 2010. A&A 512:A39
  * Meyer et al. (2022) Meyer AD, van Dyk DA, Tak H, Siemiginowska A. 2022. arXiv e-prints :arXiv:2207.09327
  * Miller et al. (2022) Miller AC, Anderson L, Leistedt B, Cunningham JP, Hogg DW, Blei DM. 2022\. arXiv e-prints :arXiv:2202.06797
  * Moore et al. (2016) Moore CJ, Berry CPL, Chua AJK, Gair JR. 2016. Phys. Rev. D 93(6):064001
  * Moreno et al. (2019) Moreno J, Vogeley MS, Richards GT, Yu W. 2019. PASP 131(1000):063001
  * Newton et al. (2022) Newton ER, Rampalli R, Kraus AL, Mann AW, Curtis JL, et al. 2022. AJ 164(3):115
  * Nicholson & Aigrain (2022) Nicholson BA, Aigrain S. 2022. MNRAS
  * Nocedal & Wright (1999) Nocedal J, Wright SJ. 1999. Numerical Optimization. Springer
  * Parthasarathy et al. (2019) Parthasarathy A, Shannon RM, Johnston S, Lentati L, Bailes M, et al. 2019\. MNRAS 489(3):3810–3826
  * Peebles (1997) Peebles PJE. 1997. ApJ 483(1):L1–L4
  * Phan et al. (2019) Phan D, Pradhan N, Jankowiak M. 2019. arXiv e-prints :arXiv:1912.11554
  * Pont et al. (2006) Pont F, Zucker S, Queloz D. 2006. MNRAS 373(1):231–242
  * Posselt et al. (2021) Posselt B, Karastergiou A, Johnston S, Parthasarathy A, Keith MJ, et al. 2021. MNRAS 508(3):4249–4268
  * Press (1997) Press WH. 1997. Understanding data better with Bayesian and global statistical methods. In Unsolved Problems in Astrophysics
  * Press et al. (1992a) Press WH, Rybicki GB, Hewitt JN. 1992a. ApJ 385:404
  * Press et al. (1992b) Press WH, Rybicki GB, Hewitt JN. 1992b. ApJ 385:416
  * Pruzhinskaya et al. (2019) Pruzhinskaya MV, Malanchev KL, Kornilov MV, Ishida EEO, Mondon F, et al. 2019. MNRAS 489(3):3591–3608
  * Quinonero-Candela & Rasmussen (2005) Quinonero-Candela J, Rasmussen CE. 2005. The Journal of Machine Learning Research 6:1939–1959
  * Rajpaul et al. (2015) Rajpaul V, Aigrain S, Osborne MA, Reece S, Roberts S. 2015. MNRAS 452(3):2269–2291
  * Rajpaul et al. (2016) Rajpaul V, Aigrain S, Roberts S. 2016. MNRAS 456(1):L6–L10
  * Rasmussen & Williams (2006) Rasmussen CE, Williams CKI. 2006. Gaussian Processes for Machine Learning. Cambridge, MA: The MIT Press
  * Rauer et al. (2014) Rauer H, Catala C, Aerts C, Appourchaux T, Benz W, et al. 2014. Experimental Astronomy 38(1-2):249–330
  * Romano & Cornish (2017) Romano JD, Cornish NJ. 2017. Living Reviews in Relativity 20(1):2
  * Rybicki & Press (1995) Rybicki GB, Press WH. 1995. Phys. Rev. Lett. 74(7):1060–1063
  * Särkkä & García-Fernández (2020) Särkkä S, García-Fernández ÁF. 2020. IEEE Transactions on Automatic Control 66(1):299–306
  * Särkkä & Solin (2019) Särkkä S, Solin A. 2019. Applied stochastic differential equations. vol. 10. Cambridge University Press
  * Seager & Mallén-Ornelas (2003) Seager S, Mallén-Ornelas G. 2003. ApJ 585(2):1038–1055
  * Seager & Sasselov (2000) Seager S, Sasselov DD. 2000. ApJ 537(2):916–921
  * Shah et al. (2014) Shah A, Wilson A, Ghahramani Z. 2014. Student-t processes as alternatives to Gaussian processes. In Artificial intelligence and statistics. PMLR
  * Shaw et al. (2022) Shaw B, Stappers BW, Weltevrede P, Brook PR, Karastergiou A, et al. 2022\. MNRAS 513(4):5861–5880
  * Skilling (2006) Skilling J. 2006. Bayesian Anal. 1(4):833–859
  * Sliski & Kipping (2014) Sliski DH, Kipping DM. 2014. ApJ 788(2):148
  * Stone et al. (2022) Stone Z, Shen Y, Burke CJ, Chen YC, Yang Q, et al. 2022. MNRAS 514(1):164–184
  * Talbot & Thrane (2020) Talbot C, Thrane E. 2020. Physical Review Research 2(4):043298
  * Tracey & Wolpert (2018) Tracey BD, Wolpert D. 2018. Upgrading from Gaussian Processes to Student’s-T Processes. In 2018 AIAA Non-Deterministic Approaches Conference. American Institute of Aeronautics and Astronautics
  * Trotta (2008) Trotta R. 2008. Contemporary Physics 49(2):71–104
  * Udalski et al. (2002) Udalski A, Paczynski B, Zebrun K, Szymanski M, Kubiak M, et al. 2002. Acta Astron. 52:1–37
  * Uhlenbeck & Ornstein (1930) Uhlenbeck GE, Ornstein LS. 1930. Physical Review 36(5):823–841
  * Unser (1984) Unser M. 1984. Signal Processing 7(3):231–249
  * van der Wilk et al. (2020) van der Wilk M, Dutordoir V, John S, Artemev A, Adam V, Hensman J. 2020. arXiv:2003.01115
  * van Haasteren & Vallisneri (2014) van Haasteren R, Vallisneri M. 2014. Phys. Rev. D 90(10):104012
  * Vanderburg & Johnson (2014) Vanderburg A, Johnson JA. 2014. PASP 126(944):948
  * Vanderriest et al. (1989) Vanderriest C, Schneider J, Herpe G, Chevreton M, Moles M, Wlerick G. 1989. A&A 215:1–13
  * Vanhatalo et al. (2009) Vanhatalo J, Jylänki P, Vehtari A. 2009. Gaussian process regression with Student-t likelihood. In Advances in Neural Information Processing Systems, eds. Y Bengio, D Schuurmans, J Lafferty, C Williams, A Culotta, vol. 22. Curran Associates, Inc.
  * Villar et al. (2021) Villar VA, Cranmer M, Berger E, Contardo G, Ho S, et al. 2021. ApJS 255(2):24
  * Villar et al. (2020) Villar VA, Hosseinzadeh G, Berger E, Ntampaka M, Jones DO, et al. 2020\. ApJ 905(2):94
  * Virtanen et al. (2020) Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, et al. 2020. Nature Methods 17:261–272
  * von der Heide (1978) von der Heide K. 1978. A&A 70(6):777–784
  * Wakeford et al. (2016) Wakeford HR, Sing DK, Evans T, Deming D, Mandell A. 2016. ApJ 819(1):10
  * Way & Srivastava (2006) Way MJ, Srivastava AN. 2006. ApJ 647(1):102–115
  * Wilson & Nickisch (2015) Wilson A, Nickisch H. 2015. Kernel interpolation for scalable structured Gaussian processes (KISS-GP). In International conference on machine learning. PMLR
  * Wilson et al. (2015) Wilson AG, Dann C, Nickisch H. 2015. arXiv e-prints :arXiv:1511.01870
  * Yang et al. (2021) Yang S, Yan D, Zhang P, Dai B, Zhang L. 2021. ApJ 907(2):105
  * Yu et al. (2022) Yu W, Richards GT, Vogeley MS, Moreno J, Graham MJ. 2022. ApJ 936(2):132
  * Zhang et al. (2022) Zhang H, Yan D, Zhang L. 2022. ApJ 930(2):157
  * Zicher et al. (2022) Zicher N, Barragán O, Klein B, Aigrain S, Owen JE, et al. 2022. MNRAS 512(2):3060–3078
  * Zu et al. (2011) Zu Y, Kochanek CS, Peterson BM. 2011. ApJ 735(2):80

## RELATED RESOURCES

SA gave a set of lectures on GPR for time-domain astronomy as part of the 2021
_Saas Fee Advanced Course in Astrophysics_ , for which PDF lecture notes are
available at shorturl.at/hrstz, along with Python exercises in the form of
Jupyter notebooks.