# On the Role of Reviewer Expertise in Temporal Review
Helpfulness Prediction

Mir Tafseer Nayeem
University of Alberta
<EMAIL_ADDRESS>
&Davood Rafiei
University of Alberta
<EMAIL_ADDRESS>

###### Abstract

Helpful reviews have been essential for the success of e-commerce services, as
they help customers make quick purchase decisions and benefit the merchants in
their sales. While many reviews are informative, others provide little value
and may contain spam, excessive appraisal, or unexpected biases. With the
large volume of reviews and their uneven quality, the problem of detecting
helpful reviews has drawn much attention lately. Existing methods for
identifying helpful reviews primarily focus on review text and ignore the two
key factors of (1) who post the reviews and (2) when the reviews are posted.
Moreover, the helpfulness votes suffer from scarcity for less popular products
and recently submitted (a.k.a., cold-start) reviews. To address these
challenges, we introduce a dataset and develop a model that integrates the
reviewer’s expertise, derived from the past review history of the reviewers,
and the temporal dynamics of the reviews to automatically assess review
helpfulness. We conduct experiments on our dataset to demonstrate the
effectiveness of incorporating these factors and report improved results
compared to several well-established baselines.

##  1 Introduction

Figure 1: A snapshot of three reviews with the reviewers’ history information:
Review a has accumulated more helpful votes but is posted almost two years
before Review b; on the other hand, Review b (a.k.a., cold-start review)
contains time-sensitive information, describing the current conditions and
Review c is likely a spam review. Photos of the reviewers are replaced with
avatars for privacy reasons.

Many customers rely on online reviews from non-professionals, on daily basis,
to decide what products to buy (e.g., Amazon), what hotels to stay at (e.g.,
TripAdvisor), what restaurants to eat (e.g., Yelp) and even what books to read
(e.g., Goodreads). A recent survey of Bizrate Insights reward members found
that approximately _98%_ of online shoppers research a vendor via online
reviews before making a purchase decision Kats (2018). Since the reviews are
expected to describe the actual experiences and opinions of users, they can
provide a reliable source of reference, improving other customers’ confidence,
comfort, and the overall shopping experience (Foo et al., 2017; Gamzu et al.,
2021). However, despite their tremendous benefits, online reviews are often of
mixed qualities. While many reviews are informative, others provide little
value and may contain excessive appraisal or spam (see Figure 1-c). There are
multiple factors that affect the quality of a review, including the reviewers’
life experience, educational background, and the motive for writing the review
(Du et al., 2021), and these factors are not usually explicit in the review
text. All these pose challenges for customers who are less experienced in a
subject area and need the reviews the most, simply because there is less
incentive for more experienced users to use the reviews. Moreover, customers
usually have limited patience for reading reviews – most customers read less
than $10$ reviews before making a purchase decision about an item (Murphy,
2016). The large volume of reviews and their unpredictable quality and the
limited customer patience demand better review utilization strategies to
manage the information overload.

One standard method to identify more informative reviews is to ask for
feedback from customers or site visitors who read them. By asking, _“Was this
review helpful to you?,”_ or _“Did you find this review helpful?”_ at the end
of each review, online platforms can crowdsource helpfulness votes from other
customers. As a result, user reviews that gain the most helpful votes are
shown first to the potential buyers to make the decision easier. However, the
voting data suffers from scarcity (Siersdorfer et al., 2010) since only a tiny
proportion of customers are willing to cast helpfulness votes. The scarcity is
even more severe in reviews of less popular products and more recently
submitted reviews (a.k.a., cold-start reviews) (Liu et al., 2008), despite the
fact that more recent reviews may in fact contain more relevant and time-
sensitive information (e.g., _"New COVID Restrictions"_ or _"Dirty Pool
Area"_) as shown in Figure 1-b but no helpfulness vote.

In this paper, we study the confluence of the reviewing history of reviewers
and the review text for helpfulness identification. First, we observe that
people who post more reviews and earn more helpful votes are more likely to be
better reviewers. Second, trustworthy reviewers (e.g., Figure 1-a) are less
likely to be posting fake or biased reviews, and their reviews are more likely
to earn more helpful votes; otherwise, they will be ruining their reputation.
Third, those who have been to more hotels or restaurants across different
cities have a better basis for comparison and writing critical reviews. To the
best of our knowledge, existing works only focus on review content and neglect
the reviewers and their reviewing history. Integrating the review text with
the reviewing history of the reviewers is the problem studied in this paper.

Our main contributions are summarized as follows:

  * •

We introduce a new dataset with both review text and reviewer’s history, to
highlight the importance of integrating the two sources for review
helpfulness.

  * •

We propose a model incorporating the reviewer’s expertise and temporal
information of reviews in helpfulness prediction.

  * •

We present a detailed case-study to interpret the model behavior and highlight
potential directions to be addressed in the future.

##  2 Related work

More traditional approaches on review helpfulness prediction focus solely on
the text of reviews, and some consider both text and images to guide the
prediction. In general, the task can be addressed using a predictive model
based on hand-crafted features such as structural (Susan and David, 2010;
Xiong and Litman, 2014), lexical (Kim et al., 2006; Xiong and Litman, 2011),
syntactic (Kim et al., 2006), emotional (Martin and Pu, 2014), semantic (Yang
et al., 2015), and arguments (Liu et al., 2017) from the review text. These
features may be fed into a conventional classifier such as SVM, Random Forest,
or gradient boosting to identify helpful reviews. These methods heavily rely
on manual feature engineering, which is labor-intensive and time-consuming.

Inspired by the remarkable progress of deep neural networks, more recent
studies make use of deep neural models, which can learn both intrinsic and
extrinsic features given labeled data. Chen et al. (2018) uses a text-based
CNN model to automatically capture the character-level, word-level, and topic-
level features for helpfulness prediction. Fan et al. (2018) uses an end-to-
end multi-task neural architecture with the help of an auxiliary task, such as
rating regression, to boost the performance of the review helpfulness
identification. Liu et al. (2021) and Han et al. (2022) use both text and
images to guide the review helpfulness prediction. Since the image field is
usually optional in reviews, a large volume of reviews contain only text, for
which these multimodal models would produce inconsistent results.

##  3 Review Helpfulness Prediction

###  3.1 Dataset

To the best of our knowledge, there is no human-annotated dataset that is
publicly available for the task of review helpfulness prediction with the
reviewers’ attributes and review date. Therefore, we build our dataset by
scraping reviews from TripAdvisor111https://www.tripadvisor.com. Out of
$225664$ reviews retrieved, close to one third have no helpful votes. We
filter such reviews, and this reduces the number of reviews to $161541$. Table
1 presents the summary of our dataset with train, validation, and test
splits222We present our dataset construction details in Section A of the
Appendix.. Following (Liu et al., 2021), we leverage a logarithmic scale
($\lfloor{\log_{2}n_{\rm{votes}}}\rfloor$) to categorize the reviews based on
the number of votes received. Specifically, we map the number of votes into
five intervals (i.e., [1,2), [2, 4), [4, 8), [8, 16), [16, $\infty$)), each
corresponding to a helpfulness score $Y$ $\in$ $\\{1,2,3,4,5\\}$, where the
higher the score, the more helpful the review.

| Train | Valid | Test
---|---|---|---
Total #Samples | 145,381 | 8,080 | 8,080
Avg. #Sentences | 7.82 | 7.80 | 7.81
Avg. #Words | 152.37 | 152.25 | 148.90
Table 1: Our dataset statistics.

###  3.2 Proposed Model

Review Helpfulness Prediction (RHP) can be modeled as a supervised machine
learning task where the input contains information about the reviews
($\mathcal{R}$) and the reviewers ($\mathcal{U}$). Let
$\mathcal{R}_{i}=([s_{1},\ldots,s_{N}],\ t_{i})$ denote a review posted at
time $t_{i}$ with sentences $s_{1},\ldots,s_{N}$, and
$\mathcal{U}_{i}=(n_{i},\ m_{i})$ denote a reviewer who posts $n_{i}$ reviews
and earns a total of $m_{i}$ helpful votes. We formulate the review
helpfulness prediction as a multi-class classification where we seek to find a
model $f$ that minimizes the loss function $\mathcal{L}$, i.e.

$min_{\theta}\mathcal{L}\ (f(\theta,\mathcal{R},\mathcal{U}),\ Y),$ (1)

where $Y$ is the ground-truth, $\theta$ is the model parameter and the output
of the model is a helpfulness class $\hat{Y}$ $\in$ $\\{1,2,3,4,5\\}$. The
learning task is to find the best parameter that minimizes the above equation.

We encode the review sentences using BERT Devlin et al. (2019); Xu et al.
(2019). We concatenate the review sentences together while inserting a [CLS]
token at the start and a [SEP] token at the end. If
$\mathbf{h}^{\texttt{[CLS]}}$ denotes the embedding vector of the special
[CLS] token and $\mathbf{h}^{(i)}$ denotes the embedding vector of the $i$-th
token, we extract the last hidden state of $\mathbf{h}_{l}^{\texttt{[CLS]}}$
to represent the review sentences and apply a linear transformation to get a
final contextualized representation $x_{h}\in{\rm I\\!R}^{K}$, where $\Theta$
is a non-linear activation function.

${{}^{(1)},\mathbf{h}^{(2)},\dots,}\begin{multlined}=\textbf{BERT}(\texttt{[CLS]}\\\
\ s_{1},\dots,s_{N}\
\texttt{[SEP]}),\end{multlined}=\textbf{BERT}(\texttt{[CLS]}\\\ \
s_{1},\dots,s_{N}\ \texttt{[SEP]}),$ (2) $x_{h}=\Theta\ (\textbf{MLP}\
(\mathbf{h}_{l}^{\texttt{[CLS]}})).$ (3)

Generally, users who post more reviews and earn more helpful votes are likely
to be better reviewers. Such users may have been to more hotels and
restaurants across the globe and have a better basis for comparison. We define
the term reviewer expertise as the mean number of helpful votes received per
review, written as $e_{s}=m/n$ for a reviewer who posts $m$ reviews and earns
$n$ overall helpfulness votes. We use a linear layer to get a weighted
representation of the expertise score ($h_{s}$).

$h_{s}=\textbf{MLP}\ (e_{s})$ (4)

Previous approaches for this task fail to consider the temporal nature of the
reviews. Older reviews are more likely to accumulate more helpfulness votes
than newer reviews but are not necessarily the most relevant describing the
current conditions (e.g., _new COVID restrictions_). One-time problems such as
broken bathrooms and dirty pool area are likely to be addressed and to be less
relevant. Let $t_{d}$ be the relative age of a review in days, for example, as
of the day the reviews are scraped. We use a linear layer to get a weighted
representation of the relative review age.

$h_{t}=\textbf{MLP}\ (t_{d}).$ (5)

It should be noted that both the review age and the reviewer expertise are
normalized to a fixed range $[a,b]$ before being used in the linear layers in
Equations 4 and 5. If $\mathcal{X}$ denotes a set of scores (e.g., reviewers
expertise score), a score $x_{i}\in\mathcal{X}$ is normalized into $z_{i}$ as
follows:

$z_{i}=(b-a)\frac{x_{i}-min(\mathcal{X})}{max(\mathcal{X})-min(\mathcal{X})}+a$
(6)

In our case, both review age and reviewer expertise are scaled into the
interval $[0,1]$.

We concatenate the textual representation ($x_{h}$), expertise representation
($h_{s}$), and temporal representation ($h_{t}$ ) to get a final embedding

$o_{final}=h_{s}\oplus x_{h}\oplus h_{t},$ (7)

where $\oplus$ is a concatenation operator. The final helpfulness prediction
layer feeds $o_{final}$ into a linear layer and use softmax activation to get
the final predicted helpfulness class $\hat{Y}$.

$\hat{Y}=\textbf{softmax}\ (W_{r}\cdot o_{final}+b_{r}),$ (8)
$\mathcal{L}=\mathcal{L}_{CE}\ (\hat{Y},Y)$ (9)

where $W_{r}\in{\rm I\\!R}^{K\times K}$ and $b_{r}\in{\rm I\\!R}^{K}$ denote
the projection parameter and a bias term respectively. We use the cross-
entropy loss function $\mathcal{L}_{CE}$ with respect to the ground truths
helpfulness class ($Y$).

###  3.3 Experiments

We evaluated the performance of the proposed model333Code, dataset, and model
checkpoints: https://github.com/tafseer-nayeem/RHP compared to well-
established baselines. We compare our system with ARH (Kim et al., 2006), UGR
+ BGR (Xiong and Litman, 2011), TextCNN (Chen et al., 2018), MTNL (Fan et al.,
2018), and BERTHelp (Xu et al., 2020). We didn’t perform any explicit
preprocessing of the review text. We discuss the baseline systems,
preprocessing, and hyperparameters used for our experiments in Appendix
(Section B & Section C).

Baseline Models | Acc. ($\uparrow$) | MAE ($\downarrow$) | MSE ($\downarrow$)
---|---|---|---
ARH | 58.73 | 0.476 | 0.619
UGR + BGR | 62.76 | 0.464 | 0.674
TextCNN | 62.82 | 0.444 | 0.608
MTNL | 62.77 | 0.458 | 0.653
BERTHelp | 63.03 | 0.432 | 0.591
Our Ablations | Acc. ($\uparrow$) | MAE ($\downarrow$) | MSE ($\downarrow$)
RHP (ours) | 65.18† | 0.393† | 0.491†
\- w/o Expertise | 63.87 | 0.421† | 0.550†
\- w/o Temporal | 63.40 | 0.437† | 0.592
\- w/o Expertise + Temporal | 62.92 | 0.446 | 0.617
Table 2:  Performance compared to our baseline models and the result of our
ablation study ($\uparrow$ indicates higher values for a better performance
and $\downarrow$ indicates lower values for a better performance). † reported
results are statistically significant in paired t-test by taking BERTHelp (Xu
et al., 2020) as a reference with the confidence of 95% ($p$-value < 0.05).

####  3.3.1 Results

As part of a detailed evaluation of our algorithm, we report our model’s
performance compared with the baselines in terms of Accuracy (Acc.), Mean
Average Error (MAE), and Mean Squared Error (MSE). As shown in Table 2, our
final model outperforms the baselines in terms of all the metrics. Our ground-
truth values consist of $5$ classes which correspond to five helpfulness
scores $\\{1,2,3,4,5\\}$, where the higher the score, the more helpful the
review. To gain more insights into the performance of our prediction model, we
also evaluate our algorithm in terms of MAE and MSE, which assess the fine-
grained differences between the ground-truth and the predicted helpfulness
scores. Our RHP model consistently outperforms the baselines with a good
margin, which means when misclassified, our model predictions are very close
to the actual helpfulness scores. We conduct detailed ablation studies to
demonstrate the effects of different components of our RHP model by removing
expertise (denoted as w/o Expertise) and removing temporal information
(denoted as w/o Temporal). The ablation test results on our dataset are
summarized in Table 2. We can observe that the temporal feature has the
largest impact on the performance of our model, and the impact of expertise is
also significant. This suggests that the reviewer’s expertise and temporal
information of the reviews play a key role in review helpfulness prediction.
Therefore, it is no surprise that combining all components achieves the best
performance on our proposed dataset.

####  3.3.2 Analysis

We also present a detailed analysis to provide more supportive evidence of our
arguments. To this end, we randomly selected $m$ examples for each class of
reviews considering helpfulness votes. Then, we extract Top $K$ (where $K$ =
$5$) $n$-grams from each class of reviews to identify the most relevant
keywords or topics in reviews to assess what aspects are most talked about the
items (e.g., hotels or restaurants).

##### Preprocessing

Our preprocessing step includes tokenization, lemmatization, removal of
stopwords, Part-Of-Speech (POS) tagging, and filtering punctuation marks. We
use the NLTK444https://www.nltk.org/ to preprocess each sentence and obtain a
more accurate representation of the information. Moreover, we also add ‘hotel’
and ‘restaurant’ in the stopwords list as they frequently occur in every
review and are not meaningful in our context.

##### Extracting Candidate $n$-grams

We remove the sentiment words and emojis using
VADER555https://github.com/cjhutto/vaderSentiment Hutto and Gilbert (2014), a
"gold-standard" sentiment lexicon especially attuned to microblog-like
contexts. As the sentiment expressed in reviews are highly subjective, we are
interested in extracting only the aspects or topics (e.g., room, location,
customer service etc.) for which the opinions are expressed. Therefore, we
keep only the nouns666As adjectives and adverbs may contain sentiment towards
aspects. (with POS tags ‘NN’ and ‘NNS’) for extracting the aspects or topics.

##### Ranking Candidate $n$-grams

We extract the unigrams and bigram collocations for each of the review
classes. Then, we rank the unigrams by counting the frequency of occurrences
and bigrams using likelihood ratios Manning and Schutze (1999) to obtain Top
$K$. We present the Top $5$ unigrams and bigrams in Table 3 grouped according
to helpfulness classes and ordered by descending ranking scores.

Helpfulness Class |  | Unigram |  | Bigram
---|---|---|---|---
|  | ‘room’ |  | ‘front desk’
|  | ‘staff’ |  | ‘coffee maker’
|  | ‘location’ |  | ‘breakfast buffet’
|  | ‘time’ |  | ‘sofa bed’
Class #1 Helpful Votes [1, 2) |  | ‘service’ |  | ‘swim pool’
|  | ‘room’ |  | ‘front desk’
|  | ‘staff’ |  | ‘shampoo conditioner’
|  | ‘service’ |  |  ‘customer service’
|  | ‘location’ |  | ‘resort fee’
Class #2 Helpful Votes [2, 4) |  | ‘time’ |  | ‘pool area’
|  | ‘room’ |  | ‘front desk’
|  | ‘staff’ |  | ‘resort fee’
|  | ‘time’ |  |  ‘customer service’
|  | ‘service’ |  | ‘coffee maker’
Class #3 Helpful Votes [4, 8) |  | ‘view’ |  | ‘city view’
|  | ‘room’ |  | ‘front desk’
|  | ‘staff’ |  | ‘resort fee’
|  | ‘service’ |  |  ‘customer service’
|  | ‘time’ |  | ‘minute walk’
Class #4 Helpful Votes [8, 16) |  | ‘pool’ |  | ‘life jacket’
|  | ‘room’ |  | ‘front desk’
|  | ‘time’ |  | ‘resort fee’
|  | ‘service’ |  | ‘bed bug’
|  | ‘staff’ |  | ‘beach chair’
Class #5 Helpful Votes [16, $\infty$) |  | ‘pool’ |  | ‘cable car’
Table 3: Top $5$ unigrams and bigrams extracted from five different classes of
reviews divided according to helpfulness votes. For each column,  green color
indicates the overlap with all $5$ classes, whereas  blue for $4$,  orange for
$3$, and  red for $2$ overlaps.

Table 3 shows a high overlap of $n$-grams among different classes of reviews,
which further strengthen our argument that helpfulness does not entirely
depend on the review text but rather the confluence of the review text,
reviewing history of reviewers (who post the reviews), review age (when the
reviews are posted). Generally, older reviews (i.e., review age) were present
longer than the newer reviews in the platform and had more time to accumulate
helpful votes.

Figure 2: Top $10$ ranked tokens of the RHP model shown in green colors with
the color intensity indicating the importance of the tokens in the overall
prediction.

###  3.4 Case Study

To gain more insights into the review helpfulness prediction task, we present
a detailed case-study to interpret the model behavior and highlight the most
important features of this task. Models are interpretable when humans can
readily comprehend the reasoning behind model predictions and decisions made
(Kim et al., 2016). To this end, we randomly selected a sample with
Helpfulness Class = $3$ from our test set and used Captum777Captum
(https://captum.ai/) is an open-source, extensible library for model
interpretability that uses the integrated gradients method (Sundararajan et
al., 2017). to interpret the words/tokens that contributed the most to the
prediction. As can be seen in Figure 2, the top-ranked words are highly
representative of the aspects/facilities listed on the restaurant page. We can
conclude from this observation that users tend to look for specific aspects in
reviews to find them helpful. We also notice that the use of personal pronouns
(e.g., I, we, they, etc.), describing personal experiences, contributes to the
helpfulness prediction. People often find reviews useful if it comes from
others’ experiences and personal pronouns are a good indicator of it.

##  4 Conclusion and Future Work

In this paper, we develop a model incorporating the reviewer’s expertise and
temporal information in reviews to predict the helpfulness, especially for
unreliable and cold-start reviews. Furthermore, we present a detailed analysis
to interpret the model behavior and provide reasoning behind model
predictions. For future work, we will look into the problem of personalized
review helpfulness prediction to model the demographics and cultural
differences of the reviewers.

## Limitations

Despite the effectiveness of incorporating the reviewer’s history and temporal
information of the reviews in helpfulness prediction, our current studies
still have several limitations, which can pave the path for future research.

For simplicity, like existing works, we assume that all the users rate reviews
unanimously. However, the diversity of demographics, age, and cultural
background also affect how users give, receive, and understand the sentiments
expressed in reviews. Users may focus on different review aspects based on
their preferences (i.e., _"5 stars, party every night"_ vs _"5 stars, always
quiet and peaceful"_). It would be interesting to see how to incorporate
personal preferences for the helpfulness prediction task.

Another limitation of our work is that we only worked with reviews written in
English. As a result, we filter out the reviews written in other languages and
notice code-switched reviews when the reviewers alternate between two or more
languages in a single review. We aim to extend this work to support more
languages.

## Ethics Statement

In our data scraping process, we took into account ethical considerations. We
obtained data at an appropriate pace, avoiding any potential DDoS attacks.
Additionally, we eliminated any Personal Identifying Information, such as
names, telephone numbers, and email addresses, from the data set.

## Acknowledgements

We thank all the anonymous reviewers for their valuable feedback and
constructive suggestions for improving this work. This research is supported
by the Natural Sciences and Engineering Research Council of Canada (NSERC) and
by a grant from Huawei. Mir Tafseer Nayeem is also supported by a Huawei
Doctoral Scholarship.

## References

  * Chen et al. (2018) Cen Chen, Yinfei Yang, Jun Zhou, Xiaolong Li, and Forrest Sheng Bao. 2018.  Cross-domain review helpfulness prediction based on convolutional neural networks with auxiliary domain discriminators.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 602–607, New Orleans, Louisiana. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du et al. (2021) Jiahua Du, Jia Rong, Hua Wang, and Yanchun Zhang. 2021.  Neighbor-aware review helpfulness prediction.  _Decision Support Systems_ , 148.
  * Fan et al. (2018) Miao Fan, Yue Feng, Mingming Sun, Ping Li, Haifeng Wang, and Jianmin Wang. 2018\.  Multi-task neural learning architecture for end-to-end identification of helpful reviews.  In _Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_ , ASONAM ’18, page 343–350. IEEE Press.
  * Foo et al. (2017) Sheng Khoo Foo, Lee Teh Phoey, and Boon Ooi Pei. 2017.  Consistency of online consumers’ perceptions of posted comments: An analysis of tripadvisor reviews.  _Journal of information and Communication Technology_ , 16(2):374–393.
  * Gamzu et al. (2021) Iftah Gamzu, Hila Gonen, Gilad Kutiel, Ran Levy, and Eugene Agichtein. 2021.  Identifying helpful sentences in product reviews.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 678–691, Online. Association for Computational Linguistics.
  * Han et al. (2022) Wei Han, Hui Chen, Zhen Hai, Soujanya Poria, and Lidong Bing. 2022.  SANCL: Multimodal review helpfulness prediction with selective attention and natural contrastive learning.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 5666–5677, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _Proceedings of the 25th International Conference on World Wide Web_ , WWW ’16, page 507–517, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Hutto and Gilbert (2014) C. Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 8(1):216–225.
  * Kats (2018) Rimma Kats. 2018.  Surprise! most consumers look at reviews before a purchase.  Accessed: May 10, 2022.
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. 2016.  Examples are not enough, learn to criticize! criticism for interpretability.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , NIPS’16, page 2288–2296, Red Hook, NY, USA. Curran Associates Inc.
  * Kim et al. (2006) Soo-Min Kim, Patrick Pantel, Tim Chklovski, and Marco Pennacchiotti. 2006.  Automatically assessing review helpfulness.  In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_ , EMNLP ’06, page 423–430, USA. Association for Computational Linguistics.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Liu et al. (2017) Haijing Liu, Yang Gao, Pin Lv, Mengxue Li, Shiqiang Geng, Minglan Li, and Hao Wang. 2017.  Using argument-based features to predict and analyse review helpfulness.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1358–1363, Copenhagen, Denmark. Association for Computational Linguistics.
  * Liu et al. (2021) Junhao Liu, Zhen Hai, Min Yang, and Lidong Bing. 2021.  Multi-perspective coherent reasoning for helpfulness prediction of multimodal reviews.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5927–5936, Online. Association for Computational Linguistics.
  * Liu et al. (2008) Yang Liu, Xiangji Huang, Aijun An, and Xiaohui Yu. 2008.  Modeling and predicting the helpfulness of online reviews.  In _2008 Eighth IEEE International Conference on Data Mining_ , pages 443–452.
  * Manning and Schutze (1999) Christopher Manning and Hinrich Schutze. 1999.  _Foundations of statistical natural language processing_.  MIT press.
  * Martin and Pu (2014) Lionel Martin and Pearl Pu. 2014.  Prediction of helpful reviews using emotions extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 28.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015\.  Image-based recommendations on styles and substitutes.  In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’15, page 43–52, New York, NY, USA. Association for Computing Machinery.
  * Murphy (2016) Rosie Murphy. 2016.  Local consumer review survey 2016.  Accessed: May 10, 2022.
  * Siersdorfer et al. (2010) Stefan Siersdorfer, Sergiu Chelaru, Wolfgang Nejdl, and Jose San Pedro. 2010.  How useful are your comments? analyzing and predicting youtube comments and comment ratings.  In _Proceedings of the 19th International Conference on World Wide Web_ , WWW ’10, page 891–900, New York, NY, USA. Association for Computing Machinery.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3319–3328. PMLR.
  * Susan and David (2010) M Mudambi Susan and Schoff David. 2010.  What makes a helpful online review? a study of customer reviews on amazon. com.  _MIS Quarterly_ , 34(1):185–200.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Xiong and Litman (2011) Wenting Xiong and Diane Litman. 2011.  Automatically predicting peer-review helpfulness.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 502–507, Portland, Oregon, USA. Association for Computational Linguistics.
  * Xiong and Litman (2014) Wenting Xiong and Diane Litman. 2014.  Empirical analysis of exploiting review helpfulness for extractive summarization of online reviews.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 1985–1995, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2324–2335, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Xu et al. (2020) Shuzhe Xu, Salvador E. Barbosa, and Don Hong. 2020.  Bert feature based model for predicting the helpfulness scores of online customers reviews.  In _Advances in Information and Communication_ , pages 270–281, Cham. Springer International Publishing.
  * Yang et al. (2015) Yinfei Yang, Yaowei Yan, Minghui Qiu, and Forrest Bao. 2015.  Semantic analysis and helpfulness prediction of text for online product reviews.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 38–44.

##  Appendix A Dataset Construction

Publicly available datasets which are mostly used for this task are
Amazon888http://jmcauley.ucsd.edu/data/amazon/index_2014.html He and McAuley
(2016); McAuley et al. (2015) and Yelp999https://www.yelp.com/dataset. In Yelp
dataset, the user votes are distributed among three categories such as
“Useful”, “Funny” or “Cool”, where “Useful” voting feature was introduced much
later than the other two categories. Therefore, many good reviews already in
the dataset may not have been marked useful. On the other hand, the Amazon
dataset does not contain the reviewers’ reviewing history and helpfulness
votes to evaluate our hypothesis studied in this paper. Moreover, for Amazon,
the samples come from various categories such as Books, Electronics, Clothing,
Beauty, Shoes and Jewelry, Grocery, Pet Supplies, etc – the total helpfulness
votes for the reviewers are coming from different categories and it’s not
explicit in the fields from Amazon website. Therefore, it’s hard to devise
expertise because of domain diversity.

We build our dataset by scraping reviews from
TripAdvisor101010https://www.tripadvisor.com, a travel site that offers online
hotel and restaurant reservations and a platform for sharing the travel
experiences of users. We take reviews from January 1st, 2015 until January
1st, 2020, and extract only those written in English. For each review, we
extract the review text, the total helpfulness votes and the posting time, and
for each reviewer, we extract the number of reviews contributed and the
cumulative helpfulness votes. The attributes we extracted are summarized as
follows:

  * •

Reviews

    * –

Review Text

    * –

Total Review Helpful Votes

    * –

Review Posting Time

  * •

Reviewers

    * –

Total Number of Reviews Contributed

    * –

Cumulative Helpful Votes

##  Appendix B Baseline Systems

We compare our system performance with the following baselines.

  * •

ARH (Kim et al., 2006) & UGR + BGR (Xiong and Litman, 2011) use machine
learning-based methods with hand-crafted features such as structural, lexical,
syntactic, emotional, semantic, and meta-data from the review text to address
this task. These features are fed into conventional classifiers such as SVM,
Random Forest, and gradient boosting to identify helpful reviews.

  * •

TextCNN (Chen et al., 2018) employs a text-based CNN model Kim (2014) to
automatically capture the character-level, word-level, and topic-level
features for helpfulness prediction.

  * •

MTNL (Fan et al., 2018) utilizes end-to-end multi-task neural learning (MTNL)
architecture for classifying helpful reviews. They take the help of an
auxiliary task, such as rating regression, to boost the performance of the
original task, which is review helpfulness identification.

  * •

BERTHelp (Xu et al., 2020) develop their helpfulness prediction model using
pre-trained BERT Devlin et al. (2019). They design a regression model using
BERT-based features extracted from review texts, star rating, and product type
information from Amazon product review dataset He and McAuley (2016).

##  Appendix C Preprocessing & Hyperparameters

##### Preprocessing

We didn’t perform any explicit preprocessing of the review text. Instead, we
use BertTokenizer to avoid the out-of-vocabulary (OOV) problem, which uses
WordPiece Wu et al. (2016) for tokenizing the sentences into words or
subwords. In addition, we add special tokens to the start (e.g., [CLS]) and
end of each review text (e.g., [SEP]) and truncate all sentences to a single
constant length (e.g., $512$).

##### Hyperparameters

We use Adam optimizer (Kingma and Ba, 2015) with a learning rate of $3\times
e^{-5}$ and a batch size of $32$. We use $\text{BERT}_{\text{BASE}}$ (Wolf et
al., 2020) pre-trained model with a fixed vocabulary. We run the training for
$5$ epochs and check the improvement of validation (dev set) loss to save the
latest best model during training.


# Fully Bayesian Forecasts with Evidence Networks

T. Gessey-Jones<EMAIL_ADDRESS>Astrophysics Group, Cavendish Laboratory,
J.J. Thomson Avenue, Cambridge, CB3 0HE, UK  Kavli Institute for Cosmology,
Madingley Road, Cambridge, CB3 0HA, UK  W. J. Handley<EMAIL_ADDRESS>Astrophysics Group, Cavendish Laboratory, J.J. Thomson Avenue, Cambridge, CB3
0HE, UK  Kavli Institute for Cosmology, Madingley Road, Cambridge, CB3 0HA, UK

###### Abstract

Sensitivity forecasts inform the design of experiments and the direction of
theoretical efforts. We argue that to arrive at representative results
Bayesian forecasts should marginalize their conclusions over uncertain
parameters and noise realizations rather than picking fiducial values.
However, this is computationally infeasible with current methods. We thus
propose a novel simulation-based forecasting methodology, which we find to be
capable of providing expedient rigorous forecasts without relying on
restrictive assumptions.

Bayesian methods, Deep learning, Cosmology

Introduction.— Forecasts are an essential part of science. Estimates of
required sensitivities guide the design of experiments, and whether these
experiments can probe different theoretical models informs which models time
and resources are spent developing. Furthermore, funding agencies use
anticipated scientific conclusions as part of their decision-making. As a
result, fast, accurate, and reliable forecasting techniques play a vital role
in the modern scientific method.

Due to its importance, many techniques exist to perform forecasts [e.g. 1, 2,
3, 4]. Within a Bayesian paradigm, arguably the most accurate forecasting
technique is to generate mock data and perform Bayesian analysis on the data
as if it were experimental data [e.g. 5, 6]. However, such an analysis is
typically computationally costly, limiting its application to a few mock data
realizations, which may lead to erroneous conclusions if the mock data is not
representative of reality. Hence, Fisher forecasts [1] are commonly employed
across astronomy and astrophysics [7, 8, 2, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18] to cover the potential data space and investigate how conclusions vary.
Unfortunately, the Gaussianity assumption Fisher forecasts rely upon does not
always hold, limiting their reliability [e.g. 19, 20]. Furthermore, Fisher
forecasts can only be used for model comparison forecasts on nested models
(when one of the models is a special case of the other), limiting their
applicability [21].

In this letter, we argue traditional Bayesian forecasting techniques break a
central tenant of Bayesian statistics by drawing conclusions based on a small
number of fiducial parameter values. To remedy this, we propose the
conclusions of traditional Bayesian forecasts should be marginalized over
uncertain parameters and noise realizations in a fully Bayesian forecast.
However, we find current forecasting techniques too slow for such forecasts to
be feasible except for rare specific cases. We thus propose an Evidence
Network [22] based methodology, which allows for expedient fully Bayesian
forecasts on any scientific question formulated as a condition on the Bayes
ratio between two models. Such questions include whether a signal can be
detected from within noise, whether two competing theories can be
distinguished by expected data, or at what sensitivity level will a piece of
additional physics be required within a model? We then demonstrate the
technique by finding the a priori chance of detection of the global 21-cm
signal by a REACH-like experiment [23], an analysis that would have been
computationally impracticable using traditional methods.

Bayesian Forecasting.— A traditional Bayesian analysis [see 24, for a more
detailed discussion] starts with a model $M$ that takes some parameters
$\theta$. From the model and experimental considerations, a likelihood of
observing data $D$ is constructed $\mathcal{L}(D|\theta,M)$. In addition, an
initial measure, the prior $\pi(\theta|M)$, is put over the parameter space to
quantify the a priori knowledge of the parameter values. Then given some
(mock) observed data, the measure over the parameter space is updated via an
application of Bayes’ theorem

$~{}\mathcal{P}(\theta|D,M)=\frac{\mathcal{L}(D|\theta,M)\pi(\theta|M)}{\mathcal{Z}},$
(1)

into the a posteriori knowledge of the parameter values
$\mathcal{P}(\theta|D,M)$, called the posterior. Here $\mathcal{Z}$ is the
Bayesian evidence

$\mathcal{Z}=\int\mathcal{L}(D|\theta,M)\pi(\theta|M)d\theta=P(D|M),$ (2)

which serves both as a normalization constant and as a natural goodness-of-fit
statistic for comparing competing models of the same data. Now suppose there
are competing models, $M_{\rm 0}$ and $M_{\rm 1}$, with corresponding Bayesian
evidences $\mathcal{Z}_{\rm i}$. If we have an a priori belief in each model
of $P(M_{\rm i})$ then for each model the a posteriori belief $P(M_{\rm i}|D)$
in the model is found using a further application of Bayes’ theorem

$~{}P(M_{\rm i}|D)=\frac{P(D|M_{\rm i})P(M_{\rm
i})}{P(D)}=\frac{\mathcal{Z}_{\rm i}P(M_{\rm i})}{P(D)}.$ (3)

Cancelling $P(D)$ between the above with $i=0$ and $i=1$ then gives the
Bayesian Model Comparison equation

$~{}\frac{P(M_{\rm 1}|D)}{P(M_{\rm 0}|D)}=\frac{\mathcal{Z}_{\rm 1}P(M_{\rm
1})}{\mathcal{Z}_{\rm 0}P(M_{\rm 0})}.$ (4)

This equation shows that after observing some data $D$ the relative belief in
the two models is updated with the ratio of their evidences, e.g. the belief
in the higher evidence model increases, and the belief in the lower evidence
model belief decreases. It is common to assume initially that the two models
are equally likely $P(M_{\rm 1})=P(M_{\rm 0})$ in which case equation 4
simplifies to

$~{}\mathcal{K}=\frac{P(M_{\rm 1}|D)}{P(M_{\rm 0}|D)}=\frac{\mathcal{Z}_{\rm
1}}{\mathcal{Z}_{\rm 0}},$ (5)

so that the Bayes ratio $\mathcal{K}$ of the model evidences is simply the
relative belief in model $1$ compared to model $0$ after observing some data.
The above can also be rearranged to find the posterior belief in a model

$~{}P(M_{\rm 1}|D)=\frac{\mathcal{K}}{1+\mathcal{K}},$ (6)

which can be compared to a statistical threshold to determine if one model is
significantly preferred over the other.

With the mathematical notation of Bayesian analysis established, we can return
to the central topic of this letter; forecasts. A traditional Bayesian
forecast for whether a proposed experiment could distinguish between two
models goes through the following steps. First, the models and corresponding
likelihoods are established. Then, mock data is simulated from one of the
models by choosing some fiducial parameter values $\tilde{\theta}$, adding
noise $\eta$, and experimental effects. $\mathcal{Z}$ is evaluated for each
model treating the mock data as you would actual observations from which
$\mathcal{K}$ (or equivalently $P(M_{\rm i}|D)$) is calculated. Finally,
$\mathcal{K}$ is compared to some condition to determine if the favouring of a
model is statistically significant and so the models are distinguishable.

The conclusion of the outlined forecast is hence based on a condition on
$\mathcal{K}$ of the form $\mathcal{K}>\mathcal{K}_{\rm crit}$ with
$\mathcal{K}_{\rm crit}$ a statistical significance threshold. However,
$\mathcal{K}$ is conditional on the fiducial parameters chosen to generate the
mock data and the noise added, so it is more accurate to state

$~{}\mathcal{K}\left(\tilde{\theta},\eta\right)>\mathcal{K}_{\rm crit}.$ (7)

Using such a criterion presents two major issues. Firstly the result is
dependent on random noise $\eta$, leading to differences in conclusion purely
based on chance. Secondly, these conclusions stem from a single point in the
parameter space $\tilde{\theta}$, leading to differences based on the choice
of fiducial model (anathema to Bayesian statistics). A central tenant of
Bayesian statistics being that particular sets of parameter values, even those
that maximize the likelihood or posterior, have zero measure and thus are
vanishingly unlikely and should not be used to draw conclusions. Instead, in
Bayesian statistics, conclusions should be reached by marginalizing over the
parameter space weighted by some measure, typically the prior or posterior.
This issue is further compounded in the case of forecasts as there is often
limited advance knowledge of the parameter values leading to the conclusion
drawn potentially varying widely between choices of $\tilde{\theta}$.

A rigorously Bayesian approach would be to marginalize the condition in
equation 7 over $\tilde{\theta}$ and $\eta$. Such a procedure would give the
expected probability of the condition holding and thus of drawing some
scientific conclusion rather than a binary yes or no answer. Since the prior
represents our knowledge of the parameters before any data is measured, it is
the natural measure for a forecasting analysis. So, for our example, we should
calculate

$~{}\mathbb{E}(\textrm{Distinguish
Models})=\left\langle\mathcal{K}\left(\tilde{\theta},\eta\right)>\mathcal{K}_{\rm
crit}\right\rangle_{\eta,\pi(\tilde{\theta})},$ (8)

to find a fully Bayesian forecast of our expected chances of distinguishing
between the models at the statistical significance set by $\mathcal{K}_{\rm
crit}$. More generally, the condition in equation 8 could be replaced with any
other condition on $\mathcal{K}$ to draw a scientific conclusion

$~{}\mathbb{E}(\textrm{Drawing
Conclusion})=\left\langle\textrm{Condition}\left[\mathcal{K}(\tilde{\theta},\eta)\right]\right\rangle_{\eta,\pi(\tilde{\theta})},$
(9)

to perform a fully Bayesian forecast on drawing said conclusion. In addition,
equation 9 can be modified to produce fully Bayesian forecasts for a broader
range of scientific questions. For example, when appropriate to the question
being posed, the marginalization should be performed over the model prior as
well (this is equivalent to marginalizing over the predictive posterior odds
distribution introduced in Trotta [2]), or over conditional priors. Thus,
fully Bayesian forecasts can in theory be used to give principled and
interpretable answers to a range of forecasting questions. However, performing
the average over the parameters and noise (and potentially models) in practice
would require thousands or millions of mock data sets. In traditional Bayesian
analysis, this would, in turn, require thousands or millions of evidence
evaluations, which with current state-of-the-art methods, e.g. Nested Sampling
[25, 26], would be prohibitively expensive even for simple problems without
dedicated supercomputing time, if at all.

The computational limitations of traditional Bayesian forecasts have led to
widespread usage of Fisher forecasts to perform forecasts over uncertain data
spaces. Fisher forecasts can provide from a small number of likelihood
evaluations analytic parameter posteriors [e.g. 7] and for nested models the
Bayes ratio [27, 28, 21]. However, the reliability of these results is
conditional on the accuracy of the implicit linearization and Gaussianity
assumptions [see 19, 20]. Thus, if we knew the assumptions of Fisher forecasts
to hold well and the forecasting question at hand was between nested models,
Fisher forecasts could form the basis of a fully Bayesian forecast (such an
analysis would be reminiscent of Trotta [2]). However, these requirements
would significantly limit the applicability of such a methodology. Reliable
and widely applicable fully Bayesian forecasts will hence require a novel
methodology that maintains the expedience of Fisher forecasts but is
applicable to non-nested models and avoids the same assumptions.

Evidence Networks.— Evidence networks are a type of classifier artificial
neural networks introduced recently in Jeffrey and Wandelt [22]. They take in
simulated data $D$ and output a single value $f_{\rm EN}(D)$, with training
performed on data generated from two models (labels $m=1$ and $m=0$). The
principal insight with these networks is that for a broad category of choices
of loss function the evidence network’s value converges toward an invertible
function of the Bayes ratio $\mathcal{K}(D)$ between model $1$ and model $0$.
Hence, if such a network is converged, the Bayes ratio between the two models
for any set of ‘observed’ data can be directly calculated from the network’s
output.

As established in the previous section, efficient evaluation of $\mathcal{K}$
for a wide range of mock data is the main obstacle to performing practicable
fully Bayesian forecasts. Since evaluating a neural network on a GPU is orders
of magnitude faster than direct Bayesian evidence calculation techniques,
utilizing evidence networks may circumvent this computational limitation. We
thus propose an alternative evidence-network-based methodology for performing
fully Bayesian forecasts:

  * •

First, create simulators of mock data from the two competing models (including
experimental considerations such as noise).

  * •

Then using the two simulators generate a training set and validation set of
labelled mock data, and train the evidence network.

  * •

Validate the evidence network using a blind coverage test [22] and, if
possible, compare its output to a sample of $\mathcal{K}$ values calculated
from traditional Bayesian techniques.

  * •

Finally, evaluate equation 9, or equivalent, using the network and previously
developed simulators to evaluate $\mathcal{K}$ over $\pi$ and $\eta$ (and
potentially $M_{\rm i}$) efficiently.

We anticipate this approach to be highly performant relative to the
traditional Bayesian approach since each $\mathcal{K}$ evaluation no longer
requires an exploration of the parameter space. Furthermore, our methodology
only needs efficient mock data simulators. As a result, it can be applied in
simulation-based inference contexts where no closed-form likelihood exists,
making it more widely applicable.

An Application to Cosmology.— We have motivated fully Bayesian forecasts and
argued that evidence networks may make performing one feasible. Here, we
demonstrate this feasibility while also illustrating some of the insights
gained from such an analysis.

The problem we tackle is determining the expected chances of a 21-cm global
signal experiment making a definitive detection. Through the redshift
evolution of the 21-cm global signal the thermal evolution of the universe
from recombination to reionization can be traced, giving insights into
cosmology and structure formation [see 29, for a more detailed introduction to
the field]. However, measurement of the signal is challenging as its expected
magnitude is five orders of magnitude below that of galactic foregrounds. As a
result, there are currently no definitive detections of the global 21-cm
signal. The EDGES 2 experiment claimed a signal detection [30], but this is
disputed [e.g. 31] due to the signal not matching theoretical expectations
[e.g. 32], being better fit by the presence of a systematic [e.g. 33], and the
null-detection from the SARAS 3 experiment [34]. Because of this lack of
definitive detection, there are several ongoing and proposed experiments to
try and measure the sky-averaged 21-cm signal, e.g. REACH [23], PRIZM [35],
and EDGES 3. The question then naturally arises, what is the a priori
expectation of a global signal experiment with given sensitivity making a
definitive detection of the uncertain 21-cm signal? Since a significant signal
detection can be determined to occur when $\mathcal{K}$ between a model with a
signal and a model without a signal exceeds some threshold $\mathcal{K}_{\rm
crit}$, this question can be rigorously answered through a fully Bayesian
forecast of the form given in equation 8.

Hence, following our fully Bayesian forecasting methodology outlined above, we
began by constructing the two mock data simulators. We imagined a REACH-like
global 21-cm signal experiment with frequency-band covering redshift $7.5$ to
$28.0$ and an analysis spectral resolution of $1$ MHz. For simplicity, we
supposed all foregrounds and systematics are marginalized away leaving a noisy
measurement of the global 21-cm signal. We modelled the residual noise as
Gaussian white noise with magnitude $\sigma_{\rm noise}=0.079$ K 111As the
noise scales with spectral resolution $\Delta\nu$ as $1/\sqrt{\Delta\nu}$ this
is equivalent to $0.250$ K noise at a resolution of $0.1$ MHz, the pessimistic
case projected for REACH [23].. For our global 21-cm signal model, we used
globalemu [37], an emulator of a more computationally costly semi-numerical
simulation code [e.g., 38, 39, 40]. This model has seven parameters, star
formation efficiency $f_{*}$, minimum circular virial velocity $V_{\rm c}$,
X-ray emission efficiency $f_{\rm X}$, optical depth to the cosmic microwave
background $\tau$, exponent $\alpha$ and lower-cutoff $E_{\rm min}$ of the
X-ray spectral energy distribution, and the root-mean free path of ionizing
photons $R_{\rm mfp}$. We specified our a priori knowledge of these parameters
through our priors listed in table 1. These priors are broad uniform or log-
uniform distributions due to the uncertainty in these parameters, except for
$\tau$ which we use a truncated Gaussian prior based on the Planck 2018
measurement of $\tau$ [41]. By combining noise generators, globalemu, and
samplers over our priors, we constructed simulators of mock global 21-cm
signal data from a model with only noise (noise-only) and a model with noise
plus a signal (noisy-signal).

Table 1: Priors on our 21-cm global signal parameters. To keep the parameter values within the training parameter ranges of globalemu we use a truncated Gaussian prior on $\tau$ derived from the Planck 2018 measurements rather than a Gaussian prior. Parameter | Prior Type | Min | Max | Mean | Std.
---|---|---|---|---|---
$f_{*}$ | Log Uniform | 0.0001 | 0.5 | - | -
$V_{\rm c}$ (km s-1) | Log Uniform | 4.2 | 30.0 | - | -
$f_{\rm x}$ | Log Uniform | 0.001 | 1000.0 | - | -
$\tau$ | Truncated Gaussian | 0.040 | 0.17 | 0.054 | 0.007
$\alpha$ | Uniform | 1.0 | 1.5 | - | -
$E_{\rm min}$ (keV) | Log Uniform | 0.1 | 3.0 | - | -
$R_{\rm mfp}$ (cMpc) | Uniform | 10.0 | 50.0 | - | -

We generated 2,000,000 (200,000) mock data sets from each simulator to form
our training (validation) set. Our evidence network was implemented in
tensorflow [42], using the same architecture and optimizer as Jeffrey and
Wandelt [22] (detailed in their appendix) and a $\alpha=2$ l-POP exponential
loss function. The network was trained with early-stopping for 20 epochs using
a batch size of 1000, after which a blind coverage test confirmed the network
had converged. We further validated our network by comparing whether a
significant detection would be concluded based on the network $\mathcal{K}$
values and $\mathcal{K}$ computed using polychord [43, 44]. For a signal
detection threshold of 3 $\sigma$ (5 $\sigma$), corresponding to
$\log(\mathcal{K}_{\rm crit})=5.91$ ($\log(\mathcal{K}_{\rm crit})=14.4$), we
found the two methods came to the same conclusion for 99.2% (98.3%) of 1000
mock data sets from our noisy-signal model.

Finally, we evaluated equation 8 using our evidence network and 1,000,000 sets
of noisy-signal model mock data. Ultimately, we found the expected chance of
the experiment detecting the global 21-cm signal at a statistical significance
of 3 $\sigma$ (5 $\sigma$) was 65.2% (52.3%). This marginal favouring of a
detection of the 21-cm signal, suggests the $0.079$ K sensitivity at 1 MHz
resolution considered here is indicative of the minimum sensitivity global
signal experiments such target.

Additionally, our 1,000,000 $\mathcal{K}$ evaluations allow for a broad range
of further analyses at little to no additional computation cost. Instead of
performing the average over $\pi$ in equation 8, we can marginalize over
conditional priors with one or two parameters fixed, giving insight into under
which early Universe astrophysical scenarios we would expect to detect the
21-cm signal. These conditional detection probabilities are depicted in figure
1 for the parameters with which strong variation in detection probability was
seen, $f_{*}$, $f_{\rm X}$, and $\tau$. We find the detection of the 21-cm
signal is more likely for high star formation efficiencies, X-ray efficiencies
below a threshold of $\sim 1$, and lower optical depths to reionization. With
an almost 100% chance of a 3 $\sigma$ detection for $f_{*}>0.001$ and $f_{\rm
X}<1$. This strong variation in definitive detection chances retroactively
provides further motivation for fully Bayesian forecasts as we see the
conclusion drawn would be highly sensitive to the fiducial global 21-cm signal
parameters chosen for a traditional Bayesian forecast.

Figure 1: Triangle plot depicting the probability of a global 21-cm signal
detection at $\geq 3$ $\sigma$ statistical significance, by an experiment
covering redshift $7.5$ to $28.0$, with frequency resolution $\Delta\nu=1$
MHz, and residual white noise of $\sigma_{\rm noise}=0.079$ K. The diagonal
shows the total probability of a detection marginalized over noise
realizations and the parameter space except for one fixed parameter, and the
below diagonal is the equivalent with two fixed parameters. The total
detection probability with no parameters fixed was $65.2$%. $V_{\rm c}$,
$\alpha$, $E_{\rm min}$, and $R_{\rm mfp}$ are not shown because the
probability of detection was found to vary little with them. We find high
$f_{\rm*}$, $f_{\rm X}\lesssim 1$, and low $\tau$ values increase the chance
of a 21-cm signal detection, with $f_{\rm X}$ the most impactful. For
different parameter combinations, we find detection probabilities varying from
$\sim 0$% to $\sim 100$%, illustrating the need for fully Bayesian forecasts
as the conclusion of detectability varies strongly over the a priori uncertain
parameter space.

There are a myriad of further analyses we could perform with our methodology.
For example, we could study the evolution of the detection probability against
the significance threshold, noise level or experiment bandwidth. Additionally,
we could consider the functional distribution of the detected and non-detected
21-cm global signals to gain insight into what differentiates these two
categories. Or alternatively, we could investigate how the chances of a 21-cm
global signal detection have changed in light of the parameter constraints
from the HERA 21-cm power spectrum limits [45, 46], or the SARAS 3 null
detection [34, 47]. However, we shall leave these analyses to future work
since the focus of this letter is on the method and its feasibility rather
than particular scientific problems.

Let us return now to the computational performance of our method. The training
data generation, network training, forecast data generation, network
$\mathcal{K}$ evaluations, and plotting of figure 1, took a combined total of
0.170 GPU hours 222On an NVIDIA A100-SXM-80GB GPU that was part of a CSD3 HPC
Ampere GPU node.. Conversely, the 1000 polychord evaluations of $\mathcal{K}$
as part of our validation process took a total of 778 CPU hours 333On 76 Intel
Ice Lake CPUs that form a CSD3 HPC Ice Lake node., from which we can estimate
the 1,000,000 $\mathcal{K}$ evaluations used in our fully Bayesian forecast
would have required 778,000 CPU hours using traditional methods. While it is
not meaningful to directly compare GPU to CPU hours we can compare the costs
of those hours. On the cluster we utilized GPU hours are charged at 50 times
the rate of CPU hours. Thus our method gives a cost-weighted performance
improvement of $10^{5.0}$. Since this performance gain was for a single
problem, and our implementation was not optimized, this level of performance
gain cannot be assumed to apply universally. However, it is indicative our
methodology is highly performant compared to traditional techniques and, as we
have directly demonstrated, facilitates analyses that were previously
computationally prohibitively expensive.

Conclusions.— We have argued that to arrive at accurate and interpretable
predictions the conclusions of scientific forecasts should be marginalized
over any uncertain model parameters and noise realizations. However, such
fully Bayesian forecasts are computationally infeasible with traditional
methods. We thus propose a novel methodology for performing fully Bayesian
forecasts based on Evidence Networks.

To illustrate our method and the insights that can be gained from fully
Bayesian forecasts, we applied it to determine the chances of a REACH-like
experiment detecting the global 21-cm signal. For a frequency resolution of
$1$ MHz and a noise level of $\sigma_{\rm noise}=0.079$ K, we find a 65.2%
(52.3%) chance of detection at 3 $\sigma$ (5 $\sigma$). Thus suggesting this
noise level is indicative of the minimum sensitivity global 21-cm signal
experiments should target. Additionally, our methodology allows us to produce
triangle plots of how this chance of detection varies when one or two model
parameters are fixed, at no extra computational cost. For this example
problem, we find a cost-weighted speed-up of $10^{5.0}$ using our approach
compared to a traditional nested-sampling-based method that would have taken
$778,000$ CPU hours.

The method we propose can be applied to any forecasting question which can be
formulated as a condition on the Bayes ratio between two models. This
includes: if a signal can be detected from within noise (e.g. gravitational
waves [50], or the 21-cm signal [29]); whether two competing theories can be
distinguished by anticipated data (e.g. MOND [51] or General Relativity [52]);
or if the inclusion of novel physics in a model will be necessary (e.g.
neutrinos in CMB experiment analysis [15]). Additionally, the method only
requires simulators of mock data, and thus can still be used in cases where
closed-form likelihoods or explicit priors are not available. As a result,
this methodology should allow for reliable and efficient fully Bayesian
forecasts on a wide range of forecasting problems.

Since the proposed methodology is simulation-based and has a low computational
cost, we anticipate it will be particularly suited to performing forecasts for
a range of potential experimental configurations. Thus allowing for the
optimization of experimental configurations to minimize cost or maximize the
chance of the detection of new physics. To facilitate the application of this
methodology by others we make public on GitHub all codes and data used in the
writing of this letter.

###### Acknowledgements.

We would like to express our gratitude to Harry Bevins for several insightful
conversations concerning this work. The authors would like to thank the
Science and Technology Facilities Council (UK) for their support of TGJ
through grant number ST/V506606/1, and the Royal Society for their support of
WJH through a Royal Society University Research Fellowship. This work was
performed using resources provided by the Cambridge Service for Data Driven
Discovery (CSD3) operated by the University of Cambridge Research Computing
Service (www.csd3.cam.ac.uk), provided by Dell EMC and Intel using Tier-2
funding from the Engineering and Physical Sciences Research Council (capital
grant EP/T022159/1), and DiRAC funding from the Science and Technology
Facilities Council (www.dirac.ac.uk). For the purpose of open access, the
author has applied a Creative Commons Attribution (CC BY) licence to any
Author Accepted Manuscript version arising from this submission.

## References

  * Fisher [1922] R. A. Fisher, Philosophical Transactions of the Royal Society of London Series A 222, 309 (1922).
  * Trotta [2007a] R. Trotta, MNRAS 378, 819 (2007a), arXiv:astro-ph/0703063 [astro-ph] .
  * Sellentin  _et al._ [2014] E. Sellentin, M. Quartin, and L. Amendola, MNRAS 441, 1831 (2014), arXiv:1401.6892 [astro-ph.CO] .
  * Ryan  _et al._ [2023] J. Ryan, B. Stevenson, C. Trendafilova, and J. Meyers, Phys. Rev. D 107, 103506 (2023), arXiv:2211.06534 [astro-ph.CO] .
  * Anstey  _et al._ [2021] D. Anstey, E. de Lera Acedo, and W. Handley, MNRAS 506, 2041 (2021), arXiv:2010.09644 [astro-ph.IM] .
  * Rieck  _et al._ [2023] S. Rieck, A. W. Criswell, V. Korol, M. A. Keim, and V. Mandic, arXiv e-prints , arXiv:2308.12437 (2023), arXiv:astro-ph/2308.12437 [astro-ph] .
  * Tegmark  _et al._ [1997] M. Tegmark, A. N. Taylor, and A. F. Heavens, ApJ 480, 22 (1997), arXiv:astro-ph/9603021 [astro-ph] .
  * Albrecht  _et al._ [2006] A. Albrecht, G. Bernstein, R. Cahn, W. L. Freedman, J. Hewitt, W. Hu, J. Huth, M. Kamionkowski, E. W. Kolb, L. Knox, J. C. Mather, S. Staggs, and N. B. Suntzeff, arXiv e-prints , astro-ph/0609591 (2006), arXiv:astro-ph/0609591 [astro-ph] .
  * Seo and Eisenstein [2007] H.-J. Seo and D. J. Eisenstein, ApJ 665, 14 (2007), arXiv:astro-ph/0701079 [astro-ph] .
  * Vallisneri [2008] M. Vallisneri, Phys. Rev. D 77, 042001 (2008), arXiv:gr-qc/0703086 [gr-qc] .
  * More  _et al._ [2013] S. More, F. C. van den Bosch, M. Cacciato, A. More, H. Mo, and X. Yang, MNRAS 430, 747 (2013), arXiv:1207.0004 [astro-ph.CO] .
  * Di Dio  _et al._ [2014] E. Di Dio, F. Montanari, R. Durrer, and J. Lesgourgues, J. Cosmology Astropart. Phys 2014, 042 (2014), arXiv:1308.6186 [astro-ph.CO] .
  * Zhai and Blanton [2017] Z. Zhai and M. R. Blanton, ApJ 850, 41 (2017), arXiv:1707.06555 [astro-ph.CO] .
  * Bonvin and Fleury [2018] C. Bonvin and P. Fleury, J. Cosmology Astropart. Phys 2018, 061 (2018), arXiv:1803.02771 [astro-ph.CO] .
  * Ade  _et al._ [2019] P. Ade, J. Aguirre, Z. Ahmed, S. Aiola, A. Ali, D. Alonso, M. A. Alvarez, K. Arnold, P. Ashton, J. Austermann, H. Awan, C. Baccigalupi, T. Baildon, D. Barron, N. Battaglia, R. Battye, E. Baxter, A. Bazarko, J. A. Beall, R. Bean, D. Beck, S. Beckman, B. Beringue, F. Bianchini, S. Boada, D. Boettger, J. R. Bond, J. Borrill, M. L. Brown, S. M. Bruno, S. Bryan, E. Calabrese, V. Calafut, P. Calisse, J. Carron, A. Challinor, G. Chesmore, Y. Chinone, J. Chluba, H.-M. S. Cho, S. Choi, G. Coppi, N. F. Cothard, K. Coughlin, D. Crichton, K. D. Crowley, K. T. Crowley, A. Cukierman, J. M. D’Ewart, R. Dünner, T. de Haan, M. Devlin, S. Dicker, J. Didier, M. Dobbs, B. Dober, C. J. Duell, S. Duff, A. Duivenvoorden, J. Dunkley, J. Dusatko, J. Errard, G. Fabbian, S. Feeney, S. Ferraro, P. Fluxà, K. Freese, J. C. Frisch, A. Frolov, G. Fuller, B. Fuzia, N. Galitzki, P. A. Gallardo, J. Tomas Galvez Ghersi, J. Gao, E. Gawiser, M. Gerbino, V. Gluscevic, N. Goeckner-Wald, J. Golec, S. Gordon, M. Gralla, D. Green, A. Grigorian, J. Groh, C. Groppi, Y. Guan, J. E. Gudmundsson, D. Han, P. Hargrave, M. Hasegawa, M. Hasselfield, M. Hattori, V. Haynes, M. Hazumi, Y. He, E. Healy, S. W. Henderson, C. Hervias-Caimapo, C. A. Hill, J. C. Hill, G. Hilton, M. Hilton, A. D. Hincks, G. Hinshaw, R. Hložek, S. Ho, S.-P. P. Ho, L. Howe, Z. Huang, J. Hubmayr, K. Huffenberger, J. P. Hughes, A. Ijjas, M. Ikape, K. Irwin, A. H. Jaffe, B. Jain, O. Jeong, D. Kaneko, E. D. Karpel, N. Katayama, B. Keating, S. S. Kernasovskiy, R. Keskitalo, T. Kisner, K. Kiuchi, J. Klein, K. Knowles, B. Koopman, A. Kosowsky, N. Krachmalnicoff, S. E. Kuenstner, C.-L. Kuo, A. Kusaka, J. Lashner, A. Lee, E. Lee, D. Leon, J. S. Y. Leung, A. Lewis, Y. Li, Z. Li, M. Limon, E. Linder, C. Lopez-Caraballo, T. Louis, L. Lowry, M. Lungu, M. Madhavacheril, D. Mak, F. Maldonado, H. Mani, B. Mates, F. Matsuda, L. Maurin, P. Mauskopf, A. May, N. McCallum, C. McKenney, J. McMahon, P. D. Meerburg, J. Meyers, A. Miller, M. Mirmelstein, K. Moodley, M. Munchmeyer, C. Munson, S. Naess, F. Nati, M. Navaroli, L. Newburgh, H. N. Nguyen, M. Niemack, H. Nishino, J. Orlowski-Scherer, L. Page, B. Partridge, J. Peloton, F. Perrotta, L. Piccirillo, G. Pisano, D. Poletti, R. Puddu, G. Puglisi, C. Raum, C. L. Reichardt, M. Remazeilles, Y. Rephaeli, D. Riechers, F. Rojas, A. Roy, S. Sadeh, Y. Sakurai, M. Salatino, M. Sathyanarayana Rao, E. Schaan, M. Schmittfull, N. Sehgal, J. Seibert, U. Seljak, B. Sherwin, M. Shimon, C. Sierra, J. Sievers, P. Sikhosana, M. Silva-Feaver, S. M. Simon, A. Sinclair, P. Siritanasak, K. Smith, S. R. Smith, D. Spergel, S. T. Staggs, G. Stein, J. R. Stevens, R. Stompor, A. Suzuki, O. Tajima, S. Takakura, G. Teply, D. B. Thomas, B. Thorne, R. Thornton, H. Trac, C. Tsai, C. Tucker, J. Ullom, S. Vagnozzi, A. van Engelen, J. Van Lanen, D. D. Van Winkle, E. M. Vavagiakis, C. Vergès, M. Vissers, K. Wagoner, S. Walker, J. Ward, B. Westbrook, N. Whitehorn, J. Williams, J. Williams, E. J. Wollack, Z. Xu, B. Yu, C. Yu, F. Zago, H. Zhang, N. Zhu, and Simons Observatory Collaboration, J. Cosmology Astropart. Phys 2019, 056 (2019), arXiv:1808.07445 [astro-ph.CO] .
  * Euclid Collaboration  _et al._ [2020] Euclid Collaboration, A. Blanchard, S. Camera, C. Carbone, V. F. Cardone, S. Casas, S. Clesse, S. Ilić, M. Kilbinger, T. Kitching, M. Kunz, F. Lacasa, E. Linder, E. Majerotto, K. Markovič, M. Martinelli, V. Pettorino, A. Pourtsidou, Z. Sakr, A. G. Sánchez, D. Sapone, I. Tutusaus, S. Yahia-Cherif, V. Yankelevich, S. Andreon, H. Aussel, A. Balaguera-Antolínez, M. Baldi, S. Bardelli, R. Bender, A. Biviano, D. Bonino, A. Boucaud, E. Bozzo, E. Branchini, S. Brau-Nogue, M. Brescia, J. Brinchmann, C. Burigana, R. Cabanac, V. Capobianco, A. Cappi, J. Carretero, C. S. Carvalho, R. Casas, F. J. Castander, M. Castellano, S. Cavuoti, A. Cimatti, R. Cledassou, C. Colodro-Conde, G. Congedo, C. J. Conselice, L. Conversi, Y. Copin, L. Corcione, J. Coupon, H. M. Courtois, M. Cropper, A. Da Silva, S. de la Torre, D. Di Ferdinando, F. Dubath, F. Ducret, C. A. J. Duncan, X. Dupac, S. Dusini, G. Fabbian, M. Fabricius, S. Farrens, P. Fosalba, S. Fotopoulou, N. Fourmanoit, M. Frailis, E. Franceschi, P. Franzetti, M. Fumana, S. Galeotta, W. Gillard, B. Gillis, C. Giocoli, P. Gómez-Alvarez, J. Graciá-Carpio, F. Grupp, L. Guzzo, H. Hoekstra, F. Hormuth, H. Israel, K. Jahnke, E. Keihanen, S. Kermiche, C. C. Kirkpatrick, R. Kohley, B. Kubik, H. Kurki-Suonio, S. Ligori, P. B. Lilje, I. Lloro, D. Maino, E. Maiorano, O. Marggraf, N. Martinet, F. Marulli, R. Massey, E. Medinaceli, S. Mei, Y. Mellier, B. Metcalf, J. J. Metge, G. Meylan, M. Moresco, L. Moscardini, E. Munari, R. C. Nichol, S. Niemi, A. A. Nucita, C. Padilla, S. Paltani, F. Pasian, W. J. Percival, S. Pires, G. Polenta, M. Poncet, L. Pozzetti, G. D. Racca, F. Raison, A. Renzi, J. Rhodes, E. Romelli, M. Roncarelli, E. Rossetti, R. Saglia, P. Schneider, V. Scottez, A. Secroun, G. Sirri, L. Stanco, J. L. Starck, F. Sureau, P. Tallada-Crespí, D. Tavagnacco, A. N. Taylor, M. Tenti, I. Tereno, R. Toledo-Moreo, F. Torradeflot, L. Valenziano, T. Vassallo, G. A. Verdoes Kleijn, M. Viel, Y. Wang, A. Zacchei, J. Zoubian, and E. Zucca, A&A 642, A191 (2020), arXiv:1910.09273 [astro-ph.CO] .
  * d’Assignies D  _et al._ [2023] W. d’Assignies D, C. Zhao, J. Yu, and J.-P. Kneib, MNRAS 521, 3648 (2023), arXiv:2301.02289 [astro-ph.CO] .
  * Mason  _et al._ [2023] C. A. Mason, J. B. Muñoz, B. Greig, A. Mesinger, and J. Park, MNRAS 524, 4711 (2023), arXiv:2212.09797 [astro-ph.CO] .
  * Perotto  _et al._ [2006] L. Perotto, J. Lesgourgues, S. Hannestad, H. Tu, and Y. Y Y Wong, J. Cosmology Astropart. Phys 2006, 013 (2006), arXiv:astro-ph/0606227 [astro-ph] .
  * Wolz  _et al._ [2012] L. Wolz, M. Kilbinger, J. Weller, and T. Giannantonio, J. Cosmology Astropart. Phys 2012, 009 (2012), arXiv:1205.3984 [astro-ph.CO] .
  * Trotta [2007b] R. Trotta, MNRAS 378, 72 (2007b), arXiv:astro-ph/0504022 [astro-ph] .
  * Jeffrey and Wandelt [2023] N. Jeffrey and B. D. Wandelt, arXiv e-prints , arXiv:2305.11241 (2023), arXiv:2305.11241 [cs.LG] .
  * de Lera Acedo  _et al._ [2022] E. de Lera Acedo, D. I. L. de Villiers, N. Razavi-Ghods, W. Handley, A. Fialkov, A. Magro, D. Anstey, H. T. J. Bevins, R. Chiello, J. Cumner, A. T. Josaitis, I. L. V. Roque, P. H. Sims, K. H. Scheutwinkel, P. Alexander, G. Bernardi, S. Carey, J. Cavillot, W. Croukamp, J. A. Ely, T. Gessey-Jones, Q. Gueuning, R. Hills, G. Kulkarni, R. Maiolino, P. D. Meerburg, S. Mittal, J. R. Pritchard, E. Puchwein, A. Saxena, E. Shen, O. Smirnov, M. Spinelli, and K. Zarb-Adami, Nature Astronomy 6, 984 (2022).
  * MacKay [2003] D. MacKay,  _Information Theory, Inference and Learning Algorithms_ (Cambridge University Press, 2003).
  * Skilling [2004] J. Skilling, in  _Bayesian Inference and Maximum Entropy Methods in Science and Engineering: 24th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering_, American Institute of Physics Conference Series, Vol. 735, edited by R. Fischer, R. Preuss, and U. V. Toussaint (2004) pp. 395–405.
  * Ashton  _et al._ [2022] G. Ashton, N. Bernstein, J. Buchner, X. Chen, G. Csányi, A. Fowlie, F. Feroz, M. Griffiths, W. Handley, M. Habeck, E. Higson, M. Hobson, A. Lasenby, D. Parkinson, L. B. Pártay, M. Pitkin, D. Schneider, J. S. Speagle, L. South, J. Veitch, P. Wacker, D. J. Wales, and D. Yallup, Nature Reviews Methods Primers 2, 39 (2022), arXiv:2205.15570 [stat.CO] .
  * Dickey [1971] J. M. Dickey, The Annals of Mathematical Statistics 42, 204 (1971).
  * Verdinelli and Wasserman [1995] I. Verdinelli and L. Wasserman, Journal of the American Statistical Association 90, 614 (1995).
  * Furlanetto  _et al._ [2006] S. R. Furlanetto, S. P. Oh, and F. H. Briggs, Phys. Rep. 433, 181 (2006), arXiv:astro-ph/0608032 [astro-ph] .
  * Bowman  _et al._ [2018] J. D. Bowman, A. E. E. Rogers, R. A. Monsalve, T. J. Mozdzen, and N. Mahesh, Nature 555, 67 (2018), arXiv:1810.05912 [astro-ph.CO] .
  * Hills  _et al._ [2018] R. Hills, G. Kulkarni, P. D. Meerburg, and E. Puchwein, Nature 564, E32 (2018), arXiv:1805.01421 [astro-ph.CO] .
  * Barkana [2018] R. Barkana, Nature 555, 71 (2018), arXiv:1803.06698 [astro-ph.CO] .
  * Sims and Pober [2020] P. H. Sims and J. C. Pober, MNRAS 492, 22 (2020), arXiv:1910.03165 [astro-ph.CO] .
  * Singh  _et al._ [2022] S. Singh, N. T. Jishnu, R. Subrahmanyan, N. Udaya Shankar, B. S. Girish, A. Raghunathan, R. Somashekar, K. S. Srivani, and M. Sathyanarayana Rao, Nature Astronomy 6, 607 (2022), arXiv:2112.06778 [astro-ph.CO] .
  * Philip  _et al._ [2019] L. Philip, Z. Abdurashidova, H. C. Chiang, N. Ghazi, A. Gumba, H. M. Heilgendorff, J. M. Jáuregui-García, K. Malepe, C. D. Nunhokee, J. Peterson, J. L. Sievers, V. Simes, and R. Spann, Journal of Astronomical Instrumentation 8, 1950004 (2019).
  * Note [1] As the noise scales with spectral resolution $\Delta\nu$ as $1/\sqrt{\Delta\nu}$ this is equivalent to $0.250$ K noise at a resolution of $0.1$ MHz, the pessimistic case projected for REACH [23].
  * Bevins  _et al._ [2021] H. T. J. Bevins, W. J. Handley, A. Fialkov, E. de Lera Acedo, and K. Javid, MNRAS 508, 2923 (2021), arXiv:2104.04336 [astro-ph.CO] .
  * Visbal  _et al._ [2012] E. Visbal, R. Barkana, A. Fialkov, D. Tseliakhovich, and C. M. Hirata, Nature 487, 70 (2012), arXiv:1201.1005 [astro-ph.CO] .
  * Fialkov  _et al._ [2014] A. Fialkov, R. Barkana, A. Pinhas, and E. Visbal, MNRAS 437, L36 (2014), arXiv:1306.2354 [astro-ph.CO] .
  * Reis  _et al._ [2020] I. Reis, A. Fialkov, and R. Barkana, MNRAS 499, 5993 (2020), arXiv:2008.04315 [astro-ph.CO] .
  * Planck Collaboration  _et al._ [2020] Planck Collaboration, N. Aghanim, Y. Akrami, M. Ashdown, J. Aumont, C. Baccigalupi, M. Ballardini, A. J. Banday, R. B. Barreiro, N. Bartolo, S. Basak, R. Battye, K. Benabed, J. P. Bernard, M. Bersanelli, P. Bielewicz, J. J. Bock, J. R. Bond, J. Borrill, F. R. Bouchet, F. Boulanger, M. Bucher, C. Burigana, R. C. Butler, E. Calabrese, J. F. Cardoso, J. Carron, A. Challinor, H. C. Chiang, J. Chluba, L. P. L. Colombo, C. Combet, D. Contreras, B. P. Crill, F. Cuttaia, P. de Bernardis, G. de Zotti, J. Delabrouille, J. M. Delouis, E. Di Valentino, J. M. Diego, O. Doré, M. Douspis, A. Ducout, X. Dupac, S. Dusini, G. Efstathiou, F. Elsner, T. A. Enßlin, H. K. Eriksen, Y. Fantaye, M. Farhang, J. Fergusson, R. Fernandez-Cobos, F. Finelli, F. Forastieri, M. Frailis, A. A. Fraisse, E. Franceschi, A. Frolov, S. Galeotta, S. Galli, K. Ganga, R. T. Génova-Santos, M. Gerbino, T. Ghosh, J. González-Nuevo, K. M. Górski, S. Gratton, A. Gruppuso, J. E. Gudmundsson, J. Hamann, W. Handley, F. K. Hansen, D. Herranz, S. R. Hildebrandt, E. Hivon, Z. Huang, A. H. Jaffe, W. C. Jones, A. Karakci, E. Keihänen, R. Keskitalo, K. Kiiveri, J. Kim, T. S. Kisner, L. Knox, N. Krachmalnicoff, M. Kunz, H. Kurki-Suonio, G. Lagache, J. M. Lamarre, A. Lasenby, M. Lattanzi, C. R. Lawrence, M. Le Jeune, P. Lemos, J. Lesgourgues, F. Levrier, A. Lewis, M. Liguori, P. B. Lilje, M. Lilley, V. Lindholm, M. López-Caniego, P. M. Lubin, Y. Z. Ma, J. F. Macías-Pérez, G. Maggio, D. Maino, N. Mandolesi, A. Mangilli, A. Marcos-Caballero, M. Maris, P. G. Martin, M. Martinelli, E. Martínez-González, S. Matarrese, N. Mauri, J. D. McEwen, P. R. Meinhold, A. Melchiorri, A. Mennella, M. Migliaccio, M. Millea, S. Mitra, M. A. Miville-Deschênes, D. Molinari, L. Montier, G. Morgante, A. Moss, P. Natoli, H. U. Nørgaard-Nielsen, L. Pagano, D. Paoletti, B. Partridge, G. Patanchon, H. V. Peiris, F. Perrotta, V. Pettorino, F. Piacentini, L. Polastri, G. Polenta, J. L. Puget, J. P. Rachen, M. Reinecke, M. Remazeilles, A. Renzi, G. Rocha, C. Rosset, G. Roudier, J. A. Rubiño-Martín, B. Ruiz-Granados, L. Salvati, M. Sandri, M. Savelainen, D. Scott, E. P. S. Shellard, C. Sirignano, G. Sirri, L. D. Spencer, R. Sunyaev, A. S. Suur-Uski, J. A. Tauber, D. Tavagnacco, M. Tenti, L. Toffolatti, M. Tomasi, T. Trombetti, L. Valenziano, J. Valiviita, B. Van Tent, L. Vibert, P. Vielva, F. Villa, N. Vittorio, B. D. Wandelt, I. K. Wehus, M. White, S. D. M. White, A. Zacchei, and A. Zonca, A&A 641, A6 (2020), arXiv:1807.06209 [astro-ph.CO] .
  * Abadi  _et al._ [2015] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems (2015), software available from tensorflow.org.
  * Handley  _et al._ [2015a] W. J. Handley, M. P. Hobson, and A. N. Lasenby, MNRAS 450, L61 (2015a), arXiv:1502.01856 [astro-ph.CO] .
  * Handley  _et al._ [2015b] W. J. Handley, M. P. Hobson, and A. N. Lasenby, MNRAS 453, 4384 (2015b), arXiv:1506.00171 [astro-ph.IM] .
  * Abdurashidova  _et al._ [2022] Z. Abdurashidova, J. E. Aguirre, P. Alexander, Z. S. Ali, Y. Balfour, R. Barkana, A. P. Beardsley, G. Bernardi, T. S. Billings, J. D. Bowman, R. F. Bradley, P. Bull, J. Burba, S. Carey, C. L. Carilli, C. Cheng, D. R. DeBoer, M. Dexter, E. de Lera Acedo, J. S. Dillon, J. Ely, A. Ewall-Wice, N. Fagnoni, A. Fialkov, R. Fritz, S. R. Furlanetto, K. Gale-Sides, B. Glendenning, D. Gorthi, B. Greig, J. Grobbelaar, Z. Halday, B. J. Hazelton, S. Heimersheim, J. N. Hewitt, J. Hickish, D. C. Jacobs, A. Julius, N. S. Kern, J. Kerrigan, P. Kittiwisit, S. A. Kohn, M. Kolopanis, A. Lanman, P. La Plante, T. Lekalake, D. Lewis, A. Liu, Y.-Z. Ma, D. MacMahon, L. Malan, C. Malgas, M. Maree, Z. E. Martinot, E. Matsetela, A. Mesinger, J. Mirocha, M. Molewa, M. F. Morales, T. Mosiane, J. B. Muñoz, S. G. Murray, A. R. Neben, B. Nikolic, C. D. Nunhokee, A. R. Parsons, N. Patra, S. Pieterse, J. C. Pober, Y. Qin, N. Razavi-Ghods, I. Reis, J. Ringuette, J. Robnett, K. Rosie, M. G. Santos, S. Sikder, P. Sims, C. Smith, A. Syce, N. Thyagarajan, P. K. G. Williams, and H. Zheng, ApJ 924, 51 (2022), arXiv:2108.07282 [astro-ph.CO] .
  * HERA Collaboration  _et al._ [2023] HERA Collaboration, Z. Abdurashidova, T. Adams, J. E. Aguirre, P. Alexander, Z. S. Ali, R. Baartman, Y. Balfour, R. Barkana, A. P. Beardsley, G. Bernardi, T. S. Billings, J. D. Bowman, R. F. Bradley, D. Breitman, P. Bull, J. Burba, S. Carey, C. L. Carilli, C. Cheng, S. Choudhuri, D. R. DeBoer, E. de Lera Acedo, M. Dexter, J. S. Dillon, J. Ely, A. Ewall-Wice, N. Fagnoni, A. Fialkov, R. Fritz, S. R. Furlanetto, K. Gale-Sides, H. Garsden, B. Glendenning, A. Gorce, D. Gorthi, B. Greig, J. Grobbelaar, Z. Halday, B. J. Hazelton, S. Heimersheim, J. N. Hewitt, J. Hickish, D. C. Jacobs, A. Julius, N. S. Kern, J. Kerrigan, P. Kittiwisit, S. A. Kohn, M. Kolopanis, A. Lanman, P. La Plante, D. Lewis, A. Liu, A. Loots, Y.-Z. Ma, D. H. E. MacMahon, L. Malan, K. Malgas, C. Malgas, M. Maree, B. Marero, Z. E. Martinot, L. McBride, A. Mesinger, J. Mirocha, M. Molewa, M. F. Morales, T. Mosiane, J. B. Muñoz, S. G. Murray, V. Nagpal, A. R. Neben, B. Nikolic, C. D. Nunhokee, H. Nuwegeld, A. R. Parsons, R. Pascua, N. Patra, S. Pieterse, Y. Qin, N. Razavi-Ghods, J. Robnett, K. Rosie, M. G. Santos, P. Sims, S. Singh, C. Smith, H. Swarts, J. Tan, N. Thyagarajan, M. J. Wilensky, P. K. G. Williams, P. van Wyngaarden, and H. Zheng, ApJ 945, 124 (2023).
  * Bevins  _et al._ [2022] H. T. J. Bevins, A. Fialkov, E. de Lera Acedo, W. J. Handley, S. Singh, R. Subrahmanyan, and R. Barkana, Nature Astronomy 6, 1473 (2022), arXiv:2212.00464 [astro-ph.CO] .
  * Note [2] On an NVIDIA A100-SXM-80GB GPU that was part of a CSD3 HPC Ampere GPU node.
  * Note [3] On 76 Intel Ice Lake CPUs that form a CSD3 HPC Ice Lake node.
  * Evans  _et al._ [2023] M. Evans, A. Corsi, C. Afle, A. Ananyeva, K. G. Arun, S. Ballmer, A. Bandopadhyay, L. Barsotti, M. Baryakhtar, E. Berger, E. Berti, S. Biscoveanu, S. Borhanian, F. Broekgaarden, D. A. Brown, C. Cahillane, L. Campbell, H.-Y. Chen, K. J. Daniel, A. Dhani, J. C. Driggers, A. Effler, R. Eisenstein, S. Fairhurst, J. Feicht, P. Fritschel, P. Fulda, I. Gupta, E. D. Hall, G. Hammond, O. A. Hannuksela, H. Hansen, C.-J. Haster, K. Kacanja, B. Kamai, R. Kashyap, J. Shapiro Key, S. Khadkikar, A. Kontos, K. Kuns, M. Landry, P. Landry, B. Lantz, T. G. F. Li, G. Lovelace, V. Mandic, G. L. Mansell, D. Martynov, L. McCuller, A. L. Miller, A. H. Nitz, B. J. Owen, C. Palomba, J. Read, H. Phurailatpam, S. Reddy, J. Richardson, J. Rollins, J. D. Romano, B. S. Sathyaprakash, R. Schofield, D. H. Shoemaker, D. Sigg, D. Singh, B. Slagmolen, P. Sledge, J. Smith, M. Soares-Santos, A. Strunk, L. Sun, D. Tanner, L. A. C. van Son, S. Vitale, B. Willke, H. Yamamoto, and M. Zucker, arXiv e-prints , arXiv:2306.13745 (2023), arXiv:2306.13745 [astro-ph.IM] .
  * Milgrom [1983] M. Milgrom, ApJ 270, 365 (1983).
  * Wald [1984] R. M. Wald,  _General Relativity_ (University of Chicago Press, 1984).