# Empirical Prior Distributions for Bayesian Meta-Analyses of Binary and Time
to Event Outcomes

František Bartoš
Department of Psychology
University of Amsterdam
Noord-Holland, The Netherlands
&Willem M. Otte
Department of Pediatric Neurology,
UMC Utrecht Brain Center
University Medical Center Utrecht
Utrecht, The Netherlands
&Quentin F. Gronau
Department of Psychology
University of Amsterdam
Noord-Holland, The Netherlands
&Bram Timmers
Department of Psychology
University of Amsterdam
Noord-Holland, The Netherlands
&Alexander Ly
Department of Psychology
University of Amsterdam
Noord-Holland, The Netherlands
&
Centrum Wiskunde & Informatica, Amsterdam
Noord-Holland, The Netherlands
&Eric-Jan Wagenmakers
Department of Psychology
University of Amsterdam
Noord-Holland, The Netherlands

###### Abstract

Bayesian model-averaged meta-analysis allows quantification of evidence for
both treatment effectiveness $\mu$ and across-study heterogeneity $\tau$. We
use the Cochrane Database of Systematic Reviews to develop discipline-wide
empirical prior distributions for $\mu$ and $\tau$ for meta-analyses of binary
and time-to-event clinical trial outcomes. First, we use 50% of the database
to estimate parameters of different required parametric families. Second, we
use the remaining 50% of the database to select the best-performing parametric
families and explore essential assumptions about the presence or absence of
the treatment effectiveness and across-study heterogeneity in real data. We
find that most meta-analyses of binary outcomes are more consistent with the
absence of the meta-analytic effect or heterogeneity while meta-analyses of
time-to-event outcomes are more consistent with the presence of the meta-
analytic effect or heterogeneity. Finally, we use the complete database - with
close to half a million trial outcomes - to propose specific empirical prior
distributions, both for the field in general and for specific medical
subdisciplines. An example from acute respiratory infections demonstrates how
the proposed prior distributions can be used to conduct a Bayesian model-
averaged meta-analysis in the open-source software R and JASP.

_Keywords_ Odds ratios, Risk ratios, Risk differences, Hazard ratios, Bayes
factor, Binary, Survival, Contingency, Frequency, Informed inference,
Historical

##  1 Introduction

Clinical trials are essential for testing whether novel therapeutic treatments
and inverventions are indeed benificial and not harmful.? Before novel
therapeutic interventions can be implemented, multiple related clinical trials
are required to accumulate evidence for the presence of a beneficial treatment
effect. This process is usually achieved by meta-analytical techniques that
allow researchers to pool estimates from individual clinical trials and obtain
an aggregated estimate of the treatment effectiveness and its uncertainty.? If
the overall uncertainty is too large, additional data collection might be
required to obtain a more precise estimate.

Aggregating data from multiple clinical trials poses a non-trivial analysis
problem as the selection of a meta-analysis model affects the treatment effect
point estimate and uncertainty jointly.? ? Uncertainty may originate from
heterogeneity within and between trial data. For example, a fixed-effects
meta-analysis model for the joint analysis of multiple clinical trials assumes
no between-trial uncertainty, whereas a random-effects model takes both within
and between-trial heterogeneity into account.? In the case with few clinical
trials, it is difficult to establish whether a fixed-effects or random-effects
model is most appropriate for the data at hand. Furthermore, the traditional
approaches struggle with obtaining a reliable between-trial heterogeneity
estimate.? ? ? The required choice between fixed or random meta-analysis model
is of limited interest to the researcher pooling multiple clinical trials, as
the concern is with obtaining the pooled effect estimate rather than the most
appropriate model.

In a previous article,? we introduced Bayesian model-averaged (BMA) meta-
analysis? ? as a coherent way of combining meta-analytic inference across
fixed- and random-effect models, and developed empirical prior distributions
for the treatment effectiveness $\delta$ and across-study heterogeneity $\tau$
parameters for Cohen’s $d$ measured continuous outcomes. The BMA meta-analysis
allows researchers to combine inferences based on a series of competing models
by taking each of them into account according to their prior predictive
performance and draw inferences that incorporate uncertainty about the data-
generating process. Furthermore, the empirical prior distributions improve
parameter estimates under small sample sizes (also see ? ? ? ? ? ) and specify
Bayes factors tests for the presence vs. absence of the treatment
effectiveness and between-study heterogeneity.? ? ? ?

In this article, we extend the previous continuous-outcome-only work by
proposing empirical prior distributions for both the treatment effectiveness
and across-study heterogeneity for binary and time-to-event outcomes based on
medical data obtained from the Cochrane Database of Systematic Reviews (CDSR).
We use the binomial-normal model for log odds ratios and a normal-normal model
for log risk ratios, risk differences, and log hazard ratios111This is also an
extension to Pullenayegum? and Turner and colleagues? who developed prior
distributions for the across-study heterogeneity $\tau$ based on an earlier
version of the database. Furthermore, we summarize information about the
evidence in favor of the effect and across-study heterogeneity in the CDSR,
implement the empirical prior distributions in the open-source software R? and
JASP,? ? and illustrate the methodology on an acute respiratory infections
example.

##  2 Meta-Analysis of Binary and Time to Event Outcomes

###  2.1 Effect size measures

We examine meta-analytic models of binary and time-to-event outcomes (see
Bartoš and colleagues? for treatment of continuous outcomes). While both
outcome types can be, in essence, addressed in the same way as continuous
data, there are additional specificities when dealing with binary outcomes.

| Events | Non-Events | N
---|---|---|---
Group 1 | a | b | $\text{n}_{1}$
Group 2 | c | d | $\text{n}_{2}$
Table 1: Characteristics of outcomes of a binary study.

By binary outcomes, we refer to studies whose endpoint of interest is either a
presence or absence of an event (e.g., death or recurrence). Consequently,
results of individual trials can be summarized by a $2\times 2$ table such as
Table 1. The rows in Table 1 correspond to the study groups (e.g., treatment
vs control arm), the first column denotes the number of observed events, the
second row denotes the number of event-free observations, and the third column
denotes the number of observation in each group.

There are multiple ways to quantify the differences between the two groups
(i.e., effect sizes), the most common being odds ratios (OR), risk ratios
(RR), and risk differences (RD). Each of the measures has its merits and the
selection of the most appropriate measure needs to be based on clinical
considerations. The most popular choices are the relative effect size measures
(i.e., OR and RR) which are less sensitive to the baseline event rate;
however, it has been argued that risk differences are better suited for
convening the clinical impact.? ? ?

| Effect Size | Standard Error
---|---|---
log OR | $\text{log}\left(\frac{\nicefrac{{a}}{{b}}}{\nicefrac{{c}}{{d}}}\right)$ | $\sqrt{\frac{1}{a}+\frac{1}{b}+\frac{1}{c}+\frac{1}{d}}$
log RR | $\text{log}\left(\frac{\nicefrac{{a}}{{n_{1}}}}{\nicefrac{{c}}{{n_{2}}}}\right)$ | $\sqrt{\frac{1}{a}-\frac{1}{n_{1}}+\frac{1}{b}-\frac{1}{n_{2}}}$
RD | $\frac{a}{n_{1}}-\frac{c}{n_{2}}$ | $\sqrt{\frac{ab}{n_{1}^{3}}+\frac{cd}{n_{2}^{3}}}$
Table 2: Effect sizes and approximate standard errors of effect size measures
of binary outcomes based on formulas from Borenstein and colleagues 2009.?

OR and RR are ratios (as suggested by the name) – they are asymmetric and can
attain positive values only. Therefore, log OR and log RR are usually used
which leads to symmetric and unbounded response variables that can be modeled
via a normal likelihood. Table 2 summarizes log OR, log RR, and RD and their
standard errors.

Table 2 also highlights an additional issue with the usage of log OR and log
RR – both are undefined when the number of events in either of the groups is
zero. This happens suprisingly often; $22.7$% of binary outcomes meta-analyses
included in CDSR contain at least one cell with zero events. One way of
dealing with the zero-cell issue is continuity corrections? which add a small
positive number to cells in Table 1 or the Mantel-Haenszel method? or
binomial-normal models for OR.

For time-to-event outcomes, it is natural to use the log hazard ratios (log
HR), which take into account the number of events, the timing of events, and
the time until the last follow-up for each trial participant without an event
(i.e., right censoring).? ?

###  2.2 Normal-normal model

Subsequently, the log OR, log RR, RD, or log HR, $\text{y}_{i}$, and their
standard errors, $\text{se}_{i}$, can be modeled using a normal-normal meta-
analytic model,

$\displaystyle\text{y}_{i}$
$\displaystyle\sim\text{Normal}(\gamma_{i},\text{se}_{i}),$ (1)
$\displaystyle\gamma_{i}$ $\displaystyle\sim\text{Normal}(\mu,\tau),$

where Normal($x$, $y$)” denotes a normal distribution with mean $x$ and
standard deviation $y$. The model parameters, $\mu$ and $\tau$ correspond to
the mean effect size and between study standard deviation (heterogeneity) and
$\gamma_{i}$ correspond to normally distributed true study effects. If we
assume the absence of heterogeneity, i.e., a fixed effect model, $\tau$ is set
to zero, and all $\gamma_{i}$ are equal to $\mu$.

###  2.3 Binomial-normal model

The continuity corrections with a normal-normal meta-analytic model can lead
to bias, especially in unbalanced designs.? Another, more elegant, solution in
the case of log OR is a binomial-normal logistic model that naturally deals
with zero events,? ?

$\displaystyle\text{a}_{i}$
$\displaystyle\sim\text{Binomial}(\pi_{1,i},\text{n}_{1,i}),$ (2)
$\displaystyle\text{c}_{i}$
$\displaystyle\sim\text{Binomial}(\pi_{2,i},\text{n}_{2,i}),$
$\displaystyle\text{logit}(\pi_{1,i})$ $\displaystyle=\beta_{i}+\gamma_{i}/2,$
$\displaystyle\text{logit}(\pi_{2,i})$ $\displaystyle=\beta_{i}-\gamma_{i}/2,$
$\displaystyle\gamma_{i}$ $\displaystyle\sim\text{Normal}(\mu,\tau),$

where Binomial($\pi$, $n$) denotes a binomial distribution with the
probability of an event $\pi$ and the number of observations $n$. The
parameter $\beta_{i}$ correspond to study specific base-rate probabilities (on
logistic scale), and parameters $\mu$, $\tau$, and $\gamma_{i}$ have the same
interpretation as in Equation 1.

##  3 Bayesian Model-Averaged Meta-Analysis

When considering the normal-normal or binomial-normal meta-analytic models, we
need to specify prior distributions for the $\mu$ and $\tau$ parameters
defining the competing hypotheses $\mathcal{H}_{\cdot}$ (and $\beta_{i}$ in
the case of the binomial-normal). Different prior distribution
specifications/assumptions about either the presence or absence of the $\mu$
and $\tau$ parameters lead to four qualitatively different models;

  1. 1.

the fixed-effect null hypothesis $\mathcal{H}_{0}^{f}$ : $\mu=0$ , $\tau=0$,

  2. 2.

the fixed-effect alternative hypothesis $\mathcal{H}_{1}^{f}$ : $\mu\sim
g(\cdot)$ , $\tau=0$,

  3. 3.

the random-effects null hypothesis $\mathcal{H}_{0}^{r}$ : $\mu=0$, $\tau\sim
h(\cdot)$,

  4. 4.

the random-effects alternative hypothesis $\mathcal{H}_{1}^{r}$ : $\mu\sim
g(\cdot)$ , $\tau\sim h(\cdot)$,

where $g(\cdot)$ denotes a prior distribution on the mean effect size
parameter assuming the presence of an effect, and $h(\cdot)$ denotes a prior
distribution on the heterogeneity parameter assuming the presence of
heterogeneity.

The goals of Bayesian model-averaged meta-analysis (BMA) are twofold: 1) to
evaluate the evidence in favor/against the specified models and hypotheses;
and 2) to combine parameter estimates from the specified models. BMA provides
a coherent way of combining results from multiple competing models (see Gronau
and colleagues? and Bartoš and colleagues? for detailed treatment). This is in
a stark difference to the classical inference that usually proceeds by
selecting one of the models (2 or 4) on either theoretical or quantitative
grounds (by the very so often under-powered test for residual heterogeneity)
and completely commits to the single selected model.

In short, in BMA, evidence in favor of the effect is quantified via an
inclusion Bayes factor which measures the change from prior to posterior
inclusion odds for the effect,

$\underbrace{\text{BF}_{10}}_{\begin{subarray}{c}\text{Inclusion Bayes
factor}\\\ {\text{for
effect}}\end{subarray}}=\;\;\underbrace{\frac{p(\mathcal{H}_{1}^{f}\mid\text{data})+p(\mathcal{H}_{1}^{r}\mid\text{data})}{p(\mathcal{H}_{0}^{f}\mid\text{data})+p(\mathcal{H}_{0}^{r}\mid\text{data})}}_{\begin{subarray}{c}\text{Posterior
inclusion odds}\\\ {\text{for
effect}}\end{subarray}}\;\;\bigg{/}\;\;\underbrace{\frac{p(\mathcal{H}_{1}^{f})+p(\mathcal{H}_{1}^{r})}{p(\mathcal{H}_{0}^{f})+p(\mathcal{H}_{0}^{r})}}_{\begin{subarray}{c}\text{Prior
inclusion odds}\\\ {\text{for effect}}\end{subarray}},$ (3)

where $p(\mathcal{H}_{\cdot})$ denotes prior model probabilities and
$p(\mathcal{H}_{\cdot}\mid\text{data})$ denotes posterior model probabilities.
Similarly, the inclusion Bayes factor for heterogeneity is given by the change
from prior to posterior inclusion odds for the heterogeneity,

$\underbrace{\text{BF}_{rf}}_{\begin{subarray}{c}\text{Inclusion Bayes
factor}\\\ {\text{for
heterogeneity}}\end{subarray}}=\;\;\underbrace{\frac{p(\mathcal{H}_{0}^{r}\mid\text{data})+p(\mathcal{H}_{1}^{r}\mid\text{data})}{p(\mathcal{H}_{0}^{f}\mid\text{data})+p(\mathcal{H}_{1}^{f}\mid\text{data})}}_{\begin{subarray}{c}\text{Posterior
inclusion odds}\\\ {\text{for
heterogeneity}}\end{subarray}}\;\;\bigg{/}\;\;\underbrace{\frac{p(\mathcal{H}_{0}^{r})+p(\mathcal{H}_{1}^{r})}{p(\mathcal{H}_{0}^{f})+p(\mathcal{H}_{1}^{f})}}_{\begin{subarray}{c}\text{Prior
inclusion odds}\\\ {\text{for heterogeneity}}\end{subarray}}.$ (4)

The posterior BMA distribution for the effect, assuming it is present, is
defined as a mixture of posterior distributions under the fixed-effect
alternative hypothesis model weighted by its posterior probability and the
random-effect alternative hypothesis model weighted by its posterior
probability,

$p(\mu\mid\text{data},\mathcal{H}_{1})\propto
p(\mu\mid\mathcal{H}_{1}^{f},\text{data})\times
p(\mathcal{H}_{1}^{f}\mid\text{data})+p(\mu\mid\mathcal{H}_{1}^{r},\text{data})\times
p(\mathcal{H}_{1}^{r}\mid\text{data}).$ (5)

Similarly, the posterior BMA distribution for the heterogeneity, assuming it
is present, is defined as a mixture of posterior distributions under the
random-effect null hypothesis model weighted by its posterior probability and
the random-effect alternative hypothesis model weighted by its posterior
probability,

$p(\tau\mid\text{data},\mathcal{H}^{r})\propto
p(\tau\mid\mathcal{H}_{0}^{r},\text{data})\times
p(\mathcal{H}_{0}^{r}\mid\text{data})+p(\tau\mid\mathcal{H}_{1}^{r},\text{data})\times
p(\mathcal{H}_{1}^{r}\mid\text{data}).$ (6)

##  4 Prior Distributions

We followed the same procedure as in the previous study where we developed
prior distributions for continuous outcomes? ; we downloaded data from the
Cochrane Database of Systematic Reviews (CDSR), we split the data equally into
a training and testing data set (50/50), we used the training set to obtain
prior distributions and the test set to select the best performing prior
distributions. Finally. we combined the data to propose prior distributions
based on the complete data set. One exception was that in the present work we
no longer consider the Cauchy prior distribution for the meta-analytic mean
parameter and the Uniform prior distribution for the heterogeneity parameter
as they were dominated by the remaining prior distributions.

###  4.1 Data set

As previously, we used trial data obtained from the CDSR which is the leading
journal and database for systematic reviews in health care. We identified all
systematic reviews in the CDSR through PubMed with the NCBI’s EUtils API
(query: “Cochrane Database Syst Rev”[journal] AND (“2000/01/01”[PDAT]:
“2021/01/31”[PDAT]). We downloaded the XML meta-analysis table file
(rm5-format) associated with the review’s latest version. We obtained a total
of 28,579 reviews, containing 163,249 comparisons, with a grand total of
788,883 estimates. We removed estimates without the control or the treatment
group and split the data according to the outcome type. This left 97,500
comparisons containing 484,128 binary trial outcomes and 1,515 comparisons
with 8,149 time-to-event outcomes.

We randomly split the binary and time-to-event data into a training and
testing data set. We used the highest level of grouping – the review level –
for the 50/50 training and testing data set split to prevent information
leakage and overfitting. We used the training data set to estimate the prior
distributions and the testing data set to assess their predictive performance.

###  4.2 Estimating prior distributions on the training data set

Initial training data set: 49,247 comparisons (243,257 estimates)6,334
comparisons (127,584 estimates) Final training data set: log OR: 6,281
comparisons (126,794 estimates) log RR: 6,215 comparisons (125,002 estimates)
RD: 6,318 comparisons (127,275 estimates) Removing comparisons with fewer than
10 estimates. Estimating frequentist random-effects meta-analytic models.

Initial training data set: 731 comparisons (3,805 estimates)100 comparisons
(1,720 estimates)Final training data set: 98 comparisons (1,692 estimates)
Removing comparisons with fewer than 10 estimates. Estimating frequentist
random-effects meta-analytic models.

Figure 1: Flowchart of the study selection procedure and data processing steps
for the training data sets. Binary outcomes (left) and time to event outcomes
(right).

Figure 1 visualizes the data processing steps applied to the training data
set. We excluded comparisons with fewer than 10 estimates to ensure that the
training set yields reliable estimates of the $\mu$ and $\tau$ parameters.
Then, we used the metafor R package? to re-estimate all comparisons using a
frequentist random-effects meta-analytic model. We used the generalized linear
mixed-effects model with fixed study effects for log OR implemented in the
rma.glmm function,222I.e., model 4 described in Jackson and colleagues? that
closely resembles the Bayesian binomial-normal model in Equation 2. and the
restricted maximum likelihood random effects model for log RR, RD, and log HR.
The last row of Figure 1 summarizes the number of converged comparisons (and
the corresponding number of estimates).

|
---|---
|
Figure 2: Frequentist effect sizes estimates and candidate prior distributions
from the training data set. Binary outcomes (log OR; first row) and time-to-
event outcomes (log HR; second row). Histogram and tick marks display the
estimated effect size estimates (left) and between-study standard deviation
estimates (right), whereas lines represent candidate prior distributions for
the population effect size parameter (left) and candidate prior distributions
for the population between-study standard deviation $\tau$ (right; see Table
3). 51 log OR outside of the $\pm 3$ range are not shown and 1 log HR outside
of the $\pm 1.5$ range is not shown. 26 $\tau$ estimates larger than 1.5 and
2,360 $\tau$ estimates lower than 0.01 of log OR are not shown. 1 $\tau$
estimates larger than 1.5 and 31 $\tau$ estimates lower than 0.01 of log HR
are not shown

We used the frequentist meta-analytic estimate obtained on the training data
set to estimate prior parameter distributions for $\mu$ and $\tau$ parameters
for the Bayesian meta-analytic models (Equations 1 and 2; we used independent
uniform distributions for the $\text{logit}(\beta_{i})$ parameter as this is
inconsequential for hypotheses regarding the mean effect and heterogeneity—as
the prior distribution is common across the specified models).333As
previously, we assumed that $\tau$ estimates lower than 0.01 are
representative of $\mathcal{H}^{f}$. Therefore these estimates were not used
to determine candidate prior distributions for $\tau$. The first row of Figure
2 visualizes the histogram of mean meta-analytic estimates $\mu$ of log OR and
heterogeneity $\tau$, and the second row of Figure 2 visualizes the histogram
of mean meta-analytic estimates $\mu$ of log HR and heterogeneity $\tau$. See
Figure 8 in Appendix B for the corresponding visualization of log RR and RD.

We considered two distributional forms for the mean effect size parameter
$\mu$: a Normal distribution centered at zero with free standard deviation
parameter, and a Student’s $t$-distribution centered at zero with free scale
and degrees of freedom parameters. Moreover, we considered three distributions
for the heterogeneity parameter $\tau$: a Half-Normal prior distribution
centered at zero with free standard deviation parameter, an Inverse-Gamma
distribution with free shape and scale parameters, and a Gamma distribution
with free scale and shape parameters. We used the MASS R package? to estimate
the free parameters via maximum likelihood.

Table 3: Candidate prior distributions for the $\delta$ and $\tau$ parameters as obtained from the training set. The Student’s $t$-distributions parameterized by location, scale, and degrees of freedom, the (half) normal distributions parameterized by mean and standard deviation, and the inverse-gamma and gamma distributions parameterized by shape and scale. See Figure 2. The gray-colored prior distribution corresponds to the Cohen $d$ to log OR transformed prior distribution from Bartoš and colleagues.? Binary outcomes as log OR |
---|---
$\mu\sim\text{Student-t}(0,0.78,5)$ | $\tau\sim\text{Inv-Gamma}(1.71,0.73)$
$\mu\sim\text{Normal}(0,0.81)$ | $\tau\sim\text{Normal}_{+}(0,0.62)$
$\mu\sim\text{Student-t}(0,0.45,2.38)$ | $\tau\sim\text{Inv-Gamma}(1.53,0.40)$
| $\tau\sim\text{Gamma}(1.99,0.25)$
Time to event outcomes as log HR |
$\mu\sim\text{Normal}(0,0.35)$ | $\tau\sim\text{Normal}_{+}(0,0.26)$
$\mu\sim\text{Student-t}(0,0.21,2.57)$ | $\tau\sim\text{Inv-Gamma}(1.80,0.21)$
| $\tau\sim\text{Gamma}(1.93,0.11)$

The estimated free parameters of the prior distributions of log OR and log HR
are summarized in Table 3. The first prior distribution for $\mu$ and $\tau$
of log OR, in gray-colored text, corresponds to the transformation of prior
distributions obtained for Cohen’s $d$ measured continuous effect sizes from
Bartoš and colleagues.? See Table 7 in Appendix B for a corresponding summary
of prior distributions of log RR and RD.

We observe a notable difference in the widths of the prior distributions
estimated on the different effect size measures obtained from the training
data set. log OR have clearly the widest prior distributions, log RR prior
distributions span approximately around a half of the log OR width, while
prior distributions of log HR and RD are much more narrowly concentrated
around zero. This is, of course, not surprising especially in the case of RD
as they are necessarily bounded to $[-1,1]$ range.

###  4.3 Assessing prior distributions on the test data set

Initial training data set: 48,253 comparisons (240,871 estimates)21,782
comparisons (206,079 estimates) Final training data set: log OR: 21,779
comparisons (205,855 estimates) log RR: 21,782 comparisons (206,079 estimates)
RD: 21,782 comparisons (206,079 estimates) Removing comparisons with fewer
than 3 estimates. Estimating Bayesian meta-analytic models.

Initial training data set: 784 comparisons (4,344 estimates)491 comparisons
(3,914 estimates)Final training data set: 481 comparisons (4,363 estimates)
Removing comparisons with fewer than 3 estimates. Estimating Bayesian meta-
analytic models.

Figure 3: Flowchart of the study selection procedure and data processing steps
for the test data sets. Binary outcomes (left) and time-to-event outcomes
(right).

Figure 3 visualizes the data processing steps applied to the test data set. In
contrast to the training data set, we included all comparisons with at least
three estimates. We specified Bayesian binomial-normal meta-analytic models
for log OR and Bayesian normal-normal meta-analytic models for log RR, RD, and
log HR. For each effect size, we created models corresponding to all
combinations of the considered prior distributions for effect size and
heterogeneity (including the null hypothesis models of no effect and no
heterogeneity). For instance, we estimated $4\times 5$ meta-analytic models
for log OR (a $\mu=0$ prior distribution assuming no effect + three informed
distributions for the $\mu$ parameter) $\times$ (a $\tau=0$ prior distribution
assuming no heterogeneity + four informed prior distributions for the $\tau$
parameter), as depicted in the first part of Table 3.

We implemented the binomial-normal model in the BiBMA function and we
estimated the normal-normal model via the NoBMA function in the RoBMA R
package.? The RoBMA R package provides implementations for a wide range of
highly modifiable Bayesian meta-analytic models estimated via MCMC using
JAGS,? computes marginal likelihoods via bridge sampling using the
bridgesampling R package,? ? and combines the models via Bayesian model-
averaging tools implemented in the BayesTools R package.? Note that only three
Bayesian binomial-normal meta-analytic models of log OR and ten normal-normal
meta-analytic models of log HR did not converge.

####  4.3.1 Predictive performance of the competing prior distributions

First, we investigated the predictive performance of the competing prior
distributions. We compared the specified prior distributions for each
parameter separately, averaging over the specified prior distributions for the
remaining parameter. For instance, when considering log OR and comparing the
predictive performance of the three competing prior distributions for the
$\mu$ parameter, for each specified prior distribution we averaged its
performance across all four prior distributions for the $\tau$ parameter.

Table 4: Ranking totals for each prior distribution in $\mathcal{H}_{1}^{r}$
based on the test set. The numbers indicate how many times a specific prior
distribution attained a specific posterior probability rank. Rank ‘1’
represents the best performance. The rankings reflect predictive adequacy that
is model-averaged across the possible prior distribution configurations of the
other parameter. Binary outcomes as log OR
---
| Rank |  |
Prior distribution | 1 | 2 | 3 | 4 | PrMP* | AV. PoMP**
Parameter $\delta$
$\text{Student-t}(0,0.78,5)$ | 1672 | 4761 | 15346 | — | 0.33 | 0.32
$\text{Normal}(0,0.81)$ | 5482 | 15947 | 350 | — | 0.33 | 0.33
$\text{Student-t}(0,0.45,2.38)$ | 14625 | 1071 | 6083 | — | 0.33 | 0.35
Parameter $\tau$
$\text{Inv-Gamma}(1.71,0.73)$ | 2011 | 1244 | 2857 | 15667 | 0.25 | 0.23
$\text{Normal}_{+}(0,0.62)$ | 2309 | 12680 | 6072 | 718 | 0.25 | 0.26
$\text{Inv-Gamma}(1.53,0.40)$ | 8915 | 2354 | 5598 | 4912 | 0.25 | 0.26
$\text{Gamma}(1.99,0.25)$ | 8544 | 5501 | 7252 | 482 | 0.25 | 0.26
Time to event outcomes as log HR
| Rank |  |
Prior distribution | 1 | 2 | 3 | 4 | PrMP* | AV. PoMP**
Parameter $\mu$
$\text{Normal}(0,0.35)$ | 214 | 267 | — | — | 0.50 | 0.49
$\text{Student-t}(0,0.21,2.57)$ | 267 | 214 | — | — | 0.50 | 0.51
Parameter $\tau$
$\text{Normal}_{+}(0,0.26)$ | 148 | 118 | 215 | — | 0.33 | 0.33
$\text{Inv-Gamma}(1.80,0.21)$ | 184 | 210 | 87 | — | 0.33 | 0.34
$\text{Gamma}(1.93,0.11)$ | 149 | 153 | 179 | — | 0.33 | 0.33
*Prior model probability
**Average posterior model probability

Table 4 summarizes the predictive performance of the competing prior
distributions for log OR and log HR in terms of ranks and the average
posterior model probability, the posterior model probabilities averaged across
comparisons, and Figure 4 visualizes the posterior model probabilities for
each prior configuration across the comparisons for log OR and log HR. See
Table 8 and Figure 9 in Appendix B for the corresponding summary of log RR and
RD.

|
---|---
Figure 4: Posterior model probability of the competing prior distributions for
log OR and log HR. For each comparison, the color gradient ranges from white
(low posterior probability) to dark red (high posterior probability). The
numbers in parentheses are the averaged posterior probabilities across all
comparisons.

Across all effect size measures, we find that the more informed Student’s
$t$-distribution for the mean effect size parameter $\mu$ outranks the less
informed normal prior distribution (most of the best ranks in Tables 4 and 8).
The dominance of the Student’s t-distribution is the most notable for binary
outcomes, especially RD, with a smaller difference in performance in the time-
to-event outcomes. The same patern is also visible from visualization of the
posterior model probabilities where the Student’s $t$-distribution is painted
in a sligly darker color signaling higher posterior model probabilities.

In the case of the heterogeneity parameter $\tau$ we find that the results are
a bit more mixed. The more informed Inverse-Gamma and Gamma distribution
attain more of the best but also the worst ranks, with an essentially
indistinguishable average posterior model probability. The similar performance
of Inverse-Gamma and Gamma distributions is not unexpected given their similar
shapes, with the Gamma distribution slightly less peaked at small values of
the heterogeneity parameter $\tau$ (Figures 2 and 8). Furthermore, we find
that the Half-Normal distribution outperforms the Gamma and Inverse-Gamma
distributions for RD, most likely due to the shorter tail in the range-
restricted effect size measure.

Finally, we find that the informed prior distributions for log OR transformed
from Cohen’s $d$ measured continuous outcomes slightly outperform the natively
estimated informed prior distributions. This might be due to inherent
differences between binary and continuous outcomes studies as well as the
larger training data set presented in the current study.

####  4.3.2 Predictive performance of model types

Second, we investigate the predictive performance of the competing model
types. Since all but the fixed effect null hypothesis model
$\mathcal{H}_{0}^{f}$ could use different (combinations of) prior
distributions, we averaged the predictive performance across all employed
prior distribution specifications. For instance, when considering the random
effects alternative hypothesis $\mathcal{H}_{1}^{r}$ for log OR, we averaged
the predictive performance across the twelve possible prior distribution
specifications (three prior distributions for the $\mu$ parameter and four
prior distributions for the $\tau$ parameter).

Table 5: Ranking totals for each model type in the test set. The numbers
indicate how many times a specific model type attained a specific posterior
probability rank. Rank ‘1’ represents the best performance. The rankings
reflect predictive adequacy that is model-averaged across the possible prior
distribution configurations. Binary outcomes as log OR
---
| Rank |  |
Model | 1 | 2 | 3 | 4 | PrMP* | AV. PoMP**
$\mathcal{H}_{0}^{f}$ | 8665 | 1704 | 1808 | 9599 | 0.25 | 0.22
$\mathcal{H}_{1}^{f}$ | 5287 | 4373 | 9715 | 2401 | 0.25 | 0.25
$\mathcal{H}_{0}^{r}$ | 2915 | 9576 | 8391 | 897 | 0.25 | 0.22
$\mathcal{H}_{1}^{r}$ | 4912 | 6126 | 1862 | 8879 | 0.25 | 0.31
Time to event outcomes as log HR
| Rank |  |
Model | 1 | 2 | 3 | 4 | PrMP* | AV. PoMP**
$\mathcal{H}_{0}^{f}$ | 101 | 36 | 51 | 293 | 0.25 | 0.13
$\mathcal{H}_{1}^{f}$ | 130 | 141 | 143 | 67 | 0.25 | 0.28
$\mathcal{H}_{0}^{r}$ | 81 | 134 | 249 | 17 | 0.25 | 0.21
$\mathcal{H}_{1}^{r}$ | 169 | 170 | 38 | 104 | 0.25 | 0.38
*Prior model probability
**Average posterior model probability
|
---|---
Figure 5: Posterior model probability of the competing prior model types for
log OR and log HR. For each comparison, the color gradient ranges from white
(low posterior probability) to dark red (high posterior probability). The
numbers in parentheses are the averaged posterior probabilities across all
comparisons.

Table 5 summarizes the predictive performance of the competing model types for
log OR and log HR in terms of ranks and the average posterior model
probability and Figure 5 visualizes the posterior model probabilities for each
model type across the comparisons. Across all effect size measures, we find
that the random effects alternative hypothesis $\mathcal{H}_{1}^{r}$ attains
the highest posterior model probability. However, the fixed effect null
hypothesis model $\mathcal{H}_{0}^{f}$ dominates in terms of the best rank in
all effect sizes measures derived from binary outcomes. The generally worst
performing model is the random effects null hypothesis $\mathcal{H}_{0}^{r}$
which receives the lowest posterior model probability in all effect size
measures but log HR. See Table 9 and Figure 10 in Appendix B for the
corresponding summary of log RR and RD.

####  4.3.3 Predictive performance of hypotheses

Third, we investigate the predictive performance of the competing hypotheses;
the presence vs absence of the effect ($\mathcal{H}_{1}$ vs $\mathcal{H}_{0}$)
and the random vs fixed effects models ($\mathcal{H}^{r}$ vs
$\mathcal{H}^{f}$). As previously, the hypotheses were composed from multiple
models and prior distributions, therefore, we averaged across the possible
combinations of prior distributions.

|
---|---
|
Figure 6: Inclusion Bayes factors in favor of the presence of a treatment
effect (left) and in favor of the presence of across-study heterogeneity
(right) in the test set. Binary outcomes (log OR; first row) and time-to-event
outcomes (log HR; second row). In log OR; for the presence of the effect, 121
log Bayes factors larger than 21 and 12 log Bayes factors lower than -3, and
for the presence of heterogeneity, 478 log Bayes factors larger than 21 and 34
log Bayes factors lower than $-3$ are not shown. In log HR; for the presence
of the effect, 9 log Bayes factors larger than 12, and for the presence of
heterogeneity, 2 log Bayes factors larger than 21 are not shown.

Figure 6 visualizes the distribution of inclusion Bayes factors for the effect
and heterogeneity for log OR and log HR. We find the usual skewed distribution
of Bayes factors, with the skew favoring inclusion of Bayes factors for the
presence of the effect and heterogeneity (as it is more difficult to obtain
evidence for absence). Nonetheless, in all effect size measures of binary
outcomes, we find that only the minority of comparisons yields with the
evidence in favor of the effect ($\text{BF}_{10}>1$: $47.0$% for log OR,
$45.1$% for log RR, and $40.1$% for RD) or heterogeneity ($\text{BF}_{rf}>1$:
$39.2$% for log OR, $34.6$% for log RR, and $36.3$% for RD). For log HR, the
reverse is true as $62.8$% of comparisons yield evidence in favor of the
effect and $55.9$% of comparisons yield evidence for heterogeneity. See Figure
11 in Appendix B for the corresponding visualization for log RR and RD.

###  4.4 Estimating prior distributions on the complete data

Finally, we used the complete data set to estimate empirical prior
distributions for the field in general and for the specific sub-disciplines.
We applied the same data processing steps as on the training data set,
resulting in 12,079 comparisons of binary outcomes (255,048 estimates) and 234
comparisons of time-to-event outcomes (3,831 estimates). After estimating the
frequentist meta-analytic models, we ended up with 11,964 log OR comparisons
(253,054 estimates), 11,862 log RR comparisons (250,331 estimates), 12,042 RD
comparisons (254,326 estimates), and 225 log HR comparisons (3,707 estimates),
which we used for constructing the data-driven general and subfield-specific
prior distributions.

For comparisons of binary outcomes, we estimate the general and subfield-
specific empirical prior distributions jointly using Bayesian hierarchical
estimation with weakly informative priors on the hyperparameters. Bayesian
hierarchical modeling allows us to shrink the estimated subfield parameter
values towards the grand mean, which is especially useful for subfields with
relatively little information and extreme values.? ? ? We implement the
hierarchical models using the rstan R package? that interfaces with the Stan
probabilistic modeling language.? We specify the models as such that all
field-specific parameters of the Student’s $t$-distributions (i.e.,
$\sigma_{i}$ and $\nu_{i}$) and the Inverse-Gamma distributions (i.e.,
$\alpha_{i}$ and $\beta_{i}$) are shrunk via a positive-only normal
distributions. For the populational parameters, we use positive-only Cauchy
distributions with scale one for the location of the Student’s
$t$-distribution ($\sigma\sim\text{Cauchy}_{+}(0,1)$), scale ten for the
degrees of freedom of the Student’s $t$-distribution
($\nu\sim\text{Cauchy}_{+}(0,10)$), and with scale one for the shape and scale
of the Inverse-Gamma distribution ($\alpha,\beta\sim\text{Cauchy}_{+}(0,1)$).

Due to the insufficient number of time-to-event comparisons to estimate prior
distributions for the subfields (i.e., 225 comparisons), we provide only a
general empirical prior distribution for log HR estimated via maximum
likelihood.

We estimate the more informed, and generally dominant, Student’s
$t$-distribution for the mean effect size parameter $\mu$ of all effect size
measures. For the heterogeneity parameter $\tau$ of log OR, log RR, and log HR
we estimate an Inverse-Gamma distribution (the performance is generally tied
with the Gamma distribution) and for RD we estimate the more dominant Half-
Normal prior distribution.

Table 6: General and subfield-specific prior distributions for log OR of individual topics from the Cochrane database of systematic reviews estimated by hierarchical regression based on the complete data set. The Student’s $t$-distributions parameterized by location, scale, and degrees of freedom and the Inverse-Gamma distribution parameterized by shape and scale. Topic | Comparisons $\mu$ | Comparisons $\tau$ | Prior $\mu$ | Prior $\tau$
---|---|---|---|---
Acute Respiratory Infections | 308 (5860) | 197 (4061) | Student-t(0, 0.48, 3) | Inv-Gamma(1.67, 0.45)
Airways | 450 (8702) | 207 (3839) | Student-t(0, 0.37, 2) | Inv-Gamma(1.35, 0.27)
Anaesthesia | 408 (9409) | 261 (6781) | Student-t(0, 0.79, 6) | Inv-Gamma(2.12, 0.86)
Back and Neck | 14 (295) | 10 (238) | Student-t(0, 0.62, 4) | Inv-Gamma(1.84, 0.68)
Bone, Joint and Muscle Trauma | 202 (3732) | 103 (1937) | Student-t(0, 0.44, 2) | Inv-Gamma(1.01, 0.36)
Breast Cancer | 137 (2646) | 113 (2296) | Student-t(0, 0.39, 3) | Inv-Gamma(1.59, 0.48)
Childhood Cancer | 1 (13) | — | Student-t(0, 0.47, 4) | —
Colorectal | 230 (4120) | 141 (2685) | Student-t(0, 0.65, 4) | Inv-Gamma(1.71, 0.60)
Common Mental Disorders | 649 (13322) | 422 (9731) | Student-t(0, 0.54, 4) | Inv-Gamma(1.63, 0.45)
Consumers and Communication | 44 (918) | 41 (875) | Student-t(0, 0.48, 4) | Inv-Gamma(2.37, 0.86)
Cystic Fibrosis and Genetic Disorders | 81 (1843) | 40 (964) | Student-t(0, 0.40, 3) | Inv-Gamma(1.82, 0.58)
Dementia and Cognitive Improvement | 127 (2097) | 78 (1278) | Student-t(0, 0.49, 4) | Inv-Gamma(1.28, 0.25)
Developmental, Psychosocial and Learning Problems | 106 (1457) | 79 (1083) | Student-t(0, 0.83, 5) | Inv-Gamma(1.83, 0.82)
Drugs and Alcohol | 103 (2027) | 73 (1545) | Student-t(0, 0.44, 4) | Inv-Gamma(1.59, 0.42)
Effective Practice and Organisation of Care | 59 (1106) | 44 (878) | Student-t(0, 0.51, 4) | Inv-Gamma(1.90, 0.68)
Emergency and Critical Care | 233 (4803) | 152 (3466) | Student-t(0, 0.35, 3) | Inv-Gamma(1.46, 0.34)
ENT | 17 (243) | 10 (136) | Student-t(0, 0.81, 4) | Inv-Gamma(1.73, 0.71)
Epilepsy | 114 (2052) | 39 (839) | Student-t(0, 0.88, 6) | Inv-Gamma(1.71, 0.43)
Eyes and Vision | 57 (1007) | 41 (815) | Student-t(0, 0.77, 5) | Inv-Gamma(2.09, 0.94)
Fertility Regulation | 62 (1072) | 46 (818) | Student-t(0, 0.46, 5) | Inv-Gamma(2.21, 0.71)
Gut | 314 (6056) | 215 (4595) | Student-t(0, 0.63, 5) | Inv-Gamma(1.94, 0.62)
Gynaecological, Neuro-oncology and Orphan Cancer | 245 (5811) | 159 (4123) | Student-t(0, 0.53, 4) | Inv-Gamma(1.80, 0.56)
Gynaecology and Fertility | 364 (6896) | 185 (3714) | Student-t(0, 0.40, 2) | Inv-Gamma(1.24, 0.28)
Haematology | 155 (4546) | 101 (3080) | Student-t(0, 0.57, 4) | Inv-Gamma(2.91, 0.66)
Heart | 979 (20760) | 579 (14085) | Student-t(0, 0.20, 2) | Inv-Gamma(1.64, 0.29)
Heart; Vascular | 8 (178) | 7 (166) | Student-t(0, 0.95, 4) | Inv-Gamma(1.64, 0.83)
Hepato-Biliary | 1042 (36665) | 706 (25668) | Student-t(0, 0.43, 3) | Inv-Gamma(1.58, 0.40)
HIV/AIDS | 32 (475) | 22 (293) | Student-t(0, 0.32, 4) | Inv-Gamma(1.76, 0.36)
Hypertension | 66 (1127) | 28 (453) | Student-t(0, 0.28, 5) | Inv-Gamma(1.25, 0.10)
Incontinence | 78 (1479) | 53 (1023) | Student-t(0, 0.75, 3) | Inv-Gamma(2.07, 1.09)
Infectious Diseases | 361 (6619) | 251 (4787) | Student-t(0, 0.66, 3) | Inv-Gamma(2.08, 0.86)
Injuries | 214 (5662) | 143 (4182) | Student-t(0, 0.60, 4) | Inv-Gamma(1.52, 0.49)
Kidney and Transplant | 479 (8828) | 274 (5226) | Student-t(0, 0.53, 4) | Inv-Gamma(1.68, 0.44)
Lung Cancer | 76 (1416) | 68 (1294) | Student-t(0, 0.61, 5) | Inv-Gamma(2.04, 0.68)
Metabolic and Endocrine Disorders | 130 (3092) | 78 (2190) | Student-t(0, 0.29, 2) | Inv-Gamma(0.92, 0.11)
Methodology | 74 (2098) | 69 (1997) | Student-t(0, 0.60, 5) | Inv-Gamma(2.04, 0.50)
Movement Disorders | 59 (1058) | 35 (708) | Student-t(0, 0.73, 5) | Inv-Gamma(2.14, 0.64)
Multiple Sclerosis and Rare Diseases of the CNS | 29 (564) | 23 (487) | Student-t(0, 0.76, 4) | Inv-Gamma(2.09, 0.71)
Musculoskeletal | 139 (2403) | 92 (1686) | Student-t(0, 0.59, 4) | Inv-Gamma(1.76, 0.59)
Neonatal | 337 (6327) | 153 (2738) | Student-t(0, 0.29, 3) | Inv-Gamma(1.80, 0.42)
Neuromuscular | 43 (739) | 20 (331) | Student-t(0, 0.70, 5) | Inv-Gamma(1.74, 0.49)
Oral Health | 63 (990) | 45 (724) | Student-t(0, 1.13, 4) | Inv-Gamma(1.85, 0.70)
Pain, Palliative and Supportive Care | 315 (6062) | 204 (4292) | Student-t(0, 1.00, 7) | Inv-Gamma(1.50, 0.40)
Pregnancy and Childbirth | 1307 (24397) | 832 (16588) | Student-t(0, 0.38, 2) | Inv-Gamma(1.73, 0.47)
Schizophrenia | 609 (13515) | 405 (9919) | Student-t(0, 0.58, 4) | Inv-Gamma(1.92, 0.69)
Sexually Transmitted Infections | 25 (318) | 19 (236) | Student-t(0, 0.61, 2) | Inv-Gamma(1.72, 0.47)
Skin | 142 (2309) | 76 (1294) | Student-t(0, 0.81, 2) | Inv-Gamma(1.64, 0.49)
Stroke | 299 (5310) | 131 (2357) | Student-t(0, 0.22, 2) | Inv-Gamma(1.36, 0.21)
Tobacco Addiction | 213 (4990) | 168 (4079) | Student-t(0, 0.49, 5) | Inv-Gamma(2.51, 0.63)
Urology | 89 (1673) | 60 (1271) | Student-t(0, 0.82, 5) | Inv-Gamma(1.72, 0.50)
Vascular | 187 (2590) | 116 (1711) | Student-t(0, 0.68, 5) | Inv-Gamma(1.70, 0.45)
Work | 14 (208) | 11 (178) | Student-t(0, 0.59, 4) | Inv-Gamma(1.79, 0.73)
Wounds | 75 (1169) | 53 (888) | Student-t(0, 0.60, 4) | Inv-Gamma(2.16, 0.86)
Pooled Estimate | 11964 (253054) | 7478 (170628) | Student-t(0, 0.58, 4) | Inv-Gamma(1.77, 0.55)

The resulting empirical prior distributions are summarized in Table 6 for log
OR, and in Table 10 and Table 11 in Appendix B for the log RR and RD. The
hierarchically pooled empirical prior distributions based on the complete data
sets are $\mu\sim\text{Student-t}(0,0.58,4)$ and $\tau\sim\text{Inv-
Gamma}(1.77,0.55)$ for log OR, $\mu\sim\text{Student-t}(0,0.32,3)$ and
$\tau\sim\text{Inv-Gamma}(1.51,0.23)$ for log RR, and
$\mu\sim\text{Student-t}(0,0.03,1)$ and $\tau\sim\text{Normal}_{+}(0,0.10)$
for RD. The maximum likelihood empirical prior distributions on the complete
data sets are $\mu\sim\text{Student-t}(0,0.13,2)$ and $\tau\sim\text{Inv-
Gamma}(2.42,0.30)$ for log HR.

We see a notable heterogeneity across topics. For instance, the prior
distribution for the meta-analytic mean $\mu$ in the “Oral Health” topic is as
much as six times wider that the prior distribution for the “Heart” topic.
These between-topic differences highlight the possibility for incorporating
domain specific information into the statistical inference.

##  5 Example: Adverse effects of honey in treating acute cough in children

We illustrate the methodology with an example from the field of acute
respiratory infections. Oduwole et al.? examined the effect of honey on
treating acute cought in children. While the main analyses focused on
reduction in Likert-scaled measured symptomatic relief of cough, finding
possible benefits of giving honey to children, we focus on the avdverse events
analysis (because it features logOR on zero cells). Specificaly, we re-examine
the comparison honey and no treatment on the presence of nervousness,
insomnia, hyperactivity. Oduwole et al.? found two elligible studies—with 5/35
and 2/40 events in the honey condition, and 0/39 and 0/40 events in the no
treatment conditions—and reported meta-analytic effect size estimate OR =
$9.40$, 95% CI [$1.16$, $76.20$], $z=2.10$, $p=0.04$ of honey on the presennce
of nervousness, insomnia, hyperactivity (“Analysis 3.5. Comparison 3 Adverse
events, Outcome 5 Honey versus no treatment”, p. 73).

We conducted a re-examination of the Oduwole et al. meta-analysis ? using the
binomial-normal model, which we implemented in the open-source statistical
software package JASP (jasp-stats.org) ? ? ? . For the same analysis in R ? ,
we utilized the RoBMA package and have included the details in Appendix A.

To perform a BMA meta-analysis using JASP, we loaded the data and activated
the “Meta-Analysis” module by clicking on the blue "+" button in the top right
corner. Then, we selected "Meta-Analysis" from the ribbon at the top, followed
by choosing "Bayesian Meta-Analysis (Binomial)" from the drop-down menu. In
the left input panel, we moved the observed number of events and the number of
participants in each group into the corresponding boxes and adjusted the prior
distributions under the “Prior” tab to match the “Acute Respiratory
Infections” subfield-specific prior distributions given in Table 6
($\delta\sim\mathcal{T}(0,0.48,3)$ and $\tau\sim\text{Inv-Gamma}(1.67,0.45)$).

Figure 7: JASP screenshot of a Bayesian model-averaged meta-analysis of the
Oduwole et al.? comparison concerning the everse events (nervousness,
insomnia, hyperactivity) of honey in comparison to no treatment in treating
acute cought in children. The left input panel shows the specification of the
“Acute Respiratory Infections” CDSR subfield-specific prior distributions for
logOR and heterogeneity $\tau$. The right output panel shows the corresponding
results.

Figure 7 shows the JASP graphical user interface. The left panel shows
settings for specifying the analysis settings and the right panel displaying
the analysis output. The JASP output panel displays the corresponding BMA
meta-analysis results. The “Model Summary” table summarizes the model-averaged
evidence for the presence vs. absence of the effect and heterogeneity. The
inclusion Bayes factors indicate weak evidence for the presence of the effect
$\text{BF}_{10}=2.64$ and virtualy no evidence for either the presence or
absence of heterogeneity $\text{BF}_{\text{rf}}=1.30$. The “Conditional
Estimates” table summarizes conditional the meta-analytic estimates (the
effect size estimate assuming presence of the effect and the heterogeneiry
estimate assuming presence of heterogeneity). We find the effect size estimate
$\text{OR}=4.25$, 95% [$0.80$, $18.20$] and heterogeneity estimate
$\tau_{\text{logOR}}=0.73$, 95% [$0.10$, $3.13$]. We can notice two
differences from the original frequentist output; 1) the effect size estimate
is smaller and the credible interval is much narrower then previously
reported, and 2) we are able to obtain a (very wide) estimate of the between-
study heterogeneity. Both are a result of incorporating the existing
information about the usual degrees of effect sizes in a given field which is
especially relevant in cases with a very few observations.

The JASP meta-analytic module also offers a multitude of visualizationoptions
(such a as forest plot, prior and posterior distributions, etc.) and
additional options and analyses such as performing one-sided hypothesis tests,
and various publication bias adjustment methods.? ?

##  6 Concluding Comments

In this article, we developed informed empirical prior distributions for the
meta-analytic mean and heterogeneity parameter for meta-analyses of binary
(log OR, log RR, and RD) and time-to-event (log HR) outcomes. We provided both
general and topic-specific prior distributions based on almost 12,000 binary
outcomes meta-analyses and general prior distributions based on 200 time-to-
event meta-analyses extracted from the Cochrane Database of Systematic
Reviews. Our results extend our previous work? where we developed empirical
prior distributions for continuous outcomes and the work of Pullenayegum? and
Turner and colleagues? who developed prior distributions for the heterogeneity
parameter of log OR based on an earlier version of the database.

The newly developed prior distributions can be combined with Bayesian model-
averaged meta-analysis to quantify the evidence in favor of or against the
presence of the mean meta-analytic effect and heterogeneity. Moreover,
Bayesian model-averaged meta-analysis does not force the analyst to base the
entire inference on a single model. Instead, Bayesian model-averaged meta-
analysis acknowledges the model uncertainty and combines the evidence and
parameter estimates from the competing models according to their predictive
performance.? ? Both the direct quantification of evidence and accounting for
model uncertainty is especially important in small sample sizes which are
typical for medical meta-analyses, as it allows the analyst to disentangle the
absence of evidence from the evidence of absence and avoid overconfident
conclusions.? ?

Our results illustrate substantial uncertainty about the most appropriate
meta-analytic model. No combination of the usual model types, the null vs the
alternative hypothesis model and the fixed vs the random effects model,
clearly dominated the other models across the examined meta-analyses.
Moreover, the meta-analyses of binary outcomes were more in line with the null
hypothesis and fixed effect models, while the opposite held true for meta-
analyses of time-to-event outcomes. Although these results go against the
common belief that the random-effects alternative model is the best-suited
model for analysing data,? ? ? ? the large uncertainly and considerable
support for all model types echo our previous findings from meta-analyses of
continuous outcomes.?

We implemented the Bayesian binomial-normal meta-analytic model for log OR
within the Bayesian model-averaged meta-analytic framework of the RoBMA R
package? (alongside the already existing normal-normal meta-analytic model).
We incorporated the framework and empirical prior distributions to into the
user-friendly graphical user interface of JASP.? ? Finally, we illustrated the
programs on and applied the empirical prior distributions to an example of
acute respiratory infections.

## Acknowledgments

This work was supported by The Netherlands Organisation for Scientific
Research (NWO) through a Research Talent grant (to QFG; 406.16.528), a Vici
grant (to EJW; 016.Vici.170.083), and an NWA Idea Generator grant (to WMO;
NWA.1228.191.045).

## Financial disclosure

None reported.

## Data availability statement

Data and analysis scripts are publicly available at: https://osf.io/v9bj6/.

## Conflict of interest

František Bartoš, Alexander Ly, and Eric-Jan Wagenmakers declare their
involvement in the open-source software package JASP (https://jasp-stats.org),
a non-commercial, publicly-funded effort to make Bayesian statistics
accessible to a broader group of researchers and students. Willem Otte is co-
founder of RCTAlert (https://rctalert.com), a commercial AI platform providing
weekly clinical trial notifications.

## Bibliography

  * [1] Friedman LM, Furberg CD, DeMets DL, Reboussin DM, Granger CB. Fundamentals of clinical trials. 4.  Springer .  2015\.
  * [2] Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis.  John Wiley & Sons .  2009\.
  * [3] Brockwell SE, Gordon IR. A comparison of statistical methods for meta-analysis. Statistics in Medicine 2001; 20(6): 825–840.
  * [4] Berkey CS, Hoaglin DC, Mosteller F, Colditz GA. A random-effects regression model for meta-analysis. Statistics in Medicine 1995; 14(4): 395–411.
  * [5] IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology 2014; 14(1): 1–12.
  * [6] Gonnermann A, Framke T, Großhennig A, Koch A. No solution yet for combining two independent studies in the presence of heterogeneity. Statistics in Medicine 2015; 34(16): 2476–2480.
  * [7] Bartoš F, Gronau QF, Timmers B, Otte WM, Ly A, Wagenmakers EJ. Bayesian model-averaged meta-analysis in medicine. Statistics in Medicine 2021; 40(30): 6743-6761.
  * [8] Gronau QF, Heck DW, Berkhout SW, Haaf JM, Wagenmakers EJ. A primer on Bayesian model-averaged meta-analysis. Advances in Methods and Practices in Psychological Science 2021; 4(3): 1–19.
  * [9] Gronau QF, Erp vS, Heck DW, Cesario J, Jonas KJ, Wagenmakers EJ. A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: The case of felt power. Comprehensive Results in Social Psychology 2017; 2(1): 123–138.
  * [10] Williams DR, Rast P, Bürkner PC. Bayesian meta-analysis with weakly informative prior distributions. Preprint at https://doi.org/10.31234/osf.io/7tbrm; 2018.
  * [11] Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172(1): 137–159.
  * [12] Chung Y, Rabe-Hesketh S, Choi IH. Avoiding zero between-study variance estimates in random-effects meta-analysis. Statistics in Medicine 2013; 32(23): 4071–4089.
  * [13] Rhodes KM, Turner RM, White IR, Jackson D, Spiegelhalter DJ, Higgins JP. Implementing informative priors for heterogeneity in meta-analysis using meta-regression and pseudo data. Statistics in Medicine 2016; 35(29): 5495–5511.
  * [14] Higgins JP, Whitehead A. Borrowing strength from external trials in a meta-analysis. Statistics in medicine 1996; 15(24): 2733–2749.
  * [15] Kass RE, Raftery AE. Bayes factors. Journal of the American Statistical Association 1995; 90(430): 773–795.
  * [16] Jeffreys H. Theory of probability.  Oxford, UK: Oxford University Press.  3rd edition ed. 1961.
  * [17] Hinne M, Gronau QF, Bergh v. dD, Wagenmakers EJ. A conceptual introduction to Bayesian model averaging. Advances in Methods and Practices in Psychological Science 2020; 3(2): 200–215.
  * [18] Pullenayegum BEM. An informed reference prior for between-study heterogeneity in meta-analyses of binary outcomes. Statistics in Medicine 2011; 30(26): 3082–3094.
  * [19] Turner RM, Jackson D, Wei Y, Thompson SG, Higgins JP. Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis. Statistics in Medicine 2015; 34(6): 984–998.
  * [20] R Core Team . R: A language and environment for statistical computing. R Foundation for Statistical Computing; Vienna, Austria: 2021.
  * [21] JASP Team . JASP (Version 0.17)[Computer software]. https://jasp-stats.org/; 2022.
  * [22] Ly A, van den Bergh D, Bartoš F, Wagenmakers EJ. Bayesian Inference With JASP. The ISBA Bulletin 2021; 28: 7–15.
  * [23] Holmberg MJ, Andersen LW. Estimating risk ratios and risk differences: alternatives to odds ratios. JAMA 2020; 324(11): 1098–1099.
  * [24] Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat—which of these should we use?. Value in Health 2002; 5(5): 431–436.
  * [25] Doi SA, Furuya-Kanamori L, Xu C, Lin L, Chivese T, Thalib L. Controversy and debate: Questionable utility of the relative risk in clinical research: Paper 1: A call for change to practice. Journal of Clinical Epidemiology 2020\.
  * [26] Plackett R. The continuity correction in 2$\times$ 2 tables. Biometrika 1964; 51(3-4): 327–337.
  * [27] Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute 1959; 22(4): 719–748.
  * [28] Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR. Practical methods for incorporating summary time-to-event data into meta-analysis. Trials 2007; 8(1): 1–16.
  * [29] Simmonds MC, Tierney J, Bowden J, Higgins JP. Meta-analysis of time-to-event data: A comparison of two-stage methods. Research Synthesis Methods 2011; 2(3): 139–149.
  * [30] J. Sweeting M, J. Sutton A, C. Lambert P. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23(9): 1351–1375.
  * [31] Günhan BK, Röver C, Friede T. Random-effects meta-analysis of few studies involving rare events. Research Synthesis Methods 2020; 11(1): 74–90.
  * [32] Smith TC, Spiegelhalter DJ, Thomas A. Bayesian approaches to random-effects meta-analysis: A comparative study. Statistics in Medicine 1995; 14(24): 2685–2699.
  * [33] Viechtbauer W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software 2010; 36(3): 1–48.
  * [34] Jackson D, Law M, Stijnen T, Viechtbauer W, White IR. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Statistics in Medicine 2018; 37(7): 1059–1085.
  * [35] Venables WN, Ripley BD. Modern Applied Statistics with S.  New York: Springer.  fourth ed. 2002.  ISBN 0-387-95457-0.
  * [36] Bartoš F, Maier M. RoBMA: An R package for robust Bayesian meta-analyses. https://CRAN.R-project.org/package=RoBMA; 2020.  R package version 3.0.0.
  * [37] Plummer M. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In: . 124. Vienna, Austria. ; 2003: 1–10.
  * [38] Gronau QF, Singmann H, Wagenmakers EJ. bridgesampling: An R package for estimating normalizing constants. Journal of Statistical Software 2020; 92(10): 1–29.
  * [39] Gronau QF, Sarafoglou A, Matzke D, et al. A tutorial on bridge sampling. Journal of Mathematical Psychology 2017; 81: 80–97.
  * [40] Bartoš F. BayesTools: Tools for Bayesian Analyses. https://CRAN.R-project.org/package=BayesTools; 2022.  R package version 0.13.4.
  * [41] Lee MD, Wagenmakers EJ. Bayesian statistical inference in psychology: Comment on Trafimow (2003). Psychological Review 2005; 112: 662–668.
  * [42] McElreath R. Statistical rethinking: A Bayesian course with examples in R and Stan.  CRC Press.  2 ed. 2020.
  * [43] Rouder JN, Lu J. An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review 2005; 12: 573–604.
  * [44] Stan Development Team . RStan: the R interface to Stan. http://mc-stan.org/3; 2018.  R package version 2.17.3.
  * [45] Carpenter B, Gelman A, Hoffman MD, et al. Stan: A probabilistic programming language. Journal of Statistical Software 2017; 76(1): 1–32.
  * [46] Oduwole O, Udoh EE, Oyo-Ita A, Meremikwu MM. Honey for acute cough in children. Cochrane Database of Systematic Reviews 2018(4).
  * [47] Doorn vJ, Bergh v. dD, Böhm U, et al. The JASP guidelines for conducting and reporting a Bayesian analysis. Psychonomic Bulletin & Review 2020: 1–14.
  * [48] Bartoš F, Maier M, Quintana D, Wagenmakers EJ. Adjusting for publication bias in JASP and R: Selection models, PET-PEESE, and robust Bayesian meta-analysis. Advances in Methods and Practices in Psychological Science 2022; 5(3): 1–19.
  * [49] Berkhout SW, Haaf JM, Gronau QF, Heck DW, Wagenmakers EJ. A tutorial on Bayesian model-averaged meta-analysis in JASP. Behavior Research Methods 2023\.
  * [50] Keysers C, Gazzola V, Wagenmakers EJ. Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence. Nature Neuroscience 2020; 23: 788–799.
  * [51] Robinson GK. What properties might statistical inferences reasonably be expected to have? – Crisis and resolution in statistical inference. The American Statistician 2019; 73: 243–252.
  * [52] National Research Council . Combining information: Statistical issues and opportunities for research.  Washington, DC: The National Academies Press .  1992
  * [53] Mosteller F, Colditz GA. Understanding research synthesis (meta-analysis). Annual Review of Public Health 1996; 17(1): 1–23.

##  Appendix A R Code for the Adverse effects of honey in treating acute cough
in children Example

This Appendix shows how to conduct the example analysis with the statistical
programming language R.? First, we need to install the RoBMA R package,? (this
command needs to be executed only if the package is not already installed):

    install.packages("RoBMA")

After the RoBMA package has been installed, we load it into the R session,

    library("RoBMA")

and specify the number of events and observations in both the experimental and
control group from both studies,

    events_experimental        <- c(5, 2)
    events_control             <- c(0, 0)
    observations_experimental  <- c(35, 40)
    observations_control       <- c(39, 40)
    study_names <- c("Paul 2007", "Shadkam 2010")

In order to use the binomial-normal BMA meta-analysis we use the BiBMA()
function. We specify the number of of events and observations in each group as
the corresponding input (x1, x2, n1, n2) and set the subfield-specific prior
distributions using the priors_effect and priors_heterogeneity arguments
accordingly to the “Acute Respiratory Infections” row in Table 6:

    fit <- BiBMA(
      x1 = events_experimental,
      x2 = events_control,
      n1 = observations_experimental,
      n2 = observations_control,
      study_names = study_names,
      priors_effect        = prior(distribution="t", parameters=list(location=0, scale=0.48, df=3)),
      priors_heterogeneity = prior(distribution="invgamma",  parameters=list(shape=1.67, scale=0.45)),
      seed = 1)

To obtain the inclusion Bayes factors and conditional posterior distribution
summaries, we use the summary function with the conditional = TRUE argument,

    summary(fit, conditional = TRUE, output_scale = "OR")

which produces output corresponds to that given by JASP (up to MCMC error):

    > summary(fit, conditional = TRUE, output_scale = "OR")
    Call:
    BiBMA(x1 = events_experimental, x2 = events_control, n1 = observations_experimental,
        n2 = observations_control, study_names = study_names, priors_effect = prior(distribution = "t",
            parameters = list(location = 0, scale = 0.48, df = 3)),
        priors_heterogeneity = prior(distribution = "invgamma", parameters = list(shape = 1.67,
            scale = 0.45)), seed = 1)

    Bayesian model-averaged meta-analysis (binomial-normal model)
    Components summary:
                  Models Prior prob. Post. prob. Inclusion BF
    Effect           2/4       0.500       0.725        2.630
    Heterogeneity    2/4       0.500       0.564        1.296

    Model-averaged estimates:
         Mean Median 0.025  0.975
    mu  3.389  1.642 0.842 15.143
    tau 0.420  0.158 0.000  2.594
    The effect size estimates are summarized on the OR scale and heterogeneity is summarized
    on the logOR scale (priors were specified on the log(OR) scale).

    Conditional estimates:
         Mean Median 0.025  0.975
    mu  4.242  2.261 0.781 17.613
    tau 0.747  0.426 0.097  3.233
    The effect size estimates are summarized on the OR scale and heterogeneity is summarized
    on the logOR scale (priors were specified on the log(OR) scale).

##  Appendix B Analysis of log RR and RD

This Appendix contains tables and figures for the log RR and RD.

|
---|---
|
Figure 8: Frequentist effect sizes estimates and candidate prior distributions
from the training data set. Binary outcomes as log RR (first row) and RD
(second row). Histogram and tick marks display the estimated effect size
estimates (left) and between-study standard deviation estimates (right),
whereas lines represent candidate prior distributions for the population
effect size parameter (left) and candidate prior distributions for the
population between-study standard deviation $\tau$ (right; see Table 3). 34
log RR outside of the $\pm 2$ range are not shown and 31 RD outside the $\pm
0.5$ range are not shown. 6 $\tau$ estimates larger than 1.5 and 2,401 $\tau$
estimates lower than 0.01 of log RR are not shown. 2,823 $\tau$ estimates
lower than 0.01 of RD are not shown. Table 7: Candidate prior distributions
for the $\delta$ and $\tau$ parameters as obtained from the training set. The
Student’s $t$-distributions parameterized by location, scale, and degrees of
freedom, the (half) normal distributions parameterized by mean and standard
deviation, and the inverse-gamma and gamma distributions parameterized by
shape and scale. See Figure 8. Binary outcomes as log RR
---
$\mu\sim\mathcal{N}(0,0.49)$ | $\tau\sim\mathcal{N}_{+}(0,0.35)$
$\mu\sim\mathcal{T}(0,0.26,2.28)$ | $\tau\sim\text{Inv-Gamma}(1.51,0.23)$
| $\tau\sim\text{Gamma}(1.96,0.14)$
Binary outcomes as RD
$\mu\sim\mathcal{N}(0,0.10)$ | $\tau\sim\mathcal{N}_{+}(0,0.10)$
$\mu\sim\mathcal{T}(0,0.02,0.85)$ | $\tau\sim\text{Inv-Gamma}(1.68,0.07)$
| $\tau\sim\text{Gamma}(1.80,0.04)$
Table 8: Ranking totals for each prior distribution in $\mathcal{H}_{1}^{r}$
based on the test set. The numbers indicate how many times a specific prior
distribution attained a specific posterior probability rank. Rank ‘1’
represents the best performance. The rankings reflect predictive adequacy that
is model-averaged across the possible prior distribution configurations of the
other parameter. Binary outcomes as log RR
---
| Rank |  |
Prior distribution | 1 | 2 | 3 | PrMP* | AV. PoMP**
Parameter $\mu$
$\text{Normal}(0,0.49)$ | 6817 | 14965 | — | 0.50 | 0.48
$\text{Student-t}(0,0.26,2.28)$ | 14965 | 6817 | — | 0.50 | 0.52
Parameter $\tau$
$\text{Normal}_{+}(0,0.35)$ | 4311 | 12760 | 4711 | 0.33 | 0.34
$\text{Inv-Gamma}(1.51,0.23)$ | 8151 | 1965 | 11666 | 0.33 | 0.33
$\text{Gamma}(1.96,0.14)$ | 9320 | 7057 | 5405 | 0.33 | 0.33
Time to event outcomes as RD
| Rank |  |
Prior distribution | 1 | 2 | 3 | PrMP* | AV. PoMP**
Parameter $\mu$
$\text{Normal}(0,0.10)$ | 6403 | 15379 | — | 0.50 | 0.42
$\text{Student-t}(0,0.02,0.85)$ | 15379 | 6403 | — | 0.50 | 0.58
Parameter $\tau$
$\text{Normal}_{+}(0,0.10)$ | 8355 | 6987 | 6440 | 0.33 | 0.37
$\text{InvGamma}(1.68,0.07)$ | 10340 | 2954 | 8488 | 0.33 | 0.31
$\text{Gamma}(1.8,23.32)$ | 3087 | 11841 | 6854 | 0.33 | 0.32
*Prior model probability
**Average posterior model probability
|
---|---
Figure 9: Posterior model probability of the competing prior distributions for log RR and RD. For each comparison, the color gradient ranges from white (low posterior probability) to dark red (high posterior probability). The numbers in parentheses are the averaged posterior probabilities across all comparisons. Table 9: Ranking totals for each model type in the test set. The numbers indicate how many times a specific model type attained a specific posterior probability rank. Rank ‘1’ represents the best performance. The rankings reflect predictive adequacy that is model-averaged across the possible prior distribution configurations. Binary outcomes as log RR |
---|---
| Rank |  |
Model | 1 | 2 | 3 | 4 | PrMP* | AV. PoMP**
$\mathcal{H}_{0}^{f}$ | 9604 | 1650 | 1776 | 8748 | 0.25 | 0.24
$\mathcal{H}_{1}^{f}$ | 5557 | 4305 | 9815 | 2101 | 0.25 | 0.26
$\mathcal{H}_{0}^{r}$ | 2348 | 9882 | 8292 | 1260 | 0.25 | 0.21
$\mathcal{H}_{1}^{r}$ | 4273 | 5945 | 1895 | 9669 | 0.25 | 0.29
Binary outcomes as RD |
| Rank |  |
Model | 1 | 2 | 3 | 4 | PrMP* | AV. PoMP**
$\mathcal{H}_{0}^{f}$ | 10644 | 1449 | 1632 | 8027 | 0.25 | 0.33
$\mathcal{H}_{1}^{f}$ | 3683 | 10165 | 5924 | 1980 | 0.25 | 0.22
$\mathcal{H}_{0}^{r}$ | 2385 | 5892 | 12774 | 731 | 0.25 | 0.18
$\mathcal{H}_{1}^{r}$ | 5070 | 4276 | 1422 | 11014 | 0.25 | 0.27
*Prior model probability
**Average posterior model probability
|
---|---
Figure 10: Posterior model probability of the competing prior model types for log OR and log HR. For each comparison, the color gradient ranges from white (low posterior probability) to dark red (high posterior probability). The numbers in parentheses are the averaged posterior probabilities across all comparisons. |
---|---
|
Figure 11: Inclusion Bayes factors in favor of the presence of a treatment effect (left) and in favor of the presence of across-study heterogeneity (right) in the test set. log RR (first row) and RD (second row). In log RR; for the presence of the effect, 140 log Bayes factors larger than 21 and 91 log Bayes factors lower than -3, and for the presence of heterogeneity, 387 log Bayes factors larger than 21 and 83 log Bayes factors lower than -3 are not shown. In RD; for the presence of the effect, 123 log Bayes factors larger than 21 and 386 log Bayes factors lower than -3, and for the presence of heterogeneity, 902 log Bayes factors larger than 21 and 41 log Bayes factors lower than -6 are not shown. Table 10: General and subfield-specific prior distributions for log RR of individual topics from the Cochrane database of systematic reviews estimated by hierarchical regression based on the complete data set. The Student’s $t$-distribution parameterized by location, scale, and degrees of freedom and the Inverse-Gamma distribution parameterized by shape and scale. Topic | Comparisons $\mu$ | Comparisons $\tau$ | Prior $\mu$ | Prior $\tau$
---|---|---|---|---
Acute Respiratory Infections | 299 (5721) | 178 (3806) | Student-t(0, 0.27, 3) | Inv-Gamma(1.58, 0.25)
Airways | 449 (8806) | 207 (3723) | Student-t(0, 0.25, 3) | Inv-Gamma(1.36, 0.15)
Anaesthesia | 409 (9471) | 231 (6158) | Student-t(0, 0.45, 5) | Inv-Gamma(1.21, 0.21)
Back and Neck | 14 (295) | 8 (186) | Student-t(0, 0.34, 3) | Inv-Gamma(1.52, 0.22)
Bone, Joint and Muscle Trauma | 198 (3616) | 86 (1455) | Student-t(0, 0.26, 2) | Inv-Gamma(0.97, 0.14)
Breast Cancer | 136 (2622) | 115 (2291) | Student-t(0, 0.21, 3) | Inv-Gamma(1.49, 0.24)
Childhood Cancer | 1 (13) | — | Student-t(0, 0.28, 3) | —
Colorectal | 235 (4157) | 133 (2523) | Student-t(0, 0.38, 4) | Inv-Gamma(1.42, 0.27)
Common Mental Disorders | 640 (13175) | 425 (9704) | Student-t(0, 0.29, 3) | Inv-Gamma(1.43, 0.20)
Consumers and Communication | 44 (918) | 40 (865) | Student-t(0, 0.13, 2) | Inv-Gamma(1.48, 0.14)
Cystic Fibrosis and Genetic Disorders | 76 (1692) | 31 (736) | Student-t(0, 0.21, 2) | Inv-Gamma(1.71, 0.24)
Dementia and Cognitive Improvement | 124 (2047) | 72 (1199) | Student-t(0, 0.29, 3) | Inv-Gamma(1.53, 0.19)
Developmental, Psychosocial and Learning Problems | 107 (1467) | 85 (1162) | Student-t(0, 0.39, 3) | Inv-Gamma(0.97, 0.11)
Drugs and Alcohol | 105 (2019) | 66 (1376) | Student-t(0, 0.22, 4) | Inv-Gamma(1.58, 0.20)
Effective Practice and Organisation of Care | 58 (1093) | 46 (904) | Student-t(0, 0.30, 4) | Inv-Gamma(1.40, 0.24)
Emergency and Critical Care | 232 (4776) | 145 (3438) | Student-t(0, 0.19, 2) | Inv-Gamma(1.65, 0.25)
ENT | 16 (233) | 5 (63) | Student-t(0, 0.40, 4) | Inv-Gamma(1.50, 0.24)
Epilepsy | 116 (2064) | 49 (1090) | Student-t(0, 0.58, 5) | Inv-Gamma(1.74, 0.27)
Eyes and Vision | 57 (984) | 38 (719) | Student-t(0, 0.34, 4) | Inv-Gamma(1.60, 0.26)
Fertility Regulation | 60 (1032) | 42 (744) | Student-t(0, 0.34, 4) | Inv-Gamma(1.59, 0.31)
Gut | 308 (5941) | 202 (4348) | Student-t(0, 0.40, 4) | Inv-Gamma(1.69, 0.29)
Gynaecological, Neuro-oncology and Orphan Cancer | 242 (5784) | 154 (3988) | Student-t(0, 0.35, 4) | Inv-Gamma(1.43, 0.25)
Gynaecology and Fertility | 373 (6967) | 181 (3666) | Student-t(0, 0.25, 2) | Inv-Gamma(1.29, 0.16)
Haematology | 153 (4472) | 95 (2958) | Student-t(0, 0.31, 3) | Inv-Gamma(1.22, 0.11)
Heart | 950 (19836) | 604 (13252) | Student-t(0, 0.17, 3) | Inv-Gamma(1.96, 0.24)
Heart; Vascular | 8 (178) | 7 (166) | Student-t(0, 0.39, 3) | Inv-Gamma(1.62, 0.25)
Hepato-Biliary | 1037 (36554) | 578 (21332) | Student-t(0, 0.23, 3) | Inv-Gamma(1.44, 0.19)
HIV/AIDS | 33 (486) | 17 (228) | Student-t(0, 0.21, 3) | Inv-Gamma(1.66, 0.23)
Hypertension | 66 (1127) | 33 (515) | Student-t(0, 0.25, 4) | Inv-Gamma(1.48, 0.11)
Incontinence | 78 (1479) | 54 (1020) | Student-t(0, 0.39, 3) | Inv-Gamma(1.53, 0.33)
Infectious Diseases | 358 (6490) | 254 (4813) | Student-t(0, 0.41, 3) | Inv-Gamma(1.30, 0.27)
Injuries | 214 (5639) | 138 (3839) | Student-t(0, 0.27, 4) | Inv-Gamma(1.46, 0.19)
Kidney and Transplant | 474 (8737) | 275 (5205) | Student-t(0, 0.35, 4) | Inv-Gamma(1.51, 0.23)
Lung Cancer | 75 (1402) | 64 (1131) | Student-t(0, 0.38, 4) | Inv-Gamma(1.49, 0.26)
Metabolic and Endocrine Disorders | 131 (3233) | 72 (1768) | Student-t(0, 0.13, 1) | Inv-Gamma(1.55, 0.19)
Methodology | 74 (2098) | 73 (2084) | Student-t(0, 0.27, 5) | Inv-Gamma(1.74, 0.23)
Movement Disorders | 58 (1042) | 40 (765) | Student-t(0, 0.46, 4) | Inv-Gamma(1.54, 0.24)
Multiple Sclerosis and Rare Diseases of the CNS | 28 (544) | 22 (467) | Student-t(0, 0.44, 3) | Inv-Gamma(1.58, 0.35)
Musculoskeletal | 138 (2393) | 84 (1541) | Student-t(0, 0.33, 3) | Inv-Gamma(1.54, 0.31)
Neonatal | 335 (6207) | 151 (2517) | Student-t(0, 0.18, 3) | Inv-Gamma(1.89, 0.30)
Neuromuscular | 40 (647) | 18 (273) | Student-t(0, 0.41, 4) | Inv-Gamma(1.44, 0.15)
Oral Health | 66 (1032) | 32 (465) | Student-t(0, 0.55, 4) | Inv-Gamma(1.50, 0.37)
Pain, Palliative and Supportive Care | 310 (5965) | 212 (4298) | Student-t(0, 0.59, 5) | Inv-Gamma(1.98, 0.39)
Pregnancy and Childbirth | 1300 (24223) | 813 (16138) | Student-t(0, 0.24, 2) | Inv-Gamma(1.40, 0.21)
Schizophrenia | 601 (13147) | 347 (8290) | Student-t(0, 0.33, 3) | Inv-Gamma(1.47, 0.26)
Sexually Transmitted Infections | 25 (318) | 18 (225) | Student-t(0, 0.39, 3) | Inv-Gamma(1.43, 0.21)
Skin | 142 (2285) | 85 (1434) | Student-t(0, 0.41, 1) | Inv-Gamma(1.49, 0.23)
Stroke | 301 (5363) | 149 (2476) | Student-t(0, 0.13, 2) | Inv-Gamma(1.52, 0.15)
Tobacco Addiction | 215 (5018) | 174 (4276) | Student-t(0, 0.36, 5) | Inv-Gamma(2.02, 0.36)
Urology | 89 (1694) | 62 (1309) | Student-t(0, 0.44, 3) | Inv-Gamma(1.37, 0.18)
Vascular | 176 (2473) | 96 (1383) | Student-t(0, 0.43, 5) | Inv-Gamma(1.26, 0.19)
Work | 14 (208) | 10 (166) | Student-t(0, 0.23, 3) | Inv-Gamma(1.50, 0.26)
Wounds | 74 (1148) | 43 (688) | Student-t(0, 0.37, 4) | Inv-Gamma(1.63, 0.38)
Pooled Estimate | 11862 (250331) | 7159 (159166) | Student-t(0, 0.32, 3) | Inv-Gamma(1.51, 0.23)
Table 11: General and subfield-specific prior distributions for RD of individual topics from the Cochrane database of systematic reviews estimated by hierarchical regression based on the complete data set. The Student’s $t$-distribution parameterized by location, scale, and degrees of freedom and the Half-Normal distribution parameterized by mean and standard deviation parametrization. Topic | Comparisons $\mu$ | Comparisons $\tau$ | Prior $\mu$ | Prior $\tau$
---|---|---|---|---
Acute Respiratory Infections | 308 (5857) | 152 (3184) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.10)
Airways | 458 (8950) | 175 (3107) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.10)
Anaesthesia | 413 (9585) | 234 (5983) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.11)
Back and Neck | 14 (295) | 10 (212) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.10)
Bone, Joint and Muscle Trauma | 206 (3781) | 86 (1476) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.12)
Breast Cancer | 137 (2646) | 105 (2143) | Student-t(0, 0.05, 2) |  $\text{Normal}_{+}$(0, 0.12)
Childhood Cancer | 1 (13) | — | Student-t(0, 0.02, 1) | —
Colorectal | 238 (4266) | 132 (2417) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.10)
Common Mental Disorders | 650 (13334) | 455 (9880) | Student-t(0, 0.05, 1) |  $\text{Normal}_{+}$(0, 0.11)
Consumers and Communication | 44 (918) | 42 (895) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.10)
Cystic Fibrosis and Genetic Disorders | 82 (1858) | 31 (786) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.09)
Dementia and Cognitive Improvement | 126 (2086) | 67 (1038) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.09)
Developmental, Psychosocial and Learning Problems | 107 (1467) | 96 (1322) | Student-t(0, 0.08, 2) |  $\text{Normal}_{+}$(0, 0.13)
Drugs and Alcohol | 107 (2065) | 70 (1445) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.10)
Effective Practice and Organisation of Care | 59 (1106) | 45 (892) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.11)
Emergency and Critical Care | 233 (4803) | 144 (3207) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.07)
ENT | 17 (243) | 9 (116) | Student-t(0, 0.04, 1) |  $\text{Normal}_{+}$(0, 0.12)
Epilepsy | 117 (2085) | 67 (1355) | Student-t(0, 0.05, 2) |  $\text{Normal}_{+}$(0, 0.09)
Eyes and Vision | 58 (1017) | 38 (666) | Student-t(0, 0.04, 1) |  $\text{Normal}_{+}$(0, 0.11)
Fertility Regulation | 62 (1072) | 34 (562) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.08)
Gut | 313 (6032) | 193 (3834) | Student-t(0, 0.05, 2) |  $\text{Normal}_{+}$(0, 0.09)
Gynaecological, Neuro-oncology and Orphan Cancer | 244 (5799) | 152 (3988) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.10)
Gynaecology and Fertility | 375 (7017) | 204 (3929) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.09)
Haematology | 155 (4546) | 82 (2740) | Student-t(0, 0.05, 1) |  $\text{Normal}_{+}$(0, 0.09)
Heart | 974 (20489) | 343 (6737) | Student-t(0, 0.00, 1) |  $\text{Normal}_{+}$(0, 0.07)
Heart; Vascular | 8 (178) | 8 (178) | Student-t(0, 0.08, 1) |  $\text{Normal}_{+}$(0, 0.12)
Hepato-Biliary | 1051 (36811) | 498 (14308) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.09)
HIV/AIDS | 34 (501) | 17 (227) | Student-t(0, 0.02, 2) |  $\text{Normal}_{+}$(0, 0.10)
Hypertension | 66 (1127) | 13 (218) | Student-t(0, 0.01, 2) |  $\text{Normal}_{+}$(0, 0.06)
Incontinence | 78 (1479) | 61 (1183) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.12)
Infectious Diseases | 361 (6614) | 228 (4141) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.10)
Injuries | 217 (5692) | 128 (3529) | Student-t(0, 0.01, 0) |  $\text{Normal}_{+}$(0, 0.14)
Kidney and Transplant | 481 (8837) | 281 (5436) | Student-t(0, 0.03, 1) |  $\text{Normal}_{+}$(0, 0.09)
Lung Cancer | 76 (1416) | 65 (1284) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.12)
Metabolic and Endocrine Disorders | 131 (3251) | 56 (1277) | Student-t(0, 0.00, 0) |  $\text{Normal}_{+}$(0, 0.09)
Methodology | 74 (2098) | 73 (2084) | Student-t(0, 0.10, 2) |  $\text{Normal}_{+}$(0, 0.10)
Movement Disorders | 59 (1058) | 48 (906) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.07)
Multiple Sclerosis and Rare Diseases of the CNS | 29 (564) | 24 (484) | Student-t(0, 0.07, 2) |  $\text{Normal}_{+}$(0, 0.10)
Musculoskeletal | 139 (2403) | 80 (1269) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.11)
Neonatal | 339 (6383) | 163 (2667) | Student-t(0, 0.02, 1) |  $\text{Normal}_{+}$(0, 0.09)
Neuromuscular | 43 (739) | 23 (395) | Student-t(0, 0.03, 1) |  $\text{Normal}_{+}$(0, 0.08)
Oral Health | 68 (1055) | 39 (541) | Student-t(0, 0.03, 1) |  $\text{Normal}_{+}$(0, 0.12)
Pain, Palliative and Supportive Care | 312 (6018) | 229 (4148) | Student-t(0, 0.11, 2) |  $\text{Normal}_{+}$(0, 0.11)
Pregnancy and Childbirth | 1310 (24437) | 724 (14321) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.08)
Schizophrenia | 614 (13574) | 400 (9463) | Student-t(0, 0.03, 1) |  $\text{Normal}_{+}$(0, 0.10)
Sexually Transmitted Infections | 25 (318) | 16 (199) | Student-t(0, 0.04, 1) |  $\text{Normal}_{+}$(0, 0.12)
Skin | 145 (2347) | 99 (1580) | Student-t(0, 0.03, 1) |  $\text{Normal}_{+}$(0, 0.12)
Stroke | 303 (5390) | 132 (2230) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.06)
Tobacco Addiction | 216 (5031) | 174 (4324) | Student-t(0, 0.04, 2) |  $\text{Normal}_{+}$(0, 0.07)
Urology | 91 (1728) | 67 (1357) | Student-t(0, 0.04, 1) |  $\text{Normal}_{+}$(0, 0.10)
Vascular | 185 (2570) | 85 (1224) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.12)
Work | 14 (208) | 9 (144) | Student-t(0, 0.06, 2) |  $\text{Normal}_{+}$(0, 0.12)
Wounds | 75 (1169) | 43 (702) | Student-t(0, 0.01, 1) |  $\text{Normal}_{+}$(0, 0.09)
Pooled Estimate | 12042 (254326) | 6749 (141733) | Student-t(0, 0.03, 1) |  $\text{Normal}_{+}$(0, 0.10)