SuppMaterial

#  Assessing replicability with the sceptical $p$-value: Type-I error control
and sample size planning

Charlotte Micheloud1111Corresponding author<EMAIL_ADDRESS>1University of Zurich
Epidemiology, Biostatistics and Prevention Institute (EBPI)
and Center for Reproducible Science (CRS)
Hirschengraben 84, 8001 Zurich, Switzerland
and
2ETH Zurich
Seminar für Statistik
Rämistrasse 101, 8092 Zürich, Switzerland Fadoua Balabdaoui2 and Leonhard
Held1 1University of Zurich
Epidemiology, Biostatistics and Prevention Institute (EBPI)
and Center for Reproducible Science (CRS)
Hirschengraben 84, 8001 Zurich, Switzerland
and
2ETH Zurich
Seminar für Statistik
Rämistrasse 101, 8092 Zürich, Switzerland

(27th August 2024)

Abstract: We study a statistical framework for replicability based on a
recently proposed quantitative measure of replication success, the sceptical
$p$-value. A recalibration is proposed to obtain exact overall Type-I error
control if the effect is null in both studies and additional bounds on the
partial and conditional Type-I error rate, which represent the case where only
one study has a null effect. The approach avoids the double dichotomization
for significance of the two-trials rule and has larger project power to detect
existing effects over both studies in combination. It can also be used for
power calculations and requires a smaller replication sample size than the
two-trials rule for already convincing original studies. We illustrate the
performance of the proposed methodology in an application to data from the
Experimental Economics Replication Project.

Key Words: Design of replication studies; Power calculations; Replicability;
Sceptical $p$-value; Two-trials rule; Type-I error control

##  1 Introduction

Replication plays a key role to build confidence in the scientific merit of
published results. The so-called replication crisis has led to increased
interest in replication studies over the last decade (Royal Netherlands
Academy of Arts and Science, , 2018; National Academies of Sciences,
Engineering, and Medicine, , 2019) with large-scale replication projects being
conducted in various fields (Open Science Collaboration, , 2015; Camerer et
al., , 2016, 2018; Errington et al., , 2021). Deciding whether a replication
is successful is, however, not a straightforward task, and different
statistical methods are currently being used. For example, the Reproducibility
Project: Cancer Biology (Errington et al., , 2021), an 8-year effort to
replicate experiments from high-impact cancer biology papers, has used no less
than seven different methods to assess replicability, including significance
of both the original and replication study, compatibility of the original and
replication effect estimates, and computation of a meta-analytic combined
effect estimate with confidence interval.

Declaring a replication as successful if both the original and replication
study are significant at level $\alpha$ (usually one-sided 0.025) is known as
the two-trials rule in drug development and serves as a useful benchmark.
Specifically, it has an overall Type-I error (T1E) rate of $\alpha^{2}$ (Senn,
, 2007) if the effect is null in both studies and the partial T1E rate, the
risk of a false claim of replication success if a true effect is present in at
most one of the two studies, is bounded by $\alpha$ (Heller et al., , 2014).
However, the “double dichotomization” of the two-trials rule has serious
limitations. Firstly, it is common practice to replicate interesting findings
even if the original study does not pass the much-critized “bright-line”
threshold of $\alpha=0.025$. For example, in the Open Science Collaboration,
(2015) Psychology replication project, four studies have been included despite
“falling a bit short of the [two-sided] 0.05 criterion – $P$ = 0.0508, 0.0514,
0.0516, and 0.0567 – but all of these were interpreted as positive effects”.
Similarly, the Experimental Economics Replication Project (Camerer et al., ,
2016) has chosen to replicate 18 studies, two of which have not been
significant at the conventional two-sided 0.05 standard. Strict application of
the two-trials rule, however, would make it pointless to try to replicate such
non-significant original studies. Secondly, the two-trials rule has been shown
to have relatively low project power, i. e. power to detect existing effects
over both studies in combination (Maca et al., , 2002; Held, 2020b, ). These
issues suggest investigating alternative methods to assess replication
success.

A recent proposal by Held, 2020a combines a reverse-Bayes approach (see Held
et al., 2022a, , for a recent review) with a prior-predictive check for
conflict (Box, , 1980) and gives rise to a quantitative measure of replication
success, the sceptical $p$-value. The sceptical $p$-value depends on the two
study-specific $p$-values, but also on the ratio of the variances of the
original and replication effect estimates. The method treats the original and
replication study not as exchangeable and specifically penalizes shrinkage of
the replication effect estimate, compared to the original one. The effect size
perspective has been further explored to propose a modification based on the
golden ratio (Held et al., 2022b, ), in the following called the golden
sceptical $p$-value. While significance of both studies is a necessary but not
sufficient success criterion in the original formulation, the golden sceptical
$p$-value also allows original studies with a “trend to significance” to be
successful at replication, but only if the effect estimate at replication is
larger than at original.

The golden sceptical $p$-value addresses some of the problems of the two-
trials rule. It can flag replication success if the original or replication
$p$-value does not meet the significance threshold $\alpha$ and provides
larger project power than the two-trials rule. However, the probability for
replication success if the observed original effect estimate is the true
effect while being non-significant is always smaller than 50%. This is less
extreme than the two-trials rule where non-significant original studies can
never lead to replication success, but precludes sample size planning for
replication studies of non-significant original findings at commonly used
power values such as 80% or 90%. Furthermore, neither the original (nominal)
nor the golden sceptical $p$-value has an exact overall T1E rate of
$\alpha^{2}$ if the effect is null in both studies. The T1E rate of the
nominal one is always below $\alpha^{2}$, whereas the T1E rate of the golden
one can exceed $\alpha^{2}$ if the variance ratio (original to replication) is
smaller than one. An alternative reverse-Bayes approach based on Bayes factors
has also been proposed, but the resulting sceptical Bayes factor can also not
be used for sample size planning if the original result was not convincing on
its own (Pawel and Held, , 2022, Section 3.3).

In this paper we study the sceptical $p$-value from a frequentist perspective
and examine its T1E rate in greater detail. We aim to control the overall T1E
rate rather than the partial T1E rate to allow for a fair comparison with the
two-trials rule (Rosenkranz, , 2023). Any other method with partial T1E rate
$<\alpha$ (such as the nominal sceptical $p$-value) will have a smaller
overall T1E rate than the two-trials rule, and its success region will be a
subset of the success region of the two-trials rule. The ultimate goal is
hence to recalibrate the sceptical $p$-value to achieve exact overall T1E
control at level $\alpha^{2}$ and to enable sample size calculations also for
non-significant original studies. However, any method with the same overall
T1E rate as the two-trials rule will have an increased partial T1E rate. This
is also the case for the sceptical $p$-value, but we will show that the
increase in conditional T1E rate, the risk of a false claim of success if the
replication study is properly powered based on the original result, is always
below $2\alpha=5$%, which is considered a ‘sensible option’ by Rosenkranz,
(2002).

In Section 2 we describe the underlying statistical framework for
replicability and consider T1E rates under two different null hypotheses, the
intersection and the union null (Heller et al., , 2014) in Section 2.1. The
two-trials rule (Section 2.2) and the harmonic mean $\chi^{2}$-test (Section
2.3) are identified as special cases of this framework with exact overall T1E
control of $\alpha^{2}$ under the intersection null. The relevant null
distribution is then derived in all other cases and the sceptical $p$-value is
recalibrated in Section 2.4 to achieve exact overall T1E control for every
possible value of the variance ratio. Limiting cases and further properties
are described in Sections 2.5 and 2.6. In Section 3, the sceptical $p$-value
is used as a dichotomous criterion for replication success with focus on the
partial T1E rate under the union null hypothesis in Section 3.1. The sceptical
$p$-value and the two-trials rule are then compared in terms of success
regions (Section 3.2), project power (Section 3.3) and for the design of
replication studies, with particular focus on the conditional T1E rate
(Section 3.4). An application to data from the Experimental Economics
Replication Project is given in Section 4. We close with some discussion in
Section 5.

##  2 A statistical framework for replicability

Let $\hat{\theta}_{i}$ denote the estimate of the unknown effect size
$\theta_{i}$ and $\sigma_{i}$ the corresponding standard error from the
original and replication study, $i\in\\{o,r\\}$. As in standard meta-analysis
we assume that the $\hat{\theta}_{i}$’s are independent and follow a normal
distribution with mean $\theta_{i}$ and known variance $\sigma_{i}^{2}$. Let
$z_{i}={\hat{\theta}_{i}}/{\sigma_{i}}$ denote the test statistic for the null
hypothesis $H^{\,i}_{0}$: $\theta_{i}=0$, $i\in{o,r}$, and
$p_{i}=1-\Phi(z_{i})$ the corresponding one-sided $p$-value for the
alternative $H^{\,i}_{1}$: $\theta_{i}>0$, here $\Phi(.)$ denotes the standard
normal cumulative distribution function. Replication success at level $\gamma$
is achieved if

$\left({z_{o}^{2}}/{z_{\gamma}^{2}}-1\right)_{+}\left({z_{r}^{2}}/{z_{\gamma}^{2}}-1\right)_{+}\geq
c$ (1)

holds, here $x_{+}=\max\\{0,x\\}$, $c=\sigma_{o}^{2}/\sigma_{r}^{2}>0$ is the
variance ratio and $z_{\gamma}=\Phi^{-1}(1-\gamma)>0$ is the threshold at
replication success level $\gamma$.

The two-sided formulation only requires (1), irrespectively of the signs of
the estimates $\hat{\theta}_{o}$ and $\hat{\theta}_{r}$, but suffers from the
“replication paradox” (Ly et al., , 2019) because replication success can
occur even if the effect estimates $\hat{\theta}_{o}$ and $\hat{\theta}_{r}$
are in opposite directions. The one-sided formulation avoids this problem with
the additional requirement that the two estimates are both in the same pre-
specified (w.l.o.g. positive) direction,

$\hat{\theta}_{o}>0\mbox{ and }\hat{\theta}_{r}>0,$ (2)

and so we usually require both (1) and (2) to achieve replication success (if
not stated otherwise).

The success conditions (1) and (2) can be motivated from a recent proposal to
define replication success with a two-step procedure (Held, 2020a, ): First, a
significant original study at one-sided level $\gamma$ is challenged by a
normal prior with mean zero modelling the belief of a hypothetical sceptic who
regards the absence of an effect to be the most likely reality (Matthews, ,
2018). The prior variance is chosen such that the posterior probability that
the effect is negative is $\gamma$. Secondly, the conflict between the
replication study result and the sceptical prior is quantified with a prior-
predictive tail probability $p_{\mbox{\scriptsize Box}}$ (Box, , 1980).
Replication success at level $\gamma$ is then achieved if
$p_{\mbox{\scriptsize Box}}\leq\gamma$, i. e. if there is more conflict
between the sceptical prior and the replication study than there was evidence
against the null hypothesis based on the original data.

We are often interested in the smallest possible value of $z_{\gamma}^{2}$
which solves (1) and denote this value as
$z_{S}^{2}\in(0,\min\\{z_{o}^{2},z_{r}^{2}\\})$, defined as the smallest
positive root of

$\left({z_{o}^{2}}/{z_{S}^{2}}-1\right)\left({z_{r}^{2}}/{z_{S}^{2}}-1\right)=c.$
(3)

This is a quadratic equation in $z_{S}^{2}$ and can be solved analytically.
Any ${z_{S}}=+\sqrt{z_{S}^{2}}\geq z_{\gamma}$ will hence lead to replication
success at level $\gamma$, so the threshold $z_{\gamma}$ in (1) serves as a
critical value for the test statistic $z_{S}$. If the effect estimates fulfill
(2), the transformation $p_{S}=1-\Phi({z_{S}})$ defines the (one-sided)
sceptical $p$-value in its original formulation and the criterion ${z_{S}}\geq
z_{\gamma}$ for replication success translates to $p_{S}\leq\gamma$. If (2)
doesn’t hold we set $p_{S}=\Phi({z_{S}})$ (Held, 2020a, , Section 3.3).

###  2.1 Null hypotheses and Type-I error rates

The T1E rate is the probability of a false claim of replication success under
a given null hypothesis. In the replication setting with two studies, this
probability can be considered under two different null hypotheses (Heller et
al., , 2014). The _intersection null hypothesis_ is a point null hypothesis,
defined as the intersection of the study-specific null hypotheses
$H^{\,i}_{0}$: $\theta_{i}=0$, $i=o,r$:

$H^{\,o}_{0}\cap H^{\,r}_{0}.$ (4)

The probability of a false claim of replication success with respect to the
intersection null (4) is the _overall_ T1E rate.

The _no-replicability_ or _union null hypothesis_ is defined as the complement
of the alternative that the effect is non-null in both studies. This is a
composite null hypothesis, which also includes the possibility that only one
study has a null effect:

$H^{\,o}_{0}\cup H^{\,r}_{0}.$ (5)

The probability of a false claim of replication success with respect to the
union null (5), the _partial_ T1E rate, depends on the values of $\theta_{o}$
and $\theta_{r}$. One of them is zero but the other one may not be zero. The
partial T1E rate has been recently investigated by Zhan et al.,  (2023) for
the two-trials rule. In Section 3.4 we also study the T1E rate under
$H_{0}^{\,r}$ only, conditional on the result of the original study. This
_conditional_ T1E rate is of primary interest because in practice the design
of the replication study depends on the result from the original study
(Anderson and Kelley, , 2022).

A necessary but not sufficient condition for the replication success criterion
(1) to hold is $\min\\{\left\lvert z_{o}\right\rvert,\left\lvert
z_{r}\right\rvert\\}>z_{\gamma}$, as otherwise the left-hand side of (1) is
zero. Combined with (2) this translates to the necessary but not sufficient
requirement $p_{\max}=\max\\{p_{o},p_{r}\\}<\gamma$. Under the union null
hypothesis, either $p_{o}$ or $p_{r}$ is uniform distributed, so $\gamma$ is a
bound on the partial T1E rate of the sceptical $p$-value for any value of the
variance ratio $c$. Likewise, the overall T1E rate is smaller than
$\gamma^{2}$ due to independence of the two studies.

This raises the question what value for $\gamma$ to use in (1). The _nominal_
success level is the standard significance level $\gamma=\alpha$, so controls
the overall and partial T1E rate at $\alpha^{2}$ respectively $\alpha$ for
every value of $c$. However, T1E control is not exact and the overall T1E rate
can be considerably smaller than $\alpha^{2}$ (Held et al., 2022b, , Section
3.2). Replication success at the nominal level is hence only possible if both
$p$-values are smaller than $\alpha$ and is much more stringent than the two-
trials rule. The _golden_ success level is
$\gamma(\alpha)=1-\Phi({z_{\alpha}}/\sqrt{\varphi})>\alpha$, where
$z_{\alpha}=\Phi^{-1}(1-\alpha)$ and $\varphi=(\sqrt{5}+1)/2$ is the golden
ratio. For example, $\gamma(\alpha=0.025)=0.062$. It is therefore less
restrictive than the nominal level in the assessment of replication success.
By construction, it controls the partial T1E rate at level $\gamma(\alpha)$
and the overall T1E rate at $\gamma(\alpha)^{2}$ and even at $\alpha^{2}$ if
$c\geq 1$ and $\alpha\leq 0.058$ (Held et al., 2022b, , Section 3.2), but the
actual overall T1E rate can be much smaller than the corresponding bound.
Comparing $p_{S}$ to the golden level $\gamma(\alpha)$ is equivalent to
comparing the golden sceptical $p$-value
$\tilde{p}_{S}=1-\Phi(\sqrt{\varphi}\,z_{S})$ to $\alpha$.

In what follows we describe how to obtain exact overall T1E control of the
sceptical $p$-value for any particular value of $c$. This is motivated through
the identification of the two-trials rule as a special case of the general
formulation (1) respectively (3) and leads to a  _controlled_ success level
$\gamma_{c}(\alpha)$ that depends on both $\alpha$ and $c$.

###  2.2 The two-trials rule

The two-trials rule requires significance of both studies at the one-sided
significance level $\alpha$, so corresponds to $z_{o}\geq z_{\alpha}$ and
$z_{r}\geq z_{\alpha}$, and translates to the single criterion
$p_{\max}\leq\alpha$. Under the intersection null (4) both $p_{o}$ and $p_{r}$
are uniformly distributed and so $p_{\max}$ follows a triangular
$\operatorname{Be}(2,1)$ distribution with cumulative distribution function
(cdf) $F(p)=p^{2}$, so that

$\Pr(p_{\max}\leq\alpha)=\alpha^{2}\mbox{ holds for all }\alpha\in(0,1).$ (6)

In the sequel, a $p$-value with this property will be said to have _exact
squared T1E control_. The square of a triangular $\operatorname{Be}(2,1)$
distribution follows a uniform distribution, so that
$p=p_{\max}^{2}=\max\\{p_{o}^{2},p_{r}^{2}\\}$ has cdf

$\Pr(p\leq\alpha)=\alpha\mbox{ for all }\alpha\in(0,1).$ (7)

In what follows, a $p$-value fulfilling (7) will be said to have _exact linear
T1E control_. Note that linear T1E control is the traditional requirement for
$p$-values (Casella and Berger, , 2002, p. 397) whereas $p$-values with
squared T1E control (such as $p_{\max}$) are useful if the $p$-value is based
on two independent studies.

Another way to obtain $p_{\max}$ as the $p$-value from the two-trials rule
with squared T1E control is by considering (3), but replacing the variance
ratio $c$ by $0$, so $z_{S}^{2}=z_{\min}^{2}=\min\\{z_{o}^{2},z_{r}^{2}\\}$.
Now the distribution of $Y=\min\\{z_{o}^{2},z_{r}^{2}\\}$ under the
intersection null is of interest. It can be shown that the random variable $Y$
has cdf

$\displaystyle F_{0}(y)$ $\displaystyle=$ $\displaystyle
1-4\left[1-\Phi(\sqrt{y})\right]^{2}\mbox{ for }y\geq 0,$ (8)

see Supporting Material (SM) A for a derivation. Now $Y$ doesn’t take into
account the direction of the effect estimates and hence the corresponding
$p$-value

$\displaystyle 4\,p=1-F_{0}(y=z_{\min}^{2})=4\left[1-\Phi(\min\\{\left\lvert
z_{o}\right\rvert,\left\lvert z_{r}\right\rvert\\})\right]^{2}$ (9)

is two-sided with exact linear T1E control. We use the notation $4\,p$, as
there are two studies (original and replication) with four possible sign
combinations of $\hat{\theta}_{o}$ and $\hat{\theta}_{r}$. If the combination
(2) is fulfilled, the one-sided $p$-value is therefore obtained from (9)
through division by $4$:

$\displaystyle
p=\left[1-F_{0}(z_{\min}^{2})\right]/4=\left[1-\Phi(\min\\{{z_{o}},{z_{r}}\\})\right]^{2}=\left(\max\\{p_{o},p_{r}\\}\right)^{2}=\max\\{p_{o}^{2},p_{r}^{2}\\}$

and so $p=p_{\max}^{2}$ respectively $p_{\max}=\sqrt{p}$.

This insight suggests a strategy to obtain a sceptical $p$-value with exact
squared overall T1E control: If we can derive the distribution function
$F_{c}(.)$ of $z_{S}^{2}$ in (3) under the intersection null for any value of
$c>0$, then we can use the transformation $p=[1-F_{c}(z_{S}^{2})]/4$, provided
(2) holds. The _controlled_ sceptical $p$-value then is $p_{S}^{*}=\sqrt{p}$
and has exact squared overall T1E control. If (2) doesn’t hold, we set
$p_{S}^{*}=1-\sqrt{p}$. For simplicity, we call $p_{S}^{*}$ the sceptical
$p$-value in the following, if no misunderstandings can arise. Figure 1
summarises the different steps from the null distribution function $F_{c}(.)$
to the assessment of replication success at overall T1E rate $\alpha^{2}$ with
the two-trials rule and the sceptical $p$-value.

$p=[1-F_{c}(z_{S}^{2})]/4$ $c=0$ (2TR) $c>0$ (sceptical $p$) $p_{S}^{*}$
$p_{\max}=\max(p_{o},p_{r})$ $p\leq\alpha^{2}\Leftrightarrow
p_{\max}\leq\alpha$ $p\leq\alpha^{2}\Leftrightarrow p_{S}^{*}\leq\alpha$ RS
$p_{\max}=\sqrt{p}$$p_{S}^{*}=\sqrt{p}$yesyes Figure 1: Replication success
(RS) with the two-trials rule (2TR) and the controlled sceptical $p$-value
$p_{S}^{*}$. The cdf $F_{c}(.)$ is based on the distribution of $z_{S}^{2}$ in
(3) under the intersection null and the $p$-values are one-sided, calculated
under the assumption that (2) holds.

###  2.3 The null distribution for equal variances

We first consider the case $c=1$, where the null distribution of $z_{S}^{2}$
is available from the harmonic mean $\chi^{2}$-test (Held, 2020b, ). The
solution of (3) then is

$z_{S}^{2}={z_{H}^{2}}/{2}$ (10)

where $z^{2}_{H}=2/(1/z_{o}^{2}+1/z_{r}^{2})$ is the harmonic mean of the
squared test statistics $z_{o}^{2}$ and $z_{r}^{2}$. It can be shown that
$z^{2}_{S}$ has a gamma $\operatorname{Ga}(1/2,2)$ distribution under the
intersection null (4), where $z_{o}^{2}$ and $z_{r}^{2}$ are independent
$\chi^{2}(1)$-distributed (Pillai and Meng, , 2016, eq. (2.3)). The cdf
$F_{c=1}(y)$ of $Y=z^{2}_{S}$ is thus readily available and a two-sided
$p$-value with exact linear T1E control can be calculated:

$4\,p=1-F_{c=1}(y=z_{S}^{2}).$ (11)

Division by 4 gives the corresponding one-sided $p$-value $p$ if (2) is
fulfilled and the square root $p_{S}^{*}=\sqrt{p}$ defines the controlled
sceptical $p$-value with exact squared T1E control for $c=1$.

###  2.4 The null distribution for unequal variances

For $c>0$ and $c\neq 1$ there is a unique solution of (3) that fulfills the
requirement $0\leq z_{S}^{2}\leq\min\\{z_{o}^{2},z_{r}^{2}\\}$:

$\displaystyle z_{S}^{2}$ $\displaystyle=$
$\displaystyle\frac{z_{A}^{2}}{c-1}\left\\{\sqrt{1+(c-1)z^{2}_{H}/z_{A}^{2}}-1\right\\},$
(12)

where $z^{2}_{A}$ is the arithmetic and $z^{2}_{H}$ the harmonic mean of
$z_{o}^{2}$ and $z_{r}^{2}$. To obtain $F_{c}(.)$, consider the probabilistic
version of equation (12), where the random variable $Y=z_{S}^{2}$ depends on
the two random variables $z_{o}^{2}$ and $z_{r}^{2}$ through $z_{A}^{2}$ and
$z_{H}^{2}$. Under the intersection null hypothesis (4), $z_{o}^{2}$ and
$z_{r}^{2}$ are independent $\chi^{2}(1)$-distributed. Then $z_{A}^{2}$ and
$z_{H}^{2}/z_{A}^{2}$ in (12) are also independent (Grimmett and Stirzaker, ,
2001, Section 4.7), which facilitates the computation of the cdf
$F_{c}(y)=\Pr(Y\leq y\,|\,c)$ of $Y$. In SM B we show that

$F_{c}(y)=1-\frac{1}{\pi}\int_{0}^{1}\frac{\exp\\{-g(y,t,c)\\}}{\sqrt{t(1-t)}}\,dt$
(13)

where

$g(y,t,c)=\frac{{(c-1)y}}{{\sqrt{1+(c-1)t}-1}}.$

Figure 2:  Cumulative distribution function $F_{c}(y)$ of $Y=z_{S}^{2}$ (main
plot) and expectation of $Y$ (inset plot) under the intersection null
hypothesis (4) for different values of $c$.

Figure 2 compares $F_{c}(y)$ and the expectation of $Y$ for different values
of $c$, including the special cases $c=0$ and $c=1$. Evaluation of (13) is
possible with numerical integration techniques and so a two-sided $p$-value
with exact linear T1E control can be calculated:

$4\,p=1-F_{c}(z_{S}^{2})$ (14)

with $z_{S}^{2}$ as defined in (12). If (2) is fulfilled, the corresponding
one-sided $p$-value has exact linear T1E control and $p_{S}^{*}=\sqrt{p}$
defines the one-sided controlled sceptical $p$-value with exact squared T1E
control.

###  2.5 Limiting cases

For $c\downarrow 0$ the two-sided $p$-value $4\,p$ in (14) converges to
$4\,p^{2}_{\max}$. This follows from the fact that $z_{S}^{2}\uparrow
z^{2}_{\min}$ for $c\downarrow 0$ (Held, 2020a, , eq. (11)) with cdf (8) under
the intersection null. The two-trials rule described in Section 2.2 is
therefore a limiting case of our framework if we are willing to ignore the
interpretation of $c$ as the variance ratio.

For $c\rightarrow\infty$ the two-sided $p$-value $4\,p$ in (14) converges to

$4\,p_{\infty}=\lim_{c\rightarrow\infty}4\,p=\frac{1}{\pi}\int_{0}^{1}\frac{\exp\left(-z_{G}^{2}/\sqrt{t}\right)}{\sqrt{t(1-t)}}\,dt,$
(15)

where $z_{G}^{2}=\sqrt{z_{o}^{2}z_{r}^{2}}=\left\lvert z_{o}z_{r}\right\rvert$
is the geometric mean of the squared test statistics $z_{o}^{2}$ and
$z_{r}^{2}$ (proof to be found in SM C). Note that $4\,p_{\infty}=1$ if either
$z_{o}=0$ or $z_{r}=0$, as $f(x)=1/\\{\pi\sqrt{x(1-x)}\\}$ is the density of a
$X\sim\operatorname{Be}(1/2,1/2)$ random variable and integrates to $1$.
Furthermore, (15) is a two-sided $p$-value with exact linear T1E control, i.
e. $4\,p_{\infty}$ is uniformly distributed on the unit interval if
$z_{o}^{2}$ and $z_{r}^{2}$ are i.i.d. $\chi^{2}(1)$, see SM D for a proof.

Other well-known methods that combine two (or more) $p$-values are Fisher’s
(Fisher, , 1958), Stouffer’s (Stouffer et al., , 1949) and Pearson’s (Pearson,
, 1933) methods. Fisher’s method is based on the product of the $p$-values,
Pearson’s method on the product of one minus the $p$-values, Stouffer’s method
on the sum of the $z$-values, whereas (15) is based on the product of the
$z$-values. We will compare the different methods to combine $p$-values in
Sections 3.1 and 3.2 in more detail.

###  2.6 Properties of the sceptical $p$-value

For fixed $p_{o}$ and $p_{r}$, the nominal and golden versions of the
sceptical $p$-value monotonically increase with increasing variance ratio $c$
(Held, 2020a, , Section 3.1) and eventually reach $0.5$ for
$c\rightarrow\infty$. The controlled sceptical $p$-value $p_{S}^{*}$ behaves
differently, as illustrated in Figure 3 (left), which shows $p_{S}^{*}$ as a
function of $c$ for selected values of $p_{o}$ and $p_{r}$.

As described in Section 2.5, $p_{S}^{*}$ converges to $p_{\max}$ for
$c\downarrow 0$. The functional behaviour of $p_{S}^{*}$ as a function of $c$
can be studied through inspection of the derivative of $p_{S}^{*}$ with
respect to $c$, see SM E and F. If $p_{o}=p_{r}$ then $p_{S}^{*}$ increases
monotonically with increasing $c$. If the difference between $p_{o}$ and
$p_{r}$ is relatively large, $p_{S}^{*}$ decreases monotonically. If
$\left\lvert p_{o}-p_{r}\right\rvert$ is relatively small but not zero, the
sceptical $p$-value first decreases and then increases with increasing $c$.
The infimum of $p_{S}^{*}$ is hence $p_{\max}$ in the first case, $p_{\infty}$
from (15) in the second, and in between those two values in the third case.

These properties allow us to compare $p_{S}^{*}$ and $p_{\max}$ for the same
values of $p_{o}$ and $p_{r}$, see Figure 3 (right). In the first case
($p_{o}=p_{r}$), $p_{S}^{*}$ is always larger than $p_{\max}$, for any value
of the variance ratio $c$. If $p_{o}$ and $p_{r}$ differ considerably,
$p_{S}^{*}$ is always smaller than $p_{\max}$ (the green region). The gray
area depicts combinations of $p_{o}$ and $p_{r}$ for which the sceptical
$p$-value $p_{S}^{*}$ is not monotone as a function of $c$, but first
decreases, then increases and eventually gets larger than $p_{\max}$ for large
$c$. Whether $p_{S}^{*}$ is smaller or larger than $p_{\max}$ now depends on
the value of $c$, as indicated with dotted and dashed lines in Figure 3
(right) for $c=0.1$ and $c=1$, respectively.

Figure 3: Left panel: Sceptical $p$-value $p_{S}^{*}$ as a function of the
variance ratio $c$ for $p_{o}=0.027$ and selected values of $p_{r}$. The dots
represent the infimum of $p_{S}^{*}$ for selected values of $p_{r}$ and the
dashed line indicates $p_{\max}$. Right panel: Combinations of $p_{o}$ and
$p_{r}$ for which $p_{S}^{*}$ is always smaller than $p_{\max}$ (green area),
for which $p_{S}^{*}$ is always larger than $p_{\max}$ (red line) and for
which it depends on the variance ratio $c$ (gray region). In the gray region
above (respectively below) the red line, $p_{S}^{*}$ is smaller than
$p_{\max}$ for combinations of $p_{o}$ and $p_{r}$ laying above (respectively
below) the black line for the corresponding $c$, and vice-versa.

##  3 Replication success rates and regions

In this section, we consider properties of the sceptical $p$-value based on
the dichotomous criterion $p_{S}^{*}\leq\alpha$ for replication success. We
start in Section 3.1 with deriving the corresponding value of the replication
success level $\gamma$ in (1) so that the overall T1E rate is exactly
$\alpha^{2}$ for a particular value of $c>0$. As discussed in Section 2.1, the
replication success level $\gamma$ is also a bound on the partial T1E rate of
the sceptical $p$-value. We will then compare success regions for different
values of $c$ in Section 3.2 and investigate project power in Section 3.3.
Finally, Section 3.4 outlines how the sceptical $p$-value can be used for
sample size calculations. This is facilitated through the interpretation of
the variance ratio $c$ as relative sample size, $c=n_{r}/n_{o}$, since the
variances of the effect estimates are usually inversely proportional to the
corresponding sample sizes $n_{o}$ and $n_{r}$, i. e.
$\sigma_{o}^{2}=\kappa^{2}/n_{o}$ and $\sigma_{r}^{2}=\kappa^{2}/n_{r}$ for
some unit variance $\kappa^{2}$ (Held, 2020a, ).

###  3.1 Partial Type-I error control

Method | Parameters | T1E control
---|---|---
|  | overall | partial
Two-trials rule |  $p_{o}$, $p_{r}$ | $=\alpha^{2}$ | $<\alpha$
Fisher |  $p_{o}$, $p_{r}$ | $=\alpha^{2}$ | $<1$
Stouffer |  $p_{o}$, $p_{r}$ | $=\alpha^{2}$ | $<1$
Pearson |  $p_{o}$, $p_{r}$ | $=\alpha^{2}$ | $<c_{P}(\alpha)$
nominal $p_{S}$ |  $p_{o}$, $p_{r}$, $c$ | $<\alpha^{2}$ | $<\alpha$
golden $\tilde{p}_{S}$ |  $p_{o}$, $p_{r}$, $c$ | $<\gamma(\alpha)^{2}$ | $<\gamma(\alpha)$
controlled $p_{S}^{*}$ |  $p_{o}$, $p_{r}$, $c$ | $=\alpha^{2}$ | $<\gamma_{c}(\alpha)$
Table 1: T1E rate of different methods to assess replication success. The
overall T1E rate is calculated under the intersection null (4). The partial
T1E rate is calculated under the union null (5).

If we want to know the bound on the partial T1E rate of the controlled
sceptical $p$-value, we need to derive the corresponding value of $\gamma$ in
(1), now denoted as $\gamma_{c}=\gamma_{c}(\alpha)$ as it depends not only on
the target overall T1E rate $\alpha^{2}$, but also the relative sample size
$c$. Comparing $p_{S}$ to $\gamma_{c}(\alpha)$ is then equivalent to comparing
$p_{S}^{*}$ to $\alpha$.

For $c=1$, we have ${\gamma_{1}}=1-\Phi(\Phi^{-1}(1-2\alpha^{2})/2)$, see
Held, 2020b (, Section 2.1). For example, $\gamma_{1}(\alpha=0.025)=0.065$, so
the partial T1E rate is bounded by 0.065 for $c=1$. The null distribution
function (13) of $z_{S}^{2}$ can be used to compute the bound $\gamma_{c}$ for
$c\neq 1$, but now numerical methods are needed. Briefly, the overall T1E rate
for any two values of $c$ and $\gamma_{c}$ can be computed with numerical
integration (Held et al., 2022b, , Section 3.2). Root-finding methods are then
used to find the value of $\gamma_{c}$ which gives the target overall T1E rate
of $\alpha^{2}$. The inset plot in Figure 4 shows the bound $\gamma_{c}$ as a
function of $c$ for exact overall T1E control at $\alpha^{2}=0.000625$. The
bound on the partial T1E rate can get quite large for large relative sample
sizes $c$. For example, it is $\gamma_{10}(\alpha=0.025)=0.14$ for $c=10$.
However, this large partial T1E rate is balanced by the fact that such large
relative sample sizes are only needed for unconvincing original studies, where
success is unlikely. The conditional T1E rate for such unconvincing original
studies combined with large relative sample sizes is in fact very small, as
shown in Section 3.4.

A summary of the overall and partial T1E rates of different methods is given
in Table 1. Fisher’s, Stouffer’s and Pearson’s methods control the overall T1E
rate exactly at significance level $\alpha^{2}$. However, as the first two
methods do not impose a threshold on the individual $p$-values $p_{o}$ and
$p_{r}$ (Rosenkranz, , 2002), the partial T1E rate is bounded by one and
replication success can occur if one of the two $p$-values is very large. The
partial T1E rate of Pearson’s method is bounded by
$c_{P}(\alpha)=1-\exp(-0.5\chi^{2}_{4}(\alpha^{2}))$, for example
$c_{P}(\alpha=0.025)=0.035$.

###  3.2 Success regions

The (one-sided) framework (1) can now be used to determine the region where
two $p$-values $p_{o}$ and $p_{r}$ lead to replication success. The main plot
in Figure 4 compares the success regions for $\alpha=0.025$ and selected
values of the variance ratio (respectively relative sample size) $c$, so the
area under each curve is equal to $\alpha^{2}=0.000625$. Each success region
is bounded on both axes by the corresponding success level $\gamma_{c}$. The
two-trials rule success region corresponds to $\gamma_{0}=0.025$ and is the
squared gray area below the black line. The success region of the sceptical
$p$-value is close to the two-trials rule’s success region for small $c$, but
becomes more and more in favour of $p$-values of different size as $c$
increases. The case $c=\infty$ is based on the one-sided $p$-value
$p_{\infty}$ available from (15). Also shown are the success regions based on
Fisher’s, Stouffer’s and Pearson’s method. The first two are even less in
favour of $p$-values of the same magnitude. For example, if both $p$-values
are equal to $\alpha/2=0.0125$ (the solid black point in Figure 4),
replication success will be flagged with the sceptical $p$-value for any value
of $c$, but not with Fisher’s nor Stouffer’s method. Pearson’s method has a
success region similar to that of the sceptical $p$-value with $c=0.1$, and
the same bound on the partial T1E rate.

Figure 4: Success region of the sceptical $p$-value as a function of $p_{o}$
and $p_{r}$ for different values of the relative sample size $c$. The colored
labels on the $y$-axis are the different values of the bound $\gamma_{c}$ on
the partial T1E rate ($\gamma_{1}=0.065$ and $\gamma_{10}=0.14$ are outside
the axis range, but can be read off the inset plot). The two-trials rule
success region is the squared gray area below the black line where $c=0$ and
$\gamma_{0}=\alpha$. Fisher’s, Stouffer’s and Pearson’s methods have been
added for comparison purposes. All methods control the overall T1E rate at
$\alpha^{2}=0.025^{2}=0.000625$, and so the area under each curve is equal to
this value.

###  3.3 Project power

Suppose none of the two studies has been conducted yet and so the probability
to declare replication success is calculated over both studies in combination
for a fixed relative sample size $c$. Using numerical integrations adapted
from Held et al., 2022b (, Section 3.3), the project power of the sceptical
$p$-value is considered in this section.

The project power is the probability to declare replication success when both
effects are equal and non-null. The distribution of $z_{o}$ is then
$\operatorname{N}(\mu=z_{\alpha}+z_{\beta},1)$, where $1-\beta$ is the power
to detect the true original effect $\theta_{o}=\mu\,\sigma_{o}$ (Matthews, ,
2006, Section 3.3), and the distribution of $z_{r}$ is
$z_{r}\sim\operatorname{N}(\sqrt{c}\mu,1)$. Figure 5 shows the project power
of the sceptical $p$-value and the two-trials rule with $\alpha=0.025$ and
original power $1-\beta=80$% (left), respectively $90$% (right). The project
power based on the two-trials rule converges to 80%, respectively 90%, for
large relative sample size $c$. The project power based on the sceptical
$p$-value is always larger than with the two-trials rule, and increases to
values close to 100% for large $c$. For example, for 80% original power and
$c=2$ the project power of the two trials rule is 78%, while the project power
of the sceptical $p$-value is already 87%.

Figure 5:  Project power as a function of the relative sample size $c$ for
$\alpha=0.025$. The original power is 80% in the left plot, and 90% in the
right one. Results are given for the sceptical $p$-value and compared with the
two-trials rule (2TR).

###  3.4 Design of replication studies

In this section we assume that the original study has already been conducted
and a replication study is planned. It has been recently argued that the
method used for sample size planning should always match the one used for the
analysis (Anderson and Kelley, , 2022). Hence, we develop methods to design
the replication study based on the sceptical $p$-value and compare it to the
design based on the two-trials rule, i. e. significance of the replication
study.

The probability to declare replication success with a particular sample size
$n_{r}$ is known as the power of the replication study and is often calculated
conditional on the effect estimate from the original study. Predictive power
(Spiegelhalter and Freedman, , 1986) can also be used and takes the
uncertainty of the original effect estimate into account. Formulas for the
power of the two-trials rule and the sceptical $p$-value at fixed success
level can be found in Micheloud and Held,  (2022, Section 2.1) and Held et
al., 2022b (, Section 3.1), respectively.

Figure 6 (left) shows the ratio of conditional power calculated with the
sceptical $p$-value versus the two-trials rule with $\alpha=0.025$ as a
function of the relative sample size $c$ and the original $p$-value $p_{o}$.
The sceptical $p$-value has larger power (ratio > $1$) if the original study
is already convincing. For example, for $c=1$, this is the case if
$p_{o}<0.01$, otherwise the two-trials rule has larger power. However, if
$p_{o}>\alpha$, the power of the two-trials rule is $0$, but not the power of
the sceptical $p$-value as long as $p_{o}<\gamma_{c}$.

Figure 6:  Ratio of conditional power (left) and replication sample sizes
(right) calculated with the sceptical $p$-value ($p_{S}^{*}$) versus the two-
trials rule (2TR) as a function of the original $p$-value $p_{o}$ and the
relative sample size $c$ (left) or the conditional power (right) for
$\alpha=0.025$.

Instead of calculating the power for a fixed replication sample size $n_{r}$,
we can also fix the power to a desired value and calculate the required sample
size $n_{r}$. Sample size calculation with the controlled sceptical $p$-value
does not have a closed-form expression, because the success level $\gamma_{c}$
in (1) depends on the relative sample size $c$. Root-finding algorithms are
therefore required to find the value of $c$ which leads to the desired power.
Importantly, sample size calculation is now possible even for non-significant
original studies, as shown in the top axis of Figure 7 for conditional power
values of 80, 90, and 95%. Note that the required relative sample size can
become quite large if $p_{o}>\alpha$.

Figure 6 (right panel) shows the ratio of the replication sample sizes
calculated with the sceptical $p$-value versus the two-trials rule. This is
done only for significant original studies, as this is required for success
with the two-trials rule. The sceptical $p$-value requires less samples than
the two-trials rule for already convincing original studies ($p_{o}<0.007$).
Similar results are obtained when predictive power is used instead of
conditional, see SM G.

The T1E rate of the sceptical $p$-value can also be considered under
$H_{0}^{\,r}$, conditional on the original study result. First, the relative
sample size $c$ to reach a certain power for a fixed original $z$-value
$z_{o}$ is calculated. These values of $z_{o}$ and $c$ are then used in (1) to
derive a lower bound for the replication $z$-value $z_{r}$ to achieve
replication success:

$z_{r}\geq z_{\gamma_{c}}\sqrt{1+c/(z_{o}^{2}/z_{\gamma_{c}}^{2}-1)}.$ (16)

Subsequent transformation of the right hand side of (16) to the corresponding
upper bound for $p_{r}$ gives the conditional T1E rate. Note that the
conditional T1E rate of the two-trials rule is constant at $\alpha$ as long as
$p_{o}\leq\alpha$.

Figure 7: Conditional T1E rate of the sceptical $p$-value as a function of the
original $p$-value $p_{o}$. The relative sample size (top axis) is calculated
with the sceptical $p$-value method to reach a conditional power of 80%, 90%,
and 95% at $\alpha=0.025$. Each dot represents the upper bound for the
conditional T1E rate with the respective power. The gray horizontal line
indicates the T1E rate of the two-trials rule.

Figure 7 shows the conditional T1E rate of the sceptical $p$-value as a
function of the $p$-value $p_{o}$ from the original study. The relative sample
size $c$ in (16) is calculated with the sceptical $p$-value method to reach a
conditional power of 80%, 90%, and 95% respectively with
$\gamma_{c}(\alpha=0.025)$. The conditional T1E rate is larger than
$\alpha=2.5$%, the conditional T1E rate of the two-trials rule, for
$p_{o}<0.008$ but bounded by $4.3$%, $4.5$% and $4.7$% for a power of 80%, 90%
and 95%, respectively. The conditional T1E rate is hence never larger than
2$\alpha=5\%$, and this also holds for other values of $\alpha$ provided that
they are not very small, see SM G.  If $p_{o}>0.008$, the conditional T1E rate
of the sceptical $p$-value is smaller than $2.5$% in all three cases. This
illustrates that the conditional T1E rate is sufficiently bounded if the
replication sample size is computed based on standard power values to detect
the observed effect from the original study.

A similar pattern can be seen if the conditional T1E rate is based on
predictive rather than conditional power, see SM G, where we also show the
conditional T1E rate of the nominal and golden sceptical $p$-values.

##  4 Application

We now illustrate the proposed methodology using all 18 pairs of studies from
the Experimental Economics Replication Project (Camerer et al., , 2016, EERP).
The different effect estimates were all transformed to correlation
coefficients, where Fisher’s $z$-transformation achieves asymptotically normal
effect estimates $\hat{\theta}_{i}$ with known standard errors. Table 2
summarizes the results for each of the 18 studies.

###  4.1 Comparison of two-trials rule versus sceptical $p$-value

The last two columns in Table 2 show the $p$-value $p_{\max}$ from the two-
trials rule and the sceptical $p$-value $p_{S}^{*}$, respectively. There is
generally a good agreement, with $p_{S}^{*}<p_{\max}$ for 12 out of the 18
studies.

Study | $\hat{\theta}_{o}$ | $\hat{\theta}_{r}$ | $p_{o}$ | $p_{r}$ |  Power ($\%$) | $c$ | $c^{\star}$ | $p_{\max}$ | $p_{S}^{*}$
---|---|---|---|---|---|---|---|---|---
de Clippel et al. (2014) | 0.12 | 0.27 | 0.0005 | < 0.0001 | 91.0 | 1.0 | 0.8 | 0.0005 | < 0.0001
Kogan et al. (2011) | 0.34 | 0.31 | < 0.0001 | 0.0005 | 93.9 | 0.7 | 0.6 | 0.0005 | 0.0002
Fudenberg et al. (2012) | 0.31 | 0.34 | 0.0003 | < 0.0001 | 93.9 | 1.0 | 0.9 | 0.0003 | 0.0003
Dulleck et al. (2011) | 0.91 | 0.93 | < 0.0001 | 0.0004 | 90.8 | 0.7 | 0.6 | 0.0004 | 0.0003
Friedman and Oprea (2012) | 0.76 | 0.47 | < 0.0001 | 0.002 | 99.6 | 0.5 | 0.4 | 0.002 | 0.0003
Bartling et al. (2012) | 0.91 | 0.79 | 0.003 | 0.0006 | 96.3 | 1.9 | 1.7 | 0.003 | 0.003
Kessler and Roth (2012) | 0.53 | 0.36 | < 0.0001 | 0.008 | 94.6 | 0.2 | 0.1 | 0.008 | 0.003
Charness and Dufwenberg (2011) | 0.40 | 0.38 | 0.005 | 0.001 | 89.0 | 1.6 | 1.5 | 0.005 | 0.005
Kirchler et al (2012) | 0.80 | 0.60 | 0.008 | 0.005 | 93.7 | 2.1 | 2.1 | 0.008 | 0.011
Fehr et al. (2013) | 0.49 | 0.32 | 0.006 | 0.013 | 92.3 | 1.8 | 1.7 | 0.013 | 0.015
Ambrus and Greiner (2012) | 0.32 | 0.23 | 0.027 | 0.006 | 93.3 | 3.2 | 4.2 | 0.027 | 0.024
Ericson and Fuster (2011) | 0.22 | 0.12 | 0.015 | 0.027 | 92.3 | 2.4 | 2.7 | 0.027 | 0.032
Huck et al. (2011) | 1.19 | 0.39 | 0.0002 | 0.082 | 99.1 | 1.4 | 1.2 | 0.082 | 0.045
Abeler et al. (2011) | 0.18 | 0.08 | 0.023 | 0.08 | 90.7 | 2.7 | 3.4 | 0.08 | 0.074
Chen and Chen (2011) | 1.23 | 0.17 | 0.017 | 0.28 | 98.3 | 3.7 | 4.1 | 0.28 | 0.24
Ifcher and Zarghamee (2011) | 0.29 | -0.01 | 0.016 | 0.53 | 90.7 | 2.3 | 2.6 | 0.53 | 0.53
Duffy and Puzzello (2014) | 1.00 | -0.12 | 0.007 | 0.66 | 95.0 | 2.2 | 2.1 | 0.66 | 0.69
Kuziemko et al. (2014) | 0.29 | -0.12 | 0.035 | 0.92 | 93.1 | 3.6 | 5.3 | 0.92 | 0.92

Table 2: Studies from the Experimental Economics Replication Project. Shown
are the original and replication effect estimates ($\hat{\theta}_{i}$) and
$p$-values ($p_{i}$), the power of the replication study with the actual
relative sample size $c$, the corresponding relative sample size $c^{\star}$
calculated with the sceptical $p$-value method, $p_{\max}$ and $p_{S}^{*}$.

The studies of Ericson and Fuster,  (2011) and Ambrus and Greiner,  (2012)
deserve closer scrutiny. While $p_{\max}=0.027$ is the same for both studies,
the values of $p_{S}^{*}$ differ. In Ericson and Fuster,  (2011),
$p_{o}=0.015$ and $p_{r}=0.027$ are relatively close to each other and the red
line in Figure 3 (left panel) at $c=1/2.4$ (because we have to reverse the
role of $p_{o}$ and $p_{r}$) explains why $p_{S}^{*}=0.032>p_{\max}$. In
contrast, there is a larger difference in $p$-values ($p_{o}=0.027$,
$p_{r}=0.006$) in Ambrus and Greiner,  (2012), and thus $p_{S}^{*}<p_{\max}$,
see the green line in Figure 3 (left) at $c=3.2$. For this combination of
$p$-values, the sceptical $p$-value is smaller than $p_{\max}$ for every value
of the relative sample size $c$. Of note, in Ambrus and Greiner,  (2012) the
two-trials rule fails because of the dichotomization at $\alpha=0.025$, but
replication success with the sceptical $p$-value is achieved
($p_{S}^{*}=0.024$).

###  4.2 Sample size calculation

In the EERP, the replication sample sizes were calculated to reach “at least
90% power […] to detect the original effect size at the [two-sided] 5%
significance level” (Camerer et al., , 2016, p.1434). We recomputed the exact
power values to then calculate the required relative sample size $c^{\star}$
with the sceptical $p$-value (see Table 2). The sceptical $p$-value would
require a smaller sample size than the two-trials rule for 12 out of 18
studies in this dataset. Two original studies were non-significant at the
$\alpha=0.025$ level (Ambrus and Greiner, , 2012; Kuziemko et al., , 2014).
Sample size calculation based on the sceptical $p$-value is also possible for
those two studies, where we obtain $c^{\star}>c$. Note that for these two
studies, the sample size $c$ is calculated to achieve significance of the
replication study, but the two-trials rule will fail anyway.

##  5 Discussion

We have described a novel statistical framework for the assessment of
replicability, stemming from a recently proposed reverse-Bayes approach to
assess replication success (Held, 2020a, ). The resulting controlled sceptical
$p$-value $p_{S}^{*}$ has exact overall T1E rate of $\alpha^{2}$ and
additionally ensures that the conditional and partial T1E rates are
sufficiently bounded. The two-trials rule can be seen as a special case of the
formulation for $c\downarrow 0$, where $p_{S}^{*}$ converges to the maximum of
the two study-specific $p$-values. The success region of the sceptical
$p$-value is smooth, shifting gradually away from the squared one of the two-
trials rule for increasing $c$, thus avoiding the “double dichotomization” and
offering larger project power. Used in the design of the replication study,
the new approach requires a smaller sample size than the two-trials rule for
already convincing original studies. In contrast to the golden version, the
controlled sceptical $p$-value allows sample size calculation for borderline
significant and even non-significant original studies.

As the $p$-value $p=(p_{S}^{*})^{2}$ is a proper $p$-value with exact linear
T1E control under the intersection null hypothesis, a $p$-value function
(Fraser, , 2019) could be computed and a “sceptical” confidence interval could
be calculated. This would address an important point raised by Diggle,  (2020)
about the need to accompany the sceptical $p$-value with suitable estimation
procedures to assess the relevance of the observed effects. We plan to
consider this in future work.

However, exact overall T1E control comes at a certain price: the explicit
penalization of small relative effect sizes in the nominal or golden versions
of the sceptical $p$-value is lost and replication success may occur even for
large shrinkage of the replication effect estimate, if the relative sample
size $c$ is large enough. Our conclusion is that exact overall T1E control and
penalization of small effect sizes are two competing goals that cannot be
achieved by a single criterion. It would therefore be interesting to extend
the recently proposed dual-criterion for replication studies (Rosenkranz, ,
2021), which simultaneously requires significance and relevance, to the
sceptical $p$-value.

This paper considers the situation where each original study only has one
replication. Further work will extend the proposed methodology to the analysis
of multiple replications per original study. A natural approach is to perform
a meta-analysis of the replication studies and use the resulting combined
effect estimate (possibly allowing for heterogeneity) as the replication
effect estimate.

#### Data and Software Availability

Software and data are available in the R-package ReplicationSuccess available
from CRAN. The data is originally from https://osf.io/pnwuz/, see Pawel and
Held,  (2020, supplement S1) for details on data preprocessing. The code to
reproduce the analysis and figures is available at
https://gitlab.uzh.ch/charlotte.micheloud/framework-for-replicability.

#### Author Contributions

CM contributed substantially to research, analysis, coding, and writing. FB
derived the required null distribution of the sceptical $p$-value, added
further proofs and contributed to writing. LH designed, performed and
supervised research and analysis, wrote parts of the code and drafts of the
paper.

#### Acknowledgments

LH thanks the University of Zurich for granting a sabbatical leave that made
this research possible. CM and LH acknowledge support by the Swiss National
Science Foundation (Project # 189295). We appreciate helpful comments by
Rachel Heyard and Samuel Pawel.

## References

  * Ambrus and Greiner, (2012) Ambrus, A. and Greiner, B. (2012).  Imperfect public monitoring with costly punishment: An experimental study.  American Economic Review, 102(7):3317–32.
  * Anderson and Kelley, (2022) Anderson, S. F. and Kelley, K. (2022).  Sample size planning for replication studies: The devil is in the design.  Psychological Methods.  https://doi.org/10.1037/met0000520.
  * Box, (1980) Box, G. E. P. (1980).  Sampling and Bayes’ inference in scientific modelling and robustness (with discussion).  Journal of the Royal Statistical Society, Series A, 143:383–430.
  * Camerer et al., (2016) Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., et al. (2016).  Evaluating replicability of laboratory experiments in economics.  Science, 351(6280):1433–1436.
  * Camerer et al., (2018) Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., et al. (2018).  Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015.  Nature Human Behaviour, 2(9):637–644.
  * Casella and Berger, (2002) Casella, G. and Berger, R. L. (2002).  Statistical Inference.  Duxbury Press, Belmont, California.
  * Craig, (1936) Craig, C. C. (1936).  On the frequency function of $xy$.  The Annals of Mathematical Statistics, 7:1–15.
  * Diggle, (2020) Diggle, P. J. (2020).  Discussion of "A new standard for the analysis and design of replication studies" by Leonhard Held.  Journal of the Royal Statistical Society, Series A, 183:450.
  * Ericson and Fuster, (2011) Ericson, K. M. M. and Fuster, A. (2011).  Expectations as endowments: evidence on reference-dependent preferences from exchange and valuation experiments.  The Quarterly Journal of Economics, 126(4):1879–1907.
  * Errington et al., (2021) Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., and Nosek, B. A. (2021).  Investigating the replicability of preclinical cancer biology.  eLife, 10:e71601.
  * Fisher, (1958) Fisher, R. A. (1958).  Statistical Methods for Research Workers.  Oliver & Boyd, Edinburgh, 13th ed.(rev.) edition.
  * Fraser, (2019) Fraser, D. A. S. (2019).  The $p$-value function and statistical inference.  The American Statistician, 73(sup1):135–147.
  * Grimmett and Stirzaker, (2001) Grimmett, G. R. and Stirzaker, D. R. (2001).  Probability and Random Processes.  Oxford University Press, Oxford, UK, 3rd edition.
  * (14) Held, L. (2020a).  A new standard for the analysis and design of replication studies (with discussion).  Journal of the Royal Statistical Society, Series A, 183:431–469.
  * (15) Held, L. (2020b).  The harmonic mean $\chi^{2}$-test to substantiate scientific findings.  Journal of the Royal Statistical Society, Series C, 69(3):697–708.
  * (16) Held, L., Matthews, R., Ott, M., and Pawel, S. (2022a).  Reverse-Bayes methods for evidence assessment and research synthesis.  Research Synthesis Methods, 13(3):295–314.
  * (17) Held, L., Micheloud, C., and Pawel, S. (2022b).  The assessment of replication success based on relative effect size.  The Annals of Applied Statistics, 16:706–720.
  * Heller et al., (2014) Heller, R., Bogomolov, M., and Benjamini, Y. (2014).  Deciding whether follow-up studies have replicated findings in a preliminary large-scale omics study.  Proceedings of the National Academy of Sciences, 111(46):16262–16267.
  * Kuziemko et al., (2014) Kuziemko, I., Buell, R. W., Reich, T., and Norton, M. I. (2014).  Last-place aversion: Evidence and redistributive implications.  The Quarterly Journal of Economics, 129(1):105–149.
  * Ly et al., (2019) Ly, A., Etz, A., Marsman, M., and Wagenmakers, E.-J. (2019).  Replication Bayes factors from evidence updating.  Behavior Research Methods, 51:2498–2508.
  * Maca et al., (2002) Maca, J., Gallo, P., Branson, M., and Maurer, W. (2002).  Reconsidering some aspects of the two-trials paradigm.  Journal of Biopharmaceutical Statistics, 12(2):107–119.
  * Matthews, (2006) Matthews, J. N. (2006).  Introduction to Randomized Controlled Clinical Trials.  Chapman and Hall/CRC.
  * Matthews, (2018) Matthews, R. A. J. (2018).  Beyond ’significance’: principles and practice of the analysis of credibility.  Royal Society Open Science, 5(1):171047.
  * Micheloud and Held, (2022) Micheloud, C. and Held, L. (2022).  Power calculations for replication studies.  Statistical Science, 37(3):369–379.
  * National Academies of Sciences, Engineering, and Medicine, (2019) National Academies of Sciences, Engineering, and Medicine (2019).  Reproducibility and Replicability in Science.  The National Academies Press, Washington, DC.
  * Open Science Collaboration, (2015) Open Science Collaboration (2015).  Estimating the reproducibility of psychological science.  Science, 349:aac4716.
  * Pawel and Held, (2020) Pawel, S. and Held, L. (2020).  Probabilistic forecasting of replication studies.  PLOS ONE, 15(4):e0231416.
  * Pawel and Held, (2022) Pawel, S. and Held, L. (2022).  The sceptical Bayes factor for the assessment of replication success.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 84(3):879–911.
  * Pearson, (1933) Pearson, K. (1933).  On a method of determining whether a sample of size n supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random.  Biometrika, 25(3-4):379–410.
  * Pillai and Meng, (2016) Pillai, N. S. and Meng, X.-L. (2016).  An unexpected encounter with Cauchy and Lévy.  Annals of Statistics, 44:2089–2097.
  * Rogozin, (2001) Rogozin, B. (2001).  Arcsine distribution.  In Encyclopedia of Mathematics. EMS Press.
  * Rosenkranz, (2002) Rosenkranz, G. (2002).  Is it possible to claim efficacy if one of two trials is significant while the other just shows a trend?  Drug Information Journal, 36(1):875–879.
  * Rosenkranz, (2021) Rosenkranz, G. (2021).  Replicability of studies following a dual-criterion design.  Statistics in Medicine, 40(18):4068–4076.
  * Rosenkranz, (2023) Rosenkranz, G. (2023).  A generalization of the two trials paradigm.  Therapeutic Innovation & Regulatory Science, 57(2):316–320.
  * Royal Netherlands Academy of Arts and Science, (2018) Royal Netherlands Academy of Arts and Science (2018).  Replication Studies – Improving Reproducibility in the Empirical Science.  KNAW, Amsterdam.
  * Senn, (2007) Senn, S. (2007).  Statistical Issues in Drug Development.  John Wiley & Sons, Chichester, U.K., second edition.
  * Spiegelhalter and Freedman, (1986) Spiegelhalter, D. J. and Freedman, L. S. (1986).  A predictive approach to selecting the size of a clinical trial, based on subjective clinical opinion.  Statistics in Medicine, 5(1):1–13.  https://doi.org/10.1002/sim.4780050103.
  * Stouffer et al., (1949) Stouffer, S. A., Suchman, E. A., Devinney, L. C., Star, S. A., and Williams, R. M., J. (1949).  The American soldier: Adjustment during army life. (Studies in social psychology in World War II).  Cambridge University Press, Princeton Univ. Press.
  * Zhan et al., (2023) Zhan, S. J., Kunz, C. U., and Stallard, N. (2023).  Should the two-trial paradigm still be the gold standard in drug assessment?  Pharmaceutical Statistics, 22(1):96–111.

Supporting materials for
Assessing replicability with the sceptical $p$-value: Type-I error control and
sample size planning Charlotte Micheloud, Fadoua Balabdaoui and Leonhard Held

##  Appendix A The distribution of $Y=\min\\{z_{o}^{2},z_{r}^{2}\\}$

Under the intersection null $X_{1}=z_{o}^{2}$ and $X_{2}=z_{r}^{2}$ are
independent $\chi^{2}(1)$ random variables with cdf
$F_{X}(x)=2\,\Phi(\sqrt{x})-1$. The cdf of the minimum
$Y=\min\\{X_{1},X_{2}\\}$ of two iid random variables $X_{1}$ and $X_{2}$ with
cdf $F_{X}(x)$ has cdf
$F_{Y}(y)=1-[1-F_{X}(y)]^{2}=1-[1-2\,\Phi(\sqrt{y})+1]^{2}=1-4\,[1-\Phi(\sqrt{y})]^{2}$,
so we obtain Equation (8) from the main manuscript.

##  Appendix B The null distribution of $Y=z_{S}^{2}$

Throughout we assume that $z_{o}$ and $z_{r}$ are independent standard normal
variables, so $X_{1}=z_{o}^{2}$ and $X_{2}=z_{r}^{2}$ are i.i.d.
$\sim\chi^{2}{(1)}\stackrel{{\scriptstyle d}}{{=}}\operatorname{Ga}(1/2,1/2)$.

###  B.1 The case $c<1$

Consider $Y=z_{S}^{2}$ to be equal to the smallest positive root of the
equation

$\left(\frac{X_{1}}{Y}-1\right)\left(\frac{X_{2}}{Y}-1\right)=c.$ (17)

After some algebra, we find for $c<1$ the solution

$\displaystyle Y$ $\displaystyle=$
$\displaystyle\frac{1}{2(1-c)}\left(X_{1}+X_{2}-\sqrt{(X_{1}+X_{2})^{2}-4(1-c)X_{1}X_{2}}\right)$
$\displaystyle=$
$\displaystyle\frac{X_{1}+X_{2}}{2(1-c)}\left(1-\sqrt{1-4(1-c)\frac{X_{1}X_{2}}{(X_{1}+X_{2})^{2}}}\right)$
$\displaystyle=$
$\displaystyle\frac{S}{2(1-c)}\left(1-\sqrt{1-4(1-c)R(1-R)}\right)$
$\displaystyle=$ $\displaystyle\frac{S}{2(1-c)}\left(1-\sqrt{1-(1-c)B}\right)$

where

$\displaystyle S=X_{1}+X_{2},\quad R=\frac{X_{1}}{X_{1}+X_{2}}\ \
\textrm{and}\ \ B=4R(1-R).$

A well-known result (Grimmett and Stirzaker, , 2001, Section 4.7, Exercise 14)
says that if $X_{1}$ and $X_{2}$ are independent such that
$X_{i}\sim\operatorname{Ga}(\alpha_{i},\beta)$, $i=1,2$, then

$\displaystyle
S=X_{1}+X_{2}\sim\operatorname{Ga}(\alpha_{1}+\alpha_{2},\beta)\ \
\perp\\!\\!\\!\perp\ \
R=\frac{X_{1}}{X_{1}+X_{2}}\sim\operatorname{Be}(\alpha_{1},\alpha_{2}).$

We have $X_{i}\sim\operatorname{Ga}(1/2,1/2)$ and therefore
$S\sim\operatorname{Ga}(1,1/2)\stackrel{{\scriptstyle
d}}{{=}}\operatorname{Exp}(1/2)$ and $R\sim\operatorname{Be}(1/2,1/2)$ are
independent. The $\operatorname{Be}(1/2,1/2)$ is also known as the
$\operatorname{Arcsine}(0,1)$ distribution on the support $[0,1]$ (Rogozin, ,
2001). The general arcsine distribution $\operatorname{Arcsine}(a,a+b)$ on the
support $[a,a+b]$, $a\in\mathbb{R}$ and $b>0$, is obtained by the linear
transformation $a+bX$ with $X\sim\operatorname{Arcsine}(0,1)$. If $b<0$, then
$a+bX\sim\operatorname{Arcsine}(a+b,a)$. The arcsine distribution has the
property

$X\sim\operatorname{Arcsine}(-1,1)\quad\Rightarrow\quad
X^{2}\sim\operatorname{Arcsine}(0,1)$

which implies that $B=4R(1-R)=1-(1-2R)^{2}\sim\operatorname{Be}(1/2,1/2)$
holds with $S$ and $B$ independent.

It is clear that $F_{c}(y)=0$ for $y\leq 0$. For $y>0$, we now obtain

$\displaystyle F_{c}(y)$ $\displaystyle=$
$\displaystyle\operatorname{\mathsf{Pr}}(Y\leq y)$ (18) $\displaystyle=$
$\displaystyle\operatorname{\mathsf{Pr}}\left(S\left(1-\sqrt{1-(1-c)B}\right)\leq
2(1-c)y\right)$ $\displaystyle=$
$\displaystyle\int_{0}^{1}\operatorname{\mathsf{Pr}}\left(S\leq\frac{2(1-c)y}{1-\sqrt{1-(1-c)t}}\right)\frac{\Gamma(1)}{\Gamma(1/2)^{2}}t^{-1/2}(1-t)^{-1/2}dt$
$\displaystyle=$ $\displaystyle
1-\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{(1-c)y}{1-\sqrt{1+(1-c)t}}\right)\frac{1}{\sqrt{t(1-t)}}dt$

using independence of $S$ and $B$ and the fact that the cdf of
$\operatorname{Exp}(1/2)$ is given by $t\mapsto 1-\exp(-t/2),t>0$ and that
$1/\pi\int_{0}^{1}1/\sqrt{r(r-1)}dr=1$ (the density of
$\operatorname{Be}(1/2,1/2)$ integrates to $1$).

###  B.2 The case $c>1$

The solution of equation (17) is in this case

$\displaystyle Y$ $\displaystyle=$
$\displaystyle\frac{S}{2(c-1)}\left(\sqrt{1+(c-1)B}-1\right)$

with $S$ and $B$ defined as before. Thus, for $y>0$,

$\displaystyle F_{c}(y)$ $\displaystyle=$ $\displaystyle
1-\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{(c-1)y}{\sqrt{1+(c-1)t}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt.$
(19)

###  B.3 The expectation of $Y$

For $c=1$ we obtain $\operatorname{\mathsf{E}}(Y)=0.25$ from
$Y\sim\operatorname{Ga}(1/2,2)$.

For $c=0$ we have

$\displaystyle\operatorname{\mathsf{E}}(Y)$ $\displaystyle=$
$\displaystyle\min\\{X_{1},X_{2}\\}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(X_{1}+X_{2}-\left\lvert X_{1}-X_{2}\right\rvert)$
$\displaystyle=$ $\displaystyle\frac{1}{2}(z_{o}^{2}+z_{r}^{2}-\left\lvert
z_{o}^{2}-z_{r}^{2}\right\rvert)$ $\displaystyle=$
$\displaystyle\frac{1}{2}(z_{o}^{2}+z_{r}^{2}-\left\lvert
z_{o}-z_{r}\right\rvert\left\lvert z_{o}+z_{r}\right\rvert).$

Now
$\operatorname{Cov}(z_{o}-z_{r},z_{o}+z_{r})=\operatorname{Var}(z_{o})-\operatorname{Var}(z_{r})=0$
and hence ${z_{o}-z_{r}}$ and ${z_{o}+z_{r}}$ are independent
$\operatorname{N}(0,2)$ variables. This implies that $\left\lvert
z_{o}-z_{r}\right\rvert$ and $\left\lvert z_{o}+z_{r}\right\rvert$ are also
independent and identically distributed according to a half-normal
distribution with expectation $2/\sqrt{\pi}$. Thus,

$\displaystyle\operatorname{\mathsf{E}}(Y)$ $\displaystyle=$
$\displaystyle\frac{1}{2}(\operatorname{\mathsf{E}}(z_{o}^{2})+\operatorname{\mathsf{E}}(z_{r}^{2})-\operatorname{\mathsf{E}}(\left\lvert
z_{o}-z_{r}\right\rvert)\operatorname{\mathsf{E}}(\left\lvert
z_{o}+z_{r}\right\rvert))$ $\displaystyle=$
$\displaystyle\frac{1}{2}(1+1-(2/\sqrt{\pi})^{2})$ $\displaystyle=$
$\displaystyle 1-2/\pi\approx 0.36.$

For $0<c<1$, the random variable $Y$ has cdf (18) with expectation

$\displaystyle E(Y)$ $\displaystyle=$
$\displaystyle\int_{0}^{\infty}(1-F_{c}(y))dy$ $\displaystyle=$
$\displaystyle\frac{1}{\pi}\int_{0}^{\infty}\int_{0}^{1}\exp\left(-\frac{(1-c)y}{1-\sqrt{1-(1-c)t}}\right)\frac{1}{\sqrt{t(1-t)}}dt\,dy$
$\displaystyle=$
$\displaystyle\frac{1}{\pi}\int_{0}^{1}\frac{1}{\sqrt{t(1-t)}}\int_{0}^{\infty}\exp\left(-\frac{(1-c)y}{1-\sqrt{1-(1-c)t}}\right)dy\,dt.$

Now

$\displaystyle\int_{0}^{\infty}\exp\left(-\frac{(1-c)y}{1-\sqrt{1-(1-c)t}}\right)dy$
$\displaystyle=$ $\displaystyle\frac{1-\sqrt{1-(1-c)t}}{1-c}$

and therefore

$\displaystyle E(Y)$ $\displaystyle=$
$\displaystyle\frac{1}{\pi(1-c)}\int_{0}^{1}\frac{1-\sqrt{1-(1-c)t}}{\sqrt{t(1-t)}}dt.$

For $c>1$ we obtain with (19)

$\displaystyle E(Y)$ $\displaystyle=$
$\displaystyle\frac{1}{\pi(c-1)}\int_{0}^{1}\frac{\sqrt{1-(c-1)t}-1}{\sqrt{t(1-t)}}dt.$

##  Appendix C The limit of the $p$-value as $c\to\infty$

Recall that for $c>1$

$Z=\frac{S}{2(c-1)}\left(-1+\sqrt{1+(c-1)B}\right)$

with $S=X_{1}+X_{2}$ and $B=4R(1-R)$. The two-sided $p$-value
$4p(c)=1-F_{c}(z_{S}^{2})$ can be factorized as

$\displaystyle 4p(c)$ $\displaystyle=$
$\displaystyle\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{(c-1)Z}{\sqrt{1+(c-1)t}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{\pi}\int_{0}^{\eta}\exp\left(-\frac{(c-1)Z}{\sqrt{1+(c-1)t}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle+$
$\displaystyle\frac{1}{\pi}\int_{1-\eta}^{1}\exp\left(-\frac{(c-1)Z}{\sqrt{1+(c-1)t}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle+$
$\displaystyle\frac{1}{\pi}\int_{\eta}^{1-\eta}\exp\left(-\frac{(c-1)Z}{\sqrt{1+(c-1)t}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$ $\displaystyle A(\eta,c)+B(\eta,c)+C(\eta,c).$

It is easy to see that

$\displaystyle\sup_{c>1}A(\eta,c)\leq\int_{0}^{\eta}\frac{1}{\sqrt{t(1-t)}}dt\searrow
0,\ \ \textrm{as $\eta\searrow 0$}.$

Similarly,

$\displaystyle\sup_{c>1}B(\eta,c)\leq\int_{1-\eta}^{1}\frac{1}{\sqrt{t(1-t)}}dt\searrow
0,\ \ \textrm{as $\eta\nearrow 1$}.$

Now, we want to show that

$\displaystyle\lim_{c\to\infty}C(\eta,c)=\frac{1}{\pi}\int_{\eta}^{1-\eta}\exp\left(-\frac{S\sqrt{B}}{2\sqrt{t}}\right)\frac{1}{\sqrt{t(1-t)}}dt.$

Let us put

$\displaystyle\Delta(\eta,c)=\frac{1}{\pi}\int_{\eta}^{1-\eta}\left\\{\exp\left(\frac{S\left(-1+\sqrt{1+(c-1)B}\right)}{2\left(\sqrt{1+(c-1)t}-1\right)}\right)-\exp\left(\frac{S\sqrt{B}}{2\sqrt{t}}\right)\right\\}\frac{1}{\sqrt{t(1-t)}}dt.$

We show now that $\lim\limits_{c\to\infty}\Delta(\eta,c)=0$. It is enough to
show that for a $\eta>0$ small enough,

$\displaystyle\lim_{c\to\infty}\sup_{t\in[\eta,1-\eta]}\left\lvert\,\exp\left(\frac{S\left(-1+\sqrt{1+(c-1)B}\right)}{2\left(\sqrt{1+(c-1)t}-1\right)}\right)-\exp\left(\frac{S\sqrt{B}}{2\sqrt{t}}\right)\right\rvert=0.$

By the mean-value theorem, $\exp(y)-\exp(x)=\exp(\theta^{*}_{x,y})(y-x)$ for
some $\theta^{*}_{x,y}$ between $x$ and $y$. Thus,
$\left\lvert\exp(y)-\exp(x)\right\rvert\leq\exp(\max(x,y))\left\lvert
y-x\right\rvert$. In what follows,

$\displaystyle x=\frac{S\sqrt{B}}{2\sqrt{t}}\ \ \ \textrm{and}\ \ \
y=\frac{S(-1+\sqrt{1+(c-1)B})}{2\left(\sqrt{1+(c-1)t}-1\right)}.$

We have

$\displaystyle y$ $\displaystyle=$
$\displaystyle\frac{S}{2}\times\frac{\sqrt{B(c-1)}\left(\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}\right)}{\sqrt{(c-1)t}\left(\sqrt{1+\frac{1}{\sqrt{(c-1)t}}}-\frac{1}{\sqrt{(c-1)t}}\right)}$
$\displaystyle=$
$\displaystyle\frac{S}{2\sqrt{t}}\times\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}$
$\displaystyle=$ $\displaystyle
x\times\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}$

where

$\displaystyle\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}$
$\displaystyle=$
$\displaystyle\frac{\sqrt{1+\frac{1}{(c-1)t}}+\frac{1}{\sqrt{(c-1)t}}}{\sqrt{1+\frac{1}{(c-1)B}}+\frac{1}{\sqrt{(c-1)B}}}$
$\displaystyle\leq$
$\displaystyle\sqrt{1+\frac{1}{(c-1)t}}+\frac{1}{\sqrt{(c-1)t}}$
$\displaystyle\leq$
$\displaystyle\sqrt{1+\frac{1}{(c-1)\eta}}+\frac{1}{\sqrt{(c-1)\eta}}$
$\displaystyle\leq$ $\displaystyle\frac{3}{2}$

for $c$ large enough. It follows that

$\displaystyle\left\lvert\exp(y)-\exp(x)\right\rvert$ $\displaystyle\leq$
$\displaystyle\exp\left(\frac{3}{2}x\right)\left\lvert y-x\right\rvert$ (20)
$\displaystyle\leq$
$\displaystyle\exp\left(\frac{3S\sqrt{B}}{4\sqrt{\eta}}\right)\left\lvert
x\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}-x\right\rvert$
$\displaystyle\leq$
$\displaystyle\exp\left(\frac{3S\sqrt{B}}{4\sqrt{\eta}}\right)\frac{S\sqrt{B}}{2\sqrt{\eta}}\left\lvert\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}-1\right\rvert$

where

$\displaystyle\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}-1=\frac{\sqrt{1+\frac{1}{(c-1)t}}+\frac{1}{\sqrt{(c-1)t}}}{\sqrt{1+\frac{1}{(c-1)B}}+\frac{1}{\sqrt{(c-1)B}}}-1$

implying that

$\displaystyle\frac{\sqrt{1+\frac{1}{(c-1)(1-\eta)}}+\frac{1}{\sqrt{(c-1)(1-\eta)}}}{\sqrt{1+\frac{1}{(c-1)B}}+\frac{1}{\sqrt{(c-1)B}}}-1\leq\frac{\sqrt{1+\frac{1}{(c-1)B}}-\frac{1}{\sqrt{(c-1)B}}}{\sqrt{1+\frac{1}{(c-1)t}}-\frac{1}{\sqrt{(c-1)t}}}-1\leq\frac{\sqrt{1+\frac{1}{(c-1)\eta}}+\frac{1}{\sqrt{(c-1)\eta}}}{\sqrt{1+\frac{1}{(c-1)B}}+\frac{1}{\sqrt{(c-1)B}}}-1.$

As the terms in both sides of the inequality converge to $0$ as
$c\nearrow\infty$ we conclude that the absolute value of the term in the
middle converges to $0$ and hence the right side of the inequality in (20) has
also to converge to $0$ where the convergence is uniform for
$t\in[\eta,1-\eta]$. It follows that
$\lim\limits_{c\nearrow\infty}\Delta(\eta,c)=0$ and hence, for $\eta$ small
enough

$\displaystyle\lim_{c\nearrow\infty}C(\eta,c)=\frac{1}{\pi}\int_{\eta}^{1-\eta}\exp\left(-\frac{S\sqrt{B}}{2\sqrt{t}}\right)\frac{1}{\sqrt{t(1-t)}}dt.$

Therefore, and since $4p(c)$ does not depend on $\eta$

$\displaystyle\lim_{c\nearrow\infty}4p(c)$ $\displaystyle=$
$\displaystyle\lim_{\eta\searrow 0}\lim_{c\nearrow\infty}4p(c)$ (21)
$\displaystyle=$ $\displaystyle
0+0+\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{S\sqrt{B}}{2\sqrt{t}}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{S\sqrt{B}}{2\sqrt{t}}\right)\frac{1}{\sqrt{t(1-t)}}dt$

which finishes the proof because

$\displaystyle W=\frac{S\sqrt{B}}{2}=\sqrt{X_{1}X_{2}}=\left\lvert
z_{o}z_{r}\right\rvert.$

$\Box$

##  Appendix D Uniform distribution of the limiting $p$-value

We want to show that the $p$-value from (21) is uniformly distributed on
$[0,1]$. To this aim, we need to determine the density of
$S\sqrt{B}/2=\left\lvert z_{o}z_{r}\right\rvert$ with $z_{o},z_{r}$ i.i.d
$\sim\mathcal{N}(0,1)$. The density of the variable $V=z_{o}z_{r}$ is

$\displaystyle f_{V}(v)=\frac{1}{\pi}K_{0}(\left\lvert v\right\rvert)$

with $K_{0}$ is second class zero order modified Bessel function (Craig, ,
1936). Now, the cumulative distribution of $W$ is given for $w>0$ by
$F_{W}(w)=2F_{V}(w)-1$, implying that density of $W$ is given by

$\displaystyle f_{W}(w)=2f_{V}(w)=\frac{2}{\pi}K_{0}(w),\ \ \textrm{for
$w>0$}.$

To show that the $p$-value from (21) is uniformly distributed on $[0,1]$ it is
enough to show that 1 minus the $p$-value from (21) is uniformly distributed
on $[0,1]$. This is equivalent to showing that the density of $W$ is also
equal to the derivative of

$\displaystyle w\mapsto
1-\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{w}{\sqrt{t}}\right)\frac{1}{\sqrt{t(1-t)}}dt.$

Since this derivative is

$\displaystyle
w\mapsto\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{w}{\sqrt{t}}\right)\frac{1}{t\sqrt{1-t}}dt,$

we need to show that

$\displaystyle\frac{1}{\pi}\int_{0}^{1}\exp\left(-\frac{w}{\sqrt{t}}\right)\frac{1}{t\sqrt{1-t}}dt=\frac{2}{\pi}K_{0}(w)$

for all $w>0$ or equivalently

$\displaystyle\int_{0}^{1}\exp\left(-\frac{w}{\sqrt{t}}\right)\frac{1}{t\sqrt{1-t}}dt=2K_{0}(w).$
(22)

It is known that for $n>-1/2$, the second class modified Bessel function of
order $n$ is for $x>0$

$\displaystyle
K_{n}(x)=\frac{\sqrt{\pi}}{\Gamma(n+1/2)}\left(\frac{1}{2}z\right)^{n}\int_{1}^{\infty}\exp(-xt)(t^{2}-1)^{n-1/2}dt.$

Thus, for $n=0$,

$\displaystyle K_{0}(x)=\int_{1}^{\infty}\frac{\exp(-xt)}{\sqrt{t^{2}-1}}dt.$

Thus, to show the identity in (22), it is enough to show that

$\displaystyle\int_{0}^{1}\exp\left(-\frac{w}{\sqrt{t}}\right)\frac{1}{t\sqrt{1-t}}dt=2\int_{1}^{\infty}\frac{\exp(-wt)}{\sqrt{t^{2}-1}}dt$

for all $w>0$. Using the change of variable $u=1/\sqrt{t}$ we obtain

$\displaystyle\int_{0}^{1}\exp\left(-\frac{w}{\sqrt{t}}\right)\frac{1}{t\sqrt{1-t}}dt$
$\displaystyle=$
$\displaystyle\int_{1}^{\infty}\exp(-wu)u^{2}\frac{1}{\sqrt{1-1/u^{2}}}\frac{2}{u^{3}}du$
$\displaystyle=$ $\displaystyle
2\int_{1}^{\infty}\frac{\exp(-wu)}{\sqrt{u^{2}-1}}du$

which completes the proof. $\Box$

##  Appendix E Derivative of the sceptical $p$-value

In the following, we write $u=c-1$. Then, we have

$\displaystyle p_{S}^{*}$ $\displaystyle=$ $\displaystyle
p_{S}^{*}(u)=\frac{1}{4\pi}\int^{1}_{0}\exp\left(-z^{2}_{A}\frac{\sqrt{1+uz_{H}^{2}/z^{2}_{A}}-1}{\sqrt{1+ut}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{4\pi}\int^{1}_{0}\exp\left(-z^{2}_{A}\frac{\sqrt{1+uB}-1}{\sqrt{1+ut}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$

with $B=z_{H}^{2}/z^{2}_{A}$. It is not difficult to show that the function on
the left can be differentiated with respect to $u$ under the sign integral.
Then, for $u\in\mathbb{R}$

$\displaystyle
p_{S}^{{}^{\prime}*}(u)=-\frac{z^{2}_{A}}{4\pi}\int^{1}_{0}\exp\left(-z^{2}_{A}\frac{\sqrt{1+uB}-1}{\sqrt{1+ut}-1}\right)\frac{d}{du}\left\\{\frac{\sqrt{1+uB}-1}{\sqrt{1+ut}-1}\right\\}\frac{1}{\sqrt{t(1-t)}}dt.$

We compute

$\displaystyle\frac{d}{du}\left\\{\frac{\sqrt{1+uB}-1}{\sqrt{1+ut}-1}\right\\}$
$\displaystyle=$
$\displaystyle\frac{B(1+ut-\sqrt{1+ut})-t(1+uB-\sqrt{1+uB})}{2(\sqrt{1+ut}-1)^{2}\sqrt{1+uB}\sqrt{1+ut}}$
$\displaystyle=$
$\displaystyle\frac{B-t-B\sqrt{1+ut}+t\sqrt{1+uB}}{2(\sqrt{1+ut}-1)^{2}\sqrt{1+uB}\sqrt{1+ut}}$
$\displaystyle=$
$\displaystyle\frac{\left(B-t-B\sqrt{1+ut}+t\sqrt{1+uB}\right)(\sqrt{1+ut}+1)^{2}}{2u^{2}t^{2}\sqrt{1+uB}\sqrt{1+ut}}.$

Thus, the derivative of the sceptical $p$-value for $c>0$ and $c\neq 1$ is
given by

$\displaystyle-\frac{z^{2}_{A}}{8\pi}\int_{0}^{1}\exp\left(-z^{2}_{A}\frac{\sqrt{1+(c-1)B}-1}{\sqrt{1+(c-1)t}-1}\right)k_{c,B}(t)dt$

with

$\displaystyle
k_{c,B}(t)=\frac{\left(B-t-B\sqrt{1+(c-1)t}+t\sqrt{1+(c-1)B}\right)(\sqrt{1+(c-1)t}+1)^{2}}{(c-1)^{2}\sqrt{1+(c-1)B}\sqrt{1+(c-1)t}}\frac{1}{t^{5/2}\sqrt{1-t}}dt.$

For $c=1$, the derivative is given by

$\displaystyle-\frac{z^{2}_{A}}{16\pi}\int_{0}^{1}\exp\left(-z^{2}_{A}\frac{B}{t}\right)\frac{t-B}{t^{3/2}\sqrt{1-t}}dt.$

##  Appendix F Monotonicity of the sceptical $p$-value for the case
$z_{o}=z_{r}$

In the case $z_{o}=z_{r}$, we have that

$\displaystyle z^{2}_{S}=\frac{z^{2}_{o}}{c-1}(\sqrt{c}-1)$

and hence the one-sided $p$-value $p$ is

$\displaystyle p=\left[1-F_{c}(z^{2}_{S})\right]/4$ $\displaystyle=$
$\displaystyle\frac{1}{4\pi}\int_{0}^{1}\exp\left(-z^{2}_{o}\frac{\sqrt{c}-1}{\sqrt{1+(c-1)t}-1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{4\pi}\int_{0}^{1}\exp\left(-z^{2}_{o}\frac{(\sqrt{c}-1)\sqrt{1+(c-1)t}+1}{(c-1)t}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{4\pi}\int_{0}^{1}\exp\left(-\frac{z^{2}_{o}}{t}\frac{\sqrt{1+(c-1)t}+1}{\sqrt{c}+1}\right)\frac{1}{\sqrt{t(1-t)}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{4\pi}\int_{0}^{1}\exp\left(-\frac{z^{2}_{o}}{t}\psi_{t}(c)\right)\frac{1}{\sqrt{t(1-t)}}dt$

with

$\displaystyle\psi_{t}(c)=\frac{\sqrt{1+(c-1)t}+1}{\sqrt{c}+1},(t,c)\in[0,1]\times[0,\infty).$

If we show that $\psi_{t}$is monotone decreasing in $c$ for all $t\in[0,1]$,
then this would imply the sceptical $p$-value $p_{S}^{*}=\sqrt{p}$ is monotone
increasing in $c$. Fix $t\in[0,1]$. The derivative of $\psi_{t}$ for $c>0$ is
given by

$\displaystyle\psi^{\prime}_{t}(c)$ $\displaystyle=$
$\displaystyle\frac{\frac{t}{2\sqrt{1+(c-1)t}}(\sqrt{c}+1)-\frac{\sqrt{1+(c-1)t}+1}{2\sqrt{c}}}{(\sqrt{c}+1)^{2}}$
$\displaystyle=$
$\displaystyle\frac{t(\sqrt{c}+1)\sqrt{c}-(\sqrt{1+(c-1)t}+1)\sqrt{1+(c-1)t}}{2\sqrt{1+(c-1)t}\sqrt{c}(\sqrt{c}+1)^{2}}$
$\displaystyle=$
$\displaystyle\frac{t(c+\sqrt{c})-(1+(c-1)t)-\sqrt{1+(c-1)t}}{2\sqrt{1+(c-1)t}\sqrt{c}(\sqrt{c}+1)^{2}}$
$\displaystyle=$
$\displaystyle\frac{t(\sqrt{c}+1)-1-\sqrt{1+(c-1)t}}{2\sqrt{1+(c-1)t}\sqrt{c}(\sqrt{c}+1)^{2}}.$

We show now that $t(\sqrt{c}+1)\leq\sqrt{1+(c-1)t}+1$ for all $c>0$. This is
equivalent to showing that $t(\sqrt{c}+1)-1\leq\sqrt{1+(c-1)t}$. If
$t(\sqrt{c}+1)-1\leq 0$, then this is obviously true. Suppose now that
$t(\sqrt{c}+1)-1>0$. Then, it is enough to show that

$\displaystyle\left(t(\sqrt{c}+1)-1\right)^{2}\leq 1+(c-1)t.$

We compute

$\displaystyle\left(t(\sqrt{c}+1)-1\right)^{2}-(1+(c-1)t)$ $\displaystyle=$
$\displaystyle t^{2}(c+2\sqrt{c}+1)-2t\sqrt{c}-2t-ct+t$ $\displaystyle=$
$\displaystyle t^{2}(c+2\sqrt{c}+1)-2t\sqrt{c}-ct-t$ $\displaystyle=$
$\displaystyle(t^{2}-t)\left(c+2\sqrt{c}+1\right)=t(t-1)(\sqrt{c}+1)^{2}\leq
0$

and the proof is completed.

##  Appendix G Design of replication studies

Figure G.1 shows the ratio of predictive power and replication sample size
calculated with the sceptical $p$-value versus the two-trials rule. Some large
predictive power values cannot be reached regardless of the sample size
(Micheloud and Held, , 2022). This explains the white area (upper-right
corner) in the right plot. Figure G.2 shows the maximum conditional T1E rate
of the controlled sceptical $p$-value as a function of $\alpha$ when the
sample size is calculated based on a conditional power of 80%, 90% and 95%,
respectively. The conditional T1E rate is smaller than $2\alpha$ for values of
$\alpha$ larger than $0.01$, $0.015$ and $0.019$, respectively. Figure G.3
shows the conditional T1E rate in the case where the replication sample size
was calculated based on predictive power. Figure G.4 shows the conditional T1E
rate based on nominal and golden sceptical $p$-values.

Figure G.1:  Ratio of predictive power (left) and replication sample sizes
(right) between the sceptical $p$-value and the two-trials rule as a function
of the original $p$-value $p_{o}$ and the relative sample size $c$ (left) or
the predictive power (right) for $\alpha=0.025$. Figure G.2: Maximum
conditional T1E rate of the controlled sceptical $p$-value as a function of
the level $\alpha$. Figure G.3: Conditional Type-I error rate as a function of
the original $p$-value $p_{o}$. The replication sample size is calculated with
the sceptical $p$-value method to reach a predictive power of 80%, 90%, and
95% with $\alpha=0.025$. Each dot represents the upper bound for the
conditional T1E rate with the respective power. The gray horizontal line
indicates the T1E rate of the two-trials rule. Figure G.4: Conditional T1E
error rate as a function of the original $p$-value $p_{o}$ for the nominal and
golden sceptical $p$-values. The replication sample size is calculated with
the sceptical $p$-value method to reach a conditional power of 80%, 90%, and
95% with $\alpha=0.025$. The relative sample size $c$ on the top axis is
calculated with the golden sceptical $p$-value. The gray horizontal line
indicates the T1E rate of the two-trials rule.