#  Bayesian questions with frequentist answers

Alan H. Guth<EMAIL_ADDRESS>Department of Physics, Laboratory for Nuclear
Science, and Center for Theoretical Physics, Massachusetts Institute of
Technology, Cambridge, MA 02139  Mohammad Hossein Namjoo<EMAIL_ADDRESS>School of Astronomy, Institute for Research in Fundamental Sciences (IPM),
Tehran, Iran, P.O. Box 19395-5531

###### Abstract

The two statistical methods, namely the frequentist and the Bayesian methods,
are both commonly used for probabilistic inference in many scientific
situations. However, it is not straightforward to interpret the result of one
approach in terms of the concepts of the other. In this paper we explore the
possibility of finding a Bayesian significance for the frequentist’s main
object of interest, the $p$-value, which is the probability assigned to the
proposition — which we call the extremity proposition — that a measurement
will result in a value that is at least as extreme as the value that was
actually obtained. To make contact with the frequentist language, the Bayesian
can choose to update probabilities based on the extremity proposition, which
is weaker than the standard Bayesian update proposition, which uses the actual
observed value. We then show that the posterior probability (or probability
density) of a theory is equal to the prior probability (or probability
density) multiplied by the ratio of the $p$-value for the data obtained, given
that theory, to the mean $p$-value — averaged over all theories weighted by
their prior probabilities. Thus, we provide frequentist answers to Bayesian
questions. Our result is generic — it does not rely on restrictive assumptions
about the situation under consideration or specific properties of the
likelihoods or the priors.

###### Contents

  1. I Introduction
  2. II A general analysis
  3. III One-sided hypothesis testing
  4. IV Two-sided hypothesis testing
  5. V Threshold testing
  6. VI Conclusion

##  I Introduction

The two major statistical schools of thought, namely the frequentist and the
Bayesian methods, have been broadly used in many scientific situations for
probabilistic inference. The advantages and disadvantages of each competing
approach have been debated for a long time. On the other hand, the connection
between the two approaches, if any, is also of great interest. Despite the
efforts put in this regard, such connections have been studied, to our
knowledge, only in some limited situations. We briefly mention a few efforts
in this direction but do not necessarily try to be comprehensive.

J. O. Berger and Sellke Berger1987  studied the point null hypothesis111A
point null hypothesis is a hypothesis $H_{0}$ with a sharp prediction about a
continuous parameter, i.e. $H_{0}:\mu=\mu_{*}$.  test. Considering a large
class of prior probability densities, they showed that the $p$-value and the
posterior probability for the null hypothesis can be very different. Thus,
they concluded that the two methods cannot be reconciled. See also an earlier
paper by Dickey Dickey  with a similar conclusion.

Inspired by (but in contrast with) Ref. Berger1987 , Casella and R. L. Berger
Casella1987  studied the connection between the $p$-value and the posterior
probabilities for the case that we call threshold testing.222We use the phrase
“threshold testing” to describe a test of a hypothesis of the form
$H_{0}:\mu<\mu_{*}$ for some particular value $\mu_{*}$. Note that in Ref.
Casella1987  this test is called “one-sided”, while we use the phrase one-
sided in a different way in our Sec. III. Since the posterior probability
depends on the choice of a prior probability while the $p$-value does not,
Ref. Casella1987  compares the $p$-value with the upper or lower bound of the
posterior probabilities for several classes of prior probability densities.
Restricting the analysis to symmetric, location likelihood densities333 A
likelihood density $p({\mu_{D}}|{\theta})$ — the probability density for
obtaining the measured value ${\mu_{D}}$, given a theory described by the
parameter $\theta$ — is called a location likelihood density if it has the
form $p({\mu_{D}}|{\theta})=f(|{\theta}-{\mu_{D}}|)$. which have a monotone
likelihood ratio,444A probability density $p({\mu_{D}}|{\theta})$ has a
monotone likelihood ratio if the ratio
$p({\mu_{D}}|{\theta}_{1})/p({\mu_{D}}|{\theta}_{2})$ is non-decreasing in
${\mu_{D}}$ whenever ${\theta}_{1}>{\theta}_{2}$ Karlin1956 . Ref. Casella1987
shows that the $p$-value is equal to the infimum of the posterior probability
for specific classes of prior probability densities and greater than or equal
to the infimum of the posterior probability for other specific classes of
prior probability densities. While this result does not show equivalence
between the two methods, it nonetheless establishes a relationship between the
two methods in certain situations.

Micheas and Dey Micheas  did a similar analysis to Ref. Casella1987 , but
studied one-sided scale parameter testing.555A scale parameter test is one for
which the likelihood density belongs to a scale parameter family, i.e., it
satisfies $p({\mu_{D}}|{\theta})=\frac{1}{{\theta}}f({\mu_{D}}/{\theta})$. The
Bayesian variables that they considered are the prior and the posterior
predictive $p$-values.666The prior predictive $p$-value is defined by $\int
P(\mu\geq{\mu_{D}}|{\theta})p({\theta})d{\theta}$, where $p({\theta})$ is the
prior probability density and ${\mu_{D}}$ is the measured value. The posterior
predictive $p$-value is defined analogously with the prior in the integrand
replaced with the posterior probability density. For several classes of prior
densities, they proved that the infimum of the prior and the posterior
predictive $p$-values are both equal to the $p$-value.

In this paper, we formulate a Bayesian question for which the answer may be
given in terms of frequentist quantities. In contrast to the papers cited
above, we consider a more general class of questions that a Bayesian analyst
could ask. A key difference between the frequentist approach and the standard
Bayesian approach is that the frequentist’s $p$-value is the probability,
according to some theory, that a typical measurement would give a result more
“extreme” than the value that is actually measured. The standard Bayesian
approach, however, depends only on the probability or the probability density,
according to the theory, of obtaining the result that was actually obtained.
To bridge this gap, we consider the possibility that a Bayesian analyst could
choose to update his probabilities not by using directly the measured value,
but instead by using the weaker statement that the measurement yielded a value
at least as “extreme” as the measured value. Using this freedom, we can
construct an expression for posterior probabilities written in terms of
$p$-values, that holds without any restrictive assumptions about the nature of
the tests or the properties of prior probability densities or likelihood
densities. This expression gives the Bayesian analyst a way to understand the
question to which the $p$-value is the answer.

##  II A general analysis

In this section we review the Bayesian method, introduce some notation, and
also obtain some general results that may be applied to specific examples that
will be given in the following sections. The theories to be tested, which we
denote by ${\cal{T}}({\theta})$, are theories that attempt to predict the
value of the measurable quantity $\mu$. More precisely, the theory
${\cal{T}}({\theta})$ predicts a probability density function (PDF)
$p(\mu|{\theta})$ that $\mu$ is observed, given ${\cal{T}}({\theta})$. The
parameter ${\theta}$ that labels the theories might be taken to be the mean,
median, or mode of $p(\mu|{\theta})$, but any parametrization can be used. The
PDF $p(\mu|{\theta})$ (which is also called the likelihood density)
incorporates the possibly probabilistic nature of the prediction as well as
any uncertainty due to the imprecision of the measurement.

A probability for a continuous variable is conventionally described by a PDF:
$p(\tilde{\mu})d\tilde{\mu}$ is the probability that the continuous variable
$\mu$ lies between $\tilde{\mu}$ and $\tilde{\mu}+d\tilde{\mu}$. In discussing
a hypothesis $H$ which has a non-zero probability, the probability is
conventionally indicated by $P(H)$. In some cases a non-zero probability might
be assigned to a particular value of a continuous parameter (e.g., we might
assign a non-zero probability to the hypothesis that the magnitude of the
cosmological dipole modulation is exactly zero Planck:2019evm ; Akrami:2014eta
). In this case, the situation can be described by a non-zero probability
$P(H)$, or we can use a probability density function $p({\theta})$ that
contains a delta function such as $P_{H}\delta({\theta}-\mu_{H})$. In what
follows we will discuss continuous variables in terms of probability density
functions, assuming that any assignment of non-zero probabilities to
particular values can be treated using delta functions.

We first obtain some general relations describing a situation in which a
single real continuous parameter $\mu$ is measured. Suppose that a measurement
is made yielding the data $D$, where we have in mind that $D$ represents a
proposition about the outcome with a non-zero probability. For example, $D$
could be the statement that the measured variable $\mu$ lies in a specified
bin, or that $\mu$ is greater than some particular value. The Bayes formula
for the posterior probability density $p({\theta}|D)$, after a measurement
yielding the data $D$, is given by:

$\displaystyle p({\theta}|D)=\dfrac{P(D|{\theta})p({\theta})}{\langle
P(D)\rangle_{\text{pr}}},$ (1)

where $p({\theta})$ is the previously assumed probability density for
${\theta}$ (i.e., the so-called prior probability density), $P(D|{\theta})$ is
the probability of $D$ given the theory ${\cal{T}}({\theta})$, and

$\displaystyle\langle p(D)\rangle_{\text{pr}}\equiv\int
P(D|{\theta})p({\theta})d{\theta}.$ (2)

Note that the denominator of Eq. (1) is fixed by the requirement that
$p({\theta}|D)$ is normalized.

In the frequentist approach, it is the so-called $p$-value that is usually
computed. The $p$-value is the probability, under the null hypothesis, that a
measurement will yield a value that is at least as “extreme” as the actual
measurement. The meaning of extremity depends on the situation under
consideration, as we shall discuss in the subsequent sections. For now, we
obtain general relations that can be applied to any definition of extremity,
allowing for the possibility that the definition of extremity might depend on
the theory ${\cal{T}}({\theta})$. Denoting the measured value by ${\mu_{D}}$,
we denote the set of extreme values of ${\mu_{D}}$ as follows:

$\displaystyle U_{\theta}({\mu_{D}})$ $\displaystyle\equiv$
$\displaystyle\\{{\mu}\,|\,\text{${\mu}$ is at least as extreme as
${\mu_{D}}$, under ${\cal{T}}({\theta})$}\\}.$ (3)

Then, the $p$-value for the measurement ${\mu_{D}}$ under the theory
${\cal{T}}({\theta})$ can be defined by

$\displaystyle
p_{\text{val},\,{\theta}}({\mu_{D}})\equiv\int_{U_{\theta}({\mu_{D}})}p({\mu}|{\theta})d{\mu}.$
(4)

Note that, as mentioned, the conventional practice for a frequentist is to
compute the $p$-value for the null hypothesis. That is, denoting the null
hypothesis by ${\cal{T}}(\theta_{s})$ — where $s$ stands for standard — the
standard $p$-value is given by

$\displaystyle p_{\text{val}}({\mu_{D}})\equiv
p_{\text{val},\,\theta_{s}}({\mu_{D}}).$ (5)

Thus, Eq. (4) is a generalization of the standard $p$-value (and we
distinguish the generalized $p$-value, Eq. (4), from the traditional one, Eq.
(5), by the subscript ${\theta}$ on the former). It is, however, a minor
generalization, as a generalized $p$-value for one fequentist might be the
standard $p$-value for another frequentist who is considering a different null
hypothesis. This generalization will allow us to relate the Bayesian to the
frequentist.

Now we consider what question a Bayesian might ask, if he is trying to
understand the language of the frequentists. In a standard Bayesian treatment,
$p({\theta})$ would be updated using the data that $\mu$ has been measured
with the result ${\mu_{D}}$. However, to make contact with the frequentist
approach, a Bayesian can instead use only the weaker statement that the
measured value is at least as extreme as ${\mu_{D}}$. That is, the Bayesian
can update the probabilities assigned to theories using the proposition
$D^{\prime}_{\theta}({\mu_{D}})$,

$\displaystyle D^{\prime}_{\theta}({\mu_{D}})\equiv\text{The hypothesis that
the measured value $\mu$ is in the set}\,U_{\theta}({\mu_{D}}),$ (6)

which we refer to as the extremity proposition. This immediately implies that

$\displaystyle
p_{\text{val},\,{\theta}}({\mu_{D}})=P(D^{\prime}_{\theta}({\mu_{D}})|{\theta}).$
(7)

As a consequence, Bayes’ theorem (Eq. (1)) becomes

$\displaystyle
p\big{(}{\theta}|D^{\prime}_{\theta}({\mu_{D}})\big{)}=\dfrac{p_{\text{val},\,{\theta}}({\mu_{D}})p({\theta})}{\langle
p_{\text{val}}({\mu_{D}})\rangle_{\text{pr}}}\,,$ (8)

where

$\displaystyle\langle p_{\text{val}}({\mu_{D}})\rangle_{\text{pr}}\equiv\int
p_{\text{val},\,{\theta}}({\mu_{D}})p({\theta})d{\theta}.$ (9)

Eq. (8) is our key relation: as a result of the specific question that was
asked, the posterior probability density can be written in terms of the prior
probability density and the generalized $p$-value.

In the following sections we will examine how Eq. (8) works in specific
situations. Different situations lead to different natural definitions of
extremity, which in turn lead to more explicit expressions for
$p\big{(}{\theta}|D^{\prime}_{\theta}({\mu_{D}})\big{)}$.

##  III One-sided hypothesis testing

We first consider one-sided hypothesis testing. An example could be the
measurement of a chemical concentration, such as the concentration of lead in
drinking water. We denote the concentration of lead by a continuous variable
$\mu$. We again consider a one parameter family of theories
${\cal{T}}({\theta})$ with ${\cal{T}}(\theta_{s})$ taken as the null
hypothesis. In this example, it is natural to take ${\theta}$ to be the actual
concentration of lead in the water, but in general we view ${\theta}$ as an
arbitrary parametrization. In this example, the spread of $p(\mu|{\theta})$
would be determined entirely by experimental uncertainties. For the case of
one-sided hypothesis testing, extremity is naturally defined by

$\displaystyle
U_{{\theta}}({\mu_{D}})\equiv\left\\{{\mu}\,|\,{\mu}\geq{\mu_{D}}\right\\}.$
(10)

According to Eq. (10), being more extreme means being farther out into the
tail. In this case the definition of extremity does not depend on ${\theta}$,
but we still use the notation $U_{{\theta}}({\mu_{D}})$ for consistency with
our general formula. Using this definition, according to Eq. (7), the
generalized $p$-value is given by

$\displaystyle
p_{\text{val},\,{\theta}}({\mu_{D}})=P({\mu}\geq{\mu_{D}}|{\theta})=\int_{{\mu_{D}}}^{\infty}p({\mu}|{\theta})d{\mu}.$
(11)

See Fig. 1 (left) for a visualization of the generalized $p$-value. Eq. (11)
can be used with Eq. (8) to obtain the posterior probability density for
${\theta}$, giving

$\displaystyle
p\big{(}{\theta}|D^{\prime}_{\theta}({\mu_{D}})\big{)}=\dfrac{p({\theta})\int_{{\mu_{D}}}^{\infty}p({\mu}|{\theta})d{\mu}}{\int_{0}^{\infty}\left[p({\theta^{\prime}})\int_{{\mu_{D}}}^{\infty}p({\mu}|{\theta^{\prime}})d{\mu}\right]d{\theta^{\prime}}}.$
(12)

In the above equation we assumed that $\theta=0$ is the lower bound, as would
be the case if $\theta=0$ corresponds to zero concentration. Alternatively,
$\theta$ could for example be identified with log(concentration), in which
case the lower limit of the integral over $\theta^{\prime}$ in Eq. (12) would
be changed to $-\infty$. In this case, it would still be natural to take Eq.
(10) as the definition of extremity, since small values of the concentration
(corresponding to large negative values of log(concentration)) are not
concerning.

Figure 1: Illustration of the generalized $p$-value, for a given theory
${\cal{T}}({\theta})$, in one-sided (left) and two-sided (right) testing.
$p(\mu|{\theta})$ is the probability distribution for measuring $\mu$
according to the theory ${\cal{T}}({\theta})$. The generalized $p$-values are
given by the areas of the shaded regions. The right graph is drawn under the
assumption that $P({\mu}\leq{\mu_{D}}|{\theta})<1/2$. $\bar{\mu}_{D}$ is
defined via Eq. (13), so that the area of the two shaded regions are equal and
the entire shaded region depicts the generalized $p$-value.

##  IV Two-sided hypothesis testing

We next consider two-sided, not necessarily symmetric, hypothesis testing. To
obtain some intuition about what extremity might mean in this situation, let
us consider the case where $P({\mu}\leq{\mu_{D}}|{\theta})<1/2$, i.e., the
measured value is located on the left half of the distribution. (The other
possibility can be studied analogously.) Clearly, any value of $\mu$ less than
or equal to ${\mu_{D}}$ can be considered at least as extreme as ${\mu_{D}}$
(just like the one-sided case). However, for a two-sided distribution, there
are values of $\mu$ on the other side which can be viewed as equally extreme.
A natural choice is to define an extreme region on the right with equal
probability to the extreme region on the left. That is, define
${\bar{\mu}}_{D}$ such that

$\displaystyle
P({\mu}\geq{\bar{\mu}}_{D}|{\theta})=P({\mu}\leq{\mu_{D}}|{\theta})\,,$ (13)

and then define the set of extreme values as
$U_{\theta}({\mu_{D}})=\left\\{{\mu}\,|\,{\mu}\geq{\bar{\mu}}_{D}\,\,\text{or}\,\,{\mu}\leq{\mu_{D}}\right\\}$.
Consequently, from Eq. (4), the generalized $p$-value is
$p_{\text{val},\,{\theta}}({\mu_{D}})=P({\mu}\leq{\mu_{D}}|{\theta})+P({\mu}\geq{\bar{\mu}}_{D}|{\theta})=2P({\mu}\leq{\mu_{D}}|{\theta})$.
Dropping the simplifying assumption that $P({\mu}\leq{\mu_{D}}|{\theta})<1/2$,
these results generalize to

$\displaystyle U_{\theta}({\mu_{D}})$ $\displaystyle=$
$\displaystyle\left\\{{\mu}\,|\,{\mu}\geq\max\\{{\mu_{D}},{\bar{\mu}}_{D}\\}\,\,\text{or}\,\,{\mu}\leq\min\\{{\mu_{D}},{\bar{\mu}}_{D}\\}\right\\}\,,$
(14) $\displaystyle p_{\text{val},\,{\theta}}({\mu_{D}})$ $\displaystyle=$
$\displaystyle
2\min\\{P({\mu}\geq{\mu_{D}}|{\theta}),P({\mu}\leq{\mu_{D}}|{\theta})\\}\,,$
(15)

where ${\bar{\mu}}_{D}$ is still defined by Eq. (13). See Fig. 1 (right) for a
visualization of the generalized $p$-value for two-sided testing. Eqs. (14)
and (15) can then be used to obtain the posterior probability density when
$D^{\prime}_{\theta}({\mu_{D}})$ is used in Bayes’ theorem, according to Eq.
(8). As a final remark, note that from Eq. (13) it follows that
$p_{\text{val},\,{\theta}}({\mu_{D}})=p_{\text{val},\,{\theta}}({\bar{\mu}}_{D})$,
$U_{\theta}({\mu_{D}})=U_{\theta}({\bar{\mu}}_{D})$ and
$D^{\prime}_{\theta}({\mu_{D}})=D^{\prime}_{\theta}({\bar{\mu}}_{D})$.

##  V Threshold testing

Finally, we consider the situation in which one is not concerned about the
actual value of a continuous parameter but instead asks whether the parameter
exceeds a particular threshold. Following the example of one-sided testing
discussed in Sec. III, an example of threshold testing is to determine whether
the concentration of lead in a drinking water supply exceeds a specific safety
guideline. The null hypothesis $H_{0}$ can be described by the inequality
${\theta}<\theta_{\text{th}}$, where ${\theta}$ denotes the actual
concentration of lead in drinking water while $\theta_{\text{th}}$ denotes the
safety guideline threshold. Our goal would be to obtain the posterior
probability for $H_{0}$ following a measurement of $\mu$ yielding the value
${\mu_{D}}$, using only the proposition that the measured value is more
“extreme” than ${\mu_{D}}$. Note that even though we are only interested in
the posterior probability for $H_{0}$, we still assume that the prior PDF for
${\theta}$ has been specified. In this situation, extremity is naturally
defined by

$\displaystyle
U_{\theta}({\mu_{D}})\equiv\\{{\mu}\,|\,{\mu}\geq{\mu_{D}}\\}\,,$ (16)

which is identical to Eq. (10) for one-sided testing. Although $\mu\geq 0$ in
our example, we might alternatively — as discussed in Sec. III — use $\mu$ to
denote, e.g., the logarithm of the concentration, which would then have no
lower bound. Eq. (16) would still be the natural definition of extremity,
because the other end of the range, $\log\text{(}\rm concentration)$ large and
negative, is consistent with the null hypothesis. As in Sec. III, we will
write the equations for this case under the assumption that ${\theta}=0$ is
the lower bound. If $\theta$ is unbounded, all the integrals over ${\theta}$
in the equations that follow should be modified by changing the lower limits
from $0$ to $-\infty$.

Using Eq. (8), we again obtain Eq. (12), the same as for one-sided testing.
However, here we need to take one further step to obtain the posterior
probability for $H_{0}$:

$\displaystyle
P\big{(}H_{0}|D^{\prime}_{\theta}({\mu_{D}})\big{)}=\int_{0}^{\theta_{\text{th}}}p\big{(}{\theta}|D^{\prime}_{\theta}({\mu_{D}})\big{)}d{\theta}=\dfrac{\int_{0}^{\theta_{\text{th}}}\left[p({\theta^{\prime}})\int_{{\mu_{D}}}^{\infty}p({\mu}|{\theta})d{\mu}\right]d{\theta^{\prime}}}{\int_{0}^{\infty}\left[p({\theta^{\prime}})\int_{{\mu_{D}}}^{\infty}p({\mu}|{\theta})d{\mu}\right]d{\theta^{\prime}}}.$
(17)

This result may be expressed in a more intuitive way by writing the prior
probability of $H_{0}$ as
$P(H_{0})=\int_{0}^{\theta_{\text{th}}}p({\theta})d{\theta}$, and denoting the
expected generalized $p$-value according to the null hypothesis by

$\displaystyle\langle
p_{\text{val}}({\mu_{D}})\rangle_{\text{pr},{H_{0}}}\equiv\dfrac{1}{P(H_{0})}\int_{0}^{\theta_{\text{th}}}p_{\text{val},\,{\theta}}({\mu_{D}})p({\theta})d{\theta}.$
(18)

Then, Eq. (17) can be rewritten as

$\displaystyle
P\big{(}H_{0}|D^{\prime}_{\theta}({\mu_{D}})\big{)}=\dfrac{\langle
p_{\text{val}}({\mu_{D}})\rangle_{\text{pr},{H_{0}}}}{\langle
p_{\text{val}}({\mu_{D}})\rangle_{\text{pr}}}P(H_{0}).$ (19)

Thus, given the data $D^{\prime}_{\theta}({\mu_{D}})$, the posterior
probability for the null hypothesis is proportional to the expected
generalized $p$-value according to the null hypothesis.

##  VI Conclusion

Both the Bayesian and frequentist approaches to assessing the implications of
measurements remain in common use, so it is certainly worthwhile to explore
the relation between them. They differ somewhat in their precise goals. The
Bayesian approach is a direct attempt to answer the fundamental question that
a scientist faces: based on some set of experiments, how should one assess the
probability that various alternative theories or hypotheses might be true? The
frequentist approach, on the other hand, addresses directly only the
probability that the observed data could arise, given a particular theory, but
makes no attempt to assign a probability to the theory itself.

Advocates of the frequentist approach sometimes fault the Bayesian approach
for its reliance on assumed prior probabilities, which are ultimately
subjective. The Bayesian might reply that the need for priors cannot be
rationally avoided. For example, if I look out my window and discover that the
ground and trees are wet, I might consider two theories to explain this: (a)
aliens from Alpha-Centauri landed and sprayed water over everything, or (b) it
rained. The two theories would account for the observations about equally
well, but undoubtedly most of us would give much higher odds to theory (b). In
Bayesian language, theory (b) would be assiged a much higher prior. Priors can
play a major role in our reasoning, whether we are formally using Bayes’
theorem, or just relying on common sense.

Given that the Bayesian insists on trying to assign actual probabilities to
theories, she might well ask whether $p$-values can be of any use in this
regard. A key obstacle to drawing connections between the two approaches is
that the standard Bayesian and frequentist approaches base their conclusions
on different quantities. In the standard Bayesian approach, the probabilities
assigned to different theories are updated on the basis of a new measurement,
making use of the “likelihood,” which is the probability or probability
density of obtaining the result that was actually obtained, according to each
theory. The standard frequentist approach, however, is to begin by defining
some notion of what it means for a result to be “extreme”. The plausibility of
any theory is then assessed by the “$p$-value,” which is the probability,
according to the theory, that a measurement of the type performed would give a
result at least as extreme as the result that was actually obtained.

To bridge this dichotomy in a way that does not depend on finding special
cases where the likelihoods and $p$-values are related, here we consider the
possibility that the Bayesian can choose to update her probabilities on the
basis of a different input. Instead of using the actual data from an
experiment, the Bayesian could update her probabilities using only the weaker
extremity proposition, the statement that the results of the experiment were
at least as extreme as the data that was obtained. The approach is still fully
Bayesian, using new information to update probabilities assigned to theories.
With this modification of the approach, we have found that the Bayesian can
update probabilities by directly using $p$-values, or at least “generalized”
$p$-values. By generalized $p$-values, we mean $p$-values calculated for
arbitrary theories, rather than just the null hypothesis.

With this modification of the update proposition, we have found that the
posterior probability assigned to theory ${\cal{T}}(\theta)$ is expressed by
Eq. (8), which is our main and generic result. In words, the posterior
probability for ${\cal{T}}(\theta)$ is equal to the prior probability,
multiplied by the ratio of the $p$-value for the data obtained, given the
theory ${\cal{T}}(\theta)$, to the mean $p$-value for the data obtained,
averaged over all theories weighted by their prior probabilities.

This relationship becomes particularly simple if one considers the Bayes
factor for two theories, ${\cal{T}}(\theta_{i})$ and ${\cal{T}}(\theta_{j})$,
which is generally defined by

$\displaystyle
B_{\theta_{i}\theta_{j}}(D)\equiv\dfrac{P(D|\theta_{i})}{P(D|\theta_{j})}=\frac{p(\theta_{i}|D)/p(\theta_{i})}{p(\theta_{j}|D)/p(\theta_{j})}\
.$ (20)

If instead of updating with the proposition $D$, the full data, the Bayesian
analyst uses the weaker extremity proposition $D^{\prime}_{\theta}({\mu_{D}})$
— the proposition that the observed data was at least as extreme as the data
actually observed — then there is a simple relationship with $p$-values:

$\displaystyle
B_{\theta_{i}\theta_{j}}({\mu_{D}})\equiv\dfrac{P\big{(}D^{\prime}_{\theta_{i}}({\mu_{D}})|\theta_{i}\big{)}}{P\big{(}D^{\prime}_{\theta_{j}}({\mu_{D}})|\theta_{j}\big{)}}=\dfrac{p_{\text{val},\,\theta_{i}}({\mu_{D}})}{p_{\text{val},\,\theta_{j}}({\mu_{D}})}\,,$
(21)

where the final equality follows from Eq. (7). Thus, by using the modified
update proposition $D^{\prime}_{\theta}({\mu_{D}})$, the Bayes factor is
precisely equal to the ratio of $p$-values. So, in this context, the Bayesian
can clearly state the question to which the $p$-value (or at least the ratio
of $p$-values) is the answer.

## References

  * (1) Berger, James O., and Thomas Sellke. “Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence”, Journal of the American Statistical Association, vol. 82, no. 397, 1987, pp. 112-122, JSTOR, https://doi.org/10.2307/2289131.
  * (2) Dickey, James M. “Is the Tail Area Useful as an Approximate Bayes Factor?” Journal of the American Statistical Association, vol. 72, no. 357, 1977, pp. 138–142, https://doi.org/10.2307/2286921.
  * (3) Casella, George, and Roger L. Berger. “Reconciling Bayesian and Frequentist Evidence in the One-Sided Testing Problem”, Journal of the American Statistical Association 82, no. 397, 1987, pp. 106–111, https://doi.org/10.2307/2289130.
  * (4) Karlin, Samuel, and H. Rubin. “Distributions Possessing a Monotone Likelihood Ratio.” Journal of the American Statistical Association, vol. 51, no. 276, 1956, pp. 637-643. JSTOR, https://doi.org/10.2307/2281477.
  * (5) Micheas, Athanasios C. and Dipak K. Dey, “Reconciling Bayesian and Frequentist Evidence in the One-Sided Scale Parameter Testing Problem”, Communications in Statistics - Theory and Methods, 36:6, 2007, pp. 1123-1138, https://doi.org/10.1080/03610920601076610.
  * (6) Akrami, Y. et al. [Planck], “Planck 2018 results. VII. Isotropy and Statistics of the CMB,” Astron. Astrophys. 641, A7, 2020, https://doi.org/10.1051/0004-6361/201935201 [arXiv:1906.02552 [astro-ph.CO]].
  * (7) Akrami, Y., Y. Fantaye, A. Shafieloo, H. K. Eriksen, F. K. Hansen, A. J. Banday and K. M. Górski, “Power asymmetry in WMAP and Planck temperature sky maps as measured by a local variance estimator,” Astrophys. J. Lett. 784, L42, 2014, https://doi.org/10.1088/2041-8205/784/2/L42 [arXiv:1402.0870 [astro-ph.CO]].