values are respectively 0.489, 0.385, and 0.256 bits, and ASI values
are respectively 0.260, 0.207, and 0.131 bits.
\label{shellsinconc}
\caption{The \textbf{bottom right} plot shows the ``conclusive''
frequentist confidence set function for $\eta=0.95$ based on
pseudo-Bayesian critical regions. TSI is 0.601 and ASI is 0.584 bits
under the first of the priors discussed in the caption to figure
\ref{shellsBayes}.
\label{shellsconc1}
\end{figure}

\subsubsection{``Conclusive'' confidence set functions}

The bottom right plot of figure \ref{shellsconc1} shows the
``conclusive'' frequentist confidence set function for the
pseudo-Bayesian critical regions considered in figure
\ref{shellspseudoBayes}. The top left plot of figure \ref{shellsconc2}
shows that for critical regions based on low likelihood under the null
hypothesis $H_0$, the top right one for those based on high likelihood
under the alternative $H_1$, while the bottom left one uses low
likelihood under $H_0$ to reject $H_0$ but high likelihood under $H_0$
to reject $H_1$, and the bottom right one is similar but for $H_1$ and
$H_0$ reversed.

Note how for both inconclusive and conclusive confidence set functions
the mapping from data to frequentist confidence set varies enormously
between the different choices of confidence set function one can make.

\begin{figure}
\begin{tabular}{cc}

\includegraphics[scale=0.5]{shellsFig5fig14} &
\includegraphics[scale=0.5]{shellsFig5fig16} \\
\includegraphics[scale=0.5]{shellsFig5fig18} &
\includegraphics[scale=0.5]{shellsFig5fig19} \\

\end{tabular}

\caption{``Conclusive'' frequentist confidence set functions for
$\eta=0.95$. In all cases brown means $f(n,\eta)=\{1\}$, dark blue
means $f(n,\eta)=\{0\}$, green means $f(n,\eta)=\{0,1\}$, and yellow
means $f(n,\eta)=\emptyset$. The \textbf{top left} plot is uses
critical regions based on low likelihoods under the null hypothesis,
the \textbf{top right} one on high likelihoods under the alternative
hypothesis, the \textbf{bottom left} one uses critical regions that
use low likelihood under $H_0$ to reject $H_0$ but high likelihood
under $H_0$ to reject $H_1$, while the \textbf{bottom right} plot is
similar but for $H_1$ and $H_0$ reversed. Evaluated only under the
first of the priors discussed in the caption of figure
\ref{shellsBayes}, TSI values are respectively 0.549, 0.282, 0.398,
and 0.420 bits, while ASI values are respectively 0.420, 0.200,
0.276, and 0.387 bits.
\label{shellsconc2}

\end{figure}

\clearpage

\subsection{Summary of TSI and ASI values for the specific shells problem}

\label{shellsinfo}

Above we considered only one particular version of the problem, namely
that with the values of $P(f|m,c)$ given in table \ref{pfgivenmc}, for
which we now summarise the ASI values in table \ref{ASItable} and the
TSI values in table \ref{TSItable}.

\begin{table}[H]
\begin{center}

\begin{tabular}{lrrrr}
\textbf{Prior $P(c)$:} & \multicolumn{2}{c}{\textbf{A}} &
\multicolumn{2}{c}{\textbf{B}} \\
\textbf{Prior $P(m|c)$:} & \multicolumn{1}{c}{\textbf{C}} & \multicolumn{1}{c}{\textbf{D}} & \multicolumn{1}{c}{\textbf{C}} & \multicolumn{1}{c}{\textbf{D}} \\
\textbf{Method}\\
Needed to be definite & 1.000 & 1.000 & 0.141 & 0.141\\
Bayes & 0.669 & 0.888 & 0.088 & 0.109\\
Pseudo-Bayes & 0.538 & 0.657 & -0.662 & 0.104\\
Far from $H_0$ & 0.118 & -0.123 & -1.196 & -1.286\\
Near $H_1$ & 0.482 & 0.718 & -0.586 & 0.060\\
Unlikely under $H_0$ & 0.070 & -0.269 & -1.149 & -1.291 \\
Likely under $H_1$ & 0.451 & 0.685 & -0.619 & 0.056 \\
IFCS by pseudo-Bayes & 0.260 & 0.119 & -1.031 & -0.928 \\
IFCS by unlikely under $H_0$ & 0.207 & 0.051 & -1.055 & -1.081 \\
IFCS by likely under $H_1$ & 0.131 & 0.046 & -1.071 & -0.931 \\
CFCS by pseudo-Bayes & 0.584 & 0.530 & -0.301 & 0.033 \\
CFCS by unlikely under $H_0$ & 0.420 & 0.384 & -0.502 & -0.221 \\
CFCS by likely under $H_1$ & 0.200 & 0.044 & -0.860 & -0.782 \\
CFCS by $H_0$ & 0.276 & 0.049 & -0.844 & -0.932 \\
CFCS by $H_1$ & 0.387 & 0.442 & -0.471 & 0.002 \\
\end{tabular}

\caption{Apparent Shannon information (ASI) values (bits) (section
\ref{shellsinfointro}) for the different methods on the shells
problem of section \ref{shells} under various different priors. IFCS
= inconclusive frequentist confidence set function; CFCS =
conclusive frequentist confidence set function.  Priors are:
\textbf{A}: $P(c)=0.5\,\forall c$; \textbf{B}:
$P(c=0)=0.98,P(c=1)=0.02$; \textbf{C}: $P(m|c)=\frac{1}{2}\,\forall
m,c$; \textbf{D}: $P(m|c)=\left\{\begin{matrix}0.01 & (m=c)\\ 0.99 &
(m\neq c)\end{matrix}\right.$. In each case the settings for
frequentist methods that use prior information matched the prior
under which ASI was assessed.
\label{ASItable}

\end{center}
\end{table}

\begin{table}[H]
\begin{center}

\begin{tabular}{lrrrr}
\textbf{Prior $P(c)$:} & \multicolumn{2}{c}{\textbf{A}} &
\multicolumn{2}{c}{\textbf{B}} \\
\textbf{Prior $P(m|c)$:} & \multicolumn{1}{c}{\textbf{C}} & \multicolumn{1}{c}{\textbf{D}} & \multicolumn{1}{c}{\textbf{C}} & \multicolumn{1}{c}{\textbf{D}} \\
\textbf{Method}\\
Needed to be definite & 1.000 & 1.000 & 0.141 & 0.141\\
Bayes & 0.669 & 0.888 & 0.088 & 0.109\\
IFCS by pseudo-Bayes & 0.489 & 0.225 & 0.056 & 0.043 \\
IFCS by unlikely under $H_0$ & 0.385 & 0.147 & 0.044 & 0.016 \\
IFCS by likely under $H_1$ & 0.256 & 0.129 & 0.029 & 0.025 \\
CFCS by pseudo-Bayes & 0.601 & 0.799 & 0.062 & 0.100 \\
CFCS by unlikely under $H_0$ & 0.549 & 0.757 & 0.061 & 0.076 \\
CFCS by likely under $H_1$ & 0.282 & 0.142 & 0.030 & 0.025 \\
CFCS by $H_0$ & 0.398 & 0.155 & 0.044 & 0.017 \\
CFCS by $H_1$ & 0.420 & 0.730 & 0.038 & 0.080 \\
\end{tabular}

\caption{True Shannon information (TSI) values (bits) (section
\ref{shellsinfointro}) for the different methods on the shells
problem of section \ref{shells} under various different priors. IFCS
= inconclusive frequentist confidence set function; CFCS =
conclusive frequentist confidence set function. TSI for the
frequentist hypothesis testing methods is the same as for Bayes.
Priors are: \textbf{A}: $P(c)=0.5\,\forall c$; \textbf{B}:
$P(c=0)=0.98,P(c=1)=0.02$; \textbf{C}: $P(m|c)=\frac{1}{2}\,\forall
m,c$; \textbf{D}: $P(m|c)=\left\{\begin{matrix}0.01 & (m=c)\\ 0.99 &
(m\neq c)\end{matrix}\right.$. In each case the settings for
frequentist methods that use prior information matched the prior
under which ASI was assessed.
\label{TSItable}

\end{center}
\end{table}

\subsection{Further assessment of other versions of the shells problem}

To see whether the same conclusions hold generally, we also ran 100
different such problems, drawing the values of $P(f|m,c)$ randomly
from a flat Dirichlet distribution on the 3-simplex, and the values of
$P(m,c)$ independently from a flat Dirichlet distribution on the

For each such sample problem, we calculated the ASI resulting from
applying the various methods of section \ref{shells}, which we plot in
figure \ref{shellsinfoplot1} (and again without the legend in
figure \ref{shellsinfoplot2}, and zoomed in to the top right corner
in figure \ref{shellsinfoplot3}). All three plots also show by the
$x$-coordinate and the black diagonal line the amount of TSI and ASI
needed to reach a definite conclusion to the User's problem. In all
cases the green cross marking the Bayesian result comes out above the
column of results of alternative methods, and usually not far below
the black line. In all three plots, many of the frequentist methods
have fallen off the bottom of the plot because of yielding ASI more
negative than -1 bit.

\clearpage

\begin{figure}[H]
\begin{center}

\includegraphics[scale=0.7]{shellsFig3fig1}

\caption{The apparent Shannon information (ASI) achieved by each
method for the Shells problem of section \ref{shells}, plotted
against the amount of information needed to give a definite answer
to the User's question on the $x$-axis. Versions of this plot
without the legend and zoomed in are in figures
\ref{shellsinfoplot2} and \ref{shellsinfoplot3} below.
\label{shellsinfoplot1}

\end{center}
\end{figure}

\begin{figure}[H]
\begin{center}

\includegraphics[scale=0.7]{shellsFig3fig2}

\caption{A zoomed in version of figure \ref{shellsinfoplot1}. The
apparent Shannon information (ASI) achieved by each method for the
Shells problem of section \ref{shells}, plotted against the amount
of information needed to give a definite answer to the User's
question on the $x$-axis. The meaning of the various dots is shown
in the legend to figure \ref{shellsinfoplot1} above.  A still more
zoomed in version is in figure \ref{shellsinfoplot3} below.
\label{shellsinfoplot2}

\end{center}
\end{figure}

\begin{figure}[H]
\begin{center}

\includegraphics[scale=0.7]{shellsFig3fig3}

\caption{A further zoomed in version of figures \ref{shellsinfoplot1}
and \ref{shellsinfoplot2}. The apparent Shannon information (ASI)
achieved by each method for the Shells problem of section
\ref{shells}, plotted against the amount of information needed to
give a definite answer to the User's question on the $x$-axis. The
meaning of the various dots is shown in the legend to figure
\ref{shellsinfoplot1} above.
\label{shellsinfoplot3}

\end{center}
\end{figure}

\subsection{Discussion of the shells example}

We have seen that there are a myriad of frequentist methods for
determining the country of origin of the shells in this problem, some
of which even depend on the choice of Bayesian prior that is used in
the calculation. There is no uniformly optimal set of critical regions
for this problem. In contrast there is a single Bayesian method, which
correctly takes prior information into account, and which maximises
both true and apparent Shannon information in the resulting answer,
conserving all the relevant information in the data. All the
frequentist methods shown lose ASI in comparison, while the
95\% frequentist confidence sets also lose TSI. Many of the
frequentist methods are indeed sufficiently misleading that they give
negative ASI, making using the results less good than just going with
the prior and ignoring the data.

Indeed the various frequentist methods give utterly different
results. If we receive the data value $n=(5,10,5)$, for example, the
frequentist methods give the following results in the order of
presentation of the figures: Frequentist confidence that $c=1: 0.999,
0.114, 0.999, 0.066, 0.994, 0.404, 0.994, 0.404$; 95\% confidence
sets: $\{1\}, \{1\}, \{0,1\}, \{1\}, \emptyset, \{0,1\}, \{1\},
\{0\}$. In other words, dependent on purely arbitrary choices of
nested sets of critical regions or confidence set functions, all four
possible frequentist confidence sets may be arrived at with 95\%
frequentist confidence (including that $c$ is in the empty set), while
we may obtain frequentist confidence levels that $c=1$ from 0.066 to
0.999 .

In contrast, depending on the prior in use, the posterior probability
that $c=1$ given by the Bayesian approach with this same data is (for
the four priors considered, in the order of presentation in figure
\ref{shellsBayes}): $0.930, 0.931, 0.213, 0.215\ $. Note that the
prior $P(c)$ for the first two of these values is $0.5$ on each
country, while for the last two it is $0.98$ on $c=0$ and $0.02$ on
$c=1$, a huge difference; it should not be surprising that if we are
almost certain that $c=0$ before collecting the data then we are less
sure that $c=1$ after collecting it than we would be with equal prior
probabilities on the two countries.

The conclusion remains that the Bayesian method retains more ASI than
any other method, and that although it usually doesn't extract
sufficient information to reach a definite answer to the User's
question, it does better than any other possible method.

Even though all the frequentist hypothesis-testing methods retain all
the TSI in their solutions for this particular problem, this is a
distinctly pyrrhic victory; we are just saying that having applied
such a method, and got a poor result with low or even negative ASI, we
needn't give up; rather the best thing to do is to use the method's
output to recover the original data, then apply the Bayesian method.

\clearpage

\section{A more realistic but less dramatic example}
\label{example3}

\subsection{Introduction}

In order to convince readers who are uncomfortable with abstract
problems (such as that presented in section \ref{example1} above) that
frequentist methods are wrong, we now consider a much more realistic
example where we still have a 2-dimensional data space to consider,
and can visualise the posterior probability and frequentist confidence
that results from different observed data. While the results are less
dramatic, they still suffice to make the necessary points in

We deal with the question of which antibiotic to use to treat a
particular disease. We suppose that we have a set of patients infected
with \textit{Microsoftus gatesii}, and we want to know whether
treatment with jobsucillin or torvaldomycin is better at curing
them. We choose to randomise $N_1+N_2=60$ patients, giving $N_1=40$
patients torvaldomycin and $N_2=20$ patients jobsucillin. Our null
hypothesis $H_0$ is that jobsucillin is better.

In terms of our standard notation for inference problems from section
\ref{infprobdef} above, we set $$
$$ $$
(with $\phi_1$, aka $p_1$, the unknown probability of cure with
torvaldomycin and $\phi_2$, aka $p_2$, the unknown probability of cure
jobsucillin), $$
(where $[\ ]$ denotes the function that takes the value 1 if the
condition inside the brackets is true and 0
otherwise), $$
$$ $$
$$ where $x=(n_1,n_2)$
means that we got $n_1$ cures with torvaldomycin and $n_2$ cures with
jobsucillin, and $$
P(n_1,n_2|θ, p_1, p_2) =
p_1^n_1(1-p_1)^N_1-n_1 p_2^n_2(1-p_2)^N_2-n_2.

As usual, we consider first the Bayesian answer, then various
frequentist answers. A key point that we want to make is that while
the Bayesian answer depends on the prior -- something that should
indeed influence what we think after collecting the data -- there are
many frequentist answers which differ from one another by making
entirely arbitrary choices, none of which is uniformly optimal.

\subsection{Bayesian solution}
\label{example3Bayes}

Here we set the prior $P(p_1,p_2)=[(p_1,p_2)\in [0,1]^2]$ (the
independent uniform prior) which automatically results in
$P(H_0)=P(H_1)=\frac{1}{2}$. (Obviously we could set different priors,
but for brevity we only explore this one which most people will find

Leaving the reader to do the calculations (or to find them in appendix
\ref{example3Bayescalcs}), the Bayesian solution is shown in figures
\ref{bcBayes} and \ref{bcBayes95}. We observe that it is totally
reasonable, and that for 283 of the possible values of $(n_1,n_2)$
(i.e. of blocks in figure \ref{bcBayes}) the probability that $H_1$
holds is at least $0.95$, and for 283 it is $\leq 0.05$.

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig1}

\caption{Bayesian posterior probability that $H_1$ holds for each
possible observed value of $(n_1,n_2)$.
\label{bcBayes}

\end{center}
\end{figure}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig2}

\caption{Bayesian posterior probability that $H_1$ holds for each
possible observed value of $(n_1,n_2)$, with each block coloured
brown or blue according to whether it is above or below 0.95
\label{bcBayes95}

\end{center}
\end{figure}

\subsection{First exact frequentist solution}
\label{bcfreq1text}

Here we use frequentist critical regions of the
form $$
C_η(k)={(n_1,n_2):n_2≠0, n_1 > kn_2}
$$ for various
values of $k$ and the uniquely determined increasing function $\eta$
to $[0,1]$ that gives the corresponding frequentist confidence. The
result is shown in figues \ref{bcfreq1} and \ref{bcfreq195}. We
observe that it is not really reasonable (e.g. we never conclude that
$H_1$ holds if $n_2=0$, as we have excluded this region of $X$ in
order to get reasonable frequentist confidence elsewhere). There are
95 blocks where frequentist confidence that $H_1$ holds is at least
$0.95$, but 460 out of 861 where it is $\leq 0.05$.

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig3}

\caption{First version of exact frequentist confidence that $H_1$
holds for each possible observed value of $(n_1,n_2)$.
\label{bcfreq1}

\end{center}
\end{figure}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig4}

\caption{First version of exact frequentist confidence that $H_1$
holds for each possible observed value of $(n_1,n_2)$, with each
block coloured brown or blue according to whether it is above or
below 0.95 respectively.
\label{bcfreq195}

\end{center}
\end{figure}

\subsection{Second exact frequentist solution}
\label{bcfreq2text}

Here we use frequentist critical regions of the
form $$
C_η(k)={(n_1,n_2):n_1 > n_2N_1/N_2+k}
$$ for
various values of $k$ and the uniquely determined increasing function
$\eta$ to $[0,1]$ that gives the corresponding frequentist
confidence. The result is shown in figues \ref{bcfreq2} and
\ref{bcfreq295}. We observe that it is not really reasonable (e.g. it
is almost zero on almost half the blocks). There are 256 blocks where
frequentist confidence that $H_1$ holds is at least $0.95$, but 420
out of 861 where it is $\leq 0.05$.

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig5}

\caption{Second version of exact frequentist confidence that $H_1$
holds for each possible observed value of $(n_1,n_2)$.
\label{bcfreq2}

\end{center}
\end{figure}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig6}

\caption{Second version of exact frequentist confidence that $H_1$
holds for each possible observed value of $(n_1,n_2)$, with each
block coloured brown or blue according to whether it is above or
below 0.95 respectively.
\label{bcfreq295}

\end{center}
\end{figure}

\subsection{$\chi^2$ test, an approximate frequentist solution, and
the corresponding \mbox{exact} frequentist confidence}
\label{bcchitext}

The $\chi^2$ test, here with 1 degree of freedom, is a standard
frequentist test for this problem, which approaches being correct
(from a frequentist point of view) as $N_1$ and $N_2$ approach
infinity (something that in real life never happens). Here we use a
directional version of it (as the standard version only tests whether
$p_1=p_2$ as $H_0$). It returns an over-optimistic estimate of the
frequentist confidence that is biased in favour of $H_1$, as shown in
figures \ref{bcchiapprox} and \ref{bcchiapprox95}; for comparison the
exact frequentist confidence corresponding to the relevant critical
regions is shown in figures \ref{bcchiexact} and \ref{bcchiexact95}.

In the approximation there are 290 blocks with frequentist confidence
above 0.95 and 290 below 0.05; in the exact version there are
respectively 271 and 371.

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig7}

\caption{Directional $\chi^2$-test version of frequentist confidence
that $H_1$ holds for each possible observed value of
$(n_1,n_2)$. Note that these values are an overoptimistic
approximation (biased in favour of $H_1$).
\label{bcchiapprox}

\end{center}
\end{figure}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig8}

\caption{Directional $\chi^2$-test version of frequentist confidence
that $H_1$ holds for each possible observed value of $(n_1,n_2)$,
with each block coloured brown or blue according to whether it is
above or below 0.95 respectively. Note that these values are an
overoptimistic approximation (biased in favour of $H_1$).
\label{bcchiapprox95}

\end{center}
\end{figure}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig9}

\caption{Directional $\chi^2$-test version of frequentist confidence
that $H_1$ holds for each possible observed value of
$(n_1,n_2)$. These values have been corrected for the errors
inherent in the $\chi^2$-test.
\label{bcchiexact}

\end{center}
\end{figure}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig10}

\caption{Directional $\chi^2$-test version of frequentist confidence
that $H_1$ holds for each possible observed value of
$(n_1,n_2)$. These values have been corrected for the errors
inherent in the $\chi^2$-test. Each block is coloured brown or blue
according to whether it is above or below 0.95 respectively.
\label{bcchiexact95}

\end{center}
\end{figure}

\subsection{Critical regions based on Fisher's exact test}
\label{bcFishertext}

``Fisher's exact test'' is a frequentist test for a slightly different
situation, namely that not only is it known in advance how many are in
each treatment group, but also it is known in advance how many get
cured in total. Equivalently, and classically, the situation we have
been addressing so far is equivalent to a lady tasting tea who has
been told that the 60 cups of tea in front of her are some of type A
and some of type B, who is asked to identify which is which; Fisher's
exact test is applicable if she is \textit{also} told in advance how
many there are of type A and how many of type B.

So application of Fisher's exact test to our present situation is
incorrect -- but nonetheless we have often seen it used in this
situation. Moreover, it does define critical regions, and we can
calculate the correct frequentist confidence values for those critical
regions. We present both the naive, incorrect, version of the
frequentist confidence and the corrected version, in figures
\ref{bcFisherapprox} to \ref{bcFishercorrect95}.

In the naive, incorrect, version there are 260 blocks with frequentist
confidence above 0.95 and 321 below 0.05; in the corrected version
there are 280 above 0.95 and 300 below 0.05; the naive version is
therefore too pessimistic (biased against $H_1$).

We also note that Boschloo first proposed using Fisher's exact test
for this present problem with this correction.

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig11}

\caption{Fisher's exact test version of frequentist confidence that
$H_1$ holds for each possible observed value of $(n_1,n_2)$, as the
test is often inappropriately used. Note that these values are too
pessimistic (biased against $H_1$).
\label{bcFisherapprox}

\end{center}
\end{figure}

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig12}

\caption{Fisher's exact test version of frequentist confidence that
$H_1$ holds for each possible observed value of $(n_1,n_2)$ as the
test is often inappropriately used. Note that these values are too
pessimistic (biased against $H_1$). Each block is coloured brown or blue
according to whether it is above or below 0.95 respectively.
\label{bcFisherapprox95}

\end{center}
\end{figure}

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig13}

\caption{Fisher's exact test version of frequentist confidence that
$H_1$ holds for each possible observed value of $(n_1,n_2)$. These
values have been corrected for the errors resulting from
inappropriate use, so are correct for the present problem, and are
identical with those resulting from using Boschloo's test.
\label{bcFishercorrect}

\end{center}
\end{figure}

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig14}

\caption{Fisher's exact test version of frequentist confidence that
$H_1$ holds for each possible observed value of $(n_1,n_2)$. These
values have been corrected for the errors resulting from
inappropriate use, so are correct for the present problem, and are
identical with those resulting from using Boschloo's test. Each
block is coloured brown or blue according to whether it is above or
below 0.95 respectively.
\label{bcFishercorrect95}

\end{center}
\end{figure}
\clearpage

\subsection{Effect of choosing which test to use \textit{after}
collecting the data}
\label{bcchooselatetext}

The experimenter, naturally, is often interested in showing a
``statistically significant'' result, i.e. one with frequentist
confidence $\geq 0.95$. In this situation it is very tempting to not
follow a cardinal rule of frequentist analysis, namely that one must
choose the nested family of critical regions to be used (or
equivalently the frequentist ``test'' to be used) \textit{before}
collecting the data -- and instead to collect the data, then choose
whichever test makes the result have highest frequentist confidence
that $H_1$ holds.

Effectively this results (if we choose from the tests listed in the
subsections above, restricting ourselves to those with correct(ed)
values of frequentist confidence) in the apparent frequentist
confidence levels shown in figures \ref{bcchooselate} and
\ref{bcchooselate95}, in which there are 283 blocks with frequentist
confidence above 0.95 and 297 below 0.05 . In particular, suppose that
we had observed $(n_1,n_2)=(3,1)$: then we would get apparent
frequentist confidence of $0.704$.

However, if we determine the critical regions that are then in effect,
and calculate their true frequentist confidence, we get the levels
shown in figures \ref{bcchooselatefixed} and
\ref{bcchooselatefixed95}, in which there are 268 blocks with
frequentist confidence above 0.95 and 298 below 0.05 . Data of
$(n_1,n_2)=(3,1)$ now only gives us frequentist confidence of

Thus choosing a frequentist ``test'' in the light of the data, without
correcting for this, is cheating. The fact that it is often so
difficult to tell whether or not somebody has chosen a frequentist
test before or after collecting the data is one of the major problems
with frequentism (as shown by its not adhering to \ref{criteria} item
\ref{whetherproceed} above).

But the problem is in reality much worse than this. Suppose in these
circumstances somebody reports that they have used a (corrected)
$\chi^2$-test and found significance at (exactly) the 0.95
level. Unless there is evidence that they had in some way restricted
themselves to using a $\chi^2$-test, and no other test, before
collecting the data, whether or not they actually have found a
statistically-significant result depends on what they \textit{would
have} done if the $\chi^2$-test had yielded say 90\% rather than
95\% frequentist confidence:

\begin{enumerate}

\item If they would then have tried to publish their paper containing
only a 90\%-confident result, then they have indeed found 95\%
frequentist confidence.

\item If, on the other hand, they would then have tried a (corrected)
Fisher exact test, and if that still didn't get the 95\% level they
wanted they would then have tried the test of section
\ref{bcfreq1text}, and if that still didn't get the 95\% level they
wanted they would then have tried the test of section
\ref{bcfreq2text}, and would have reported the highest level of
frequentist confidence of these, then they have not actually found
95\% frequentist confidence, but some lower level -- even though
none of these hypothetical events actually happened, and even if one
(or all !)  of these other tests would in fact have given them
apparent 95\% frequentist confidence.

\end{enumerate}

This is all akin to frequentism's violation of criterion
\ref{whetherproceed} of section \ref{criteria}, even though those are
not the exact circumstances applying here.

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig15}

\caption{The apparent frequentist confidence that $H_1$ holds that
results from choosing the ``test'' in the light of the data, for
each possible observed value of $(n_1,n_2)$. These values are
over-optimistic and biased in favour of $H_1$.
\label{bcchooselate}

\end{center}
\end{figure}

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig16}

\caption{The apparent frequentist confidence that $H_1$ holds that
results from choosing the ``test'' in the light of the data, for
each possible observed value of $(n_1,n_2)$. These values are
over-optimistic and biased in favour of $H_1$. Each block is
coloured brown or blue according to whether it is above or below
0.95 respectively.
\label{bcchooselate95}

\end{center}
\end{figure}

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig17}

\caption{The actual frequentist confidence that $H_1$ holds that
results from choosing the ``test'' in the light of the data, for
each possible observed value of $(n_1,n_2)$. These values have been
corrected for the late choice of test.
\label{bcchooselatefixed}

\end{center}
\end{figure}

\begin{figure}[p]
\begin{center}

\includegraphics[scale=0.5]{Fig21fig18}

\caption{The actual frequentist confidence that $H_1$ holds that
results from choosing the ``test'' in the light of the data, for
each possible observed value of $(n_1,n_2)$. These values have been
corrected for the late choice of test. Each block is coloured brown
or blue according to whether it is above or below 0.95 respectively.
\label{bcchooselatefixed95}

\end{center}
\end{figure}

\subsection{Conclusions from this example}

We draw a number of conclusions; since the various plots are
moderately difficult to distinguish from each other by naked eye
assessment of colour shading, the reader is encouraged to consider
specifically the counts of blocks (i.e. data values) which lead to
posterior probabilities or frequentist confidences above 0.95 or below
0.05 (table \ref{numbersabove}):

\begin{table}[hp]
\begin{tabular}{lrr}
\textbf{Test type} & \textbf{Number giving $>0.95$} &
\textbf{Number giving $<0.05$}\\
Bayes & 283 & 283\\
Frequentist section \ref{bcfreq1text} & 95 & 460\\
Frequentist section \ref{bcfreq2text} & 256 & 420\\
Directional $\chi^2$ naive & (290) & (290)\\
Directional $\chi^2$ corrected & 271 & 371\\
Fisher exact naive & (260) & (321)\\
Fisher exact corrected (Boschloo's test) & 280 & 300\\
Late choice (apparent) & (283) & (297)\\
Late choice (corrected) & 268 & 298
\end{tabular}
\caption{
\label{numbersabove}
Performance of various tests for comparison of jobsucillin and
torvaldomycin for treatment of \textit{Microsoftus gatesii}
infection. The numbers given are the numbers of data values resulting
in frequentist confidence or posterior probability $>0.95$ or $<0.05$
that $H_1$ holds as indicated. Numbers in brackets are for tests that
are either only an approximation or are being misused.}
\end{table}

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig31fig1}

% The above is a correction; the previous version had been made using
% N1=20,N2=10, whereas it should have been (and now is) N1=40,N2=20.

\caption{The frequentist test described that gives greatest
resp. least probability of concluding with 95\% frequentist
confidence that $H_1$ holds when it does resp. doesn't among those
considered given each value of $(p_1,p_2)$: 3 denotes that of
section \ref{bcfreq1text}, 4 that of section \ref{bcfreq2text}, 6
the corrected $\chi^2$-test, 8 the corrected Fisher exact test, and
10 the corrected frequentist confidence from choosing after
collecting the data among the others of these tests. (The omitted
comparisons and numbers correspond to Bayes against thresholds 0.5
and 0.95 and the inexact frequentist tests, none of which were
included in this comparison.)
\label{optifreq}

\end{center}
\end{figure}

\begin{enumerate}

\item The Bayesian solution gives an entirely reasonable and symmetric
result; none of the exact solutions in this selection of
frequentist solutions does.

\item The Bayesian test is more likely to conclude that it is 95\%
sure that $H_1$ holds than any of the exact frequentist
tests considered. Moreover the symmetry of the Bayesian test means
that there are the same number of data values for which it will
conclude that it is 95\% sure that $H_0$ holds as there are for
which it will conclude that it is 95\% sure that $H_1$ holds.

\item The correct(ed) frequentist tests considered are all biased in
favour of $H_0$ in the sense that there are always more data values
for which they will conclude that frequentist confidence of $H_1$ is
below 0.05 than that it is above 0.95, as well as in the sense that
most of the time the Bayesian posterior probability of $H_1$ is
greater than the frequentist confidence.

\item The commonly used $\chi^2$ test, in the uncorrected form in
which it is usually used, is biased in favour of $H_1$ in comparison
with exact frequentist inference using the same critical regions
(whether in the directional version we used above or in the standard
version where $H_1 = \{h\in H : p_1 \neq p_2\}$).

\item The commonly used Fisher's exact test is both inappropriate for
this problem, and, as usually used, biased against $H_1$ in
comparison with exact frequentist inference using the same critical

\item None of these frequentist tests is uniformly optimal (no such
test exists for this problem).

Considering parameter space (as in the usual definition of uniformly
optimal), figure \ref{optifreq} shows that the best of the
considered exact frequentist tests to use varies with the actual
value of $(p_1,p_2)$ applying. Even when $p_1>p_2$, sometimes the
corrected version of Fisher's exact test does best, sometimes one
does best using the corrected frequentist confidence resulting from
choosing among the other exact tests listed after collecting the data.

Considering data space, if $x=(n_1,n_2)=(8,1)$ then the solution of
section \ref{bcfreq1text} concludes with $95\%$ frequentist
confidence that $H_1$ holds, but none of the other exact frequentist
solutions do. Equally if $x=(n_1,n_2)=(40,18)$ then the (corrected)
solution of section \ref{bcchitext} so concludes, but none of the
other exact frequentist solutions do. Or if $x=(n_1,n_2)=(24,7)$ then
the solutions of sections \ref{bcfreq2text} and (corrected)
\ref{bcFishertext} so conclude, but the others don't, not even the
(corrected) one of section \ref{bcchooselatetext}.

\item The Bayesian test reaches its conclusion based on the prior and
the data, independent of the intentions of the statistical
analyst. The frequentist one, however, as can be seen from section
\ref{bcchooselatetext} above, depends on what the intentions of the
analyst were (unless he can prove he made his choice of test before
collecting the data).

\item The frequentist test depends on a totally arbitrary choice of
test, i.e. of nested set of critical regions. The Bayesian test, on
the other hand, depends on the knowledge possessed before collecting
the data, and can be reanalysed to evaluate the effect of differing
assumptions about that knowledge \textit{ad lib}.

\end{enumerate}

In our view the Bayesian test is the only one that makes any sense; it
is also the only one that takes appropriate account of the relevant

\section{Examples with long sequences of i.i.d. random variables}
\label{example2}

\subsection{Introduction}
\label{statintro}

For those readers who are more used to seeing inference applied to
problems involving long sequences of independent and identically
distributed random variables, we now consider a different type of
example problem, in two flavours and in two different settings. Here
we concentrate on showing the advantages of the Bayesian approach
rather than showing that the frequentist approach is wrong, as the
latter has been dealt with in sections \ref{critsection} to
\ref{example3} above.

In these examples an experimenter is aiming to show that $h>h_0$ for
some fixed value of $h_0$, so he sets $H_0=\{h\in H : h \leq
h_0\}$. He would like to collect some number $K$ of samples of data
$x_1,x_2,...,x_K$ and reach the conclusion that $h\in H_1$ with some
predetermined level of sureness $\eta$, usually 0.95 (assuming of
course that in truth $h>h_0$). However he does not know the true value
of $h$, and of course the closer $h$ is to $h_0$, the larger $K$ will
have to be to get a high probability that he can conclude that he is
$\eta$-sure that $h\in H_1$.

In the first (Bernoulli) flavour of this example we will assume that
each $x_k$ is independently Bernoulli distributed with parameter $h$,
in other words it is 0 or 1 with probability $h$ that it is 1 (so that
$H=[0,1]$), and take $h_0=0.975$. One might think of this as wanting
to be sure that if we drop a product object on the floor, the
probability of it remaining intact is at least $h_0$. So our data will
consist of a sequence of breakages or survivals of a length somehow

In the second (Gaussian) flavour we will assume that each $x_k$ is an
independently Gaussianly distributed real number with unit variance
and mean $h$ (so that $H=\mathbb{R}$) and take $h_0=0$. One might
think of this as wanting to be sure that some measurement gives a
positive result on average. So our data will consist of some number of
measurements and the resulting values.

Thus in terms of our standard notation for inference problems from
section \ref{infprobdefsub} we are
setting $$
$$ $$
Φ=[0,1] or
$$ $$
H={(θ,ϕ)∈Θ×Φ: θ=
$$ $$
H_k=({k}×Φ)∩H  for
$$ $$
X=⋃_k=0^∞{0,1}^k or
and $$
P(x_k|θ,ϕ)=ϕ^x_k(1-ϕ)^1-x_k or
$$ respectively for
the two flavours, and we can identify $h$ with $\phi$ without
ambiguity. Here saying that we have observed $\mathbf{x}\in
\{0,1\}^K\subset X$ means not only that we have observed the length
$K$ sequence $x_1,x_2,...,x_K$ but also that we have decided to stop
data collection at that point; note that then in the notation of
section \ref{criteria} criterion \ref{unifpow} the likelihood is then
given by $$
$$ where
the $q_k$ depend on the data collection plan. Of course, in the
Bayesian solution all the factors involving the $q_k$ are the same in
numerator and denominator of Bayes' theorem, so cancel. Similarly in
the fixed $K$ frequentist approach only $q_K$ is non-zero (indeed it
is 1) and it doesn't depend on the $x_k$, so these factors
disappear. In the general frequentist case, however, the situation is
more complicated.

Note that although the first flavour is reasonably like real-life, the
second is not, as both known variance and Gaussianity are only rarely
encountered in practice; we use this oversimplistic second flavour as
it is then simple enough for the reader to check our calculations
without too much effort.

To give us some lifelike vocabulary we will consider that we are
testing whether or not a particular factory meets some specification
on the products it produces. In the first setting we will assume that
only one aspect of the products produced has to be checked. In the
second setting we will take it that $N=224$ independent similar
aspects of the products have to be checked, and that the factory will
be approved if and only if for all $N$ aspects we are $95\%$-sure
that $h_n > h_0$. Any reader who thinks that one would never need to
check as many as $224$ aspects might be interested to read ISO 20072,
and in particular the example treated in [12], where 16
different aspects of product performance have each to be assessed
under 14 different sets of environmental conditions (although they are
not all of the same flavour).

Whether we are considering a frequentist or a Bayesian approach, we
will consider that the value of $h$ is drawn from a specific input
distribution giving a probability of 0.9 that the factory is good
(i.e. has $h>h_0$ or all $h_n>h_0$ as appropriate) as specified in
table \ref{inputtable}, while for the Bayesian approach we will use a
non-informative broad prior for the actual inference. As a measure of
performance of our inference method we will consider how small $K$ can
be while achieving a suitably high probability (e.g. $0.9$ or $0.99$)
that a good factory drawn from the input distribution will be

\begin{table}[hp]
\begin{tabular}{llllr}
& \multicolumn{2}{c}{\textbf{Setting}}\\
\textbf{Flavour} & \textbf{Single aspect} & \textbf{224 aspects} &
\textbf{Prior used for Bayesian inference} & $h_0$\\
Bernoulli & Beta(155,2) & Beta(400,2) & Beta(1,1)=Uniform & 0.975\\
Gaussian & Gaussian(1.3,1) & Gaussian(3.33,1) & Gaussian(0,$10^{-4}$) & 0
\end{tabular}
\caption{
\label{inputtable}
The input distributions used for the $h$ or $h_n$ parameters of
simulated factories to be tested. Parameters of the Gaussian are mean
and scale (=1/variance). All four cases give a probability of
approximately 0.9 that a factory with $h$ or each $h_n$ drawn from the
given distribution will have $h>h_0$ or each $h_n > h_0$ as appropriate.
\end{table}

Our aim then will be to be able to approve 90\% (or 99\%) of good
factories drawn from the input distribution. We will consider three
different approaches: the frequentist approach using a predetermined
fixed value of $K$; a  pseudo-Bayesian frequentist approach designed to
be suitable for the target factory pass probability; and the Bayesian
approach using either the uniform prior on $h\in [0,1]$ or a Gaussian
prior of zero mean and standard deviation 100 on $h\in \mathbb{R}$ as
appropriate to the problem flavour. There are of course infinitely
many other possible frequentist approaches, but time and space
restrict us to consider just these here.

We will then examine what average value of $K$ is needed for each of
the three approaches. We first specify the method of calculation for
each method, and then summarise the necessary values of $K$ in a joint
table. All simulations use factories drawn from the input
distribution, discarding any that are not good, until we have 100,000
factories to test. In the case of the Bayesian approach we will also
separately consider the average value of $K$ needed when the bad
factories from the input distribution are also included.

Note that both the frequentist approaches make use of knowledge of the
input distribution (the ``true prior'') in their design (at least as a
worst case that they have to deal with), while the Bayesian approach
\textit{does not} -- it would be exactly the same whatever input
distribution were expected, so long as it is intended to be executed
without knowledge of that distribution, and its parameters depend only
on the nature of the likelihood, on $h_0$, on $N$, and on the desired
probability of passing a good factory. Of course, the Bayesian
approach uses a prior in its execution, namely the flat uniform prior
or a very broad Gaussian prior as appropriate to the problem flavour.

Further, both the frequentist approaches are designed for the purpose
of passing a specific fraction of the given input distribution; the
Bayesian approach, however, requires no such determination in advance
-- it can just be run for as long as it takes to pass any particular
factory (in real life there is only one factory under consideration at
any one time).

\subsection{Frequentist approach with fixed predetermined sample size}

We search for the appropriate value of $K$ by binary chop on a
suitable range of values.

For each potential value of $K$ we calculate the appropriate uniformly
optimal critical region to achieve, for each aspect, 95\% frequentist
confidence that $h\in H_1$. For each good sample factory $f$ drawn
from the input distribution as above we calculate the probability
$p_f$ that the data for a single aspect will lie in the critical
region. The probability of a factory being approved is given
by $$
(_f p_f)^N,
$$ where $N$ is the number of aspects in the
given setting.

\subsection{Bayesian approach}

We simulate on the case $N=1$ (i.e. on a single aspect), calculating
the corresponding figures for the $N=224$ case afterwards. For each
good sample factory we collect one data point at a time, each time
calculating the posterior probability that $h>h_0$ given the data so
far. When that probability first rises above $p_1$ (resp. falls below
$p_0$) we pass (resp. fail) that aspect, noting the number of
datapoints that have been required. The values of $p_0$ and $p_1$ are
as shown in table \ref{Bayesparmtable}. The manufacturer is left to
decide when to give up testing if the posterior probability passes
neither limit, which time will depend on the cost of the factory, the
cost of testing, and the current posterior probability. In the
simulations, if wanting to pass a good factory with probability
$p=0.9$ say, we will need to pass each aspect with probability

\begin{table}[hp]
\begin{tabular}{lrr|r|r}
& \textbf{Setting} & \textbf{Fraction of}\\
\textbf{Flavour} & \textbf{($N$)} & \textbf{good to pass} &
$p_0$ & $p_1$\\
\hline
Bernoulli & 1 & 0.9 & $10^{-2}$ & 0.95 \\
& & 0.99 & $10^{-4}$ & 0.95 \\
\hline
Bernoulli & 224 & 0.9 & $10^{-4}$ & 0.95 \\
& & 0.99 & $10^{-5}$ & 0.95 \\
\hline
Gaussian & 1 & 0.9 & $10^{-1}$ & 0.95 \\
& & 0.99 & $10^{-2}$ & 0.95 \\
\hline
Gaussian & 224 & 0.9 & $10^{-4}$ & 0.95 \\
& & 0.99 & $10^{-5}$ & 0.95
\end{tabular}
\caption{
\label{Bayesparmtable}
The settings of the various parameters used for the various Bayesian
tests reported in table \ref{statresults}.}
\end{table}

\subsection{Pseudo-Bayesian frequentist approach}

We set $h_0'>h_0$ such that roughly the desired fraction of good
factories to be passed from the input distribution have $h_n>h_0'$ for
all $n$. We then set $p_0$, $p_1$, $K_{\text{min}}$, and
$K_{\text{max}}$ such that the probability of a factory with $h=h_0$
passing in a single aspect is $\approx 0.05$ under the Bayesian
approach that has exit barred before $K_{\text{min}}$ and after
$K_{\text{max}}$ samples have been collected and otherwise fails if
posterior probability of $H_1$ falls below $p_0$ and passes if it
rises above $p_1$ (as estimated by $10^5$ simulations). We then
proceed as in the Bayesian approach but with the modifications implied
by $h_0'$, $p_0$, $p_1$, $K_{\text{min}}$, and $K_{\text{max}}$.

The specific values used are given in table
\ref{pseudosettings}. Because of the difficulty of achieving exactly
0.95 frequentist confidence, we bracket this figure with two different
sets of values to give a range of expected $K$ required.

Note that many other choices are possible in designing such a
pseudo-Bayesian test, and we know no way of finding the most efficient
such solution (or whether or not a most efficient one exists), so we are
not claiming that these choices give the most efficient
basic pseudo-Bayesian approach possible. The resulting test is then at least
adaptive to the nature of the actual factory and aspect that it is
currently considering.

\begin{table}[!htp]
\begin{tabular}{lrr|r|r|r|r|r|r}
& \textbf{Setting} & \textbf{Fraction of}\\
\textbf{Flavour} & \textbf{($N$)} & \textbf{good to pass} & $h_0'$ &
$p_0$ & $p_1$ & $K_{\text{min}}$ & $K_{\text{max}}$ & \textbf{conf} \\
\hline
Bernoulli & 1 & 0.9 & 0.979 & 0.001 & 0.95 & 400 & $1 \times 10^5$ & 0.955 \\
& & & 0.979 & 0.001 & 0.95 & 320 & $1 \times 10^5$ & 0.949 \\
\hline
Bernoulli & 1 & 0.99 & 0.9756 & 0.001 & 0.991 & 1500 & $5 \times 10^6$ & 0.956 \\
& & & 0.9756 & 0.001 & 0.991 & 800 & $5 \times 10^6$ & 0.946 \\
\hline
Bernoulli & 224 & 0.9 & 0.97695 & 0.001 & 0.96 & 1500 & $2 \times 10^6$ & 0.953\\
& & & 0.97695 & 0.001 & 0.96 & 1300 & $2 \times 10^6$ & 0.949\\
\hline
Bernoulli & 224 & 0.99 & 0.9752 & 0.0005 & 0.985 & 50000 & $4 \times 10^7$ & 0.959 \\
& & & 0.9752 & 0.0005 & 0.985 & 30000 & $4 \times 10^7$ & 0.948 \\
\hline
Gaussian & 1 & 0.9 & 0.36 & 0 & 0.95 & 0 & 125 & 0.951 \\
& & & 0.35 & 0 & 0.95 & 0 & 125 & 0.949 \\
\hline
Gaussian & 1 & 0.99 & 0.04 & 0.001 & 0.96 & 150 & 4000 & 0.952 \\
& & & 0.04 & 0.001 & 0.96 & 130 & 10000 & 0.947 \\
\hline
Gaussian & 224 & 0.9 & 0.19 & 0.001 & 0.96 & 4 & 1000 & 0.9505 \\
&  &  & 0.19 & 0.001 & 0.96 & 3 & 1000 & 0.943 \\
\hline
Gaussian & 224 & 0.99 & 0.02 & 0.001 & 0.96 & 600 & 25000 & 0.9504 \\
&  &  & 0.02 & 0.001 & 0.96 & 500 & 25000 & 0.944 \\
\end{tabular}
\caption{
\label{pseudosettings}
The values of the various parameters used for the various
pseudo-Bayesian tests reported in table \ref{statresults}. The column
headed ``conf'' gives the frequentist confidence achieved with these
values, based on $10^5$ simulations.
\end{table}

\subsection{Results}
\label{resultssection}

Table \ref{statresults} gives the values of $K$ resulting for each
approach to each flavour and setting.

\begin{table}[!h]
\begin{tabular}{lrr|r|r|r|r}
&&& \multicolumn{4}{c}{\textbf{Approach}} \\ \cline{4-7}
& \textbf{Setting} & \textbf{Fraction of} & \multicolumn{2}{c|}{\textbf{Frequentist}} &
\multicolumn{2}{c}{\textbf{Bayesian}} \\ \cline{4-7}
\textbf{Flavour} & \textbf{($N$)} & \textbf{good to pass} & \textbf{Fixed
$K$} & \textbf{ Pseudo-Bayesian} & \textbf{Good} & \textbf{All}\\
\hline
Bernoulli & 1 & 0.9 & 3000 & 1464 - 1509 & 472 & 568 \\
& & 0.99 & 130000 & 27965 - 29415 & 1145 & 2392 \\
& & 0.996 & & & 2091 & 5573 \\
\hline
Bernoulli & 224 & 0.9 & 5600000 & 323142 - 384169 & 45483 & 46106 \\
& & 0.99 & $>$ 200000000 & 6770414 - 11288141 & 49720 & 57616 \\
\hline
Gaussian & 1 & 0.9 & 19 & 15 - 18 & 3.2 & 3.5\\
& & 0.99 & 1000 & 178 - 187 & 9.8 & 16\\
& & 0.996 & & & 28 & 60\\
\hline
Gaussian & 224 & 0.9 & 16352 & 758 - 972 & 273 & 278\\
& & 0.99 & 672000 & 112092 - 134638 & 285 & 324\\
& & 0.998 & & & 294 & 426\\
\end{tabular}
\caption{
\label{statresults}
The expected number of datapoints needed for all $N$ aspects of a good
factory drawn from the relevant input distribution to be approved,
under the various methods. Values were estimated from 100,000 samples
of good factories from the input distribution, except that the last
column gives the corresponding number for the Bayesian test if the bad
factories are also included in the factories being tested (for
fixed-$K$ this makes no difference, and for  pseudo-Bayesian
these values are not shown but would be higher than those for good
factories). For the  pseudo-Bayesian tests the first figure
resulted from the run for the lower bracketing frequentist confidence
from table \ref{pseudosettings} and the second for the higher. For the
Bernoulli 224-aspect fixed-$K$ case the number of datapoints total
required to get 99\% of the good factories to pass was in the range
$2\times 10^8$ to $2.5\times 10^8$.}
\end{table}

\subsection{Discussion of results}
\label{statresdisc}

\subsubsection{General}
\label{statresgen}

We first remind ourselves of what the different methods are supposed
to achieve:

\begin{itemize}

\item Bayesian approaches that result in a factory being given 0.95
posterior probability of being good are supposed to result in not
more than 0.05 of such approved factories actually being bad.

If the prior matches the actual distribution of factories being
tested, then that will always be the case; however in the examples
here the two distributions are very different, and we would like to
know what fraction of the approved factories are in fact bad.

\item Frequentist approaches that result in a factory getting 95\%
frequentist confidence that it is good are supposed to have the
effect that any bad factory gets such approval with probability not
more than 0.05, irrespective of the input distribution. This will be
the case with these frequentist solutions in particular.

\end{itemize}

Of course, what actually matters to the users of these factories'
products is whether the ones that are approved meet specification or
not. If most of the factories being tested are in fact bad, then the
frequentist achievement could be that the total number of factories
getting approval is low compared with the number tested, but most of
the approved ones are bad. Moreover, since the frequentist criterion
makes no promises that it won't fail good factories with arbitrarily
high probability, it is even possible in principle that the fraction
of bad factories in the frequentist-approved set is higher than it was
in the original distribution of factories being tested, although this
cannot happen in this specific example.

So we now review from table \ref{statresults} how much data is
required to achieve various fractions of good factories passing, and
then turn to ask what happens to bad factories in this example, which
apart from the oversimplification of the second flavour is very

\subsubsection{What happens to good factories with each approach ?}

It is striking how many fewer drops or measurements respectively are
needed to approve any given fraction of good factories by the Bayesian
approach than by either of the other two. The smallest ratio of sample
numbers needed by any of these frequentist solutions to get 90\% passes
of good factories to the corresponding Bayesian number is 2.5 (for the
Bernoulli $N=1$ case, pseudo-Bayes / Bayes), and the second smallest
such number is 2.7 (for the Gaussian $N=224$ case, pseudo-Bayes /
Bayes). The highest similar ratio (for 90\% passes of good factories)
observed here is 121 (for the Bernoulli $N=224$ case, fixed-$K$ /

For 99\% pass rate of good factories, we observe that the fixed-$K$,
Gaussian $N=224$ version requires over 2000 times as much data as the
Bayesian version does, and the corresponding Bernoulli case requires
over 3000 times as much data as the Bayesian version. Even for the
comparison between  pseudo-Bayesian and Bayesian tests, in the
Gaussian $N=224$ case the  pseudo-Bayesian requires more than 300
times as much data. These comments apply even if the appropriate
fraction of bad factories are included for the Bayesian case, and have
used the lowest of the two bounding figures for the  pseudo-Bayesian
case. (Of course this is not to say that one might not be able to
design a different frequentist solution that did better.)

One should also note that for the Bayesian method, the amount of data
required rises less fast than $N$ (bearing in mind the input
distributions were adjusted to make 90\% of the input factories good
in both cases), while the frequentist versions have data demands which
increase much faster than $N$.

Finally we note that the frequentist test designers need to have some
idea of the input distribution (or prior !) to be able to design their
tests with adequate probability of passing good factories. What
evidence would a frequentist fixed-$K$ designer need to be reasonably
sure that a reasonable prior to design to was as used here, if he
optimally used a flat prior and Bayes (!) to find out ?

For the Bernoulli flavour, for $N=1$, he would need to drop $155$
product samples and observe only one breakage; for $N=224$ he would
need to drop $224\times 400=89600$ devices, observing only one
breakage in each of the 224 sets; in the first case he has already
consumed a quarter as much data as the Bayesian is expected to
need\footnote{Recall that the Bayesian needs no such preparatory data
as he is using a flat prior.} to pass 90\% of good devices, and in
the second nearly twice as much -- and that's before he's even started
his test proper. For the Gaussian flavour (variance being known) a
single measurement might suffice (depending on its value), but that
single measurement has already used over a quarter of the expected
number of measurements needed by the Bayesian to pass 90\% of good
devices in the $N=1$ case.

\subsubsection{What happens to bad factories with each approach ?}

But we must also consider what happens to bad factories with each
approach, because they are not treated the same, as mentioned in
section \ref{statresgen} above. The frequentist case is easy: no bad
factory has probability greater than 0.05 of passing a test of an
aspect on which it is bad.

In the Bayesian case the situation needs more careful
consideration. First let us consider what the Bayesian method does
from a frequentist point of view.

\begin{figure}[!htp]
\begin{center}

\includegraphics[scale=0.5]{experimentsFig1fig1}

\caption{The probability of a factory passing a single
Gaussian-flavour aspect by the Bayesian solution as a function of its
mean measurement. Here the Bayesian solution was set to fail the
factory on this aspect if after 20,000 measurements it had not
passed. (An alternative criterion for giving up might be that the
posterior probability that the factory was \textit{bad} rose above
0.95; that would then result in a symmetric curve about zero.)
\label{bayesbads}

\end{center}
\end{figure}

Figure \ref{bayesbads} shows the situation in the Gaussian $N=1$
case, where the Bayesian solution was set to abandon the factory (and
fail it) if it had not passed after 20,000 measurements. A consequence
of the property that Bayes will (eventually) pass any good factory is
that as $h$ approaches $h_0$ from below, the probability of passing
approaches 1 (if arbitrarily many measurements are allowed). So one
might reasonably worry that factories that were only just bad might
contaminate the output to the extent that one gets more than 5\% of
the approved factories actually being bad. So does that happen here ?

\begin{table}[h]
\begin{tabular}{lrccc}
\textbf{Flavour} & \textbf{Setting ($N$)} & \textbf{Bads/Total in
input} & \textbf{Pass rate / Bads} & \textbf{Bads passed / Total passed}\\
Bernoulli & 1 & 0.096 & $0.11$  & $1.02 \times 10^{-2}<0.05$\\
Bernoulli & 224 & $4.4\times 10^{-4}$ & 0.16 & $7 \times 10^{-5} <
2.2\times 10^{-4} =0.05 / 224$\\
Gaussian & 1 & 0.097 & 0.08 & $8.1\times 10^{-3}<0.05$\\
Gaussian & 224 & $4.3\times 10^{-4}$ & 0.11 & $4.8\times
10^{-5}<2.2\times 10^{-4}=0.05/224$ \\
\end{tabular}
\caption{
The probability that an aspect of a factory is bad in the
given input distribution, the pass rate with the Bayesian solution on
those bad aspects, and the resulting fraction of the apparent
passed aspects that are actually bad.
\label{statresbads}
\end{table}

The answer is No. Table \ref{statresbads} shows the fraction of the
input distribution that is bad (for a single aspect), the fraction of
those bad factories that pass this single aspect, and the resulting
fraction $f$ of the passed population that are actually bad. In all
cases $Nf$, an overestimate of the population of factories that are
bad, is below 0.05 of the passed population of factories.

For that not to be the case, we would need \textit{both} the fraction
of bads in the input distribution to be higher, \textit{and} the bads
to be concentrated exactly at or just below $h_0$. A factory designer
would not be able to achieve that if he wanted to. Given that of
course in real life we do not have an input population but just a
single factory that we are testing, the corresponding proposition is
that we believe with high probability that our factory has $h$
\textit{just} below $h_0$. In the highly unlikely event that that is
the case, the situation can of course be fixed by using a prior that
matches what we actually think about the factory under test.

\subsubsection{What is the meaning of a bad factory passing in the
Bayesian case ?}

Now let us think about the meaning of a bad factory passing the test
from a Bayesian point of view. This means that the dataset that has
been collected is at least 0.95 likely to have come from a good
factory, assuming the prior we have used for inference. Given that
that is the only dataset we have to consider on this point, if the
prior we have used reflects our beliefs about how likely different
values of $h$ are, then it is therefore entirely appropriate that it
be approved.

\section{How should we set a prior with a regulatory authority ?}
\label{regulatorprior}

\subsection{Scenario}

If then we are going to use Bayesian methods, we will need to be
prepared to express what we know before collecting the data as a prior
probability distribution on the unknowns. Where only one party at a
time needs to use the deductions or predictions that result, this may
be relatively straightforward.

However, one situation where two parties with different interests may
need to be able to reach a common conclusion about some unknown
parameters is when a regulatory agency needs to be convinced by a
manufacturer that his product satisfies necessary legal
requirements. For example, an asthma inhaler may have a requirement
that the dose delivered is within some particular range for a given
fraction of the doses delivered; or there may be a requirement that
when a device is dropped onto a hard floor from a given height it has
at least a given probability of surviving undamaged. In this situation
a regulator might want a prior on the parameters that was sceptical
that the device meets the given requirements, while the manufacturer
might want one that even before collecting the data considers it very
likely that the device meets the requirements.

It is also possible that regulator and manufacturer even differ on the
type of distribution of the regulated attribute; for example the
manufacturer may think that his inhaler delivers log doses distributed
according to a Gaussian
distribution $$
$$ with suitable
prior distributions on the parameters $\mu$ and $\sigma$, while the
regulator thinks the log doses are distributed according to a Student
distribution $$
r^m/(r + 1/2(x-μ)^2)^m+1/2,
$$ again with
suitable priors on the parameters $m,r,\mu$.

\subsection{Proposed way forward -- ``Let the data decide''}

\label{datadecide}

In setting a prior that is acceptable to both parties, the first step
should obviously be to attempt to reach agreement through discussion
after sharing all available data. If unsuccessful, it may then be
necessary that a formal mechanism is available for the parties to
reach a prior by a mechanism that automatically mediates between their
two desired priors and/or models. Moreover knowing that such a
mechanism exists may also facilitate agreement actually being reached
at the initial discussions.

We therefore propose a mechanism for resolving such disputes about
priors as follows, to which we give the moniker ``Let the data
decide'', following MacKay[14].

Let us denote by $M_1$ the model desired by the regulator and by $M_2$
the model desired by the manufacturer, in each case complete with
priors on the parameters ($\theta_1, \theta_2$ respectively) deemed
suitable by the relevant party. Let us denote by $A$ the event that
the actual distribution meets the desired criteria (e.g. has 90\% of
log doses between $a$ and $b$), so that when the data $D$ is
collected, $P(A|M_k,D)$ is the probability that if model $M_k$ holds
the legal requirements are satisfied.

Our proposal, then, is very simple: we make no decision on whether
$M_1$ or $M_2$ is to be used, but instead assume that the test data
$D$ arises as a result of parameters and likelihood taken from one or
other of these models with probability $\frac{1}{2}$ on each.

\subsection{General effect of this proposal}

One might be forgiven for thinking that then we simply
get $$
P(A|D)=1/2P(A|M_1,D) + 1/2P(A|M_2,D).
$$ But this
is \textit{not} what happens. Rather, according to the marginalisation
rule and the chain rule of probability,
\begin{IEEEeqnarray*}{rCl}
P(A|D) &=& P(A,M_1|D) + P(A,M_2|D)\\
&=& P(A|M_1,D)P(M_1|D) + P(A|M_2,D)P(M_2|D).
\end{IEEEeqnarray*}

In other words, each model contributes to the confidence that the
legal requirements are met in proportion to the probability that that
model did indeed give rise to the data.

\subsection{Examples of the use of this proposal}

\subsubsection{Introduction to examples}

For each of these examples, for simplicity, we will consider the
situation that the legal requirement to be satisfied is that when a
device is dropped onto a hard floor a single time from a defined
height the probability $p$ that it survives intact must be at least
$\tau = 0.9$. We further suppose that both parties agree that, given
that no device will be tested more than once, the outcomes of the
various drops will be independent, so that after $N$ drops, the number
of intact devices will be binomially distributed according to
P(n|N,p) = N!/n!(N-n)!p^n(1-p)^N-n.

For all examples we used the optimum Bayesian data collection plan as
in appendix \ref{bayesproofs} criterion \ref{appunifpow}, except that
we stopped each simulation if any of the following occurred:
\begin{itemize}
\item $P(A|D) > 0.95$ (concluding requirements probably met, as in the
optimum Bayesian data collection plan));
\item $P(A|D) < 0.01$ (concluding requirements probably not met);
\item more than $10,000$ devices had been dropped (leaving the
decision whether or not to continue up to the manufacturer).
\end{itemize}

We first look at three unrealistic examples simply for the purpose of
showing that ridiculous settings don't lead to ridiculous results. We
will then turn to look at a more realistic scenario, first from a
frequentist point of view then from a Bayesian point of view, before
exploring the effects of mismatch between prior and the actual
distribution of regulatory submissions on a range of priors.

\subsubsection{Both parties pulling in opposite directions, version 1}

\label{bothobstructive}

Suppose the regulator decides to make it as difficult as possible for
the manufacturer to pass the test (but without actually making it
impossible), and sets the prior on $p$ to be given by $$
P(p|M_1) =
{1-δ    (p = τ- ϵ)
δ    (p =
τ+ ϵ)
0     (otherwise).
$$ where
$\delta>0,\epsilon>0$ are small positive numbers perhaps in the region
of $0.001$.

The manufacturer, on the other hand, perhaps misguidedly, attempts to
abuse the system in his own favour by setting the prior $$
P(p|M_2) =
{1     (p = 1)

What then happens as data is collected using the optimal data
collection plan of appendix \ref{bayesproofs} criterion
\ref{appunifpow} ?

First, if every device dropped survives intact, the probability that
the device factory is legally acceptable $P(A|D)$ passes $0.95$ when
$N=\lceil 27.946\rceil = 28$. For comparison, the uniformly optimal
frequentist test for $H_0=(p\in [0,0.9])$ reaches $95\%$ frequentist
confidence when $N=\lceil 28.433\rceil=29$, hardly a large difference.

Second, as soon as a single device fails to survive intact, $P(M_2|D)$
becomes zero, so that the regulator's obstructive prior takes full
effect, and depending on the values of $\epsilon$ and $\delta$ it may
become arbitrarily difficult to pass the test even after dropping many
thousands of devices.

\subsubsection{Both parties pulling in opposite directions, version 2}

If the regulator is still obstructive using the same prior as in
section \ref{bothobstructive}, a manufacturer who is slightly wiser
but still biased in his own favour might choose to set the
prior $$
P(p)=[τ< p ≤1]/1-τ,
$$ i.e. the prior that is
uniform on the interval $(\tau,1]$, again expressing $100\%$ prejudice
that his factory produces legally acceptable devices. Again using
the uniformly optimal data collection plan of appendix
\ref{bayesproofs} criterion \ref{appunifpow}, the number of tests
needed before $P(A|D)=0.95$ is now a random variable that depends on
the true value of $p$. If in truth $p=1$, so that no devices break,
we reach this posterior probability when $N=42$ (the answer to life,
the universe, and everything according to Douglas Adams\footnote{In
``The Hitch-hiker's guide to the Galaxy''}), a number which is
even larger than the frequentist's 29, while if $p<1$ then it will
be expected to take longer still. Passing the test is now almost
synonymous with showing that $M_2$ holds. A full exploration of what
happens when $p<1$ will be found below in section \ref{examplesims}.

\subsubsection{Both parties pulling in opposite directions, version 3}

Keeping again the same prior for the regulator, the biased
manufacturer might try doing exactly the opposite of what the
regulator does but more so, by setting his prior according to $$
P(p) =
[p=τ+ ϵ],
$$ expressing total prejudice that $p$ is just on
the right side of $\tau$. Now it becomes very difficult to get
$P(M_2|D)$ to move upwards from $0.5$, as both models are essentially
saying that $p=\tau$ to a good (but slightly different)
approximation. Given these particular choices, again passing the test
is almost synonymous with showing that $M_2$ holds, which is extremely

\subsubsection{Realistic regulator but manufacturer calls upon
his past experience to bias prior in his favour}

\label{reasonableregvpast}

In this more realistic situation, the regulator sets the
prior $$
which is a Beta prior with its $0.025$ and $0.975$ quantiles at
$0.676$ and $0.953$, also implying that there is only a $0.2$
probability that the factory's devices meet requirements. On the other
hand the manufacturer has dropped $100$ similar (but not quite
identical) devices without breakage before, so wants to set the prior
$P(p) = 101p^{100}$, i.e. to say that $p$ is
Beta(101,1)-distributed. We note to start with that under the
manufacturer's prior the probability that the factory is good is
already around $0.99998$, so he would have no need to collect any data
at all were his prior to be accepted by the regulator. As it is, if in
fact $p=1$, then after dropping $17$ devices he will get no breakages,
and the posterior probability that $p>\tau$ will be about $0.956$;
this is because the data ($17$ drops with no breakages) has allowed
the inference that the data arose from $M_2$ rather than $M_1$ with
posterior probability

\begin{IEEEeqnarray*}{rClr}
P(M_2|D) &=& \frac{P(M_2)P(D|M_2)}{P(M_1)P(D|M_1) +
P(M_2)P(D|M_2)}\\ &=&\frac{P(D|M_2)}{P(D|M_1)+P(D|M_2)} &
(P(M_1)=P(M_2)=\frac{1}{2})\\ &=&\frac{\int{P(p|M_2)P(D|p)\,dp}}{\int{P(p|M_1)P(D|p)\,dp}
\int{P(p|M_2)P(D|p)\,dp}}\\ &=&\frac{\int{(101p^{100})p^{17}\,dp}}{\int{(\frac{\Gamma(25)}{\Gamma(21)\Gamma(4)}p^{20}(1-p)^{3})p^{17}\,dp}
+ \int{(101 p^{100})p^{17}\,dp}}\\ &=&\frac{\frac{101}{118}}
+ \frac{101}{118}}\\ &\approx& 0.891
\end{IEEEeqnarray*}
so that
\begin{IEEEeqnarray*}{rCl}
P(A|D) &=& P(M_1|D)P(A|M_1,D) + P(M_2|D)P(A|M_2,D)\\
&\approx& 0.109\times 0.597 + 0.891 \times 0.999994\\
&\approx& 0.956
\end{IEEEeqnarray*}
while in contrast, had the regulator's prior been accepted, one would
have needed $28$ drops without breakage to have reached the same

Let us also consider the same pair of priors and see what happens if
in fact $p=0.85$, i.e. the devices produced by the factory are
unacceptable ($\tau$ being $0.9$).

In this case the number of drops until we know whether or not the
devices produced by the factory are acceptable is a random variable,
so to find out what happens we conducted simulations. We think of each
simulation as the testing of a factory producing devices.

Note that the manufacturer was allowed to set a more definite
confidence threshold for failure than the regulator sets for success,
because it is the manufacturer who has to pay for more devices to be
tested -- and in any case, this only affects when he gives up, and
does not affect how sure we have to be that his product is good before
passing it.

Now, of the $100$ runs total, none required more than $10,000$ drops
to reach a decision, $93$ concluded that with probability $>0.99$ the
devices did not meet the requirements after an average of $141$
devices were dropped, and $7$ concluded that with probability $>0.95$
the devices did meet the requirements (although in fact they didn't).

In contrast, if the regulator's prior had been used unmodified, all
$100$ runs would have concluded that with probability $>0.99$ the
devices did not meet the requirements after an average of $145$
devices had been dropped.

We note that it is difficult to compare this with what would happen
with a frequentist test because there are so many possible choices of
critical regions and data collection plans, none of which is in any
sense optimal.

Turning back to devices that ought to pass the test, with $p=0.95$,
with the prior containing both regulator's and manufacturer's
components, of $100$ runs all concluded that the devices met the
requirements after on average $99$ devices had been dropped. In
contrast using only the regulator's prior the same conclusions were
reached, but for those concluded to meet the requirements on average
$162$ devices needed to be dropped.

The astute reader may by now have noticed that we have so far been
testing these priors using a frequentist point of view, contrary to
the spirit of this paper. Let us now consider what happens from a
Bayesian point of view. Here we will look at inference from each
possible point in the space of pairs $(n_{\text{good}},n_\text{bad})$.
In a later section of this paper (\ref{examplesims}) we will visit a
number of priors using factory parameters drawn from the prior in use
and from other priors.

Note that we cannot display the full data space in only $2$
dimensions. Although pairs $(n_{\text{good}},n_\text{bad})$ represent
a sufficient statistic for inference at that point in time (i.e. they
contain all the information for inference at that point without
collecting any more data), the outcome of a test run depends also on
the order in which the results come in -- e.g. $10$ breakages
followed by $90$ survived drops will have a different outcome from the
reverse order.

Figure \ref{bvreg21c4man101c1} shows what happens with two typical
time-courses in data space; the black trace uses devices that do not
meet the requirement as they have $p=0.85<\tau$, while the white one
uses devices that do meet the requirement with $p=0.95>\tau$. Each
time a device is dropped the trace moves either one unit (block) to
the right if it survives, or one block up if it breaks. The posterior
probability $P(A|D)$ is shown in colour, and the testing terminates
when either the time-course enters the light grey lower region, when
it concludes the devices do meet the requirements, or when the
time-course enters the dark grey upper region, when it concludes that
the devices do not meet the requirements.

\begin{figure}[htp]
\begin{center}

\includegraphics[scale=0.5]{Fig15fig1}

\caption{Posterior probability that devices meet legal requirements as
a function of the numbers of devices that are intact $n_\text{good}$
and broken $n_\text{bad}$ using the regulator's prior that is
Beta(21,4) and the manufacturer's prior that is Beta(101,1). The top
plot uses the regulator's prior alone, the bottom one the
manufacturer's prior alone, and the middle one the proposed mixture
prior. The starting point at $(0,0)$ has been marked as $(0.2,0.2)$
on these log scales. The white track shows a device with
$p=0.95>\tau$ and the black track one with $p=0.85<\tau$. With the
manufacturer's prior both are deemed good without collecting any
data. The other two get the same results as each other, though it is
clear that if there are no or few breakages, the mixed prior will
conclude sooner than the regulator's prior.
\label{bvreg21c4man101c1}

\end{center}
\end{figure}

As can be seen, these two time-courses each reach the correct
conclusion with the regulator's prior or the mixed prior, despite the
manufacturer's prior concluding that both sets of devices meet the
requirements without collecting any data.

Figure \ref{bvmreg21c4man101c1} then shows what happens to $P(M_2|D)$
as the two test runs proceed.

From a Bayesian point of view the key observation, however, is not the
time courses of these two individual tests, but the underlying colour
plot. As can immediately be seen from the three underlying colour
plots of figure \ref{bvreg21c4man101c1}, the effect of admixing 50\%
of the regulator's prior to the manufacturer's prior is to
dramatically change the inference pattern; in general the less
restrictive (i.e. less informative) prior dominates the inference
pattern in such a mixture.

\begin{figure}[hpt]
\begin{center}

\includegraphics[scale=0.5]{Fig15fig2}

\caption{Posterior probability that manufacturer's model and priors
are correct as a function of the numbers of devices that are intact
$n_\text{good}$ and broken $n_\text{bad}$ using the regulator's
prior that is Beta(21,4) and the manufacturer's prior that is
Beta(101,1). The starting point at $(0,0)$ has been marked as
$(0.2,0.2)$ on these log scales. The white track shows a device with
$p=0.95>\tau$ and the black track one with $p=0.85<\tau$.
\label{bvmreg21c4man101c1}

\end{center}
\end{figure}

\subsubsection{Flat prior from regulator, manufacturer calls on past

\label{flatregvpast}

Revisiting the results of the previous section
\ref{reasonableregvpast}, we modify the regulator's prior to be flat
on $[0,1]$, while leaving the manufacturer using $$
p^100 [0≤p ≤1].

If the devices under test are good with in truth $p=0.95$, all of the
$100$ runs drew the correct conclusion after an average of $33$
devices were dropped.

The time-course plots for this case are shown in figures
\ref{bvreg1c1man101c1} and \ref{bvmreg1c1man101c1}. As can be seen the
inference pattern is again dominated by the regulator's prior, though
perhaps to a lesser extent than in figure \ref{bvreg21c4man101c1}.

\begin{figure}[hp]
\begin{center}

\includegraphics[scale=0.5]{Fig16fig1}

\caption{Posterior probability that devices meet legal requirements as
a function of the numbers of devices that are intact $n_\text{good}$
and broken $n_\text{bad}$ using the regulator's prior that is
Beta(1,1) (i.e. flat) and the manufacturer's prior that is
Beta(101,1). The top plot uses the regulator's prior alone, the
bottom one the manufacturer's prior alone, and the middle one the
proposed mixture prior. The starting point at $(0,0)$ has been
marked as $(0.2,0.2)$ on these log scales. The white track shows a
device with $p=0.95>\tau$ and the black track one with
$p=0.85<\tau$. With the manufacturer's prior both are deemed good
without collecting any data. The other two get the same results as
each other, though it is clear that if there are no or few
breakages, the mixed prior will conclude sooner than the regulator's
prior. (In the middle plot, the brief excursion of the black track
into the dark grey area without terminating is due to inadequate
resolution in the color scale.)
\label{bvreg1c1man101c1}

\end{center}
\end{figure}

\begin{figure}[htp]
\begin{center}

\includegraphics[scale=0.5]{Fig16fig2}

\caption{Posterior probability that manufacturer's model and priors
are correct as a function of the numbers of devices that are intact
$n_\text{good}$ and broken $n_\text{bad}$ using the regulator's
prior that is Beta(1,1) and the manufacturer's prior that is
Beta(101,1). The starting point at $(0,0)$ has been marked as
$(0.2,0.2)$ on these log scales. The white track shows a device with
$p=0.95>\tau$ and the black track one with $p=0.85<\tau$.
\label{bvmreg1c1man101c1}

\end{center}
\end{figure}

But for a frequentist the point of interest here is what happens to a
factory producing devices with $p=0.85$ i.e. that do not meet the
requirements, and comparing it with the corresponding result in
section \ref{reasonableregvpast}. Of $100$ runs, although $70$ draw
the correct conclusion after an average of $124$ drops, there are now
$30$ that conclude wrongly that the devices do meet the requirements
(with probability $>0.95$) after an average of just $10$ devices are
dropped. Why is this ?

Essentially what is going on here is that the regulator has set a
prior that reckons that a manufacturer aiming to get $90\%$ of devices
to survive a drop unscathed has a probability as high as $0.5$ of
producing devices that only survive $50\%$ of the time. If in truth
the devices have a probability of survival only slightly below the
acceptable value of $0.9$, the regulator's prior is easily
rejected. We will discuss this further below; for now we just note
that the regulator changing his prior to be more in the regulator's
favour has the opposite effect of that desired -- it makes it
\textit{more} likely that a bad device will pass, by making it more
likely that his prior will be rejected by the data.

\subsubsection{Results of runs with true parameters drawn from various

\label{examplesims}

We now investigate what happens using this approach both testing using
a distribution equal to the prior, and testing with various other
distributions. The definitions of the various distributions are in
table \ref{priordefstable}, while the results of the various runs are
in table \ref{examplestable}.

\begin{table}[hp]
\begin{center}
\begin{tabular}{lllrrr}
\textbf{Number} & \textbf{Name} & \textbf{Type} &
\multicolumn{2}{c}{\textbf{Parameters}} & \textbf{Mixture fraction} \\
\hline
1 & Flat & Beta & 1.0 & 1.0 & 1.00\\
\hline
2 & Somewhat good & Beta & 2.0 & 1.0 & 1.00\\
\hline
3 & Reasonable & Beta & 21.0 & 4.0 & 1.00\\
\hline
4 & Obstructive & Point & 0.89 &  & 0.95\\
& & Point & 0.91 &  & 0.05\\
\hline
5 & Past experience & Beta & 101.0 & 1.0 & 1.00\\
\hline
6 & Uniform prejudiced & Uniform & 0.90 & 1.00 & 1.00\\
\hline
7 & Extremely prejudiced & Uniform & 0.99 & 1.00 & 1.00\\
\hline
8 & Frequentist bad & Point & 0.89 &  & 1.00\\
\hline
9 & Frequentist good & Point & 0.91 &  & 1.00\\
\hline
10 & Perfect & Point & 1.00 &  & 1.00\\
\hline
11 & Catastrophic & Point & 0.00 &  & 1.00\\
\hline

\end{tabular}

\caption{
\label{priordefstable}
Definitions of the various priors and test distributions investigated
in table \ref{examplestable}. The parameters given are $\alpha$ and $\beta$
for Beta distributions, the range for uniform distributions, and the
location of the point mass for point masses. Plots of the various Beta
priors are in figure \ref{showpriors}.}
\end{center}
\end{table}

\begin{figure}[hpt]
\begin{center}

\includegraphics[scale=0.5]{Fig18fig1}

\caption{
\label{showpriors}
Plots of the various Beta priors defined in table \ref{priordefstable}.

\end{center}
\end{figure}

\begin{table}[hp]
\begin{tabular}{llllrrrrr}
\textbf{Test} & \multicolumn{1}{p{2.5 cm}}{\textbf{Prior used}} &
\multicolumn{1}{p{2.5 cm}}{\textbf{Input \mbox{distribution}}} &
\textbf{Truth} & \multicolumn{3}{c}{\textbf{Outcome}} &
\multicolumn{2}{p{2.5 cm}}{\textbf{Mean number of devices dropped}}\\ & Regulator /
Manufacturer & & & \multicolumn{1}{r}{Bad} & \multicolumn{1}{r}{Good}
& \multicolumn{1}{r}{Unkn} & \multicolumn{1}{r}{Bad} &
\multicolumn{1}{r}{Good}\\ \hline 1 & R: Reasonable & Reasonable & Bad: & 339 & 30 & 10 & 307.1 & 139.9 \\
& M: Past experience & Past experience & Good: & 4 & 612 & 5 & 306.8 & 94.2 \\
\hline
2 & R: Reasonable & Reasonable & Bad: & 340 & 33 & 14 & 439.3 & 72.4 \\
& M: Uniform prejudiced & Uniform prejudiced & Good: & 3 & 605 & 5 & 41.7 & 172.2 \\
\hline
3 & R: Reasonable & Reasonable & Bad: & 350 & 26 & 11 & 333.2 & 72.2 \\
& M: Past experience & Uniform prejudiced & Good: & 9 & 594 & 10 & 131.2 & 266.1 \\
\hline
4 & R: Reasonable & Reasonable & Bad: & 344 & 26 & 9 & 403.1 & 94.8 \\
& M: Uniform prejudiced & Past experience & Good: & 1 & 618 & 2 & 8967.0 & 104.3 \\
\hline
5 & R: Flat & Reasonable & Bad: & 284 & 92 & 3 & 198.6 & 14.3 \\
& M: Past experience & Past experience & Good: & 7 & 610 & 4 & 154.7 & 46.7 \\
\hline
6 & R: Reasonable & Flat & Bad: & 431 & 5 & 3 & 97.4 & 17.0 \\
& M: Past experience & Past experience & Good: & 0 & 561 & 0 & NaN & 43.9 \\
\hline
7 & R: Reasonable & Somewhat good & Bad: & 388 & 7 & 1 & 92.9 & 146.6 \\
& M: Past experience & Past experience & Good: & 0 & 601 & 3 & NaN & 49.1 \\
\hline
8 & R: Obstructive & Obstructive & Bad: & 453 & 27 & 9 & 2083.5 & 738.9 \\
& M: Uniform prejudiced & Uniform prejudiced & Good: & 7 & 502 & 2 & 1187.3 & 493.5 \\
\hline
9 & R: Flat & Obstructive & Bad: & 97 & 366 & 26 & 3480.3 & 89.2 \\
& M: Uniform prejudiced & Uniform prejudiced & Good: & 2 & 509 & 0 & 9.5 & 49.1 \\
\hline
10 & R: Obstructive & Obstructive & Bad: & 459 & 30 & 0 & 693.0 & 30.7 \\
& M: Extremely prejudiced & Uniform prejudiced & Good: & 33 & 478 & 0 & 1072.5 & 583.0 \\
\hline
11 & R: Reasonable & Reasonable & Bad: & 362 & 5 & 12 & 337.6 & 832.8 \\
& M: Reasonable & Past experience & Good: & 5 & 610 & 6 & 1665.8 & 154.0 \\
\hline
12 & R: Obstructive & Reasonable & Bad: & 376 & 3 & 0 & 216.1 & 3669.0 \\
& M: Obstructive & Past experience & Good: & 15 & 605 & 1 & 1087.7 & 476.4 \\
\hline
13 & R: Obstructive & Reasonable & Bad: & 370 & 9 & 0 & 214.0 & 779.4 \\
& M: Past experience & Past experience & Good: & 15 & 605 & 1 & 1087.7 & 183.6 \\
\hline
14 & R: Reasonable & Catastrophic & Bad: & 521 & 0 & 0 & 4.0 & NaN \\
& M: Past experience & Perfect & Good: & 0 & 479 & 0 & NaN & 17.0 \\
\hline
15 & R: Reasonable & Frequentist bad & Bad: & 377 & 106 & 38 & 2454.7 & 74.6 \\
& M: Past experience & Frequentist good & Good: & 16 & 459 & 4 & 58.3 & 1353.5 \\
\hline

\end{tabular}

\caption{
\label{examplestable}
A range of simulations done to investigate inference using the
proposed mixtures of regulator's and manufacturer's prior (in equal
parts), and the two listed input distributions (in equal parts). Each
of $1000$ runs were classified using the optimal Bayesian algorithm of
appendix \ref{bayesproofs} item \ref{appunifpow} giving the Outcome,
while the status given by the value of $p$ for the particular factory
tested is as under the Truth column. ``Bad'' denotes $p<\tau$ (if as
an Outcome then with posterior probability $> 0.99$) while ``Good''
denotes $p>\tau$ (and if as an Outcome then with posterior probability
$>0.95$). ``Unknown'' indicates that the trial was stopped after
$10,000$ devices had been dropped with neither of the other possible
conclusions reached. For a full discussion of the results see section
\ref{examplesims}, and for definitions of the priors and input
distributions see table \ref{priordefstable}.}
\end{table}

The first thing to notice is that where the test distribution is the
same as the prior in use (as in tests 1, 2, and 8), the fraction of
runs with Good outcome that should have been classified as Bad is
e.g. $30$ out of $642$ (for test 1), or about $5\%$, in keeping with
the posterior probability required for concluding Good of
$0.95$. Similarly the fraction of runs with Bad outcome that should
have been classified as good is around $1\%$, in keeping with the
$0.01$ posterior probability required before concluding Bad. Note in
particular that we are talking in the first place of $612$ of $642$
Good \textit{outcomes} being correctly classified, not $612$ of $616$
truly Good factories, as a frequentist assessment would report -- we
are interested in what \textit{did} happen in each individual factory
considered alone, not in what \textit{might} have happened when
multiple (e.g. bad) factories considered as a group.

Second, in tests 3 and 4, where the manufacturer mis-guesses either
mildly over-optimistically (test 3) or pessimistically (test 4),
classification performance remains good.

We first notice rather worse performance in test 5 when the regulator
tries too hard to counteract the manufacturer's optimistic prior, and
says that when aiming to design a factory making devices with at least
a $0.9$ probability of surviving being dropped, half of manufacturers
would instead make devices with at least a $0.5$ probability of
breaking. The other way round, though, in tests 6 and 7, where the
prior expects something better than actually turns up, there is no

Similarly, let us consider tests 8, 9, and 10, in which half of the
input distribution comes from the obstructive distribution and the
other half is uniformly distributed over the good range. In test 8 the
prior is exactly matched to the distribution of factories tested, with
the expected good performance, although the number of devices to be
dropped for each factory becomes rather large (due to the obstructive
nature of this prior, which is designed to make it difficult for the
manufacturer). If the regulator moves his prior yet further to low
values of $p$ as in test 9, the performance he is interested in (what
happens to truly bad factories) gets dramatically worse, while if the
manufacturer tries to boost his own chances by making his own prior
even more prejudiced in his favour (as in test 10) the performance
that he is interested in (what happens to truly good factories) gets
worse, as does his workload.

We think that the moral of the story is not to try to influence the
results by how you set your prior - rather, your prior will be
preferred if it is more realistic, so set it to reflect what you
actually think might be the case. Maybe both parties would then more
easily reach a common mind without resort to this proposed mechanism.

Tests 11 and 12 show what happens without the proposed ``let the data
decide'' mechanism: here we set pure regulator priors that are
unmixed reasonable (test 11) or obstructive (test 12). Comparing test
11 with test 1, the classification performance of test 1 is still up
to specification (5\% and 1\% error rates under good and bad outcomes
respectively), but the manufacturer's testing workload for good
devices is reduced by 39\% compared with test 11. Comparing test 12
with test 13, we see that test 13 with the mixed prior proposed has
just as good classification performance as test 12 with only the
regulator's prior, but the manufacturer's workload is reduced by
62\%. We also note that the regulator's obstructive prior has led to
around a 5\% rate of Bad outcomes that should have been classified as
Good (due to the mismatch with the actual incoming distribution),
whether or not the present proposal is used.

Test 14 shows what happens when either every device survives being
dropped or every device breaks; we get perfect classification, with
respectively 17 or 4 devices needing to be used. The corresponding
numbers for a frequentist test using minimal sample size are 29 and 1,
so on this count also the Bayesian test does better.

But the two odd men out, with spectacularly bad performance, are tests
15 and 9. With test 15, among the 1000 simulated factories being
tested, half \textit{just} failed to meet the requirements with
$p=0.89$ while the other half \textit{just} met the requirements with
$p=0.91$. This is, of course, an extremely unrealistic scenario
(except when doing frequentist testing) -- no manufacturer could
achieve a set of factories that did this if they wanted to. Test 9 has
a very similar issue: here we again have unrealistic spikes in the
input distribution just below and just above $\tau$, and the regulator
has shot himself in the foot by putting in a prior that is far worse
than reality.

\subsubsection{Conclusions from these examples}

We can draw several conclusions from these examples, as follows:

\begin{enumerate}

\item The nature of Bayesian inference is clear from those tests (1,
2, and 8) where the input distribution exactly matches the prior:
false positives are $\approx5\%$ of the apparent postives at the
$95\%$ level and false negatives are $\approx 1\%$ of the apparent
negatives at the $99\%$ level.

\item When both parties set realistic priors, that good classification
performance is maintained in the face of mismatch in the form of
significantly worse input distribution than expected.

\item Important savings of workload for the manufacturer can be
achieved using the proposed ``let the data decide'' model for
setting a joint prior for the two parties over simply using the
regulator's prior.

\item \label{idealaim} That saving is compatible with maintained good
classification performance if both parties aim to set priors that
reflect what they really think rather than setting a prior that will
create prejudice in their own favour. Indeed, as we see from the
comparisons discussed above, moving a prior away from reality in the
direction that would be expected to favour the outcome you prefer
tends to have the opposite effect from that desired.

\item As a result of knowing about point \ref{idealaim} above, there
is an increased chance of informal discussion reaching an agreed
prior that is prejudicial to neither side.

\item If a factory does happen to be perfect then that conclusion will
be reached with $0.95$ posterior probability with fewer devices
dropped than the frequentist test of minimum sample size.

\item Frequentist testing (with input populations of factories right
on the borderline of acceptability) does not represent reality, and
is not an appropriate method of assessing either a Bayesian test
solution or a real-life test solution.

\end{enumerate}

We note also that thanks to the availability of a uniformly optimal
Bayesian data collection plan a manufacturer:

\begin{itemize}

\item does not need to know exactly how good the devices made by his
factory are before starting the test; and

\item does not need to adjust the sample size to match the actual
error rate; and

\item does not run any risk of a truly good factory failing its tests
through mis-guessing \textit{exactly} how good it is; and

\item does not have to take into account all the other parameters of
the devices that need to be tested when designing the tests for
``this'' parameter.

\end{itemize}

\section{Can frequentist results be intuitively post-processed to give
something just as good as Bayesian results ?}
\label{cannotpostfix}

\subsection{Introduction}

We now turn to consider an idea that has been put forward by some
supporters of frequentist methods, who suggest that frequentist
confidence levels are simpler to calculate than Bayesian posterior
probabilities, and can then be intuitively interpreted, perhaps using
intuitively Bayesian thinking, to give results that are just as good
as Bayesian posterior probabilities.

We can easily show that this is not in general the case, although
there is always a particular choice of a nested set of critical
regions for which the resulting frequentist confidence can be so
post-processed\footnote{Such a choice, however, usually requries
Bayesian calculations to be done first, and does not provide any
specific algorithm for doing the post-processing.}. Such a proof
rests on two fundamental pillars: the concept of Shannon information,
and the Data Processing Theorem ([15]).

\subsection{Intuitive argument}

At one level, it is obvious that we cannot in general recover the
Bayesian posterior probability itself from the frequentist confidence:
compare figures \ref{postprobeven} and \ref{freqxonly}; for any one
value of frequentist confidence in figure \ref{freqxonly} there are
many possible values of posterior probability that may correspond in
figure \ref{postprobeven}. (On the other hand comparing figures
\ref{postprobeven} and \ref{unifopt} we see that sometimes we are able
to recover posterior probability from frequentist confidence, at least
if we exclude some things that happen only with total probability

However, something much deeper is true: in general, the frequentist
confidence may tell us strictly less about the unknowns than does the
Bayesian posterior probability, although there are some cases where
both carry the same amount of information about the unknowns.

Intuitively, the argument goes like this (a detailed precise version
follows in subsequent subsections): We want to know $\theta$, and Shannon
has enabled us to define and calculate the amount of information about
$\theta$ contained in the observed data $x$ (or in any other variable). The
posterior probability distribution $P(\theta|x)$ can be shown to contain
exactly the same amount of information about $\theta$ as $x$ does. The Data
Processing Theorem tells us that any other variable that is a function
of $x$ contains no more information about $\theta$ than $x$ does; so
frequentist confidence cannot contain any \textit{more} information
about $\theta$ than the Bayesian posterior probability, and \textit{nor can
anything deduced from the frequentist confidence}. Moreover, any
function $z(x)$, for which the probability that $P(\theta|z(x))$ is not
equal to $P(\theta|x)$ is greater than zero, contains strictly less
information about $\theta$ than $x$ does. Since this applies to the
frequentist confidence calculated from many choices of nested sets of
critical regions, not only may frequentist confidence contain strictly
less information about $\theta$ than the Bayesian posterior, but there is
nothing one can then post-process out of the frequentist confidence
that does any better -- and in particular one cannot in general
post-process frequentist confidence into Bayesian posterior

Indeed, the argument can be taken further. If we are given a
probability distribution $Q(\theta|x)$ on $\Theta$, that varies with $x$, we can
define the Apparent Shannon Information (ASI) in $Q$ about $\theta$, which
measures what $Q$ tells us about $\theta$ \textit{without} any further
(intuitive or precise) post-processing. We can show that the ASI never
exceeds the (true) Shannon information in $x$ about $\theta$, and that if
we treat frequentist confidence $c$ as $Q(H_1|x)$ (setting
$Q(H_0|x)=1-c$) then in many cases the ASI is even negative, showing
that frequentist confidence can be downright misleading.

The following subsections make this argument precise.

\subsection{Shannon information}
\label{Shannon}

Suppose we have two random variables $\theta$ and $x$. Then the (true)
Shannon information (TSI) contained in $x$ about $\theta$ is defined
by $$
$$ where $\E_{P(x,\theta)}$ denotes
the expectation using the probability measure $P$ on the random
variables $x$ and $\theta$. If $x$ and $\theta$ are continuous random
variables with density functions (also denoted by $P$) this can also
be written
as $$
I(θ;x)=∫P(x,θ)logP(θ|x)/P(θ) d(x,θ),
and if they are discrete random variables then the integral can be
replaced with a summation. In both cases the range of integration or
summation is over all the possible values of $\theta$ and $x$.

If the logarithm is taken to base 2 then $I(\theta;x)$ will be measured in
bits; if taken to base $e$ then $I(\theta;x)$ will be measured in ``nats''
or ``nepers''.

To see that Shannon information is a sensible way of measuring
information content the reader is invited to consider the case that
$\theta$ is a uniform random number from $[0,1]$ that we do not know, and
that $x$ is a statement by a somewhat dishonest politician who does
know $\theta$ telling us whether it is bigger or smaller than
$\frac{1}{2}$, or whether it is bigger or smaller than $\frac{1}{4}$,

\subsection{Basic facts relevant to Shannon information}
\label{basicSi}

If $P$ and $Q$ are two probability measures on a random variable $x$,
a simple application of Jensen's inequality to the concave function
$\log$ tells us that\footnote{Here $P(x)$ and $Q(x)$ may be both
probabilities, both probability densities with respect to the same
underlying measure, or $\frac{Q(x)}{P(x)}$ may be a Radon-Nikodym
derivative of the absolutely continuous part of $Q$ with respect to
$P$ (measurable on the same $\sigma$-algebra as $P$) evaluated at
$x$. Any set on which $P$ is zero contributes nothing to the
expectation. If there is any set of positive $P$-probability but
zero $Q$-probability then the expectation is $-\infty$. Similar
considerations apply with opposite sign to the following displayed
equation.} $$
$$ with equality
if and only if $P$ and $Q$ are equal except on a set of
$P$-probability zero. In
consequence $$
Since $$
$$ we can also
deduce that $I(\theta;x)\geq 0$, and by subtraction that for any
probability distribution $Q(\theta|x)$ on $\theta$ that varies with
$x$, $$
$$ and hence that
for any random variable $z$ that is a function of
$x$, $$
i.e. $$
I(θ;x)≥I(θ; z),
$$ which is the Data Processing

Further, since\footnote{To see this, let $A$ be any measurable subset
of $\Theta$, and denote the random variable $P(\theta|x)$ (measurable
on the $\sigma$-field generated by $x$ and taking values in the space
of probability measures on $\Theta$) by $Q$, and $P(\theta\in A|x)$ by
$R$. Then
\begin{IEEEeqnarray*}{rClr} P(\theta\in A|P(\theta|x)) &=&
P(\theta\in A|Q)\\
&=& \int{P(\theta\in A,x|Q)\,dx} & \text{(marginalisation)}\\
&=& \int{P(x|Q)P(\theta\in A|x,Q)\,dx} & \text{(chain rule)}\\
&=& \int{P(x|Q)P(\theta\in A|x)\,dx} & \text{($x$ known implies $Q$ known)}\\
&=& E(P(\theta\in A|x)|Q) & \text{(definition of conditional expectation)}\\
&=& E(P(\theta\in A|x)|P(\theta|x)) & \text{(definition of $Q$)}\\
&=& E(P(\theta\in A|x)|P(\theta\in A|x) & \text{(rest of $Q$ makes no
&=& E(R|R) & \text{(definition of $R$)}\\
&=& R & \text{(expectation of a r.v. given itself is itself)}\\
&=& P(\theta\in A|x) & \text{(definition of $R$)}.\\
\end{IEEEeqnarray*}} $P(\theta|P(\theta|x)) = P(\theta|x)$, we note that the Shannon
information about $\theta$ contained in the Bayesian posterior is equal to
that contained in the observed value $x$, i.e. nothing has been lost,
thus proving the ``Information Optimality'' criterion.

We note also that $I(\theta;x)=0$ if $\theta$ and $x$ are independent, since
then $P(\theta|x)=P(\theta)$. By the initial comments about when Jensen's
inequality gives an equality, we note that this only happens if
$P(\theta|x)=P(\theta)$ except perhaps on a set of $P$-probability zero; but
then $x$ and $\theta$ are independent. Thus $x$ and $\theta$ being independent
is equivalent to there being no Shannon information about $\theta$ in $x$.

\subsection{Apparent Shannon information (ASI)}
\label{ASI}

For two variables, such as $\theta$ and $x$, the Shannon information about
$\theta$ in $x$ tells us how much we can learn about $\theta$ by applying
optimal data processing methods to $x$. But if instead we consider
variables such as $\theta$ and some arbitrary probability distribution
$Q(\theta|x)$ on $\theta$ which varies with $x$, we may also want to know how
much $Q$ tells us about $\theta$ if taken at face value without further

For this we define the Apparent Shannon Information (ASI) by $$
= _P(x,θ)logQ(θ|x)/P(θ).
$$ (In contrast the (true)
Shannon information in $Q$ about $\theta$ would be defined by $$
I(θ;Q) =

Using the same arguments as in section \ref{basicSi} we find that ASI
is maximised by setting $Q(\theta|x)=P(\theta|x)$, the Bayesian posterior.

\subsection{Application: The frequentist confidence does
not in general contain sufficient information to be able to recover
the Bayesian posterior from it}

Now consider figure \ref{freqxonly}. It is clear that the frequentist
confidence (in the case of this particular choice of nested critical
regions) is a function of the $x$-coordinate of where the bullet
lands, and that this function is injective. Accordingly the
$x$-coordinate (or the fact that the bullet didn't land) can be
recovered from the frequentist confidence as well as vice versa, and
hence by the data processing theorem the information contained in the
frequentist confidence about $h$ is equal to that contained in the
$x$-coordinate, namely 0.233 bits (as discussed in section
\ref{commonpartsall}). But the information contained in the Bayesian
posterior (making use of both coordinates) is the same as that
contained in the $x$ and $y$ coordinates together, or about 0.258
bits. Thus by the data processing theorem there is no way of
recovering either the Bayesian posterior or anything containing the
same amount of information from the frequentist confidence (for this
particular choice of critical regions, and hence in general).

\subsection{Special cases where the frequentist confidence does
contain as much information as the Bayesian posterior}

Now consider figure \ref{freqpseudoB} and compare it with figure
\ref{postprobeven}. In this case the Bayesian posterior probability is
a bijective function of the frequentist confidence, and therefore the
Shannon information about $h$ contained in the two are equal. But this
is, of course, a special case - and it is certainly \textit{not}
simpler to use the pseudo-Bayesian critical regions than to use the
Bayesian method.

Moreover, if we interpret the frequentist confidence here as a
probability that $H_1$ holds, we can assess the Apparent Shannon
Information (ASI) in the frequentist confidence and compare it with
that of the Bayesian posterior (the latter being 0.258 bits). The ASI
in the frequentist confidence is defined
as $$
$$ where for $h=1$, $c(h|x)$
denotes the frequentist confidence that $H_1$ holds given $x$ has been
observed, and for $h=0$ it denotes one minus that value. For these
nearly pseudo-Bayesian critical regions, the ASI in the frequentist
confidence turns out to be $-1.69$ bits -- in other words the
frequentist confidence is downright misleading as it stands (despite
the fact that it can be post-processed to give the Bayesian posterior
probability) -- one would be better off ignoring the frequentist
confidence and sticking with the (even) prior probability.

But if you need to use pseudo-Bayesian critical regions to get a
version of frequentist confidence from which it is possible to recover
the Bayesian posterior, why not just calculate the Bayesian posterior
in the first place ? It would certainly be simpler to do so.

Moreover, we should caution the reader that there are examples in
which a basic pseudo-Bayesian frequentist confidence \textit{cannot}
be post-processed to give the Bayesian posterior, and indeed contains
strictly less Shannon information about $\theta$. Indeed in the next
section we give an example of a problem where deterministic
frequentist solutions only ever give zero information about $\theta$.

\subsection{An inference problem on which deterministic frequentist
solutions \mbox{provide} no information at all}
\label{zeroinfo}

We return to using the notation of section \ref{infprobdef}.

Let $$
$$ $$
$$ $$
H={(0,0), (0,1),
$$ $$
$$ $$
$$ (where $[\ ]$ denotes the
function which takes the value 1 when the expression
inside the $[\ ]$ is true and 0 otherwise),
H_0={(0,0),(0,1)}=({0}×Φ)∩H =
$$ and $$

Putting that in words, there are three possible underlying states A =
$(0,0)$, B = $(0,1)$, and C = $(1,2)$, of which A and B are in $H_0$;
the observed data $x$ is always 1 if B or C is the underlying state,
and always 0 if A is the underlying state.

First consider the Bayesian position. We assume that the Bayesian
chooses an arbitrary prior on $H$ which puts non-zero probability on
each of the three possible underlying states. Then if $x=0$ is
observed, we get $P((0,0)|x) = 1$, i.e. state A becomes
certain\footnote{Technically ``almost certain'', i.e. has probability
1.}, as does $\theta=0$, which they weren't before. Thus $h$ and $x$
are not independent, and therefore $I(h;x)>0$ by the remarks of
section \ref{basicSi} above. Similarly $\theta$ and $x$ are not
independent, and therefore $I(\theta;x)>0$.

In particular, to give a specific example, suppose that the Bayesian
prior is $\frac{1}{3}$ on each of the three possibilities in $H$,
i.e. the Bayesian considers all three underlying states equally likely
\textit{a priori}. Then, considering the possible states
$(0,0),(0,1),(1,2)$ of $h$ and their associated (only possible) $x$
values of $0,1,1$ in order, we have $$
I(θ; x) =
1/3log1/2/3 + 1/3log1/2/2/3 +
1/3log1/2/1/3 = 1/3log27/16,
$$ which
works out at 0.174 nats or 0.252 bits. Moreover if different priors
are in force, the information about $\theta$ contained in the Bayesian
posterior can be anything up to 1 bit (which is the most you can ever
expect about a variable which has only two possible values).

But now consider what nested sets of critical regions the frequentist
could deploy. An empty critical region is no use, as $x$ can never end
up in it. The critical region $\{0\}$ has probability $1$ of $x$ being
in it when $h=(0,0)\in H_0$, so it would only give frequentist
confidence of zero. Similarly $\{1\}$ has probability $1$ of $x$ being
in it when $h=(0,1)\in H_0$, so too would only give frequentist
confidence of zero. The same is true for the critical region
consisting of the whole of $X$ -- and there are no other subsets of
$X$ to consider.

Thus whatever nested set of critical regions the frequentist might
deploy, he will always end up with frequentist confidence $c=0$ that
$H_1$ holds. Then $I(\theta;c)=0$, i.e. the frequentist confidence
contains no information at all about $\theta$, our wanted variable,
whichever choice of nested critical regions we adopt. Moreover
$J(\theta;c)=-\infty$ because $h=(1,2)$ occurs with non-zero
probability, and then the frequentist method always gives zero
confidence that $h\in H_1$ when actually (with the even prior above)
the data allows us to have posterior probability of $\frac{1}{2}$ that
indeed $h=(1,2)$ and $\theta=1$. Nonetheless the Bayesian posterior
does contain a positive amount of information about $\theta$. So for
this problem, no matter what choice of critical regions is made, the
Bayesian posterior can never be deduced from the frequentist
confidence when this is obtained from deterministic critical regions.

\subsection{What about non-deterministic critical regions ?}

However, the reader may wonder why we have not mentioned
non-deterministic critical regions here. It turns out that we can
always make a non-deterministic nested set of critical regions such
that the resulting frequentist confidence contains all the available
information in the Bayesian posterior about whether $H_0$ or $H_1$ is
true. Moreover, we can do so in such a way that the set of critical
regions in question are arbitrarily close to a set of basic pseudo-Bayesian
critical regions in an appropriate sense. Nonetheless, as we shall
see, there is really no point using such critical regions -- it is far
easier to just use the Bayesian method. Moreover designing such
critical regions requires use of the Bayesian prior, thus negating one
of the ``advantages'' that frequentists see in avoiding Bayes.

Take then any hypothesis testing problem, so that $\Theta=\{0,1\}$ and
$H_0=\{(\theta,\phi)\in H:\theta=0\}$, i.e. $\theta$ is either 0 or 1
and tells us whether $H_0$ or $H_1$ holds.

Now choose any $\alpha\in (0,1]$; the smaller the choice of $\alpha$,
the closer our constructed set of critical regions will be to a
pseudo-Bayesian set.

If we now consider the basic pseudo-Bayesian critical regions (for
some particular prior and data collection plan) defined
by $$
D_p={x∈X : P(H_1|x)≥p},
$$ then we may extend the data
space from $X$ to $X\times [0,1]$ with $P(x,u|h)=P(x|h)[u\in [0,1]]$,
and set $$
$$ for some
$\eta:[0,1]\to[0,1]$ a non-strictly increasing function such that
$\eta(p)$ is the frequentist confidence corresponding to $C_{\eta(p)}$
that $H_1$ holds. Then $D_p$ and $C_{\eta(p)}$ are decreasing as $p$
increases. Moreover it is clear that as $\alpha\to 0$, $C_{\eta(p)}
\to D_p\times (0,1]$ (and, in an appropriate sense, uniformly in $p$).

Now let $p$, except when acting as a dummy variable, denote the random
variable $P(h\in H_1|x)$.

If $\eta(p)$ is not to contain all the information about $\theta$
contained in $p$, we need $\eta$ to not have a left inverse, hence not
be injective, on any set of probability $1$. We will show that there
is in fact some set of zero probability off which $\eta$ is strictly
increasing and injective, whence the information content
$I(\theta;\eta(p))$ is equal to $I(\theta;p)$ as required.

By the definitions of $C$ and $\eta$, we have, for all $p\in
[0,1]$, $$
η(p) = 1 - sup_h∈H_0((1-αp)P(x∈D_p|h)) =
1 - (1-αp) sup_h∈H_0P(x∈D_p|h),
$$ which is strictly
increasing so long as $$
$$ for all
$p<1$. Suppose then that this is not the case, and note that this
quantity is a non-strictly decreasing function of $p$. Then
let $$
p_0=sup( {p∈[0,1]:sup_h∈H_0P(x∈D_p|h)>0}∪{0}),
$$ and
define $$
R={(p_0,1)     (sup_h∈H_0P(x∈D_p_0|h)>0)
[p_0,1)     (sup_h∈H_0P(x∈D_p_0|h)=0).
$$ and $$
$$ Then $\eta(p)$ is strictly increasing when $p\notin R$,
i.e. off $D_R\setminus D_1$, so it remains only to show that
$P(D_R\setminus D_1)=0$.

To avoid measure-theoretic complications we will assume that $x$ and
$h$ are variables having density functions with respect to an
underlying measure on $H\times X$ (this indeed also addresses the
general case since we may take the underlying measure to be $P$ and
the density function $P(h,x)=1$ everywhere).

Choose then any $p\in R$. Then for all $h\in H_0$, $$
$$ Also, for all $x\in D_p$, $$
P(H_1|x)P(x) =
$$ and hence $$
P(H_1|x)∫_HP(h)P(x|h)dh =
$$ and
therefore $$
∫_D_pP(H_1|x)∫_HP(h)P(x|h)dh dx =
∫_D_p∫_H_1P(h)P(x|h)dh dx.

Since the integrands are non-negative we may apply Tonelli's theorem
to get $$
A:=∫_H_0∪H_1P(h)∫_D_pP(H_1|x)P(x|h)dx dh =
∫_H_1P(h)∫_D_pP(x|h)dx dh=:B.

Since $P(H_1|x)\leq 1$, we note that $$
A≤∫_H_1P(h)∫_D_pP(H_1|x)P(x|h)dx dh +
∫_H_0P(h)∫_D_pP(x|h)dx dh.
$$ But the inner integral in
the last term is just $P(x\in D_p|h)$ for some $h\in H_0$, which we
know is zero. Therefore $$
A≤∫_H_1P(h)∫_D_pP(H_1|x)P(x|h)dx dh ≤B = A,
$$ so both
inequalities are equalities,
and $$
∫_H_1P(h)∫_D_pP(H_1|x)P(x|h)dx dh =
∫_H_1P(h)∫_D_pP(x|h)dx dh.
$$ Therefore $$
_(x,h)∈D_p×H_1P(H_1|x) = 1.
$$ But we already know
that $$
P((x,h)∈D_p×H_0) = 0,
$$ so actually $$
$$ or in other words for almost all $x \in D_p$,
$P(H_1|x)=1$, i.e. $x\in D_1$, so $P(D_p\setminus D_1)=0$. But
$D_R\setminus D_1$ is a countable union of sets of the form $D_p
\setminus D_1$ for various $p\in R$, so also $P(D_R\setminus D_1)=0$
as required, whence frequentist confidence derived from the
non-deterministic nested set of critical regions $C_{\eta(p)}$ carries
all the information in the Bayesian posterior about $\theta$.

Translating this general method onto the specific problem in section
\ref{zeroinfo} above, the solution arrived at (for the case
$\alpha=1$) is equivalent to the following recipe for calculating
frequentist confidence: \begin{quote}If $x=0$ set $c=0$ that $H_1$
holds, otherwise pick a random number $u$ uniformly from $[0,1]$ and
set $c=u$.\end{quote} It is then clear that $P(H_1|x)$ can be
recovered from $c$ with probability 1 by the following recipe
(assuming the prior of $\frac{1}{3}$ on each of the three possible
values of $h$): \begin{quote} If $c=0$ set $P(H_1|x)=0$, otherwise
set $P(H_1|x)=\frac{1}{2}$.\end{quote} Nonetheless, the ASI about
$\theta$ in this non-deterministic $c$
is $$
J(θ;c)=1/3log1/2/3 +
1/3∫_0^1logu/2/3 du +
1/3∫_0^1logu/1/3 du
$$ which works out at
$-0.030$ nats or $-0.044$ bits, showing that unless one does such
post-processing the frequentist confidence itself, even in this case,
is more misleading than just taking the prior.

But why would one bother to first choose a Bayesian prior, then use it
to calculate the Bayesian posterior in order to set up such a
complicated nested set of non-deterministic critical regions to obtain
a version of frequentist confidence from which the Bayesian posterior
could be recovered by some procedure yet to be determined ? We can see
no reason to -- it is, after all, so much easier just to calculate and
use the Bayesian posterior in the first place.

\section{Discussion of other counter-arguments}
\label{counterarguments}

At this point it seems worth listing and commenting on the
counter-arguments in favour of frequentist hypothesis testing and
confidence sets that we have encountered.

\subsection{The Bayesian method is equivalent to the frequentist

This one is so obviously false that it is surprising that anybody
would say it. What is true is that for problems satisfying certain
nice conditions (which many do, but not all), as the amount of data
approaches infinity the difference in the conclusions of the two
methods approaches zero in some sense. But we have yet to meet a
real-life inference problem in which the amount of data available
approaches infinity -- and the results reported in table
\ref{statresults} make clear that even in real-life problems for which
the said ``certain nice conditions'' \textit{do} apply, frequentist
and Bayesian methods behave very differently in important ways even
when the amount of data is in the thousands or millions of items.

\subsection{Bayesian results depend on the choice of prior}

This is true -- and so they should, as can clearly be seen from the
case where the quantity of data is zero, or where the data is
independent of the wanted unknowns, when what is known about the
wanted unknowns afterwards is exactly the same as what is known

\subsection{Bayesian methods don't control the type I error rate}

This is also true. However:
\begin{enumerate}
\item as we have seen in the example of section \ref{example1}
(e.g. in figure \ref{freqct1} or in table \ref{resultstable}),
control of the type I error rate doesn't actually guarantee that
type I errors will be uncommon in any real-world sense; and
\item as we have seen in the examples of section \ref{examplesims},
frequentist testing of type I error rate is highly unrepresentative
of real-world test scenarios; and
\item controlling the type I error rate, at least in the frequentist
ways discussed here, leads to the resulting frequentist confidence
not behaving like what most people understand by ``confidence'' at
all, as seen in section \ref{criteria}, however much frequentists
might like us to think that ``confidence'' is a good way to describe
this concept.
\end{enumerate}

\subsection{Choosing a good critical region involves more than just
controlling the type I error rate}

Various other desirable attributes of critical regions beyond those
given in section \ref{freqmeth} are sometimes listed, e.g.
\begin{enumerate}
\item \label{morelikely} it should be more likely that $x\in C_\eta$
if $h\in H_1$ than if $h\in H_0$;
\item \label{highposs} $P(x \in C_\eta|h\in H_1)$ should be as high as possible;
\item \label{wantunifopt} $C_\eta$ should be uniformly optimal.
\end{enumerate}

However of these:
\begin{itemize}

\item \ref{morelikely} is unclear unless $H_0$ and $H_1$
are singleton sets. Do we mean that $$\forall h_0\in H_0,\forall h_1\in H_1,
P(x\in C_\eta|h_1)>P(x\in C_\eta|h_0),$$ or that $$P(x\in C_\eta|h\in
H_1)>P(x\in C_\eta|h\in H_0) ?$$ The former may not be achievable, and
the latter depends on the prior distribution of $h$, something
frequentists don't want to consider;

\item moreover \ref{morelikely} is achieved in all of figures
\ref{freqmaxarea}, \ref{freqct1}, \ref{freqxonly},
\ref{freqdirection}, \ref{freqpseudoBayesfig}, and it doesn't help
us choose between them;

\item similarly \ref{highposs} depends on the prior on $h$;

\item and for most problems \ref{wantunifopt} is unachievable, while
even for those for which it is achievable, the resulting solution is
often nonsense (e.g. figures \ref{unifopt} and \ref{freqpseudoB}).
\end{itemize}

\subsection{Frequentist methods are easier to understand}

We believe that this depends entirely on whether one was first taught
frequentist statistics or Bayesian inference. The na\"ive novice
learning frequentist methods for the first time usually starts with a
natural intuition that is based around Bayesian posterior
probabilities and has to learn to invert their thinking to understand
frequentist methods. Such a person then understandably finds it hard
to revert to their original way of thinking after many years of
frequentism. On the other hand those who start by learning Bayesian
methods rarely have any difficulty understanding the basic principles.

\subsection{Bayesian solutions are harder to compute}

Or alternatively ``Frequentist solutions are easier to compute'' --
indeed so, but often because they are using approximations that are
only valid as the number of independent and identically distributed
data samples approaches infinity (which never happens in real
life). In most non-Gaussian, non-uniform cases accurate calculation of
frequentist confidence is every bit as difficult or more than
calculation of posterior probability -- consider, for example, the
calculations involved in producing figure \ref{unifopt}.

\subsection{Frequentist methods give more opportunities for research}

Again, this is true -- but we do not believe that such opportunities
are valuable contributions to knowledge of inference methods in the
real world, even though they may be good mental exercise.

As we have seen in the example of section \ref{example1}, frequentist
hypothesis testing and confidence sets require arbitrary choices of
(nested families of) critical regions to be made, and require checks
to be made on the chosen critical regions that they do indeed satisfy
the defining conditions. In 2 dimensions there is a huge choice of
potential critical regions to work on; in hundreds of dimensions there
are even more (in a non-technical sense), so there is an enormous
amount of work that could be done.

But the \textit{right} answer to the actual real-life problem is
already known -- it is the Bayesian answer, not any of the
many possible frequentist wrong answers.

Moreover to see that plenty of work can be done on Bayesian methods,
some of it hugely important, the reader is referred to
[16, 17, 18, 19],
later work by these authors, and to the numerous recent preprints by
e.g. Amos Storkey and various co-authors on
\url{https://www.arxiv.org}.

\subsection{Regulators require the use of frequentist methods}

Yes, at present they do -- and in some cases they even try to
obfuscate the issue (see [3]). But colluding with
those in power when they are wrong advances science not one whit. This
needs to change.

\subsection{Frequentist methods provide incentives for manufacturers
to produce better products}

In the context of approvals testing of a factory, it is sometimes
claimed by regulators that insisting on frequentist hypothesis tests
provides an incentive for manufacturers to design better products, as
they are then more likely to pass approvals tests than if they only
just satisfy the approvals criteria.

Let us first point out that Bayesian testing also leads to less
testing when each $h_n$ is substantially greater than $h_0$ than when
it is only just above $h_0$ -- so the incentive is not lost with

Then, let us consider the Bernoulli $N=224$ scenario of table
\ref{statresults} as discussed in section \ref{statresdisc}. Here a
manufacturer forced to do frequentist testing has done his best to
ensure that his factory satisfies all the requirements and wants his
factory, providing it is good, to pass testing with probability at
least $0.99$. However it is hard for him to be sure that his factory
is good before doing the potentially very costly approvals tests;
indeed his preliminary Bayesian scouting run takes nearly ninety
thousand devices to confirm that of the $224$ aspects less than one in
two thousand are likely to be below the quality threshold of
$h_0=0.975$. But in then designing his frequentist approval test he is
forced to do between six million and 200 million individual drops,
each on a new device, potentially using up more than a year's worth of
production. This is unreasonable, given that Bayesian testing will
achieve the goals of \textit{both} the scouting run and the final
approval test in less than sixty thousand drops, and will
automatically adjust the number of tests on each of the $N$ aspects
according to how far that particular $h_n$ is above $h_0$.

What the manufacturer using frequentist testing loses if he goes for
only a $0.9$ probability of passing a good factory, or if he omits the
scouting run, is the risk of losing the factory altogether -- for
frequentist tests cannot just be repeated once all the type I error
probability has been used up. This is usually an unacceptable risk for
such a manufacturer. So what actually risks happening is not that the
manufacturer executes this excessively demanding frequentist test, but
that he decides to assume that his factory is better than it is (or
even that it is perfect) in order to design a less demanding test,
tacitly planning to resort to cheating\footnote{There are many ways,
e.g. repeating the test of an aspect that fails hoping it won't next
time, trying a different frequentist test hoping it will give higher
confidence, claiming that a particular failed drop wasn't done
properly so replacing it with one that works, collecting more data
and pretending that one always planned to, discarding results one
doesn't like as ``outliers'', or even just testing the whole factory
again from scratch. An even more inventive way is reported in
[12] section 5.3.1 .}[12] if in consequence some aspect
of the test fails. Indeed, the need to cheat risks becoming so
commonplace that it is considered totally normal.

So, far from making it more likely that manufacturers will be
motivated to make near-perfect product, insistence on frequentist
testing is likely to result in undetected cheating and complete
invalidity of the approvals testing done.

\subsection{``I define a `good' frequentist solution to be one that
agrees with Bayes...''}

``I define a `good' frequentist solution to be one that gives the same
answer as an appropriate Bayesian solution; then I'm happy to use
\textit{good} frequentist solutions.'':
\begin{itemize}
\item How would you know that the `good' frequentist solution agrees
with Bayes unless you've run the Bayesian solution ?
\item And if you have, why not then just report the Bayesian solution
and forget the frequentist one ?
\item And if you're going to do that, why not skip the obfuscation and
just say that the frequentist way of designing inference methods is
wrong ?
\end{itemize}

\subsection{``I refuse to engage in this discussion''}

Very sadly, this is by far the most common response to attempts to
debate these matters. Refusal by those currently in the majority to
discuss where they might be wrong is not helpful, and nor is running a
truce between right and wrong.

\section{Conclusion}
\label{conclusion}

First, it is clear that the pseudo-Bayesian method cannot be better
than the best frequentist methods, as it is itself a frequentist
method. However, whether the Bayesian method is (or is not) better
than the frequentist method is a totally different question.

Failure to adhere to any one of the criteria of section \ref{criteria}
above would imply that the concept of being ``$\eta$-sure'' does not
match the real-world understanding of such an expression; this is
especially so of criterion \ref{containment}. We have already seen
that the Bayesian method adheres to all these criteria, while the
frequentist method violates them all.

In addition we note that what we conclude after collecting some
(inevitably finite amount of) data should depend not only on the data
but \textit{also} on what we knew before collecting the data. In
contrast, what we conclude should \textit{not} depend on arbitrary
choices of family of critical regions $C_\eta$. We further note that
although in a small subset of problems to be solved there is a
uniquely determined ``uniformly optimal'' critical region, in the
majority of problems there is no special critical region and there is
an infinite choice of (nested families of) critical regions satisfying
the necessary conditions.

As we have seen in figure \ref{unifopt} of the example of section
\ref{example1}, even a uniformly optimal nested set of critical
regions (where it exists) does not necessarily give remotely sensible
answers. We conclude that FHT can therefore not be considered a
reliable method of solving inference problems.

Moreover use of frequentist confidence sets in place of frequentist
hypothesis testing solves only one of the problems with the
frequentist approach (namely the lack of symmetry). We saw in section
\ref{conclusive} how confidence sets can be made to treat $H_0$ and
$H_1$ symmetrically, at least when symmetry between $H_0$ and $H_1$
exists to start with. But we also saw how there are many, many
possible choices of confidence set functions, and that not all of them
conserve symmetry (see e.g. figure \ref{freqmixedf2}). We therefore
conclude that FCS also can not be considered a reliable method of
solving inference problems.

Returning to our original concept of an admissible method, it is clear
that any Bayesian method satisfies the conditions for it to be
admissible, and that frequentist methods in general do not.

Even the issue of the choice of priors to please more than one party
can be addressed, as we have seen in section \ref{regulatorprior}
above, and in such a way that it is the data rather than either
party's particular status, opinion, or interests that determines which
party's prior prevails.

In short, this is a case of \textit{res ipsa loquitur} -- the thing
speaks for itself. It should by now be obvious both that the Bayesian
method is mathematically and logically correct and (as far as we can
tell\footnote{As G\"odel has taught us, we cannot prove it.}) consistent, and
that the frequentist method both requires arbitrary and inappropriate
choices and violates the basic properties required of any inference