# The distribution of Bayes’ ratio

Luca Amendola1,a, Vrund Patel1,a, Ziad Sakr1,111L.A., V.P., Z.S. contributed
equally to this work. 1Institut für Theoretische Physik, Universität
Heidelberg, Philosophenweg 16, 69120 Heidelberg, Germany  Elena Sellentin2,3,
Kevin Wolz4 2 Mathematical Institute, Leiden University, Gorleaus Building,
Einsteinweg 55, NL-2333 CA Leiden, The Netherlands  3 Leiden Observatory,
Leiden University, Gorleaus Building, Einsteinweg 55, NL-2333 CA Leiden, The
Netherlands  4Department of Physics, University of Oxford, Denys Wilkinson
Building, Keble Road, Oxford OX1 3RH, United Kingdom

###### Abstract

The ratio of Bayesian evidences is a popular tool in cosmology to compare
different models. There are however several issues with this method: Bayes’
ratio depends on the prior even in the limit of non-informative priors, and
Jeffrey’s scale, used to assess the test, is arbitrary. Moreover, the standard
use of Bayes’ ratio is often criticized for being unable to reject models. In
this paper, we address these shortcoming by promoting evidence ratios to
frequentist statistics and deriving their sampling distributions. By comparing
the evidence ratios to their sampling distributions, poor fitting models can
now be rejected. Our method additionally does not depend on the prior in the
limit of very weak priors, thereby safeguarding the experimenter against
premature rejection of a theory with a uninformative prior, and replaces the
arbitrary Jeffrey’s scale by probability thresholds for rejection. We provide
analytical solutions for some simplified cases (Gaussian data, linear
parameters, and nested models), and we apply the method to cosmological
supernovae Ia data. We dub our method the FB method, for Frequentist-Bayesian.

##  I Introduction

Modern cosmology is entering an era where large amounts of data will be
forthcoming from cosmic microwave background, large scale structure, distance
indicators, and gravitational waves. This requires a thorough understanding of
the employed statistical methodology in order to make robust inferences from
the data. Often investigations are performed in terms of a Bayesian study
where the analysis of competing models is typically performed in three steps:
first, the confidence region of a model is produced; secondly, two or more
competing models are compared via Bayes’ ratios; and third, the quality of fit
of the models is assessed via a simple $\chi^{2}$ test Dinda et al. (2023);
Reeves et al. (2023); Alonso et al. (2023); McDonough et al. (2023); Kou et
al. (2023); Yang et al. (2023); Holm et al. (2023); Sakr et al. (2023);
Colgáin et al. (2023); Sakr et al. (2022); Serra et al. (2007); Kurek and
Szydłowski (2008); John and Narlikar (2002); Shafer (2015); Heavens et al.
(2017) These three tools are based on heterogeneous assumptions: for instance,
the $\chi^{2}$ quality of fit is a frequentist test that does not take into
account priors, and the parameter confidence region cannot tell, per se, if an
alternative point hypothesis outside this region is excluded, until we perform
a specific model comparison. Moreover, concerning Bayes’ ratio test
limitations, its interpretation relies on arbitrary criteria to map its value
onto the commonly used Jeffreys Jeffreys (1961); Robert et al. (2008) or
Raftery-Kass scale Kass and Raftery (1995). Another important issue is that
the result depends on the considered priors, even in the weak limit or in the
case where an informative flat prior is used, since the evidence is averaged
over the prior volume which can impact differently each model. Finally, the
Bayesian comparison cannot reject models: it can only determine which model is
relatively better, but not whether either model is a good fit to the data on
its own.

Several other issues have been discussed in the literature. For instance, Ref.
Efstathiou (2008) showed that computing Bayesian evidence using cosmological
data could yield, wrongly, a statistically insignificant preference for a
model over a physically poorly motivated alternative model. Ref. Nesseris and
Garcia-Bellido (2013) showed that when comparing nested models, the Jeffreys
scale can fail to penalize extra degrees of freedom when mock data are
generated using the simpler model. Ref. Jenkins and Peacock (2011) noted that
the performance of Bayes’ factor test is strongly affected by the signal to
noise ratio in the data along with the non comprehensiveness of the models
under consideration.

There have been some attempts, still within the Bayesian framework, to remedy
for some of these limitations, in particular for the models comparison or
rejection issue, by proposing alternatives such as the Akaike information
criterion (AIC) test, the Bayesian information criterion (BIC) test or the
deviance information criterion (DIC). In that regard, Rezaei and Malekjani
(2021) performed a comparison of the above tests with Bayes’ factor using
cosmological data and found different levels of agreement between the outcome
for certain models and probes. Hoeting (1999) Hoeting et al. (1999), or
Parkinson and Liddle (2013) for a review, proposed the Bayesian Model
Averaging method where each individual posterior, generated under the
assumption of a particular model, is weighted by the model likelihood and then
combined with the other posteriors to give a model-independent posterior. This
method is especially useful in case the data are not strong enough to
distinguish decisively between the models. Ref. Kunz et al. (2006) introduced
a statistical measure of the effective model complexity that assesses how many
effective parameters a set of data can support and demonstrated that it can be
used as a useful complement to Bayes’ factor in model selection questions.
Ref. Raveri and Hu (2019) develops a full set of estimators of internal and
mutual agreement and disagreement between datasets whose strengths complement
each other. This allows to take into account the effect of prior information
and the statistical significance of discrepancies and unknown systematics.
Ref. Heavens et al. (2023) studied the sampling properties of Bayesian
evidences as a frequentist statistic when compressing the data from its
original size down to the number of free parameters. Finally, Koo et al.
(2022) proposed a method in which the likelihood distribution of the
difference in $\chi^{2}$ between a smoothed function and that of the best-fit
of the model being tested is calculated. They then use the values that enclose
95% and 99% of the likelihood volume as criterion to test whether data support
a given model, independent of how well other models perform, though they do
not assess the effect of the prior on the final decision.

In this work we propose instead the use of Bayes’ factor as a statistic to
perform a frequentist $p$-value test. Moreover, we implement a frequentist
version of Bayesian model comparison by using the odds ratio, i.e. the ratio
of probabilities, between two competing models. These two complementary tools,
$p$-value and frequentist model comparison, have interesting properties: they
are entirely based on the same statistic; they generally depend on the prior
but not in the limit of weak priors; they can assess both the quality of the
fit and the relative probability. Assuming two competing models $A$ and $B$,
the final result can give one of the following answers: the data are
compatible with $A$ but not with $B$; they are compatible with $B$ but not
with $A$; they are compatible with both; they are incompatible with both.
Finally, regardless of the above, the model comparison test can tell us
whether $A$ is favored over $B$ or viceversa. We illustrate the whole
procedure with two complementary types of plots that provide a full analysis
of the problem: one comparing the Bayes’ factor distributions with the real
data Bayes’ factor, and another one comparing the model comparison odds ratio
to the real data odds ratio.

Perhaps the first suggestion to perform a frequentist test with Bayes’ ratio
is in Ref. Good (1957) (see review in Good (1992)). A study of the frequentist
distribution of Internal Robustness (Amendola et al. (2013)), which is formed
with ratio of evidences, was carried out in Wolz (2018); Wolz et al. (2024).
The approach of this paper is quite close to Keeley and Shafieloo (2021)
Keeley and Shafieloo (2022). They generated a distribution of evidences from
datasets assuming a model and showed that the model is ruled out if its
evidence lies outside that distribution. Here, in addition, we will assess the
impact of the prior on the final decision and put forward a frequentist analog
of the Bayesian model comparison. We also provide the analytical Bayes’ ratio
distribution for correlated Gaussian data with linear nested models (thereby
extending Heavens et al. (2023) who studied model rejection under data
compression). In this paper, we apply our method, to be denoted as FB method,
to a simple polynomial fit and to supernovae Ia data.

##  II Bayes’ ratio for Gaussian data and linear models

Bayes’ ratio is used in general to compare two competing models, where the
fiducial model is compared to a new one by computing the ratio $b$ defined as

$b=\frac{E({D}|{M}_{A})}{E({D}|{M}_{B})},$ (1)

where $E$ is the evidence

$E({D}|{M})=\int d\theta\ {\cal L}({D}|\theta,{M})\,{\cal P}(\theta,{M})\,,$
(2)

with ${\cal L}(D|M_{A,B})$ the probability of the data $D$ given model
$M_{A,B}$ with parameters $\theta_{\alpha}$ and a prior ${\cal P}$. Model
$M_{A}$ is favored over model $B$ if $b>1$, and $M_{B}$ is favored otherwise.

In this and the next section we will find the analytical distribution of
Bayes’ ratio in a particular, but important, case, namely Gaussian data with
linear nested models. Let us start with introducing the general problem.

Linear models can be described by a sum $f_{i}=\theta_{\alpha}g_{\alpha i}$
where $\theta$ is the parameter vector and $g_{\alpha i}$ denotes the linear
mapping between the $\alpha$-th model parameter and the $i$th datapoint. For
example, the parameters may act as the linear coefficients of a polynomial in
a deterministic variable $x$,
$f_{i}=\theta_{0}+\theta_{1}x_{i}+\theta_{2}x_{i}^{2}+...$. We use Greek
subscripts referring to parameters, lowercase Latin subscripts referring to
data, and capital Latin subscripts to refer to models. Moreover, we adopt the
convention that repeated indexes are summed over. The likelihood for $N$ data
points $d_{i}$ is

${\mathcal{L}}=(2\pi)^{-N/2}|\Sigma|^{-1/2}\exp-\frac{1}{2}(d_{i}-f_{i})\Sigma^{-1}_{ij}(d_{j}-f_{j})$
(3)

where $\Sigma$ is the data covariance matrix. We also assume a Gaussian prior
with mean vector $\tilde{\theta}$

${\mathcal{P}}=(2\pi)^{-N_{M}/2}|P|^{1/2}\exp-\frac{1}{2}(\theta_{\alpha}-\tilde{\theta}_{\alpha})P_{\alpha\beta}(\theta_{\beta}-\tilde{\theta}_{\beta})$
(4)

where $N_{M}$ is the number of parameters for model M. The posterior is
therefore $\mathcal{LP}/E$, where $E$ is the evidence. In the limit of weak
prior, $P_{\alpha\beta}\to 0$. We will often adopt this limit to obtain
simpler expressions. Frequently used symbols are listed in Table 1.

The bestfit (maximum likelihood) parameters are found by imposing
$\partial{\mathcal{L}}/\partial\theta_{\alpha}=0$, i.e. solving the equation

$g_{\alpha i}\Sigma^{-1}_{ij}(d_{i}-\hat{\theta}_{\beta}g_{\beta j})=0$ (5)

from which $\hat{\theta}_{\beta}=D_{\alpha}L^{-1}_{\alpha\beta}$ where
$L_{\alpha\beta}=g_{\alpha i}\Sigma_{ij}^{-1}g_{\beta j}$ is the Fisher
matrix, and $D_{\alpha}=d_{i}\Sigma_{ij}^{-1}g_{\alpha j}$. In the limit of
weak prior, $\hat{\theta}$ coincides with the maximum of the posterior, and
therefore gives the Bayesian best fit solution. For general Gaussian priors,
the posterior maximum is

$\theta^{*}=(\hat{\theta}L+\tilde{\theta}P)F^{-1}\,.$ (6)

For linear models and Gaussian data, the maximum likelihood parameters are
Gaussian random variables and the Fisher matrix represents their exact inverse
covariance. In Sec. IV we briefly discuss the Fisher approximation for non-
linear models.

We define the matrix $G$ as $g_{\alpha i}L_{\alpha\beta}^{-1}g_{\beta j}\equiv
G_{ij}$. In index-free notation, $\hat{\theta}^{T}=D^{T}L^{-1}$,
$D^{T}=d^{T}\Sigma^{-1}g^{T}$, $G=g^{T}L^{-1}g$ and $L=g\Sigma^{-1}g^{T}$. We
will use both indexes and index-free notation, whichever is clearer in each
case. It follows

$f=\hat{\theta}^{T}g=D^{T}L^{-1}g=d^{T}\Sigma^{-1}g^{T}L^{-1}g=d^{T}\Sigma^{-1}G$
(7)

We see that $C\equiv\Sigma^{-1}G$ is the matrix that projects the information
from $d$ to the theory best-fit $g^{T}\hat{\theta}$. (Note that $C^{2}=C,$ so
it’s indeed a projector; it’s also a symmetric matrix). Note also that

$\displaystyle\mathrm{Tr}\,C={\mathrm{T}r}(\Sigma^{-1}G)={\mathrm{T}r}(\Sigma^{-1}g^{T}L^{-1}g)={\mathrm{T}r}(g\Sigma^{-1}g^{T}L^{-1})={\mathrm{T}r}(LL^{-1})=N_{M}$
(8)

where $N_{M}$ is the number of parameters of model $M$. Then on the best fit
one has (see App. A)

$\displaystyle\hat{\chi}^{2}$
$\displaystyle=(d^{T}-\hat{\theta}^{T}g)\Sigma^{-1}(d-g^{T}\hat{\theta})$ (9)
$\displaystyle=d^{T}\Sigma^{-1}(I-C)d$ (10)

We now rewrite the likelihood as a function of the parameters, by writing
$f_{i}=\hat{\theta}_{\alpha}g_{\alpha
i}+(\theta_{\alpha}-\hat{\theta}_{\alpha})g_{\alpha i}$ and using the fact
that

$\displaystyle\frac{1}{2}(d_{i}-f_{i})\Sigma^{-1}_{ij}(d_{j}-f_{j})$
$\displaystyle=\frac{1}{2}(d_{i}-\hat{\theta}_{\alpha}g_{\alpha
i})\Sigma^{-1}_{ij}(d_{j}-\hat{\theta}_{\beta}g_{\beta
i})-(\theta_{\alpha}-\hat{\theta}_{\alpha})g_{\alpha
i}\Sigma^{-1}_{ij}(d_{j}-\hat{\theta}_{\beta}g_{\beta
i})+\frac{1}{2}(\theta_{\alpha}-\hat{\theta}_{\alpha})g_{\alpha
i}\Sigma^{-1}_{ij}g_{\beta j}(\theta_{\beta}-\hat{\theta}_{\beta})$
$\displaystyle=\frac{1}{2}\hat{\chi}^{2}+\frac{1}{2}(\theta_{\alpha}-\hat{\theta}_{\alpha})L_{\alpha\beta}(\theta_{\beta}-\hat{\theta}_{\beta})$
(11)

since $(\theta_{\alpha}-\hat{\theta}_{\alpha})g_{\alpha
i}\Sigma^{-1}_{ij}(d_{j}-\hat{\theta}_{\beta}g_{\beta i})=0$ on account of Eq.
(5). Therefore

${\mathcal{L}}=(2\pi)^{-N/2}|\Sigma|^{-1/2}e^{-\frac{1}{2}\hat{\chi}^{2}}\exp-\frac{1}{2}(\theta_{\alpha}-\hat{\theta}_{\alpha})L_{\alpha\beta}(\theta_{\beta}-\hat{\theta}_{\beta})$
(12)

Using this expression, one finds that the evidence for Gaussian data, linear
model and Gaussian priors is

$E=\int{\mathcal{L}\mathcal{P}}d^{N}\theta=(2\pi)^{-N/2}|\Sigma|^{-\frac{1}{2}}e^{-\frac{1}{2}\hat{\chi}^{2}}\frac{|P|^{1/2}}{|F|^{1/2}}\exp\left[-\frac{1}{2}(\hat{\theta}-\tilde{\theta})^{T}LF^{-1}P(\hat{\theta}-\tilde{\theta})\right]\,.$
(13)

symbol | meaning
---|---
$\Sigma_{ij}$ | data covariance matrix
$d_{i}$ | data points
$g_{\alpha i}$ | fit functions matrix (design matrix)
$P_{\alpha\beta}$ | prior precision matrix (inverse of the parameter covariance matrix)
$L_{\alpha\beta}$ | likelihood parameter Fisher matrix
$F_{\alpha\beta}=L_{\alpha\beta}+P_{\alpha\beta}$ | posterior parameter Fisher matrix
$N$ | number of data points
$N_{M}$ | number of parameters for model M
$\hat{\theta}_{\alpha}$ | vector of best-fit parameters (dependent on the data)
$\tilde{\theta}_{\alpha}$ | vector of prior means
$\hat{\chi}^{2}$ | minimum $\chi^{2}$ (dependent on the data)
$\beta=2\log b$ | log of ratio of evidences $b$ (Bayes’ ratio)
Table 1:  Frequently used symbols.

In the linear fit case, the data (our random variables) are only in
$\hat{\theta}$ and in $\hat{\chi}^{2}$; all the other quantities are not
random variables.

Bayes’ ratio for models $A$ (with $N_{A}$ free parameters) and $B$ (with
$N_{B}>N_{A}$ free parameters) defined as the ratio of evidences, becomes then

$\displaystyle b$
$\displaystyle\equiv\frac{E_{A}}{E_{B}}=\frac{|\Sigma|^{-\frac{1}{2}}e^{-\frac{1}{2}{\hat{\chi}}_{A}^{2}}\frac{|P_{A}|^{1/2}}{|F_{A}|^{1/2}}\exp\left[-\frac{1}{2}(\hat{\theta}_{A}-\tilde{\theta}_{A})^{T}L_{A}F_{A}^{-1}P_{A}(\hat{\theta}_{A}-\tilde{\theta}_{A})\right]}{|\Sigma|^{-\frac{1}{2}}e^{-\frac{1}{2}\hat{\chi_{B}}^{2}}\frac{|P_{B}|^{1/2}}{|F_{B}|^{1/2}}\exp\left[-\frac{1}{2}(\hat{\theta}_{B}-\tilde{\theta}_{B})^{T}L_{B}F_{B}^{-1}P_{B}(\hat{\theta}_{B}-\tilde{\theta}_{B})\right]}$
(14)
$\displaystyle=\exp\left[-\frac{1}{2}(\hat{\chi}^{2}_{A}-\hat{\chi}_{B}^{2})-\frac{1}{2}(\chi_{P,A}^{2}-\chi_{P,B}^{2})\right]\frac{|P_{A}F_{B}|^{1/2}}{|P_{B}F_{A}|^{1/2}}$
(15)

where

$\chi_{P,M}^{2}=(\hat{\theta}_{M}-\tilde{\theta}_{M})^{T}L_{M}F_{M}^{-1}P_{M}(\hat{\theta}_{M}-\tilde{\theta}_{M})$
(16)

We have then

$\displaystyle\beta$ $\displaystyle\equiv 2\ln
b=\chi_{B}^{2}-\chi_{A}^{2}+\chi_{P,B}^{2}-\chi_{P,A}^{2}+Y_{B}-Y_{A}$ (17)

where

$Y_{M}=\ln\frac{|F_{M}|}{|P_{M}|}$ (18)

with $M=A,B$. In Eq. (17), the first pair of terms gives the distance between
the likelihood maxima, the second pair measures how close are the best-fits to
the prior mean, and the third pair measures how new data improve upon the
prior. In the standard Bayesian approach, model A is preferred if $\beta$ is
positive and large, that is, when it is a good fit (small $\chi^{2}_{A}$),
which does not depart too much from the prior (small $\chi^{2}_{P,A}$), and
when the gain from the prior to the final uncertainty is minimal (small
$Y_{A}$). From Eq. (17) we already intuitively see that a ‘robust’ rejection
of one model (i.e. a rejection that holds up under repeated draws of the data)
depends on the relative magnitude of the $\chi^{2}_{M}$ terms with respect to
the $Y_{M}$ terms. If $Y_{M}\gg\chi^{2}_{M}$, then $\beta$ will hardly scatter
under repetitions of the experiment. However, $Y_{M}$ sets the experiments
measurement precision with respect to the experiments prior range. We
therefore see that an experiment that fails to strongly update the prior
indeed also produces a $\beta$ that strongly scatters, and hence warrants a
comparison to the sampling distribution of $\beta$ for reliable model
selection.

We removed the hat from $\chi^{2}$ since from now on we interpret $\beta$ as a
frequentist statistic, i.e. as a random variable which can assume any value,
while $\hat{\chi}^{2}$ was the specific value obtained with the real dataset.

For nested models, there will be a number of common parameters, and some
additional parameters for model B (we call them ”common parameters” and ”extra
parameters”, respectively, in the following). We assume that the prior means
for the common parameters are the same for both models, and zero for the extra
ones (this is not restrictive: one can always redefine the extra parameters so
that when they vanish model B reduces to model A). The elements $g_{\alpha i}$
will then be the same for the common parameters, and the prior mean vector
$\tilde{\theta}$ for model B will be the same as for model A, plus as many
zeros as the extra parameters.

In the limit of weak prior, since

$\displaystyle L_{A}F_{A}^{-1}P_{A}$
$\displaystyle=L_{A}(L_{A}+P_{A})^{-1}P_{A}$ (19)
$\displaystyle=(P_{A}^{-1}+L_{A}^{-1})^{-1}\approx P_{A}$ (20)

vanishes, also $\chi^{2}_{P,M}$ vanishes, and we have

$\beta={\chi}^{2}_{B}-{\chi}_{A}^{2}+Y_{B}-Y_{A}\,.$ (21)

In this limit, the statistic $\beta$ has the same distribution as the log of
the likelihood ratio (up to the constant $Y_{B}-Y_{A}$).

If, in the same weak prior limit, for simplicity one assumes that both $F$ and
$P$ are diagonal matrices with entries $\sigma^{-2}$ and $\sigma_{P}^{-2}$,
respectively, for each parameter, and $\sigma_{P}=\alpha\sigma$, then one has
$|F_{M}|=\sigma^{-2N}$ and $|P|=\sigma_{P}^{-2N}$, and therefore

$\beta={\chi}^{2}_{B}-{\chi}_{A}^{2}+2r\log(\alpha)$ (22)

where $r=N_{B}-N_{A}$, which always favors $A$ for $\alpha\to\infty$. This
argument can be directly extended to any likelihood, not just Gaussian. In
fact, if the prior is very broad with respect to the likelihood, then its
value can be taken approximately constant over a large volume $V_{M}$ in
parameter space, so that ${\mathcal{P}}\approx 1/V_{M}$. Then one has

$b\approx\frac{\int{\mathcal{L}}{\mathcal{P}}d^{N_{A}}\theta_{A}}{\int{\mathcal{L}}{\mathcal{P}}d^{N_{B}}\theta_{B}}\to\frac{V_{B}\int{\mathcal{L}}d^{N_{A}}\theta_{A}}{V_{A}\int{\mathcal{L}}d^{N_{B}}\theta_{B}}$
(23)

If models are nested, then $V_{B}=V_{A}V_{e}$, where $V_{e}$ is the volume of
the extra parameters. Then again one sees that $b$ increases arbitrarily for
large prior volume $V_{e}$. The unappealing consequences of this behavior in a
simple but extreme example are illustrated in App. D.

Note that the artificial suppression of the new model arises because the prior
was subjectively chosen as one that is effectively a flat prior with wide
boundaries. If the prior is derived by sound physical or mathematical
principles (see e.g. Heavens and Sellentin (2018) who computed an
uninformative prior that does not disturb the low signal-to-noise measurements
of elusive neutrino physics), then of course the prior is part of the model
and no issue arises.

##  III Analytical distribution of Bayes’ ratio

Our next goal now is to answer this question: given that the data are Gaussian
random variables distributed around a given model, what is the distribution of
$\beta$ ? We always assume the models to be nested, i.e. model B has a subset
of parameters that coincide with those of model A, and with the same prior,
plus a number of extra parameters with their corresponding prior. In other
words, model A is a constrained version of model B. Moreover, without lack of
generality, model B reduces to model A if the extra parameters are put to
zero. This applies, for instance, to $\Lambda$CDM and $w$CDM, which is the
real data example we investigate later on, provided one takes as extra
parameter $w^{\prime}=w-1$.

It is important to realize that the random variables (the data) enter
quadratically both in $\chi^{2}_{A/B}$ and in $\chi^{2}_{P,B/A}$. A positive-
defined quadratic form of Gaussian variables $x_{i}M_{ij}x_{j}$, as $\beta$ in
Eq. (21), is distributed as a $\chi^{2}$ variable. There are however three
types of $\chi^{2}$ variables: standard, non-central, and generalized. If the
Gaussian variables $x$ have zero mean, the $\chi^{2}$ distribution is the
standard one, defined uniquely by the degrees of freedom (dof) $\nu$. If the
Gaussian variables have mean $\mu_{i}$ such that

$\lambda=\mu_{i}M_{ij}\mu_{j}$ (24)

then they are non-central $\chi^{2}$ variables, defined by $\nu$ and
$\lambda$. Finally, a linear combination of non-central $\chi^{2}$ variables
is distributed as a generalized $\chi^{2}$ variable. This distribution is
defined by the $\nu,\lambda$ of each $\chi^{2}$ variables, and by their
coefficients of the linear combination.

As we will see, the standard case applies to nested models with weak priors
when assuming A to be true and the prior mean to coincide with the best fit
values. The non-central case applies to nested models with weak priors when
the prior mean does not coincide with the best fit values, or with any prior
but the two models differ by just one parameter. The generalized case,
finally, applies in all other situations. We need also consider variables that
are shifted and scaled versions of $\chi^{2}$ variables, i.e.
$z\equiv\alpha\chi^{2}+a$, and their distributions are trivially obtained by
shifting and scaling the argument of the original $\chi^{2}$ distributions.
More detailed information in App. C.

We need then to show that the combination
$(\chi_{B}^{2}+\chi_{P,B}^{2})-(\chi_{A}^{2}+\chi_{P,A}^{2})$, which are the
terms that depend on data, is a quadratic form and find out its distribution.
For each term we have (see Appendix A for the detailed calculation; we
temporarily suppress here the subscript A or B)

$\chi^{2}+\chi_{P}^{2}=\Xi$ (25)

where

$\Xi\equiv(d_{m}-u_{m})T_{mk}(d_{k}-u_{k})$ (26)

and (see App. A for the next four equations)

$\displaystyle u_{k}$
$\displaystyle=T_{kn}^{-1}\Sigma_{n\ell}^{-1}g_{\alpha\ell}(F^{-1}P)_{\alpha\gamma}\tilde{\theta}_{\gamma}=W_{k\gamma}\tilde{\theta}_{\gamma}$
(27) $\displaystyle T$ $\displaystyle=\Sigma^{-1}(I-g^{T}F^{-1}g\Sigma^{-1})$
(28)

It turns out that

$T^{-1}=\Sigma+g^{T}P^{-1}g$ (29)

and therefore

$u=g^{T}\tilde{\theta}$ (30)

i.e., $W=g$. Since $\Xi$ is a quadratic form of Gaussian variables, it is
distributed as one of the three types of $\chi^{2}$ variable.

Let us denote with $\mu_{m}$ the expected value of $d_{m}-u_{m}$

$\mu=\langle d-u\rangle=g^{T}(\hat{\theta}-\tilde{\theta})$ (31)

We also have $\langle d\rangle=f=\hat{\theta}g$ and, by definition of
covariance matrix,

$\langle(d-g^{T}\hat{\theta})(d^{T}-\hat{\theta}^{T}g)\rangle=\Sigma$ (32)

so that

$\displaystyle\langle\Xi\rangle$
$\displaystyle=\langle(d-g^{T}\hat{\theta})+\mu)^{T}T((d-g^{T}\hat{\theta})+\mu)\rangle$
$\displaystyle=\langle(d-g^{T}\hat{\theta})^{T}T(d-g^{T}\hat{\theta})\rangle+\mu^{T}T\mu-2\mu^{T}T\langle(d-g^{T}\hat{\theta})\rangle=T\Sigma+\mu^{T}T\mu$
(33)

From now on we reinsert the subscripts $A$ or $B$. Now we can consider the
combination of $A$ and $B$, i.e. $\beta=\Xi_{B}-\Xi_{A}+Y_{B}-Y_{A}$. We
assume that the prior mean vector $\tilde{\theta}$ is the same for the common
parameters, and zero for the additional B parameters. This is where the
condition of nestedness enters and simplifies considerably the problem. Then
clearly $u_{A}=u_{B}=u$. We have then

$\displaystyle(d_{m}-u_{m})T_{B,mk}(d_{k}-u_{k})-(d_{m}-u_{m})T_{A,mk}(d_{k}-u_{k})=(d_{m}-u_{m})T_{mk}(d_{k}-u_{k})$
(34)

where

$\displaystyle T$
$\displaystyle=T_{B}-T_{A}=\Sigma^{-1}(g^{T}F^{-1}g|_{A}-g^{T}F^{-1}g|_{B})\Sigma^{-1}$
(35)

In order to define the mean

$\mu=\langle d-u\rangle=\langle d\rangle-u$ (36)

we need to specify whether we assume $A$ or $B$ to be true. Then we have

$\mu_{M}=g^{T}_{M}(\hat{\theta}_{M}-\tilde{\theta}_{M})$ (37)

In the following, $\mu_{M}$ will take different values depending on whether
one assumes $M=A$ or $M=B$ to be true. In the general case we have the
following mean and variance:

$\displaystyle\langle\beta\rangle$
$\displaystyle=\langle\Xi_{B}\rangle-\langle\Xi_{A}\rangle+Y_{B}-Y_{A}=\mathrm{Tr}(T\Sigma)+\mu_{M}^{T}T\mu_{M}+Y_{B}-Y_{A}$
(38) $\displaystyle V[\beta]$
$\displaystyle=\sigma_{\beta}^{2}=2\mathrm{Tr}(T\Sigma)^{2}+4\mu_{M}^{T}T\Sigma
T\mu_{M}$ (39)

Let us first consider the weak prior limit. In this case, $F\approx L$, and
$T_{M}\Sigma\approx$
$\Sigma^{-1}(I-g_{M}^{T}L^{-1}g_{M}\Sigma^{-1})\Sigma=I-\Sigma^{-1}G_{M}=I-C_{M}$
222 One can also simplify the expression for the variance by noting that
$(T_{M}\Sigma)^{2}=T_{M}\Sigma$ and therefore $T_{M}\Sigma T_{M}=T_{M}\Sigma
T_{M}\Sigma\Sigma^{-1}=(T_{M}\Sigma)^{2}\Sigma^{-1}=T_{M}\Sigma\Sigma^{-1}=T_{M}$.
Since $C_{M}$ is a projector, its eigenvalues are 1 or 0. Because of Eq. (8),
we see that $C_{M}$ has $N_{M}$ unit eigenvalues and $N-N_{M}$ zero
eigenvalues. Then the matrix

$-T\Sigma=-I+C_{B}+I-C_{A}=C_{B}-C_{A}$ (40)

has $r=N_{B}-N_{A}$ unit eigenvalues and the rest are zero. From App. C, we
conclude that $-\beta+(Y_{B}-Y_{A})$ is a non-central $\chi^{2}$ variable with
$\nu=r$ degrees of freedom and non-central parameter

$\lambda_{M}=-\mu_{M}^{T}T\mu_{M}$ (41)

That is, $\beta$ is distributed as

$P(x;a,\nu,\lambda)=\frac{1}{2}e^{-(a-x+\lambda)/2}\left(\frac{a-x}{\lambda}\right)^{\nu/4-1/2}I_{\nu/2-1}(\sqrt{\lambda(a-x)})$
(42)

with $a=Y_{B}-Y_{A}$, and where $I$ is a modified Bessel function of the first
kind. In the special case in which the elements of the prior mean vector
$\tilde{\theta}$ coincide with model A’s best fit parameters
$\hat{\theta}_{A}$ for the common parameters, and vanishes for the extra B
parameters, then $\mu_{A}=0$, so that $\lambda_{A}=0$, and $\beta$ is a
standard $\chi^{2}$ variable up to shift and inversion, so its distribution is

$P(x;a,\nu)=\frac{(a-x)^{\nu/2-1}e^{-(a-x)/2}}{2^{\nu/2}\Gamma(\nu/2)}$ (43)

Finally, if the prior is not negligible, then the eigenvalues of $-T\Sigma$
are no longer 1 or 0’s, and $\beta$ can be written as a sum of non-central
$\chi^{2}$ variables. This sum follows a generalized $\chi^{2}$ distribution.
In this case, , as we explain in App. C, there is a simple analytical
expression for the moment-generating function, but no closed form of the
distribution, which has then to be computed by an inverse Fourier transform.
So, in general, the logarithm of Bayes’ factor is one of the three kinds of
$\chi^{2}$ variable in the Gaussian-data, linear-parameter case. 333Clearly,
the logarithm of the evidence is also a $\chi^{2}$ variable, now with
$N-N_{M}$ dofs. One could also envisage a frequentist test based on the
evidence of a single model, without comparing it to an alternative. In this
paper, however, we explore only the properties of Bayes’ ratio.

The Bayes’ distributions can now be employed to build our FB method, i.e. to
test the hypothesis that, given that model M is true, the data are consistent
with M rather than with an alternative M’. Since a $p$-value test can only
reject hypotheses, and not assess whether they are true, we need to make two
tests when comparing A versus B. In the first, we assume A to be true, so that
the null hypothesis is $H_{0}:$ (A is true), and we either reject or non-
reject A; in the second test, we assume B to be true, so $H_{0}:$ (B is true),
and again we can reject or non-reject B. Together, the two tests can produce
four distinct outcomes: reject both models, reject neither of them, reject
only A or only B. One can consider both one-tailed tests, as commonly done for
likelihood ratio tests, or two-tailed tests, as is typically the case for
$\chi^{2}$ tests. In this second case, the point of view is that too good fits
are rejected because incompatible with the assumed data errors. Here, however,
we only discuss one-tailed tests for simplicity. The entire procedure will be
illustrated below in Sec. VI and VII.

##  IV Fisher approximation

In many realistic applications, parameters do not enter linearly in the
likelihood. The fitting function $f$ can however be linearized near the best
fit as

$f(x,\theta)\approx f(x)|_{\theta=\hat{\theta}}+\sum_{\alpha}\frac{\partial
f}{\partial\theta_{\alpha}}|_{\theta_{\alpha}=\hat{\theta}_{\alpha}}(\theta_{\alpha}-\hat{\theta}_{\alpha})$
(44)

The Fisher matrix for Gaussian data with parameters only in the mean vector
$f_{i}$ is then

$F_{\alpha\beta}=\left(\frac{\partial
f_{i}}{\partial\theta_{\alpha}}\Sigma^{-1}_{ij}\frac{\partial
f_{j}}{\partial\theta_{\beta}}\right)_{\theta_{\alpha}=\hat{\theta}_{\alpha}}$
(45)

Then we see that we can identify

$g_{\alpha i}=\frac{\partial
f(x_{i})}{\partial\theta_{\alpha}}|_{\theta_{\alpha}=\hat{\theta}_{\alpha}}$
(46)

We can then use a Fisher matrix approximation to obtain the matrix $L$:

$L_{\alpha\beta}=g_{\alpha i}\Sigma^{-1}_{ij}g_{\beta j}$ (47)

and the problem is reduced to the one of the previous sections. Whether this
approximation is acceptable, however, should be analysed on a case-by-case
basis. We apply this Fisher approximation in Sec. VII.

##  V A frequentist model-comparison test

The use of the $\beta$ distribution to see whether models $A$ or $B$ are
compatible with the data is an absolute test of compatibility of data with
models (goodness of fit). But we can also directly compare $A$ with $B$ and
see which one is to be considered a better fit to the data (model comparison).
The result of this model comparison will be a statement about which model
better explains the data, even if both are, on absolute terms, unacceptable
fits. These two tools, together, form the content of our FB method.

In Bayesian context, $b=e^{\beta/2}$ is not a random variable. Its value
expresses the odds ratio of $A$ versus $B$, i.e. the ratio of the
probabilities that, given the observed data $D$, the model $A$ or the model
$B$ is to be preferred,

$b=e^{\beta/2}=\frac{P(A)}{P(B)}$ (48)

The value of $\beta$ is then compared to an empirical scale (Jeffrey’s scale
or one of its variants) to convert the number into an expression of ordinary
language.

Alternatively, we can also convert $b$ into the probability of $A$ given that
the possible models are only $A$ or $B$ as follows (the subscript $b$ here
stands for Bayesian):

$R_{b}(A)\equiv
P(A|A\mathtt{or}B)=\frac{P(A\mathtt{and}(A\mathtt{or}B))}{P(A\mathtt{or}B)}=\frac{P(A)}{P(A)+P(B)}=\frac{b}{1+b}=(1+e^{-\beta/2})^{-1}$
(49)

In this way, Bayes’ ratio can be interpreted as the probability of “event” $A$
within a set of possible disjoint events composed only by $A$ and $B$. This
probability over the space of events is well normalized: in fact, one also has

$R_{b}(B)\equiv P(B|A\mathtt{or}B)=\frac{P(B)}{P(A)+P(B)}=1-R_{b}(A)$ (50)

Following this point of view, Jeffreys’ scale can be replaced by ordinary
probability thresholds. For instance we can accept model $A$ versus $B$ at 68%
c.l. if

$R_{b}(A)=\frac{b}{1+b}\geq 0.68$ (51)

or $b\geq 2.125$ (“barely worth mentioning” in Jeffreys’ language), at $95\%$
c.l. if $b\geq 19$ (“strong”) and at $99.7\%$ if $b\geq 332$ (“decisive”).
Naturally, if there are other models to consider, and not just $A$ and $B$ (as
is almost always the case), then this scale is hardly meaningful because it
depends on the arbitrary number of models we decide to include. The main
problem, however, as we already emphasized, is that $R_{b}$ depends on the
prior also in the limit of weak priors:

$R_{b}=(1+e^{-\beta/2})^{-1}=(1+e^{-(\chi^{2}_{B}-\chi^{2}_{A}+Y_{B}-Y_{A})/2})^{-1}=(1+\sqrt{\frac{|F_{A}||P_{B}|}{|P_{A}||F_{B}|}}e^{-\frac{1}{2}(\chi_{B}^{2}-\chi_{A}^{2})})^{-1}$
(52)

For instance, let us assume that model $B$ is identical to model $A$ except
for an additional parameter that has an uncertainty $\sigma_{B}$ and a prior
with a very large error $\sigma_{P,B}\to\infty$. Then, if both models have a
diagonal Fisher matrix, we can write $F_{B}=F_{A}\sigma_{B}^{-2}$ and
$P_{B}=P_{A}\sigma_{P,B}^{-2}$, and one gets

$R_{b}(A)\approx
1-(\frac{\sigma_{B}}{\sigma_{P,B}})^{2}e^{-\frac{1}{2}(\chi^{2}_{B}-\chi^{2}_{A})}\to
1$ (53)

This shows that model $A$ is always favoured in this limit, regardless of its
quality. That is, the addition of a parameter with a very weak prior, maybe
because $B$ is a new model with little experimental history, is discarded a
priori. This carries the risk of artificially enhancing trust on established
models when confronted with unbeaten paths (see App. D for an example).

In the frequentist interpretation, the Bayesian odds ratio should be replaced
by the ratio of the distributions of $\beta$ when assuming either $A$ or $B$.
We can then form the frequentist (subscript $f$) analog of $R_{b}$ as

$R_{f}(A)=\frac{P_{A}(\alpha_{A}\beta+a)}{P_{A}(\alpha_{A}\beta+a)+P_{B}(\alpha_{B}\beta+a)}$
(54)

and of course its complement, $R_{f}(B)=1-R_{f}(A)$. We denoted with
$P(\alpha_{M}\beta+a)$ a distribution ($\chi^{2}$ or otherwise) for the
variable $\chi^{2}=\alpha_{M}\beta+a$ with $M=A,B$. Since in the weak prior
limit $\alpha_{M}\beta+a=\chi^{2}_{A}-\chi^{2}_{B}$, we see that $R_{f}$,
contrary to $R_{b}$, becomes independent of the prior. A value of $R_{f}(A)$
larger than $1/2$ means that $A$ is favoured with respect to $B$; viceversa,
$R_{f}(A)<1/2$ means that $B$ is favoured. However, here we encounter the same
caveat as in the Bayesian case, namely, that $R_{f}$ can be directly compared
to a Gaussian scale only if the models $A$ and $B$ are exhaustive of the model
landscape.

##  VI Applications: polynomial fits

Now that we know the distribution of the statistic $\beta$, we can form an
analytical $p$-value test. Let us assume as null hypothesis $H_{0}:$ model $A$
is better than B in explaining the data, and as alternative hypothesis
$H_{1}$: model $B$ is better than $A$. We can then find a specific value of
$\beta$ from a given dataset and compare it to the $\beta$ distribution.

As a first illustration of the method, we define two models: model A (a
straight line $f=a+bx$) and model B (a quadratic function $f=a+bx+cx^{2}$,
with the same $a,b$ as for model A). We take as prior for $a,b,c$ a Gaussian
with mean $a=1,b=1$ and two values of $c$, and diagonal prior covariance with
a very broad variance, $\sigma_{P}=10$ for each parameter (to be compared to
errors between 0.1 and 1 for the three parameters). Then we generate many sets
of Gaussian data points distributed around model A with a given data
covariance matrix $\Sigma$ (we put for simplicity $\Sigma=I$, but we tested
also correlated cases). Every time a data set is generated, the best fit
parameters $\hat{a},\hat{b},\hat{c}$ are obtained by fitting first with a
straight line and then with a quadratic function, and the resulting $\beta$ is
calculated. This gives the distribution of $\beta$ assuming A is correct. Then
we repeat generating many data sets around model B (with the same prior and
data covariance matrix), and obtain the distribution of $\beta$ when B is
assumed correct. The parameter $c$ is a measure of how different the two
models are, and therefore of how much the two distributions overlap. Putting
as means $a=b=1$, we find the distributions as in Figs. 1 (with $c=1$) and 2
(with $c=0.6$).

We now assume that the data give a particular value of $\beta$, say
$\beta=\beta_{i}$. As illustrated in the figures, there are four
possibilities: A is rejected ($\beta_{1}$), B is rejected ($\beta_{3}$), both
models are rejected ($\beta_{2}$), neither model is rejected ($\beta_{0}$ in
Fig. 2). In this last case, Fig. 2, if the data give $\beta=0$, i.e. equal
evidences for the two models, the Bayesian test would be inconclusive, while
here we see that A is rejected but B is not. In terms of $p$-values, we obtain
$p\sim 10^{-7}$ for $\beta_{1}$ assuming A is true; we obtain $p\sim 10^{-6}$
for $\beta_{3}$ assuming B is true; and we obtain $p=0.00063\,\,(0.0017)$ for
$\beta_{2}$ assuming A (respectively B). Notice that the $p$-values are
obtained integrating the distribution in the interval $(-\infty,\beta_{i})$
when assuming A, and in the interval $(\beta_{i},\infty)$ when assuming B.

In Figs. 3 and 4 we illustrate the model comparison test for the same settings
as the previous figures. Clearly, $\beta_{1}$ favors $B$, $\beta_{3}$ favors
$A$, while $\beta_{0},\beta_{2}$ are non-committal. If we had obtained the
value $\beta=0$ for the case of Fig. 3, then the Bayesian model comparison
would have been inconclusive, while in our FB test it would have been quite in
favor of $B$ because essentially incompatible with the data, as can be seen in
Fig. (2).

Plots (1-4) represent therefore a full analysis of the models. The $p$-value
test gives the absolute quality of the fits, while the probability ratio
$R_{f}$ expresses the relative advantage of a model over another one,
regardless of whether they are a good fit or not.

Further, in Fig. 5 we show how the non-central $\chi^{2}$ distribution of App.
C reproduces the numerical distribution in the same case as Fig. 2 but now
with a tight prior ($\sigma_{P}=1$). Since the prior is strong and centered on
model A parameters, the range of $\beta$ that favors model B is now much
reduced with respect to Fig. 2. Finally, in Fig. 6, we modified model B to a
cubic function ($f=a+bx+cx^{2}+dx^{3}$), and the corresponding numerical
distribution was generated as described earlier. In this case, however, the
analytical distribution takes the shape of a generalized $\chi^{2}$ (as
discussed in App. C).

Figure 1:  Distribution of $\beta$ assuming model $A$, a straight line with
two parameters (right) or model $B$, a quadratic polynomial with three
parameters (left), with $c=1$. The prior has been taken to be very broad. The
histograms have been obtained by a numerical simulation, the red and green
curves are the $\chi^{2}$ analytical results. If an experiment finds a value
$\beta_{1}$, $A$ is rejected; if it finds $\beta_{2}$, both $A$ and $B$ are
rejected; if it finds $\beta_{3}$, $B$ is rejected.  Figure 2: Same as the
previous plot, but for two models that are closer ($c=0.6$). If an experiment
finds a value $\beta$ as marked by the dashed vertical line, neither model can
be rejected to a high significance. Figure 3:  For the same settings as in
Fig. 1, here we plot $R_{f}$ (theoretical distribution: red line; numerical
distribution: green line) and $R_{b}$ (blue line) versus $\beta$.  Figure 4:
For the same settings as in Fig. 2, here $R_{f}$ (theoretical distribution:
red line; numerical distribution: green line) and $R_{b}$ (blue line) are
shown versus $\beta$. Figure 5:  Same as Fig. 2, but for a strong prior
$\sigma_{P}=1$. The red curves are the analytical non-central distributions.
Figure 6:  Distribution of $\beta$ with modified model B, a cubic function,
with $c=0.04$ and $d=0.06$. The prior used here is $\sigma_{P}=0.5$. In this
case, the distribution is a generalized $\chi^{2}$.

##  VII Applications: supernovae

In this section, we apply the FB approach to real supernovae data. We have
used supernovae data from the Union2.0 catalogue Amanullah et al. (2010),
which consists of a total of 557 supernovae observations. The models we are
comparing are $\Lambda$CDM and $w$CDM.

The absolute magnitude of a supernova can be related to the luminosity
distance using the equation:

$\mu=5\log_{10}(d_{L})+25=m-M$ (55)

Here, apparent magnitudes ($M$) and absolute magnitudes ($m$) are the
observables for a supernova, and the luminosity distance $d_{L}$ is measured
in Mpc. The luminosity distance $d_{L}$ can be given as (we assume flat space
and adopt units such that $c=1$):

$d_{L}(z)=\frac{(1+z)}{H_{0}}\int_{0}^{z}\frac{dz^{\prime}}{E(z^{\prime})},$
(56)

where $E^{2}(z)=\Omega_{m}(1+z)^{3}+\Omega_{\Lambda}$ for $\Lambda$CDM and
$E^{2}(z)=\Omega_{m}(1+z)^{3}+\Omega_{de}(1+z)^{3(1+w)}$ for $w$CDM and where
$H_{0}$ is the Hubble constant, $w$ the dark energy equation of state, and
$\Omega_{m},\Omega_{\Lambda},\Omega_{de}$ the present matter, cosmological
constant, and dark energy density parameters, respectively.

Now, we can formulate the likelihood Eq. (3) for our analysis as:

$\mathcal{L}=M\exp[-\frac{1}{2}(m_{\rm
data}-5\log_{10}d_{L}^{\ast}+\alpha)_{i}\Sigma_{ij}^{-1}(m_{\rm
data}-5\log_{10}d_{L}^{\ast}+\alpha)_{j}]$ (57)

where $d^{*}_{L}=H_{0}d_{L}$, the indexes $i,j$ run over the data points, and
$\alpha=5\log_{10}H_{0}+25-M$. We can marginalize over $\alpha$ (as shown in
App. B), and the new likelihood can be written as:

$\mathcal{L}=M^{\ast}\exp[-\frac{1}{2}(m_{\rm
data}-5\log_{10}d_{L}^{\ast})_{i}K_{ij}^{-1}(m_{\rm
data}-5\log_{10}d_{L}^{\ast})_{j}]$ (58)

Here, $M^{\ast}$ and $K_{ij}^{-1}$ are, respectively, a new normalization
constant and inverse of the data covariance matrix for our marginalized
likelihood. Bayes’ ratio was computed for the data using this likelihood and
two different priors: a strong prior case, in which the variances were set at
$\sigma_{\Omega_{m}}^{2}=0.005$ and $\sigma_{w}^{2}=0.08$, and a weak prior
case, with $\sigma_{\Omega_{m}}^{2}=0.5$ and $\sigma_{w}^{2}=8$. We obtained a
value of $\beta_{\rm data}=0.79$ for strong priors and $\beta_{\rm data}=4.30$
for weak priors (see Figs. 7 and 8). In standard Bayesian analysis, a
$\beta_{\rm data}$ value of $0.79$ for strong priors would be considered
“barely worth mentioning” evidence in favour of $\Lambda$CDM, while a value of
$4.30$ for weak priors would be deemed “substantial” evidence. Our method
aligns with the standard test in the case of strong priors. However, we
contend that in the case of weak priors, both models should be regarded on a
similar footing, contrary to what the standard test suggests.

For the $\beta$ distribution, first, a model was supposed to be true; let us
say $\Lambda$CDM. Subsequently, a synthetic dataset was created by selecting
random $\mu$ values from a multivariate Gaussian distribution assuming
$\Lambda$CDM. The mean of this distribution was determined using Eq. (56), and
the covariance matrix was taken from the Union2.0 catalogue. From Eq. (58), a
likelihood and, consequently, $\beta$ are computed for this simulated dataset.
This procedure was repeated 100000 times, resulting in a distribution of
$\beta$ values denoted as $P(\beta)$. A similar procedure is then replicated,
assuming the $w$CDM model to be true.

Our analytical approach works for linear models, but the parameters enter the
models $\Lambda$CDM and $w$CDM in a non-linear way. Linearization of our
models can be done by approximating them by first-order Taylor expansion at
the parameter best-fit values Eq. (44),

$d_{L}^{*}(z;\theta_{i})\approx
d_{L}^{*}(z)|_{\theta=\hat{\theta}}+\sum_{i}\frac{\partial
d_{L}^{*}}{\partial\theta_{i}}|_{\theta_{i}=\hat{\theta_{i}}}(\theta_{i}-\hat{\theta_{i}})$
(59)

where $\theta_{i}$ are $(\Omega_{m})$ and $(\Omega_{m},w)$ for $\Lambda$CDM
and $w$CDM, respectively. This is a good approximation: the relative error
between $d_{L,\rm exact}^{\ast}$ (Eq.56) and $d_{L,\rm approx}^{\ast}$ (Eq.59)
for both $\Lambda$CDM and $w$CDM models is less $1\%$ for $z\leq 2.5$.

In this scenario, where we have two linear models with the number of
parameters $N_{\Lambda\rm CDM}=1$ and $N_{w\rm CDM}=2$, the analytical
approximation of the $\beta$ distribution can be represented by a non-central
$\chi^{2}$ distribution with $\nu=N_{w\rm CDM}-N_{\Lambda\rm CDM}=1$ degrees
of freedom and non-centrality parameter $\lambda$, as derived in App. C. This
analytical distribution is shown in Figs. 7 and 8.

The best-fit value of the additional parameter in the $w$CDM model
($\hat{w}=-1.15$) is very close to the value for the $\Lambda$CDM model
($w=-1$), making both distributions very similar to each other. This close
alignment between the distributions makes it challenging to conclusively
comment on the accuracy of either cosmological model at present. However, we
anticipate that future supernovae observations, with reduced observational
errors, will enable a more distinct separation between two distributions. As a
purely illustrative example, if the SN magnitude errors are decreased by a
factor of 20, the new $\beta_{\rm data}=-2.98$ would imply that $w$CDM is more
likely to be the true model of the Universe than $\Lambda$CDM (see Fig. 9).
Finally, Fig. 10 highlights how the value of $\beta$ depends on the prior
width. Relying on Jeffrey’s scale in such cases might result in inaccurate
predictions for the correct cosmological model.

Figure 7:  Distribution of $\beta$ for supernovae Ia. For this simulation, a
strong prior configuration was employed with $\sigma_{\Omega_{m}}^{2}=0.005$
and $\sigma_{w}^{2}=0.08$.  Figure 8: Similar to Fig. 7, but for weak priors:
$\sigma_{\Omega_{m}}^{2}=0.5$ and $\sigma_{w}^{2}=8$. Figure 9:  Similar to
Fig. 7, but now with the supernovae covariance reduced by a factor of 20
(i.e., $C_{ij}/20)$. The two distributions are now quite different, and it
would be possible to clearly differentiate the two competing models.  Figure
10:  Dependence of $\beta$ on different priors. The Gaussian variance of
priors for each $\beta$ is taken as $\sigma_{\Omega_{m}}^{2}=n\times 0.005$
and $\sigma_{w}^{2}=n\times 0.08$, where $n=$ prior broadness.

##  VIII Conclusions

The decisive role of statistics in testing physical models cannot be
overestimated, especially in view of the very abundant data that are expected
to be generated by current and future experiments. Having a number of
alternative and complementary tools at our disposal to compare theoretical
models is therefore a pressing issue. In this paper we argued that some
shortcomings of standard Bayesian analysis can be overcome in the framework of
a mixed frequentist/Bayesian approach, denoted as FB method, in which Bayes’
ratio is employed as a frequentist statistic (see also Keeley and Shafieloo
(2022) for a similar point of view). In particular, we have emphasized the
dependence of Bayes’ ratio on the prior, even in the limit of weak priors, and
have shown that this issue is solved in our mixed approach.

The frequentist distribution of Bayes’ factor can be employed both as a
quality-of-fit test and as model comparison, thereby producing an exhaustive
overview of competing models. We also found the analytical expression of such
distribution in the case of correlated Gaussian data with linear nested models
and arbitrary Gaussian priors, and shown that it provides a good approximation
also in a realistic setting of interest to cosmology. In such case, the FB
method can be carried out without the need of data simulations.

## Acknowledgments

LA thanks Juan Garcia-Bellido, Valerio Marra, Savvas Nesseris, and Björn M.
Schäfer for useful discussions on these topics. ZS and LA acknowledge support
from DFG project 456622116.

##  Appendix A Detailed calculations

Eq. (9):

$\displaystyle\hat{\chi}^{2}$
$\displaystyle=(d^{T}-\hat{\theta}^{T}g)\Sigma^{-1}(d-g^{T}\hat{\theta})$ (60)
$\displaystyle=d^{T}\Sigma^{-1}d+\hat{\theta}^{T}L\hat{\theta}-2d^{T}\Sigma^{-1}g^{T}\hat{\theta}$
(61)
$\displaystyle=d^{T}\Sigma^{-1}d+D^{T}L^{-1}LL^{-1}D-2d^{T}\Sigma^{-1}g^{T}L^{-1}D$
(62)
$\displaystyle=d^{T}\Sigma^{-1}d+d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d-2d^{T}\Sigma^{-1}g^{T}L^{-1}D$
(63)
$\displaystyle=d^{T}\Sigma^{-1}d+d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d-2d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d$
(64)
$\displaystyle=d^{T}\Sigma^{-1}d-d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d$
(65) $\displaystyle=d^{T}\Sigma^{-1}(I-C)d$ (66)

Eq. (25):

$\displaystyle\chi^{2}+\chi_{P}^{2}$
$\displaystyle=(d^{T}-\hat{\theta}^{T}g)\Sigma^{-1}(d-g^{T}\hat{\theta})+(\hat{\theta}^{T}-\tilde{\theta}^{T})LF^{-1}P(\hat{\theta}-\tilde{\theta})$
(67)
$\displaystyle=(d^{T}-d^{T}\Sigma^{-1}g^{T}L^{-1}g)\Sigma^{-1}(d-g^{T}L^{-1}g\Sigma^{-1}d)+(d^{T}\Sigma^{-1}g^{T}L^{-1}-\tilde{\theta}^{T})LF^{-1}P(L^{-1}g\Sigma^{-1}d-\tilde{\theta})$
(68)
$\displaystyle=d^{T}\Sigma^{-1}d+d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d-2d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d$
(69)
$\displaystyle+d^{T}\Sigma^{-1}g^{T}L^{-1}LF^{-1}PL^{-1}g\Sigma^{-1}d-2d^{T}\Sigma^{-1}g^{T}L^{-1}LF^{-1}P\tilde{\theta}+\tilde{\theta}^{T}LF^{-1}P\tilde{\theta}$
(70)
$\displaystyle=d^{T}\Sigma^{-1}d+d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d-2d^{T}\Sigma^{-1}g^{T}L^{-1}g\Sigma^{-1}d+d^{T}\Sigma^{-1}g^{T}F^{-1}PL^{-1}g\Sigma^{-1}d$
(71)
$\displaystyle-2d^{T}\Sigma^{-1}g^{T}F^{-1}P\tilde{\theta}+\tilde{\theta}^{T}LF^{-1}P\tilde{\theta}$
(72)
$\displaystyle=d^{T}[\Sigma^{-1}(I-g^{T}L^{-1}g\Sigma^{-1})]d+d^{T}\Sigma^{-1}g^{T}F^{-1}PL^{-1}g\Sigma^{-1}d-2d^{T}\Sigma^{-1}g^{T}F^{-1}P\tilde{\theta}+\tilde{\theta}^{T}LF^{-1}P\tilde{\theta}$
(73)
$\displaystyle=d^{T}[\Sigma^{-1}(I-g^{T}(L^{-1}-F^{-1}PL^{-1})g\Sigma^{-1})]d-2d^{T}\Sigma^{-1}g^{T}F^{-1}P\tilde{\theta}+\tilde{\theta}^{T}LF^{-1}P\tilde{\theta}$
(74)
$\displaystyle=d^{T}[\Sigma^{-1}(I-g^{T}F^{-1}g\Sigma^{-1})]d-2d^{T}TT^{-1}\Sigma^{-1}g^{T}F^{-1}P\tilde{\theta}+\tilde{\theta}^{T}LF^{-1}P\tilde{\theta}$
(75)
$\displaystyle=d^{T}Td-2d^{T}Tu+u^{T}Tu-u^{T}Tu+\tilde{\theta}^{T}LF^{-1}P\tilde{\theta}$
(76) $\displaystyle=(d-u)^{T}T(d-u)-\tilde{\theta}^{T}(W^{T}TW-
LF^{-1}P)\tilde{\theta}$ (77) $\displaystyle=\Xi+\Pi$

where

$\displaystyle\Xi$ $\displaystyle\equiv(d_{m}-u_{m})T_{mk}(d_{k}-u_{k})$ (78)
$\displaystyle\Pi$
$\displaystyle\equiv-\tilde{\theta}_{\gamma}(W_{m\gamma}T_{mk}W_{k\alpha}-L_{\gamma\beta}F_{\beta\delta}^{-1}P_{\delta\alpha})\tilde{\theta}_{\alpha}$
(79)

However, it turns out that $\Pi=0$:

$\displaystyle\Pi$
$\displaystyle=-\tilde{\theta}^{T}(gTg^{T}-LF^{-1}P)\tilde{\theta}$ (80)
$\displaystyle=-\tilde{\theta}^{T}(g(\Sigma^{-1}(I-g^{T}F^{-1}g\Sigma^{-1}))g^{T}-LF^{-1}P)\tilde{\theta}$
(81)
$\displaystyle=-\tilde{\theta}^{T}(g\Sigma^{-1}g^{T}-g\Sigma^{-1}g^{T}F^{-1}g\Sigma^{-1}g^{T}-LF^{-1}P)\tilde{\theta}$
(82) $\displaystyle=-\tilde{\theta}^{T}(L-LF^{-1}L-LF^{-1}P)\tilde{\theta}$
(83) $\displaystyle=-\tilde{\theta}^{T}(L-LF^{-1}F)\tilde{\theta}=0$ (84)

Eq. (27):

$\displaystyle T$
$\displaystyle=\Sigma^{-1}(I-g^{T}(L^{-1}-F^{-1}PL^{-1})g\Sigma^{-1})$ (85)
$\displaystyle=\Sigma^{-1}(I-g^{T}F^{-1}(F-P)L^{-1}g\Sigma^{-1})$ (86)
$\displaystyle=\Sigma^{-1}(I-g^{T}F^{-1}LL^{-1}g\Sigma^{-1})=\Sigma^{-1}(I-g^{T}F^{-1}g\Sigma^{-1})$
(87)

Eq. (29):

$\displaystyle
T^{-1}T=(\Sigma+g^{T}P^{-1}g)\Sigma^{-1}(I-g^{T}F^{-1}g\Sigma^{-1})$
$\displaystyle=$ (88) $\displaystyle
I-g^{T}F^{-1}g\Sigma^{-1}+g^{T}P^{-1}g\Sigma^{-1}(I-g^{T}F^{-1}g\Sigma^{-1})$
$\displaystyle=$ (89) $\displaystyle
I-g^{T}F^{-1}g\Sigma^{-1}+g^{T}P^{-1}g\Sigma^{-1}-g^{T}P^{-1}g\Sigma^{-1}g^{T}F^{-1}g\Sigma^{-1}$
$\displaystyle=$ (90) $\displaystyle
I-g^{T}(F^{-1}-P^{-1}+P^{-1}LF^{-1})g\Sigma^{-1}$ $\displaystyle=$ (91)
$\displaystyle I-g^{T}(F^{-1}-P^{-1}+P^{-1}(F-P)F^{-1})g\Sigma^{-1}$
$\displaystyle=$ (92) $\displaystyle
I-g^{T}(F^{-1}-P^{-1}+P^{-1}FF^{-1}-F^{-1})g\Sigma^{-1}$ $\displaystyle=$ (93)
$\displaystyle I-g^{T}(-P^{-1}+P^{-1})g\Sigma^{-1}$ $\displaystyle=I$ (94)

$\displaystyle u$
$\displaystyle=T^{-1}\Sigma^{-1}g^{T}F^{-1}P\tilde{\theta}=(\Sigma+g^{T}P^{-1}g)\Sigma^{-1}g^{T}F^{-1}P\tilde{\theta}$
(95)
$\displaystyle=(g^{T}F^{-1}P+g^{T}P^{-1}g\Sigma^{-1}g^{T}F^{-1}P)\tilde{\theta}$
(96) $\displaystyle=(g^{T}F^{-1}P+g^{T}P^{-1}LF^{-1}P)\tilde{\theta}$ (97)
$\displaystyle=g^{T}(F^{-1}+P^{-1}LF^{-1})P\tilde{\theta}$ (98)
$\displaystyle=g^{T}(F^{-1}+P^{-1}(F-P)F^{-1})P\tilde{\theta}=g^{T}\tilde{\theta}$
(99)

##  Appendix B Marginalization of an overall additive constant

We need to perform this integration within $(-\infty,\infty)$

$\displaystyle\int
dA\exp-\frac{1}{2}(d_{i}-f_{i}+Au_{i})C_{ij}^{-1}(d_{j}-f_{j}+Au_{j})$
$\displaystyle=$ (100) $\displaystyle\int
dA\exp-\frac{1}{2}[(d_{i}-f_{i})C_{ij}^{-1}(d_{j}-f_{j})+2Au_{i}C_{ij}^{-1}(d_{j}-f_{j})+A^{2}S_{0}]$
$\displaystyle=$ (101)
$\displaystyle\exp-\frac{1}{2}(d_{i}-f_{i})C_{ij}^{-1}(d_{j}-f_{j})\int
dA\exp-\frac{1}{2}[2AS_{1}+A^{2}S_{0}]$ $\displaystyle=$ (102)
$\displaystyle\exp-\frac{1}{2}(d_{i}-f_{i})C_{ij}^{-1}(d_{j}-f_{j})\exp\frac{1}{2}S_{1}^{2}S_{0}^{-1}\int
dA\exp-\frac{1}{2}[S_{1}^{2}S_{0}^{-1}+2AS_{1}+A^{2}S_{0}]$ $\displaystyle=$
(103)
$\displaystyle\exp-\frac{1}{2}[(d_{i}-f_{i})C_{ij}^{-1}(d_{j}-f_{j})-S_{1}^{2}S_{0}^{-1}]\int
dA\exp-\frac{1}{2}[S_{1}S_{0}^{-1/2}+AS_{0}^{1/2}]^{2}$ $\displaystyle=$ (104)
$\displaystyle(2\pi)^{1/2}S_{0}^{-1/2}\exp-\frac{1}{2}[(d_{i}-f_{i})C_{ij}^{-1}(d_{j}-f_{j})-S_{1}^{2}S_{0}^{-1}]$
$\displaystyle=$ (105) $\displaystyle
M\exp-\frac{1}{2}[S_{2}-\frac{S_{1}^{2}}{S_{0}}]$ (106)

where $M$ is an unimportant normalization factor, $u_{i}=\\{1,1,1...\\}$ and

$\displaystyle S_{0}$
$\displaystyle=u_{i}C_{ij}^{-1}u_{j}=\sum_{ij}C_{ij}^{-1}$ (107)
$\displaystyle S_{1}$ $\displaystyle=u_{i}C_{ij}^{-1}(d_{j}-f_{j})$ (108)
$\displaystyle S_{2}$ $\displaystyle=(d_{i}-f_{i})C_{ij}^{-1}(d_{j}-f_{j})$
(109)

This can be written as a new Gaussian

$\displaystyle M\exp-\frac{1}{2}[S_{2}-\frac{S_{1}^{2}}{S_{0}}]$
$\displaystyle=$ (110)
$\displaystyle=M\exp-\frac{1}{2S_{0}}[(d_{i}-f_{i})S_{0}C_{ij}^{-1}(d_{j}-f_{j})-(d_{j}-f_{j})u_{i}C_{ij}^{-1}u_{n}C_{nm}^{-1}(d_{m}-f_{m})]$
$\displaystyle=$ (111) $\displaystyle
M\exp-\frac{1}{2S_{0}}[(d_{i}-f_{i})S_{0}C_{ij}^{-1}(d_{j}-f_{j})-(d_{i}-f_{i})u_{p}C_{pi}^{-1}u_{n}C_{nj}^{-1}(d_{j}-f_{j})]$
$\displaystyle=$ (112) $\displaystyle
M\exp-\frac{1}{2S_{0}}[(d_{i}-f_{i})[S_{0}C_{ij}^{-1}-u_{p}C_{pi}^{-1}u_{n}C_{nj}^{-1}](d_{j}-f_{j})]$
$\displaystyle=$ (113) $\displaystyle
M\exp-\frac{1}{2}(d_{i}-f_{i})K_{ij}^{-1}(d_{j}-f_{j})$ (114)

where the new inverse of the covariance function is

$K_{ij}^{-1}\equiv C_{ij}^{-1}-\frac{u_{p}C_{pi}^{-1}u_{n}C_{nj}^{-1}}{S_{0}}$
(115)

Note that the matrix $K^{-1}$ is singular, so one cannot invert it to obtain
$K$. This causes no problem, however, since we only need $K^{-1}$.

Now the Fisher matrix becomes

$F_{\alpha\beta}=g_{\alpha i}K^{-1}_{ij}g_{\beta j}=\frac{\partial
f_{i}}{\partial\theta_{\alpha}}K^{-1}_{ij}\frac{\partial
f_{j}}{\partial\theta_{\beta}}$ (116)

where $\theta_{\alpha}$ is the parameter vector. The parameter distribution is
then approximated by a Gaussian centered on the best fit
$\hat{\theta}_{\alpha}$ and with covariance matrix $F^{-1}$.

##  Appendix C General distribution

The distribution of a quadratic form

$Q=(d_{m}-u_{m})T_{mk}(d_{k}-u_{k})=X^{T}TX$ (117)

can be obtained as follows (see IMHOF (1961); Duchesne and Lafaye De Micheaux
(2010)). The mean of $X$ is $\mu=\langle
X\rangle=g^{T}(\hat{\theta}-\tilde{\theta})=f-u$. The covariance matrix of $X$
is

$\displaystyle\Sigma$ $\displaystyle=\langle(d-u-\mu)(d-u-\mu)^{T}\rangle$
(118) $\displaystyle=\langle(d-f)(d-f)^{T}\rangle$ (119)

Let now $C$ be a matrix such that (Cholesky decomposition)

$CC^{T}=\Sigma$ (120)

and $P$ an orthonormal matrix (i.e., $PP^{T}=I)$ that diagonalizes $CTC^{T}$,
i.e.

$PCTC^{T}P^{T}=D=\mathrm{diag}(\alpha_{1},\alpha_{2}..\alpha_{p})$ (121)

where the eigenvalues of $CTC^{T}$ $\alpha_{i}$ are non-zero for $i=1...r$ and
zero for $i=r,..,p$. The number of non-zero eigenvalues should be
$r=N_{B}-N_{A}$. Let also define the two vectors $Y=P(C^{T})^{-1}X$ and
$\xi=P(C^{T})^{-1}\mu$. Then one can see that $Y$ is a vector of Gaussian
variables with mean vector $\xi$ and covariance matrix $\Sigma_{Y}=I$.
Therefore $Q$ can be written as a linear combination of non-central,
uncorrelated $\chi^{2}$ variables,

$Q=X^{T}TX=\sum_{i}^{r}\alpha_{i}Y_{i}^{2}=\sum_{i}^{r}\alpha_{i}\chi_{1}^{2}(\lambda_{i})$
(122)

where $\chi_{1}^{2}(\lambda)$ are non-central $\chi^{2}$ variables with one
degrees of freedom and non-central parameter

$\lambda_{i}=\xi_{i}^{2}$ (123)

Then, in general, $Q$ is distributed as a generalized $\chi^{2}$ variable. The
characteristic function (CF) for the generalized $\chi^{2}$ variable can be
obtained analytically IMHOF (1961). The CF for eq. 122 can be expressed as:

$\phi(t)=\frac{\exp\left\\{it\sum_{i}^{r}\frac{\lambda_{i}\alpha_{i}}{1-2i\alpha_{i}t}\right\\}}{\prod_{i}^{r}(1-2i\alpha_{i}t)^{\frac{1}{2}}}$
(124)

However, one cannot derive analytically neither the probability density
function (PDF) nor the cumulative distribution function. One can numerically
determine the PDF performing an inverse Fourier transform or utilizing
existing codes (e.g. Duchesne and Lafaye De Micheaux (2010)). If the
eigenvalues $\alpha_{i}$ are all unit, $Q$ is a non-central variable with
$\lambda=\xi^{T}\xi$ and $r$ dofs. Finally, if the eigenvalues are unity and
$\lambda=0$, $Q$ is a standard $\chi^{2}$ variable with $r$ dofs. We have
shown in the main text that in the limit of weak prior, the eigenvalues are
indeed unity.

In the case $r=1$, $Q$ is a single non-central variable with one dof and

$\lambda=\xi^{2}$ (125)

In this case, the distribution of $\beta=\alpha\chi_{1}^{2}(\lambda)+a$ is
obtained from the non-central distribution $P(\chi_{1}^{2}(\lambda))$ as

$P_{\beta}(\beta)=P(\chi_{1}^{2}(\lambda))|\frac{d\chi_{1}^{2}(\lambda)}{d\beta}|=\frac{1}{|\alpha|}P(\frac{\beta-a}{\alpha})$
(126)

##  Appendix D A simple example

Fig. 11 shows an extremely simple example of how misleading weak priors can
be. Here, model A is a horizontal straight line $y=a$ with just $a$ as free
parameter, while model B is a general straight line $y=a+bx$ with two free
parameters. The data points have been created by sampling a Gaussian
distribution around the line $y=1+0.6x$, with uncertainty $\sqrt{1/2}$. The
blue dashed line is the best fit for model A, while the continuous orange line
is the best fit for B. The Gaussian prior standard deviation for the common
parameter $a$ is 10, while for the extra parameter $b$ is huge, $10^{6}$.
Clearly, model B appears strongly favoured by this data set. However, the very
weak prior shifts $\beta$ to roughly 9.8, which corresponds to a Bayes’ ratio
$\approx 130$, well into the ”decisive” range in favor of model A of Jeffrey’s
scale. So a pure Bayesian analysis of this simple case would produce the
puzzling result that model A is to be much preferred over B. Bayes’ factor
becomes smaller than unity (i.e., model B is favoured) only if the prior
uncertainty on $b$ is smaller than 7000.

Figure 11:  This plot appear to show that model B is a better model than A in
explaining the data. However, if the extra B parameter has an extremely weak
prior, Bayes’ factor $b$ would strongly favor model A.

## References

  * Dinda et al. [2023] Bikash R. Dinda, Haveesh Singirikonda, and Subhabrata Majumdar.  Constraints on cosmic curvature from cosmic chronometer and quasar observations.  _arXiv e-prints_ , art. arXiv:2303.15401, March 2023.  doi: 10.48550/arXiv.2303.15401.
  * Reeves et al. [2023] Alexander Reeves, Laura Herold, Sunny Vagnozzi, Blake D. Sherwin, and Elisa G. M. Ferreira.  Restoring cosmological concordance with early dark energy and massive neutrinos?  _Mon. Not. Roy. Astron. Soc._ , 520(3):3688–3695, 2023.  doi: 10.1093/mnras/stad317.
  * Alonso et al. [2023] David Alonso, Giulio Fabbian, Kate Storey-Fisher, Anna-Christina Eilers, Carlos García-García, David W. Hogg, and Hans-Walter Rix.  Constraining cosmology with the Gaia-unWISE Quasar Catalog and CMB lensing: structure growth.  _JCAP_ , 11:043, 2023.  doi: 10.1088/1475-7516/2023/11/043.
  * McDonough et al. [2023] Evan McDonough, J. Colin Hill, Mikhail M. Ivanov, Adrien La Posta, and Michael W. Toomey.  Observational constraints on early dark energy.  _arXiv e-prints_ , art. arXiv:2310.19899, October 2023.  doi: 10.48550/arXiv.2310.19899.
  * Kou et al. [2023] Raphaël Kou, Calum Murray, and James G. Bartlett.  Constraining f(R) gravity with cross-correlation of galaxies and cosmic microwave background lensing.  _arXiv e-prints_ , art. arXiv:2311.09936, November 2023.  doi: 10.48550/arXiv.2311.09936.
  * Yang et al. [2023] Jing Yang, Xin-Yan Fan, Chao-Jun Feng, and Xiang-Hua Zhai.  Latest Data Constraint of Some Parameterized Dark Energy Models.  _Chin. Phys. Lett._ , 40(1):019801, 2023.  doi: 10.1088/0256-307X/40/1/019801.
  * Holm et al. [2023] Emil Brinch Holm, Laura Herold, Théo Simon, Elisa G. M. Ferreira, Steen Hannestad, Vivian Poulin, and Thomas Tram.  Bayesian and frequentist investigation of prior effects in EFT of LSS analyses of full-shape BOSS and eBOSS data.  _Phys. Rev. D_ , 108(12):123514, 2023.  doi: 10.1103/PhysRevD.108.123514.
  * Sakr et al. [2023] Ziad Sakr, Ana Carvalho, Antonio Da Silva, Juan Garcia Bellido, Jose P. Mimoso, David Camarena, Savvas Nesseris, Carlos J. A. P. Martins, Nelson J. Nunes, and Domenico Sapone.  Constraining LLTB models with galaxy cluster counts from next generation surveys.  _arXiv e-prints_ , art. arXiv:2309.17151, September 2023.  doi: 10.48550/arXiv.2309.17151.
  * Colgáin et al. [2023] Eoin Ó. Colgáin, Saeed Pourojaghi, M. M. Sheikh-Jabbari, and Darragh Sherwin.  MCMC Marginalisation Bias and $\Lambda$CDM tensions.  7 2023.
  * Sakr et al. [2022] Ziad Sakr, Stephane Ilic, and Alain Blanchard.  Cluster counts - III. $\Lambda$CDM extensions and the cluster tension.  _Astron. Astrophys._ , 666:A34, 2022.  doi: 10.1051/0004-6361/202142115.
  * Serra et al. [2007] P. Serra, A. Heavens, and A. Melchiorri.  Bayesian evidence for a cosmological constant using new high-redshift supernova data.  _Monthly Notices of the Royal Astronomical Society_ , 379(1):169–175, July 2007.  ISSN 1365-2966.  doi: 10.1111/j.1365-2966.2007.11924.x.  URL http://dx.doi.org/10.1111/j.1365-2966.2007.11924.x.
  * Kurek and Szydłowski [2008] Aleksandra Kurek and Marek Szydłowski.  The $\Lambda$CDM Model in the Lead—A Bayesian Cosmological Model Comparison.  _Astrophys. J._ , 675(1):1–7, March 2008.  doi: 10.1086/526333.
  * John and Narlikar [2002] Moncy V. John and J. V. Narlikar.  Comparison of cosmological models using Bayesian theory.  _Phys. Rev. D_ , 65(4):043506, February 2002.  doi: 10.1103/PhysRevD.65.043506.
  * Shafer [2015] Daniel L. Shafer.  Robust model comparison disfavors power law cosmology.  _Phys. Rev. D_ , 91(10):103516, 2015.  doi: 10.1103/PhysRevD.91.103516.
  * Heavens et al. [2017] Alan Heavens, Yabebal Fantaye, Elena Sellentin, Hans Eggers, Zafiirah Hosenie, Steve Kroon, and Arrykrishna Mootoovaloo.  No evidence for extensions to the standard cosmological model.  _Phys. Rev. Lett._ , 119(10):101301, 2017.  doi: 10.1103/PhysRevLett.119.101301.
  * Jeffreys [1961] H. Jeffreys.  _Theory of probability_.  Oxford Classics series (reprinted 1998). Oxford University Press, Oxford, UK, 3rd edn edition, 1961.
  * Robert et al. [2008] Christian P. Robert, Nicolas Chopin, and Judith Rousseau.  Harold Jeffreys’s Theory of Probability Revisited.  _arXiv e-prints_ , art. arXiv:0804.3173, April 2008.  doi: 10.48550/arXiv.0804.3173.
  * Kass and Raftery [1995] Robert E. Kass and Adrian E. Raftery.  Bayes Factors.  _J. Am. Statist. Assoc._ , 90(430):773–795, 1995.  doi: 10.1080/01621459.1995.10476572.
  * Efstathiou [2008] G. Efstathiou.  Limitations of Bayesian Evidence Applied to Cosmology.  _Mon. Not. Roy. Astron. Soc._ , 388:1314, 2008.  doi: 10.1111/j.1365-2966.2008.13498.x.
  * Nesseris and Garcia-Bellido [2013] Savvas Nesseris and Juan Garcia-Bellido.  Is the Jeffreys’ scale a reliable tool for Bayesian model comparison in cosmology?  _JCAP_ , 08:036, 2013.  doi: 10.1088/1475-7516/2013/08/036.
  * Jenkins and Peacock [2011] C. R. Jenkins and J. A. Peacock.  The power of Bayesian evidence in astronomy.  _Mon. Not. Roy. Astron. Soc._ , 413:2895, 2011.  doi: 10.1111/j.1365-2966.2011.18361.x.
  * Rezaei and Malekjani [2021] Mehdi Rezaei and Mohammad Malekjani.  Comparison between different methods of model selection in cosmology.  _Eur. Phys. J. Plus_ , 136(2):219, 2021.  doi: 10.1140/epjp/s13360-021-01200-w.
  * Hoeting et al. [1999] Jennifer A. Hoeting, David Madigan, Adrian E. Raftery, and Chris T. Volinsky.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors.  _Statistical Science_ , 14(4):382 – 417, 1999.  doi: 10.1214/ss/1009212519.  URL https://doi.org/10.1214/ss/1009212519.
  * Parkinson and Liddle [2013] David Parkinson and Andrew R. Liddle.  Bayesian Model Averaging in Astrophysics: A Review.  _Statist. Anal. Data Mining_ , 6:3–14, 2013.  doi: 10.1002/sam.11179.
  * Kunz et al. [2006] Martin Kunz, Roberto Trotta, and David Parkinson.  Measuring the effective complexity of cosmological models.  _Phys. Rev._ , D74:023503, 2006.  doi: 10.1103/PhysRevD.74.023503.
  * Raveri and Hu [2019] Marco Raveri and Wayne Hu.  Concordance and Discordance in Cosmology.  _Phys. Rev. D_ , 99(4):043506, 2019.  doi: 10.1103/PhysRevD.99.043506.
  * Heavens et al. [2023] Alan F. Heavens, Arrykrishna Mootoovaloo, Roberto Trotta, and Elena Sellentin.  Extreme data compression for Bayesian model comparison.  _jcap_ , 2023(11):048, November 2023.  doi: 10.1088/1475-7516/2023/11/048.
  * Koo et al. [2022] Hanwool Koo, Ryan E. Keeley, Arman Shafieloo, and Benjamin L’Huillier.  Bayesian vs frequentist: comparing Bayesian model selection with a frequentist approach using the iterative smoothing method.  _JCAP_ , 03(03):047, 2022.  doi: 10.1088/1475-7516/2022/03/047.
  * Good [1957] I. J. Good.  Saddle-point Methods for the Multinomial Distribution.  _The Annals of Mathematical Statistics_ , 28(4):861 – 881, 1957.  doi: 10.1214/aoms/1177706790.  URL https://doi.org/10.1214/aoms/1177706790.
  * Good [1992] I. J. Good.  The bayes/non-bayes compromise: A brief review.  _Journal of the American Statistical Association_ , 87(419):597–606, 1992.  ISSN 01621459.  URL http://www.jstor.org/stable/2290192.
  * Amendola et al. [2013] Luca Amendola, Valerio Marra, and Miguel Quartin.  Internal Robustness: systematic search for systematic bias in SN Ia data.  _Mon.Not.Roy.Astron.Soc._ , 430:1867–1879, 2013.  doi: 10.1093/mnras/stt008.
  * Wolz [2018] Kevin Wolz.  Bayesian and frequentist methods to investigate tensions between data sets.  Msc thesis, University of Heidelberg, Heidelberg, Germany, October 2018.
  * Wolz et al. [2024] Kevin Wolz, Luca Amendola, Elena Sellentin, et al.  in prep.  2024\.
  * Keeley and Shafieloo [2022] Ryan E. Keeley and Arman Shafieloo.  On the distribution of Bayesian evidence.  _Mon. Not. Roy. Astron. Soc._ , 515(1):293–301, 2022.  doi: 10.1093/mnras/stac1851.
  * Heavens and Sellentin [2018] Alan F. Heavens and Elena Sellentin.  Objective Bayesian analysis of neutrino masses and hierarchy.  _jcap_ , 2018(4):047, April 2018.  doi: 10.1088/1475-7516/2018/04/047.
  * Amanullah et al. [2010] Amanullah et al.  Spectra and Hubble Space Telescope Light Curves of Six Type Ia Supernovae at 0.511 ¡ z ¡ 1.12 and the Union2 Compilation.  _Ap.J._ , 716:712–738, June 2010.  doi: 10.1088/0004-637X/716/1/712.
  * IMHOF [1961] J. P. IMHOF.  Computing the distribution of quadratic forms in normal variables.  _Biometrika_ , 48(3-4):419–426, 12 1961.  ISSN 0006-3444.  doi: 10.1093/biomet/48.3-4.419.  URL https://doi.org/10.1093/biomet/48.3-4.419.
  * Duchesne and Lafaye De Micheaux [2010] Pierre Duchesne and Pierre Lafaye De Micheaux.  Computing the distribution of quadratic forms: Further comparisons between the liu-tang-zhang approximation and exact methods.  _Computational Statistics & Data Analysis_, 54(4):858–862, 2010.  ISSN 0167-9473.  doi: https://doi.org/10.1016/j.csda.2009.11.025.  URL https://www.sciencedirect.com/science/article/pii/S0167947309004381.


# Improving Question Answering with Generation of NQ-like Questions

Saptarashmi Bandyopadhyay
University of Maryland, College Park
<EMAIL_ADDRESS>Shraman Pal
IIT Kharagpur
<EMAIL_ADDRESS>Hao Zou
University of Minnesota
<EMAIL_ADDRESS>Abhranil Chandra
IIT Kharagpur
<EMAIL_ADDRESS>Jordan Boyd-Graber
University of Maryland, College Park
<EMAIL_ADDRESS>
###### Abstract

Question Answering (QA) systems require a large amount of annotated data which
is costly and time-consuming to gather. Converting datasets of existing QA
benchmarks are challenging due to different formats and complexities. To
address these issues, we propose an algorithm to automatically generate
shorter questions resembling day-to-day human communication in the Natural
Questions (NQ) dataset from longer trivia questions in Quizbowl (QB) dataset
by leveraging conversion in style among the datasets. This provides an
automated way to generate more data for our QA systems. To ensure quality as
well as quantity of data, we detect and remove ill-formed questions using a
neural classifier. We demonstrate that in a low resource setting, using the
generated data improves the QA performance over the baseline system on both NQ
and QB data. Our algorithm improves the scalability of training data while
maintaining quality of data for QA systems.

##  1 Introduction

Large-scale data collection is a challenging process in the domain of Question
Answering and Information Retrieval due to the necessity of high-quality
annotations which are scarce and expensive to generate. There are several QA
datasets Joshi et al. (2017), Rajpurkar et al. (2016), Yang et al. (2018),
Kwiatkowski et al. (2019), Rodriguez et al. (2021) with significantly
different structure and complexity. Large quantities of high quality data are
effective in training a more efficacious Machine Learning system.

In this paper, we focus on converting questions normally spoken in a trivia
competition to questions resembling day-to-day human communication. Trivia
questions have multiple lines, consist of multiple hints as standalone
sentences to an answer, and players can buzz on any sentence in the question
to give an answer. In contrast, questions used in daily human communication
are shorter (often a single line). We propose an algorithm to generate
multiple short natural questions from every long trivia question by converting
each of the sentences with multiple hints to several shorter questions. We
also add a BERT Devlin et al. (2019) based quality control method to filter
out the ill-formed questions and retain the well-formed questions. We show
that our algorithm for generating questions improves the performance of two
question answering (QA) systems in a low resource setting. We also demonstrate
that concatenation of original natural questions with generated questions
improves the QA system performance. Finally, we prove that by using such a
method to generate synthetic data, we can achieve higher scores than a system
that uses only NQ data.

##  2 Dataset and Data Extraction

We use two popular datasets- the Quizbowl Rodriguez et al. (2021), henceforth
referred to as QB dataset, and NQ-Open dataset Lee et al. (2019), derived from
Natural Questions Kwiatkowski et al. (2019). QB has a total of 119247
question/answer samples and NQ has 91434 total question/answer samples. For
the NQ dataset, we use the same 1800 dev and 1769 test question/answer splits
as used in the EfficientQA Competition Min et al. (2021).

As our task involves transforming QB questions to NQ-like questions, we
extract pairs of questions that are semantically similar. We first extract
every possible question-question pair with the same answer using string
matching resulting in 95651 question-question pairs. From this parallel
corpus, we extract the last sentence of the QB questions and pass them through
a pre-trained Sentence-BERT Reimers and Gurevych (2019) model along with the
corresponding NQ question. We take the cosine similarity between the [CLS]
embedding to find pairs that are semantically equivalent by setting the
threshold to 0.5. From this we extract 19439 question-question pairs having
moderate semantic equivalence. We also use the same index from the last
sentence of QB questions paired with NQ questions to retrieve corresponding QB
full questions in paragraph form. Out of the extracted corpora, we create a
smaller dataset of the last sentence of every QB question (and corresponding
QB full paragraph) paired with NQ questions for our low resource setting. This
paired dataset has a total of 1218 training samples, 93 validation samples and
563 test samples which are semantically similar. We outline the statistics of
the baseline and generated datasets in Table 1.

##  3 Methods to generate NQ-like Questions

We use the following NLP techniques to generate NQ-like questions from our QB
dataset

  * •

Tokenization

  * •

Coreference resolution

  * •

Parse tree output

  * •

Bag of words based question generation

We outline our methods in Algorithm 1. At first, we tokenize the long
paragraph of a QB question into individual sentences with hints to an answer
providing sufficient context. Using every sentence annotated by the question
id was not sufficient due to annotation errors of sentence delimiters like a
‘.’.

  * •

Initial: … and "k." For 10 points, …

  * •

Tokenized:

    * –

… and " k. "

    * –

For 10 points , …

However, each sentence contains 2-3 clues about the answer whereas NQ
questions generally are in 1 sentence with 1 clue. To resolve this issue, we
use Coreference Resolution on every sentence to obtain clusters of nouns and
pronouns referring to those nouns Kirstain et al. (2021), and Parse Trees.

We then split the sentences to get parts of a sentence with similar syntactic
structure using Parse Tree based on ADVCL (adverbial clause) and CC (conj)
tags. Any pronoun that is present in a split is replaced with the noun from
the clusters found using Coreference Resolution. After breaking into splits,
we run a check on the number of words in every split and append sentences with
smaller number of words (less than 8) back into the original sentence. We
finally clean the split sentences of trailing punctuation marks and words that
should not be at the sentence ending like ‘and’, ‘but’ etc. This gives us
sentences or phrases that usually have one clue similar to NQ.

To make the outputs question-like, we use a bag of words approach by replacing
words like ‘this’ with ‘which’, ‘it’ with ‘what’ for all but the last
sentence. The last sentence in QB has a specific syntactic structure where the
sentence contains, ‘For x points , name this’ followed by the identifier to
the answer. We replace this whole phrase according to whatever comes after.
For example,

  * •

‘name this author’ : ‘who is the author’

  * •

‘name this 1985 event’ : ‘which is the 1985 event’

  * •

‘name this phenomenon’ : ‘what is the phenomenon’

Algorithm 1 NQ-like Question Generation

1:procedure NQlikegen($sentences$)

2: $Individual(sentences)$ $\triangleright$ split full QB question into single
sentences

3: $Pre\\_processing(sentences)$ $\triangleright$ remove punctuations,
conjunctions or any other word not expected at the end of the sentence.

4: $Coreference\\_clusters(sentences)$ $\triangleright$ forms clusters like
[“this country”, “its”]

5: $Parse\\_Tree(sentences)$ $\triangleright$ Split the conjoined clues based
on similar syntactic structure

6: if $Noun,Pronoun$ not in $same\\_parse\\_split$ then

7: $Replace$ $\triangleright$ replace the pronoun in the other split with the
NP

8: for $QB\\_last\\_sentences$ do

9: $String\\_Replacement(last\\_sentence)$ $\triangleright$ Replace ‘name
this’ with the annotated ‘wh’ question by NOUNS vocabulary formed

10: for $QB\\_non\\_last\\_sentences$ do

11: $Bag\\_of\\_Words$ $\triangleright$ use frequency based vocabulary table
to replace “this” to “which”

12: return $nq\\_like\\_questions$

To achieve this, we extract noun phrases following ‘name this’ from 1000
samples and form a vocabulary to manually annotate them with ‘who is the’,
‘which is the’, ‘what is the’, and their plural counterparts. This gives us
questions that resemble NQ questions.

System Description | Data Size | Mean | Median | Mode
---|---|---|---|---
QB system with full questions | 1874 | 4.39 | 4.0 | 4.0
Generated NQ-like full questions | 11062 | 5.9 | 6.0 | 6.0
Filtered generated NQ-like full questions | 2075 | 1.7 | 1.0 | 1.0
QB system with full questions | 119247 | 6.2 | 6.0 | 6.0
Generated NQ-like full questions | 772456 | 6.5 | 6.0 | 6.0
Table 1: Statistics of number of sentences per Quizbowl question Figure 1: Generation of NQ-like Questions from the original QB Questions along with answers of the QA systems (Gold Answer: pennsylvania) System | Total samples | Accuracy | Precision | Recall | F1 | SacreBLEU
---|---|---|---|---|---|---
NQ | 1874 | 0.490 | 0.550 | 0.540 | 0.540 | 29.57
NQ + NQ-like from QB full | 9140 | 0.390 | 0.420 | 0.410 | 0.420 | 27.95
NQ + Quality controlled NQ-like from QB full | 3319 | 0.467 | 0.505 | 0.503 | 0.502 | 34.03
NQ + QB full | 3092 | 0.465 | 0.503 | 0.502 | 0.499 | 27.62
NQ + QB last sentence | 3092 | 0.516 | 0.558 | 0.560 | 0.556 | 37.37
NQ + NQ-like from QB last sentence | 3259 | 0.500 |  0.557 |  0.556 |  0.553 | 34.98
NQ + Quality controlled NQ-like from QB last sentence | 2967 | 0.518 | 0.567 | 0.564 | 0.562 | 37.46
QB full | 1874 | 0.232 | 0.261 | 0.259 | 0.259 | 12.17
NQ-like from QB full | 11062 | 0.171 | 0.188 | 0.188 | 0.188 | 12.45
Quality controlled NQ-like from QB full | 2075 | 0.176 | 0.192 | 0.191 | 0.191 | 14.11
QB last sentence | 1874 | 0.44 | 0.495 | 0.488 | 0.488 | 22.18
NQ-like from QB last sentence | 2112 | 0.421 | 0.454 | 0.456 | 0.453 | 34.49
Quality controlled NQ-like from QB last sentence | 1632 | 0.441 | 0.495 | 0.488 | 0.488 | 38.25
Table 2: Question Answering Metric values with RAG system System | Total samples | Accuracy | Precision | Recall | F1 | SacreBLEU
---|---|---|---|---|---|---
NQ | 1874 | 0.211 | 0.243 | 0.242 | 0.241 | 8.74
NQ + NQ-like from QB full | 9140 | 0.243 | 0.271 | 0.270 | 0.269 | 10.49
NQ + Quality controlled NQ-like from QB full | 3319 | 0.227 | 0.256 | 0.254 | 0.254 | 11.01
NQ + QB full | 3092 | 0.229 | 0.257 | 0.258 | 0.256 | 10.97
NQ + QB last sentence | 3092 | 0.257 | 0.291 | 0.295 | 0.290 | 9.65
NQ + NQ-like from QB last sentence | 3259 | 0.253 |  0.281 |  0.276 |  0.277 | 10.20
NQ + Quality controlled NQ-like from QB last sentence | 2967 | 0.234 | 0.268 | 0.267 | 0.266 | 8.91
QB full | 1874 | 0.191 | 0.251 | 0.228 | 0.233 | 7.17
NQ-like from QB full | 11062 | 0.027 | 0.085 | 0.085 | 0.085 | 2.23
Quality controlled NQ-like from QB full | 2075 | 0.197 | 0.224 | 0.218 | 0.219 | 8.71
QB last sentence | 1874 | 0.328 | 0.394 | 0.384 | 0.383 | 9.66
NQ-like from QB last sentence | 2112 | 0.310 | 0.363 | 0.358 | 0.357 | 25.40
Quality controlled NQ-like from QB last sentence | 1632 | 0.367 | 0.428 | 0.429 | 0.424 | 23.16
Table 3: Question Answering Metric values with DrQA system

###  3.1 Quality Control

We trained a BERT based Devlin et al. (2019) classifier using the pretrained
bert-base-uncased model for detecting well-formed questions and improving the
data quality. We use the dataset from Faruqui and Das (2018) which has
questions annotated with scores from 0 to 1 signifying how well the questions
are structured. With the help of this classifier, we generate the scores for
our training set (7266 samples) of generated NQ like questions. We only take
the questions with a score greater than 0.5 to maintain a good balance between
quality and quantity. This provides us with 1445 synthetic good quality
samples. We add these good-quality NQ-like questions to our original NQ
training set thereby increasing the number of samples to 2663 QA pairs. In the
example shown in Figure 1, the well-formed questions detected by the
classification model are:

  * •

Chris Carney represents which state ’s 10th district in congress , which
includes Snyder and Wyoming counties

  * •

What includes Raystown Lake ; the Monongahela ends in which state , where what
meets the Allegheny river

  * •

With capital at Harrisburg , what northeastern state has Philadelphia as its
metropolis , and is named after its Quaker founder ?

##  4 Experiments

We use a smaller subset (1874 samples) from the total paired samples to form
our paired dataset containing NQ and QB questions. The train, dev, and test
splits are 1218, 93, and 563 respectively. We provide generated samples sizes
using our algorithm and the scores of two QA systems; DrQA and RAG which
trained on these varying experiments.

###  4.1 Generated Data

We originally possess 1874 total samples [1218 train, 93 dev, 563 test] for
both the NQ dataset and QB full dataset consisting of QB questions having
multiple sentences. We extract only the last sentence from the QB questions
(QB last sentence) with the same split [1218 train, 93 dev, 563 test]. We
generate NQ-like questions from QB questions with all sentences (NQ-like from
QB full) [7266 train, 517 dev, 3279 test] and questions in the last sentence
of QB questions (NQ-like from QB last sentence) [1385 train, 105 dev, 622
test], as outlined in Algorithm 1. We then extract only the well-formed
questions by utilizing a quality control classifier as outlined in Section
3.1. From this we obtain the filtered NQ-like questions from the QB questions
with multiple sentences (Quality controlled NQ-like from QB full) dataset
[1445 train, 113 dev, 517 test], and the filtered NQ-like questions from the
last sentence of the QB questions (Quality controlled NQ-like from QB last
sentence) [1093 train, 74 dev, 465 test]. The NQ dataset is concatenated with
the 6 different types of datasets outlined above. Assuming a generic [Dataset
Name] for the 6 datasets, we name the augmented system like (NQ + [Dataset
Name]) shown in Tables 2 and 3.

###  4.2 Question Answering Systems

After reviewing several QA systems Zhu et al. (2021), we finally used the
Retrieval Augmented Generation (RAG) Lewis et al. (2021) and DrQA Chen et al.
(2017) as our baseline QA systems. We used a small subset of the Wikipedia
dump as our retrieval dump for the low resource setting.

#### RAG QA System

We retrieve $5$ documents using the RAG retriever and use a batch size of $2$
for our training. We train our systems for $5$ epochs by using AdaGrad Duchi
et al. (2011) as the optimizer with a learning rate of $1e^{-4}$.

#### DrQA System

We retrieve $10$ documents for gold passage computation. We use SGD as the
optimizer with $0.1$ learning rate, and pre-trained word vector GloVe with 840
billion tokens and 300 dimensional vector embeddings Pennington et al. (2014).
3-layer bidirectional long short-term memory network (LSTM) with $128$ hidden
units is used for both paragraph and question encoding. The numbers of
document layers and answer layers are both set to $3$. We train all our
systems for $50$ epochs with batch size $64$ and max length $30$.

##  5 Results

Our research work demonstrates a clear improvement in QA performance for RAG
and DrQA systems as shown in Tables 2 and 3. We see an improvement in QA
evaluation scores when trivia questions with multiple sentences in QB dataset
can be used to automatically generate NQ-like questions which are then
concatenated with the original NQ questions. For the RAG QA system, the
baseline score was improved by 3 of our proposed systems, out of which
generated outputs from the last sentence concatenated with original NQ data
has seen the most success with an improvement of 2.8 points in accuracy and an
increase of 1.7, 2.4, 2.2 and 7.89 points on Precision, Recall, F1 measure,
and BLEU respectively. The performance of the QA system trained on NQ-like
filtered data concatenated with the NQ data is also better than the system
based on the concatenation of QB dataset with NQ dataset. For DrQA, all of our
proposed systems have done better than the baseline with NQ data and the
system with concatenated NQ and QB data. The generated outputs from the last
sentence of the QB concatenated with NQ have shown an improvement of 4.6
points in accuracy and 4.8, 5.3, 4.9, 0.91 points in Precision, Recall, F1
measure, and BLEU respectively on the DrQA system. Most of our systems exhibit
an increase in BLEU score over the baseline. We also observe that the
performance of RAG and DrQA systems trained on only NQ like data generated
from QB dataset is better than the baseline systems on QB dataset.

This is understandable as the last sentence usually has the easiest clue. On
top of which, it has a regular structure. These characteristics make the last
sentence from the QB more semantically aligned with NQ-like data after
conversion using our algorithm.

Our proposed algorithm as outlined in Algorithm 1 and quality filtering
technique (1093 extra samples) as mentioned in Section 3.1 can be effective in
a low resource setting with a lack of sufficiently well annotated QA data.
This also helps in scaling the training data by converting multiple datasets
automatically.

A broad example of NQ-like question generation and subsequent answer
generation has been provided in Figure 1.

##  6 Conclusion and Future Work

We clearly observe from the results that adding filtered NQ-like questions
from the QB data has given a boost over using only NQ questions. In finer
detail, we observe that questions from the last sentence of the QB are of
higher quality than from intermediate sentences and therefore provide a higher
boost to performance even with less samples. Even by simply adding questions
generated from last sentence, we increase the exact match accuracy by nearly 2
points. We also observe that the BLEU score of answers generated from quality
controlled NQ like system is 16 points more than the BLEU score of the
baseline QB system for the RAG system and by 13 points for the DrQA system.
This shows that our algorithm to generate NQ-like questions has been effective
in improving the quality of the training dataset.

We are working to extend this system from the smaller datasets to the entire
119247 QB and 91494 NQ samples by using the generated 772456 unfiltered NQ-
like questions from the entire QB data, retain the well-formed questions and
improve the performance over the baseline system trained on NQ dataset.

We observe that QB trivia questions are in passive voice while NQ questions
are in active voice. So Neural Machine Translation can help in converting the
style of QB questions. Our dataset pairing method automatically filters out
adversarial samples and improves the quality of the generated NQ-like corpus.
This opens the door towards a possible research area where multiple non-
adversarial datasets can be used to filter the NQ dataset from adversarial
samples. We plan on improving the filtering process by generating our own
annotated dataset regarding well-formed and ill-formed questions in order to
detect well formed questions and merge ill formed questions with better
quality questions.

Specifically, through our manual transformation for NQ-like questions, we are
specifying three different answer types like WHO/WHICH/WHAT, which might be
useful to propagate to intermediate parse outputs during NQ-like question
generation in order to improve answer generation.

Other information like difficulty and year associated with QB questions can be
leveraged to improve quality or filtering and can even be used to predict
extra information on NQ questions.

Further work can also be done on augmenting other datasets to NQ-like
questions to generate more synthetic samples which may improve question
answering performance.

## References

  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011.  Adaptive subgradient methods for online learning and stochastic optimization.  _Journal of Machine Learning Research_ , 12(Jul):2121–2159.
  * Faruqui and Das (2018) Manaal Faruqui and Dipanjan Das. 2018.  Identifying Well-formed Natural Language Questions.  In _Proc. of EMNLP_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
  * Kirstain et al. (2021) Yuval Kirstain, Ori Ram, and Omer Levy. 2021.  Coreference resolution without span representations.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021.  Retrieval-augmented generation for knowledge-intensive nlp tasks.
  * Min et al. (2021) Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen tau Yih. 2021.  Neurips 2020 efficientqa competition: Systems, analyses and lessons learned.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , page 1532–1543.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.
  * Rodriguez et al. (2021) Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber. 2021.  Quizbowl: The case for incremental question answering.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.