11institutetext:  CERN, Information Technology Department, CH-1211 Geneva 23,
Switzerland

#  Optimising HEP parameter fits via Monte Carlo
weight derivative regression

Andrea Valassi 11<EMAIL_ADDRESS>
###### Abstract

HEP event selection is traditionally considered a binary classification
problem, involving the dichotomous categories of signal and background. In
distribution fits for particle masses or couplings, however, signal events are
not all equivalent, as the signal differential cross section has different
sensitivities to the measured parameter in different regions of phase space.
In this paper, I describe a mathematical framework for the evaluation and
optimization of HEP parameter fits, where this sensitivity is defined on an
event-by-event basis, and for MC events it is modeled in terms of their MC
weight derivatives with respect to the measured parameter. Minimising the
statistical error on a measurement implies the need to resolve (i.e. separate)
events with different sensitivities, which ultimately represents a non-
dichotomous classification problem. Since MC weight derivatives are not
available for real data, the practical strategy I suggest consists in training
a regressor of weight derivatives against MC events, and then using it as an
optimal partitioning variable for 1-dimensional fits of data events. This
CHEP2019 paper is an extension of the study presented at CHEP2018: in
particular, event-by-event sensitivities allow the exact computation of the
“FIP” ratio between the Fisher information obtained from an analysis and the
maximum information that could possibly be obtained with an ideal detector.
Using this expression, I discuss the relationship between FIP and two metrics
commonly used in Meteorology (Brier score and MSE), and the importance of
“sharpness” both in HEP and in that domain. I finally point out that HEP
distribution fits should be optimized and evaluated using probabilistic
metrics (like FIP or MSE), whereas ranking metrics (like AUC) or threshold
metrics (like accuracy) are of limited relevance for these specific problems.

##  1 Introduction

The point estimation of physics parameters, such as the measurement of a cross
section or of a particle’s mass or couplings, is an important category of data
analysis problems in experimental High Energy Physics (HEP). Optimizing these
measurements ultimately consists in minimizing the combined statistical and
systematic errors on the measured parameters. In this paper, I only discuss
the minimization of the statistical error $\Delta\theta$, in the measurement
of a single parameter $\theta$ from the binned fit of a multi-dimensional
distribution of selected events. This implies the optimization of two analysis
handles: event selection, i.e. the criteria for signal-background
discrimination, and event partitioning, i.e. the choice of binning variables.

This article follows up on that I presented at CHEP2018 bib:chep2018av . As in
that occasion, two central points of my study are a discussion of evaluation
and training metrics for the data analysis tools used in the measurement, and
a comparison of these metrics to those used in other scientific domains. The
starting point of this analysis is, again, the calculation of the statistical
error $\Delta\theta$ in a binned fit for the parameter $\theta$ and its
comparison to the minimum error $\Delta\theta^{\smash{\text{(ideal})}}$ which
could be achieved in an “ideal” case. Minimizing $\Delta\theta$ is equivalent
to maximizing
$(\Delta\theta^{\smash{\text{(ideal})}})^{2}/(\Delta\theta)^{2}$, a metric in
[0,1] that I refer to as “Fisher Information Part” (FIP).

This research differs from and extends my CHEP2018 work in two respects.
First, it shifts the focus from event selection, which is a binary
classification problem, to event partitioning, and it shows that the latter
can be addressed as a non-binary regression problem. The key improvement is
the derivation of $\Delta\theta$ in terms of the event-by-event sensitivity
$\gamma_{i}$ of each event $i$ to the parameter $\theta$, rather than in terms
of the bin-by-bin sensitivity in bin $k$ (which is simply the average event-
by-event sensitivity $\langle\gamma\rangle_{k}$ in that bin). I show that the
optimal partitioning strategy consists in binning events according to their
sensitivity $\gamma_{i}$, and I use this to derive the minimum error
$\Delta\theta^{\smash{\text{(ideal})}}$ achievable with an ideal detector and
an ideal analysis method. While $\gamma_{i}$ can be computed for Monte Carlo
(MC) events from the derivative of their MC weight with respect to $\theta$,
however, $\gamma_{i}$ is not available for real data events: the practical
strategy I suggest consists in training a regressor $\mathpzc{q}_{i}$ of
$\gamma_{i}$ on MC events, and using it as an optimal partitioning variable
for a 1-dimensional fit of data events. The FIP metric can be used both for
evaluating the quality of the result, and as a loss function for training the
regressor $\mathpzc{q}_{i}$. In this context, where only statistical errors
are considered, event partitioning can be seen as a generalization of event
selection, which is a simpler, binary, sub-case. Rather than simply separating
signal events, which are sensitive to $\theta$, from background events, which
are not, the problem to address is how to resolve, i.e. separate, events with
different sensitivities to $\theta$: this ultimately represents a non-
dichotomous classification problem.

The second new contribution of this research is the comparison to other non-
HEP scientific domains, beyond those I had previously considered. In my
CHEP2018 study, I had mainly considered the evaluation metrics for binary
classification problems in Medical Diagnostics, Information Retrieval, and
Machine Learning research. I had also briefly discussed a few metrics used in
those fields to go beyond a strictly dichotomous categorization of the true
event categories, or to take into account the ranking of events when a scoring
classifier is used instead of a binary discrete classifier. In this paper, I
extend this comparative analysis by pointing out the close relationship
between FIP and two metrics commonly used in Meteorology (the “Brier score”
bib:brier1950  and the “Mean Squared Error” or MSE), and the importance of
“sharpness” both in HEP and in that domain. More generally, I suggest that HEP
distribution fits should be optimized and evaluated using probabilistic
metrics (like MSE, or FIP) as is commonly the case in Meteorology and Medical
Prognostics, whereas ranking metrics (like the “Area Under the ROC Curve” or
AUC) or threshold metrics (like “accuracy”), which are widely used in Medical
Diagnostics, are of limited relevance for these specific problems.

The outline of this paper is the following. Section 2 describes a mathematical
framework for discussing statistical error minimizations in HEP parameter
fits, and the use of MC weight derivative regression to optimize event
partitioning. It also discusses the relationship between FIP and MSE as
training metrics for Decision Tree regressors, using a decomposition of MSE
into calibration and sharpness that is copied from Meteorology. Section 3
points out the relevance of probabilistic metrics, more than threshold or
ranking metrics, in both HEP and Meteorology. An outlook for this research and
some conclusions are given in Section 4.

##  2 Statistical errors in HEP binned fits of a parameter
$\boldsymbol{\theta}$

Binned fits for a HEP parameter $\theta$ rely on splitting all selected events
into $K$ disjoint partitions, or “bins”, according to the values of one or
more variables that are computed as functions of the observed properties
$\mathbf{x}_{i}$ of each event $i$. When only statistical errors are
considered, the Fisher information $\mathcal{I}_{\theta}$ about $\theta$ which
is gained from its measurement, i.e. the inverse of the square of the
statistical error $\Delta\theta$, is easily shown bib:chep2018av  to be the
sum of the information contributions from the independent, and a fortiori
uncorrelated, measurements of $\theta$ in these $K$ bins,

$\mathcal{I}_{\theta}=\frac{1}{(\Delta\theta)^{2}}=\sum_{k\\!=\\!1}^{K}\frac{1}{(\Delta\theta)_{k}^{2}}=\sum_{k\\!=\\!1}^{K}n_{k}\left(\frac{1}{n_{k}}\frac{\partial
n_{k}}{\partial\theta}\right)^{2}\,.$ (1)

where $n_{k}(\theta)\\!=\\!s_{k}(\theta)\\!+\\!b_{k}$ is the number of
selected events in bin $k$. This is the sum of the number of signal events
$s_{k}$, which depends on $\theta$, and that of background events $b_{k}$,
which does not.

### MC reweighting and event-by-event sensitivities

In practice, HEP fits of a parameter $\theta$ rely on the theoretical
prediction of the number of signal events $s_{k}(\theta)$ in bin $k$ as a
function of $\theta$, obtained through MC simulations. A relatively standard
practice to derive $s_{k}(\theta)$ is the MC reweighting technique, which, for
instance, was used extensively by the LEP experiments in the late 1990s, for
measurements of both particle masses bib:opal-ww-1998 ; bib:aleph172  and
particle couplings bib:aleph-lemaitre-2000 . This technique is also applicable
to hadron colliders bib:gainer2014 , where it has been shown that is generally
feasible also at NLO accuracy bib:mattelaer2016 : it has been pointed out
bib:gainer2014 , in particular, that it is conceptually and practically
simpler than the Matrix Element Method bib:kondo1988 ; bib:dalitz1992 , which
has been extensively used at hadron colliders bib:me-dz2004 ; bib:kondo2006 ;
bib:me-cdf2007 , because it does not imply the time-consuming integration over
undetermined momenta which is necessary in that method, and which can be
performed by tools such as MadWeight bib:madweight2008 .

Monte Carlo reweighting essentially consists in the following three steps.
First, a sample of MC events for the signal process is generated at a
reference value ${\theta_{\text{ref}}}$ of the parameter $\theta$, and a
weight $w_{i}({\theta_{\text{ref}}})$ is assigned to each event $i$; if
unweighted events are generated, they all have the same
$w_{i}({\theta_{\text{ref}}})$, but this is not strictly needed. Second,
generator-level events are passed through full detector simulation. Third,
each detector-level event $i$ is assigned a weight $w_{i}(\theta)$ at another
value of the parameter $\theta$; this is done by rescaling
$w_{i}({\theta_{\text{ref}}})$ by the ratio of the predicted probabilities for
$\theta$ and ${\theta_{\text{ref}}}$ of event $i$, as described by its MC
truth (generator-level) properties $\mathbf{x}^{\smash{\text{(true})}}_{i}$.
The probability ratio is typically just a ratio of squared matrix elements,

$w_{i}(\theta)=\left(\frac{\text{Prob}_{(\theta)}(\mathbf{x}^{\smash{\text{(true})}}_{i})}{\,\text{Prob}_{({\theta_{\text{ref}}})}(\mathbf{x}^{\smash{\text{(true})}}_{i})\,}\right)\,w_{i}({\theta_{\text{ref}}})=\left(\frac{|\mathcal{M}(\theta,\mathbf{x}^{\smash{\text{(true})}}_{i})|^{2}}{\,|\mathcal{M}({\theta_{\text{ref}}},\mathbf{x}^{\smash{\text{(true})}}_{i})|^{2}\,}\right)\,w_{i}({\theta_{\text{ref}}})\,.$
(2)

The above description applies to signal MC events, but each background MC
event is also assigned a weight $w_{i}$, with the important difference that,
by definition, it does not depend on $\theta$. Assuming that all weights
$w_{i}$ take into account a normalization factor to the luminosity of the
data, the expected number of selected signal and background events
$n_{k}(\theta)$ in bin $k$, as a function of $\theta$, can be written as the
sum of the event weights $w_{i}$ for all MC events $i$ in bin $k$,

$n_{k}(\theta)=\sum_{i\in k}w_{i}(\theta)=\sum_{i\in
k}^{\text{Sig}}w_{i}(\theta)+\sum_{i\in
k}^{\text{Bkg}}w_{i}=s_{k}(\theta)+b_{k}\,.$ (3)

The bin-by-bin sensitivity of $n_{k}$ to $\theta$ which appears in Eq. 1 can
then be written as

$\displaystyle\frac{1}{n_{k}}\frac{\partial
n_{k}}{\partial\theta}=\frac{\sum_{i\in k}\frac{\partial
w_{i}}{\partial\theta}}{\sum_{i\in k}w_{i}}=\frac{\sum_{i\in
k}w_{i}(\frac{1}{w_{i}}\frac{\partial w_{i}}{\partial\theta})}{\sum_{i\in
k}w_{i}}=\frac{\sum_{i\in k}w_{i}\gamma_{i}}{\sum_{i\in
k}w_{i}}=\langle\gamma\rangle_{k}\,,\,$ (4)

i.e. as the weighted average over all MC events $i$ in bin $k$, of the event-
by-event sensitivity

$\gamma_{i}=\left.\frac{1}{w_{i}}\frac{\partial
w_{i}}{\partial\theta}\right|_{\theta_{\mathcal{I}}}\,.$ (5)

Note that all $\gamma_{i}$ (and hence $\mathcal{I}_{\theta}$) depend on the
value ${\theta_{\mathcal{I}}}$ of $\theta$ where $w_{i}$ and $\partial
w_{i}/\partial\theta$ are computed (typically, ${\theta_{\text{ref}}}$). In a
given binning scheme, the information $\mathcal{I}_{\theta}$ of Eq. 1 can then
be written as

$\mathcal{I}_{\theta}=\sum_{k\\!=\\!1}^{K}n_{k}\left(\frac{1}{n_{k}}\frac{\partial
n_{k}}{\partial\theta}\right)^{2}=\sum_{k\\!=\\!1}^{K}n_{k}\langle\gamma\rangle_{k}^{2}\,.$
(6)

### Beyond the signal-background dichotomy

For individual signal events $i$, the event-by-event sensitivity $\gamma_{i}$
may be positive or negative, and the absolute value of $\gamma_{i}$ may also
be significantly different from one event to another. Background events,
conversely, all have a zero event-by-event-sensitivity, because these events,
by definition, are produced by processes that are insensitive to the parameter
$\theta$:

$\begin{array}[]{lr}\displaystyle\gamma_{i}\\!=\\!\left(\frac{1}{w_{i}}\frac{\partial
w_{i}}{\partial\theta}\right)\\!\in\\![-\infty,\\!+\infty]\,,\,\hskip
2.84544pt\text{if}\,i\in\\{\text{Signal}\\}\,;&\displaystyle\gamma_{i}\\!=\\!\left(\frac{1}{w_{i}}\frac{\partial
w_{i}}{\partial\theta}\right)\\!=\\!0\,,\,\hskip
2.84544pt\text{if}\,i\in\\{\text{Background}\\}\,.\end{array}$ (7)

Equation 6 shows that the largest contributions to the information
$\mathcal{I}_{\theta}$ come from the bins with the largest average event-by-
event sensitivities. As discussed more in detail later on, a good measurement
is therefore one satisfying two criteria: first, the event selection accepts
the events with sensitivities $\gamma_{i}$ that are significantly different
from zero, whether positive or negative, i.e. those with high absolute values
of $\gamma_{i}$ (in the following I will refer to these as events with high
sensitivities, but it should be implicitly understood that I refer to their
absolute values); second, the event partitioning resolves events with very
different sensitivities into separate bins, as it is the average bin-by-bin
sensitivity that determines the contribution to $\mathcal{I}_{\theta}$.

As an example, consider the measurement of a particle mass $M$ from the fit to
the distribution of the invariant mass $m$ of the decay products of that
particle. The sensitivity $\gamma_{i}$ to $M$ is positive for the signal
events on the right of the mass peak ($m\\!>\\!M$) and negative for those on
its left ($m\\!<\\!M$). The events with the highest sensitivity (in absolute
value) are those on the steep ascending and descending slopes to the left and
to the right of the peak. Conversely, the events below the peak or on the
tails far away from it have a sensitivity that is close to 0. These low-
sensitivity signal events are not very different from background events, as
the information about $\theta$ that they provide is extremely limited, and it
is important to separate both of these types of events from high-sensitivity
signal events, so as not to dilute their sensitivity.

In spite of its limitations, a dichotomous categorization of events as signal
or background is still useful (especially when considering systematic errors).
Using the symbols $\rho_{k}\\!=\\!s_{k}/n_{k}$ to indicate the selection
purity and $\phi_{k}$ to indicate the sensitivity of signal events alone in
bin $k$,

$\phi_{k}=\langle\gamma\rangle_{k,\text{Sig}}=\frac{\sum_{i\in
k}^{\text{(Sig)}}w_{i}\gamma_{i}}{\sum_{i\in
k}^{\text{(Sig)}}w_{i}}=\frac{\sum_{i\in k}^{\text{(Sig)}}\frac{\partial
w_{i}}{\partial\theta}}{\sum_{i\in
k}^{\text{(Sig)}}w_{i}}=\frac{1}{s_{k}}\frac{\partial
s_{k}}{\partial\theta}\,,$ (8)

it is easy to see that $\langle\gamma\rangle_{k}\\!=\\!\rho_{k}\phi_{k}$: the
net effect of background is to dilute the overall bin-by-bin sensitivities by
a factor $\rho_{k}\\!\leq\\!1$, with respect to that computed from signal
events alone. The same is also true for the bin-by-bin contributions to
information, which can be written as:

$\mathcal{I}_{\theta}=\frac{1}{(\Delta\theta)^{2}}=\sum_{k\\!=\\!1}^{K}n_{k}\langle\gamma\rangle_{k}^{2}=\sum_{k\\!=\\!1}^{K}n_{k}(\rho_{k}\phi_{k})^{2}=\sum_{k\\!=\\!1}^{K}\rho_{k}(s_{k}\phi_{k}^{2})\,.$
(9)

For simplicity, I will assume $w_{i}({\theta_{\mathcal{I}}})\\!=\\!\\!1$ for
all signal and background events in the following. This implies that
$\langle\gamma\rangle_{k}\\!=\\!(\sum_{i\in k}\\!\gamma_{i})/n_{k}$ and
$\phi_{k}\\!=\\!(\sum_{i\in k}^{\text{(Sig)}}\\!\gamma_{i})/s_{k}$ in the rest
of this paper.

### An ideal measurement with an ideal detector, and a realistic analysis with
a limited detector

In my previous paper bib:chep2018av , I had shown that the optimal
partitioning in a fit of $\theta$ consists in separating events into bins with
different values of the bin-by-bin sensitivity $(1/n_{k})(\partial
n_{k}/\partial\theta)$. Event-by-event sensitivities make it possible to go to
a much finer granularity.

If only two selected events ${i_{1}}$ and ${i_{2}}$ are expected, the
“information inflow” bib:vdbos  in keeping them in separate one-event bins,
rather than mixing them together in a single two-event bin,

$\Delta\mathcal{I}_{\theta}=\gamma_{i_{1}}^{2}\\!+\\!\gamma_{i_{2}}^{2}\\!-\\!2\left(\\!\frac{\gamma_{i_{1}}\\!+\\!\gamma_{i_{2}}}{2}\\!\right)^{2}\\!=\frac{1}{2}(\gamma_{i_{1}}\\!-\\!\gamma_{i_{2}})^{2}\,,$
(10)

is zero if $\gamma_{i_{1}}$ and $\gamma_{i_{2}}$ are equal, whereas it is
strictly positive if they are different. In other words, in the “ideal” case
where all true values of the event-by-event sensitivities $\gamma_{i}$ were
known, the optimal way to measure $\theta$ would be a fit of the one-
dimensional distribution of $\gamma$. The maximum information
$\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}$ that is theoretically
achievable in this ideal case is simply

$\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}\\!=\\!\frac{1}{(\Delta\theta^{\smash{\text{(ideal})}})^{2}}\\!=\\!\sum_{i=1}^{N_{\mathrm{tot}}}\gamma_{i}^{2}\\!=\\!\sum_{i=1}^{S_{\\!\mathrm{tot}}}\gamma_{i}^{2}\,,$
(11)

where the sum over all
$N_{\mathrm{tot}}\\!=\\!S_{\\!\mathrm{tot}}\\!+\\!B_{\mathrm{tot}}$ events
includes $S_{\\!\mathrm{tot}}$ signal and $B_{\mathrm{tot}}$ background
events, but the contribution from the latter is 0 because they have
$\gamma_{i}\\!=\\!0$ as described in Eq. 7.

As in Ref. bib:chep2018av , I suggest to evaluate the quality of a measurement
using the “Fisher Information Part”, a dimensionless scalar metric in [0,1],
defined as the ratio between the information which was actually achieved, in
Eq. 9, and that achievable in an ideal case, in Eq. 11:

$\displaystyle\text{FIP}_{3}\\!=\\!\displaystyle\frac{\,\mathcal{I}_{\theta}\hphantom{{}^{\smash{\text{(ideal})}}}\,}{\,\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}\,}\\!=\\!\displaystyle\frac{\sum_{k\\!=\\!1}^{K}n_{k}\langle\gamma\rangle_{k}^{2}}{\sum_{i=1}^{S_{\\!\mathrm{tot}}}\gamma_{i}^{2}}\\!=\\!\displaystyle\frac{\sum_{k\\!=\\!1}^{K}s_{k}\rho_{k}\phi_{k}^{2}}{\sum_{i=1}^{S_{\\!\mathrm{tot}}}\gamma_{i}^{2}}\,.$
(12)

In Eq. 12, the numerator is a sum over bins, based on metrics derived from the
$N_{\mathrm{sel}}\\!=\\!\sum_{k\\!=\\!1}^{K}n_{k}$ selected events in those
bins (where
$N_{\mathrm{sel}}\\!=\\!S_{\\!\mathrm{sel}}\\!+\\!B_{\mathrm{sel}}$, including
$S_{\\!\mathrm{sel}}$ signal and $B_{\mathrm{sel}}$ background events), while
the denominator is a sum over the $S_{\\!\mathrm{tot}}$ signal events in a
given data sample. The main difference between this metric and that I had
previously presented bib:chep2018av  is that in the past I only used FIP to
evaluate the quality of event selection and signal-background discrimination
in a fit with a given binning, while now I redefine it to also evaluate the
quality of the binning.

FIP is a valuable metric because it is simple to use and interpret both
qualitatively and quantitatively, in statistically-limited measurements:
qualitatively, in that an analysis should be optimized to achieve the highest
value of FIP; quantitatively, in that its numerical value is proportional to
$1/\Delta\theta^{2}$, where $\Delta\theta$ is the statistical error on the
measurement. Another useful feature is that, since it is a ratio between 0 and
1, FIP can be decomposed as the product of several independent metrics which
are also ratios between 0 and 1. In particular, I propose to distinguish
between three effects which can result in information loss, and I decompose
$\text{FIP}_{3}$ in Eq. 12 as the product of three ratios, each taking values
between 0 and 1:

$\displaystyle\text{FIP}_{3}=\displaystyle\frac{\sum_{k\\!=\\!1}^{K}s_{k}\rho_{k}\phi_{k}^{2}}{\sum_{i=1}^{S_{\\!\mathrm{tot}}}\gamma_{i}^{2}}=\displaystyle\frac{\sum_{i=1}^{S_{\\!\mathrm{sel}}}\gamma_{i}^{2}}{\sum_{i=1}^{S_{\\!\mathrm{tot}}}\gamma_{i}^{2}}\\!\times\\!\displaystyle\frac{\sum_{k\\!=\\!1}^{K}s_{k}\phi_{k}^{2}}{\sum_{i=1}^{S_{\\!\mathrm{sel}}}\gamma_{i}^{2}}\\!\times\\!\displaystyle\frac{\sum_{k\\!=\\!1}^{K}s_{k}\rho_{k}\phi_{k}^{2}}{\sum_{k\\!=\\!1}^{K}s_{k}\phi_{k}^{2}}=\text{FIP}_{\\!\text{efS}}\\!\times\\!\text{FIP}_{\\!\text{shS}}\\!\times\\!\text{FIP}_{\\!\text{shB}}\,.$
(13)

The symbols $\text{FIP}_{\\!\text{efS}}$, $\text{FIP}_{\\!\text{shS}}$ and
$\text{FIP}_{\\!\text{shB}}$ denote that these ratios represent effective
measures of signal efficiency and of signal and background “sharpness”. The
concept of sharpness (also known as "resolution", a more familiar term in HEP)
describes the effectiveness at separating different categories of events from
one another. $\text{FIP}_{\\!\text{efS}}$ is an information-weighted signal
selection efficiency, describing the loss of information in rejecting some
events: it is the ratio between the $S_{\\!\mathrm{sel}}$ selected and
$S_{\\!\mathrm{tot}}$ total signal events, where each event is weighted by its
information contribution $\gamma_{i}^{2}$, the square of its event-by-event
sensitivity. $\text{FIP}_{\\!\text{shS}}$ measures the sharpness at resolving
selected signal events with different sensitivities $\gamma_{i}$, i.e. at
partitioning them into different bins of the distribution fit,
$S_{\\!\mathrm{sel}}\\!=\\!\sum_{k}\\!s_{k}$: it is the ratio of the
information achieved in the chosen binning $\mathcal{K}$, to that
theoretically achievable if it were possible to partition signal events
according to the true value $\gamma_{i}$ of their sensitivity to $\theta$.
$\text{FIP}_{\\!\text{shB}}$ is an information-weighted signal selection
purity, describing the loss of information due to an imperfect background
rejection, in a given binning scheme $\mathcal{K}$: it too measures a
“sharpness”, that at resolving background events (with $\gamma_{i}\\!=\\!0$)
from signal events (of any sensitivity $\gamma_{i}$).

${\begin{array}[]{c}{S_{\\!\mathrm{ALL}},\gamma_{i},\delta_{i}}\\\\[2.84544pt]
\boxed{{\mathcal{I}_{\theta}^{\smash{\text{(ideal,\,$S_{\\!\mathrm{ALL}}$})}}\\!=\\!\sum_{i=1}^{S_{\\!\mathrm{ALL}}}\gamma_{i}^{2}}}\end{array}}$${\begin{array}[]{c}\\\\[8.5359pt]
{\text{FIP}_{\\!\text{ALL}}=\text{FIP}_{\\!\text{ACC}}\times\text{FIP}_{3}}\\\\[2.84544pt]
{\text{FIP}_{3}=\text{FIP}_{\\!\text{efS}}\times\text{FIP}_{\\!\text{shS}}\times\text{FIP}_{\\!\text{shB}}}\\\\[2.84544pt]
{0\leq\text{FIP}_{3}\leq\text{FIP}_{3}^{\text{(max)}}\leq 1}\\\\[-28.45274pt]
\end{array}}$${\begin{array}[]{c}{S_{\\!\mathrm{tot}},\gamma_{i},\delta_{i}}\\\\[2.84544pt]
\boxed{{\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}\\!=\\!\sum_{i=1}^{S_{\\!\mathrm{tot}}}\gamma_{i}^{2}}}\end{array}}$${\begin{array}[]{c}{S_{\\!\mathrm{sel}},\gamma_{i},\delta_{i}}\\\\[2.84544pt]
\boxed{{(\mathcal{I}_{\theta}\\!=\\!\sum_{i=1}^{S_{\\!\mathrm{sel}}}\gamma_{i}^{2})}}\end{array}}$${\begin{array}[]{c}{S_{\\!\mathrm{tot}},\phi(\mathbf{x}),\delta_{i}}\\\\[2.84544pt]
\boxed{{(\mathcal{I}_{\theta}\\!=\\!\int
s(\mathbf{x})\phi(\mathbf{x})^{2}d\mathbf{x})}}\end{array}}$${\begin{array}[]{c}{S_{\\!\mathrm{sel}},\phi_{k},\delta_{i}}\\\\[2.84544pt]
\boxed{{(\mathcal{I}_{\theta}\\!=\\!\sum_{k\\!=\\!1}^{K}s_{k}\phi_{k}^{2})}}\end{array}}$${\begin{array}[]{c}{S_{\\!\mathrm{tot}},\phi(\mathbf{x}),\rho(\mathbf{x})}\\\\[2.84544pt]
\boxed{{\mathcal{I}_{\theta}^{\text{(max)}}\\!=\\!\int
s(\mathbf{x})\phi(\mathbf{x})^{2}\rho(\mathbf{x})d\mathbf{x}}}\end{array}}$${\begin{array}[]{c}{S_{\\!\mathrm{sel}},\phi_{k},\rho_{k}}\\\\[2.84544pt]
\boxed{{\mathcal{I}_{\theta}\\!=\\!\sum_{k\\!=\\!1}^{K}s_{k}\phi_{k}^{2}\rho_{k}}}\end{array}}$$\scriptstyle{\text{FIP}_{\\!\text{ACC}}}$$\scriptstyle{\text{FIP}_{\\!\text{efS}}}$$\scriptstyle{\text{FIP}_{3}=\frac{\raisebox{1.0pt}{$\scriptstyle\mathcal{I}_{\theta}$}}{\raisebox{-5.0pt}{$\scriptstyle\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}$}}}$$\scriptstyle{\text{FIP}_{3}^{\text{(max)}}=\frac{\raisebox{1.0pt}{$\scriptstyle\mathcal{I}_{\theta}^{\text{(max)}}$}}{\raisebox{-5.0pt}{$\scriptstyle\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}$}}}$$\scriptstyle{\text{FIP}_{\\!\text{shS}}}$$\scriptstyle{\text{FIP}_{\\!\text{shB}}}$
Figure 1:  Graphical representation of FIP metrics and of their inter-
relationships. For each of the seven scenarios considered, the number of
signal events used and the resolution on signal sensitivity $\gamma_{i}$ and
on signal/background classification $\delta_{i}$ (where $\delta_{i}\\!=\\!1$
and $\delta_{i}\\!=\\!0$ for true signal and background events, respectively)
are reported, as well as the information $\mathcal{I}_{\theta}$ which can be
achieved from the measurement. The main FIP metric discussed in this paper is
$\text{FIP}_{3}\\!=\\!\mathcal{I}_{\theta}/\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}$,
which is the product of $\text{FIP}_{\\!\text{efS}}$,
$\text{FIP}_{\\!\text{efS}}$ and $\text{FIP}_{\\!\text{shB}}$. While
theoretically $\text{FIP}_{3}$ is a metric in [0,1], for a realistic detector
$\text{FIP}_{3}\leq\text{FIP}_{3}^{\text{(max)}}$.

While I suggest the use of $\text{FIP}_{\\!\text{efS}}$,
$\text{FIP}_{\\!\text{shS}}$ and $\text{FIP}_{\\!\text{shB}}$ as figures of
merit for the information-weighted efficiency and signal and background
sharpness achieved by the final analysis stage of a measurement, it is
important to point out that, for all these three effects, the maximum
achievable figure of merit with a realistic detector may be lower than 1 even
if the best possible analysis method is used. Some loss of information may in
fact be inevitable given the limitations of the detector, but also those of
the computing and data processing chain which precedes the final analysis
stage of a measurement. This is shown schematically in Fig. 1. To start with,
the $S_{\\!\mathrm{tot}}$ signal events in a final analysis sample may be
fewer than the $S_{\\!\mathrm{ALL}}$ signal events produced in beam collisions
in the given data taking period, because of detector acceptance, trigger
decisions and preselection cuts: this may be taken into account by another
ratio $\text{FIP}_{\\!\text{ACC}}$, analogous to $\text{FIP}_{\\!\text{efS}}$
and lower than 1, by which the analysis-level $\text{FIP}_{3}$ should be
multiplied to obtain the overall $\text{FIP}_{\\!\text{ALL}}$ metric for the
measurement. $\text{FIP}_{\\!\text{shB}}$, or $\text{FIP}_{\\!\text{shS}}$,
respectively, may be lower than 1 because the limited resolution of the
detector mixes together signal events with different sensitivities
$\gamma_{i}$, or mixes together signal events and background events,
respectively, making them experimentally indistinguishable. For a real
detector, even the best possible analysis method can at most try to determine,
at each point $\mathbf{x}$ of the observable phase space, the average local
sensitivity of signal events
$\phi(\mathbf{x})\\!=\\!\langle\gamma\rangle_{\text{Sig}}(\mathbf{x})\\!=\\!(1/s(\mathbf{x}))(\partial
s(\mathbf{x})/\partial\theta)$ and the average local purity
$\rho(\mathbf{x})\\!=\\!(s(\mathbf{x}))/(s(\mathbf{x})\\!+\\!b(\mathbf{x}))$
that the detector resolution effectively establishes. In these expressions,
$s(\mathbf{x})$ and $b(\mathbf{x})$ indicate the differential distributions of
signal and background events in $\mathbf{x}$-space, with
$\int\\!s(\mathbf{x})d\mathbf{x}\\!=\\!S_{\\!\mathrm{tot}}$ and
$\int\\!b(\mathbf{x})d\mathbf{x}\\!=\\!B_{\mathrm{tot}}$.

While the framework I propose describes the general case where signal events
have different sensitivities $\gamma_{i}$ to $\theta$ and are thus not all
equivalent to one another, it also describes a much simpler case where signal
events all have the same sensitivity $\gamma_{i}$, namely the measurement of a
total signal cross section $\sigma\\!_{s}$. In this case, which I discussed in
Ref. bib:chep2018av , the only challenge is the classic binary classification
problem of signal-background discrimination in the presence of strictly
dichotomous true categories. As there is no need to resolve signal events from
one another, $\text{FIP}_{\\!\text{shS}}$ is always 1 in this case. If
$\sigma\\!_{s}$ is measured by a counting experiment (i.e. using a single
bin), $\text{FIP}_{3}$ reduces to to
$\text{FIP}_{1}\\!=\\!\epsilon_{s}\varrho$ bib:chep2018av , the product of the
global signal selection efficiency
$\text{FIP}_{\\!\text{efS}}\\!=\\!\epsilon_{s}$ and purity
$\text{FIP}_{\\!\text{shB}}\\!=\\!\varrho$, a metric that has been widely used
in HEP already since the late 1990s bib:yellow1996 ; bib:valassi-phd ;
bib:opal-ww-1998 ; bib:aleph-lemaitre-2000 ; bib:delphi-ww-2000 . Another
common way to measure $\sigma\\!_{s}$ is the fit of a scoring classifier
distribution: examples include fits of Neural Network or Rarity distributions
at LEP bib:aleph161  and fits of Boosted Decision Trees at the Tevatron
bib:d0-singletop-2008 ; bib:cdf-singletop-2009  and LHC bib:cms-singletop-2011
. In this case, $\text{FIP}_{\\!\text{efS}}\\!=\\!1$ because all pre-selected
events are included in the fit, while $\text{FIP}_{\\!\text{shB}}$ reduces to
$\text{FIP}_{2}\\!=\\!(\sum_{k}\\!s_{k}\rho_{k})/(\sum_{k}\\!s_{k})$
bib:chep2018av , because $\gamma_{i}$ is the same for all signal events.

### Monte Carlo weight derivative regression

To optimize the measurement of $\theta$ from a sample of $N_{\mathrm{tot}}$
events, it would then be enough to know a single property of all events, their
sensitivity $\gamma_{i}$ to $\theta$. The fit of the one-dimensional
distribution of $\gamma_{i}$ would provide optimal partitioning and background
rejection, and achieve the minimum statistical error
$\Delta\theta^{\smash{\text{(ideal})}}$. The challenge to address is that,
while $\gamma_{i}$ can be computed for MC events, $\gamma_{i}$ is not
available for real data. The practical strategy I suggest is to train a
regressor $\mathpzc{q}_{i}$ of $\gamma_{i}$ on MC events, i.e. a regressor of
the MC weight derivatives $(1/w_{i})(\partial w_{i}/\partial\theta)$ computed
from the generator-level properties $\mathbf{x}^{\smash{\text{(true})}}_{i}$
of MC events, and use it to fit $\theta$ from the one-dimensional distribution
of $\mathpzc{q}(\mathbf{x}_{i})$ on data events, computed from their detector-
level properties $\mathbf{x}_{i}$. I refer to this approach as “Weight
Derivative Regression” (WDR).

In such a crude form, this method is probably of little applicability in many
practical situations, and more refined variations should be used to overcome
some of its limitations. The main issue is that the MC weight derivatives
$(1/w_{i})(\partial w_{i}/\partial\theta)$ depend on the value
${\theta_{\mathcal{I}}}$ of $\theta$ at which they are computed: this
dependency may be weak in fits of particle couplings, but is certainly strong
in fits of particle masses. It may be necessary to compute these derivatives
at more than one value of ${\theta_{\mathcal{I}}}$, and possibly train more
than one regressor, using them to measure $\theta$ from a multi-dimensional
fit. A separate binary classifier for background rejection may also be useful,
especially to handle systematic errors. A more detailed discussion of the
limitations of this method, and practical examples of its use, are foreseen
for later publications.

I stress that the method I suggest has clear similarities with, and was
strongly inspired by, the “Optimal Observables” (OO) approach bib:optobs92 ;
bib:optobs93-tau ; bib:optobs93-tgc ; bib:optobs05 . There is, however, an
important difference, which schematically is the following: the WDR method
consists in computing the true sensitivity $\gamma_{i}$ of each MC event $i$
from its generator-level properties $\mathbf{x}^{\smash{\text{(true})}}_{i}$,
and training the regressor $\mathpzc{q}_{i}\\!=\\!\mathpzc{q}(\mathbf{x}_{i})$
against these true $\gamma_{i}$, to obtain an estimate
$\mathpzc{q}(\mathbf{x})$ of the functional dependency of the local average
sensitivity
$\langle\gamma\rangle(\mathbf{x})\\!=\\!\phi(\mathbf{x})\rho(\mathbf{x})$ on
the detector-level properties $\mathbf{x}$ for real data events; the OO method
approximately consists, instead, in analytically computing the functional
dependency of $\gamma_{i}$ on $\mathbf{x}^{\smash{\text{(true})}}_{i}$, and
applying that same functional dependency on the observed $\mathbf{x}$ to
obtain an estimate of $\langle\gamma\rangle(\mathbf{x})$ for real data events.
As a consequence, the results that can be obtained through the OO method are
significantly degraded by the effect of the experimental detector resolution,
which is not properly accounted for.

The regressor $\mathpzc{q}_{i}\\!=\\!\mathpzc{q}(\mathbf{x}_{i})$ of the
sensitivity $\gamma_{i}$ may be implemented in many different ways. Selecting
a specific algorithm essentially means choosing two things: the
parametrization of the $\mathpzc{q}(\mathbf{x})$ function, and the metric to
use for training the regressor. As in Ref. bib:chep2018av , I focus on
Decision Tree (DT) algorithms bib:cart , and I suggest that the maximization
of $\text{FIP}_{3}$ should be used both for evaluating the measurement and for
training the regressor. In a DT, the space of detector-level event properties
$\mathbf{x}$ is split into $K$ disjoint nodes, such that
$\mathpzc{q}(\mathbf{x})\\!=\\!\mathpzc{q}_{(k)}$ is a constant in each node
$k$. Taking into account that each node of the tree may be used as a bin in
the fit, the goal is to split all
$N_{\mathrm{sel}}\\!\\!=\\!\\!N_{\mathrm{tot}}$ events in the training sample
into $K$ nodes/bins, with $n_{k}$ events in node/bin $k$, so as to maximize
$\text{FIP}_{3}$ in Eq. 12. It is extremely interesting to see that this is
equivalent to using a much more common criterion, the minimization of the Mean
Squared Error (MSE). It is easy to prove, in fact, that the MSE can be
decomposed as follows,

$\displaystyle{\text{MSE}}\\!=\\!\frac{1}{N_{\mathrm{tot}}}\\!\sum_{i\\!=\\!1}^{N_{\mathrm{tot}}}(\mathpzc{q}_{i}\\!-\\!\gamma_{i})^{2}$
$\displaystyle\\!=\\!\frac{1}{N_{\mathrm{tot}}}\\!\left[\sum_{k\\!=\\!1}^{K}\\!n_{k}\\!\left(\mathpzc{q}_{(k)}\\!-\\!\langle\gamma\rangle_{k}\right)^{2}\right]\\!+\\!\frac{1}{N_{\mathrm{tot}}}\\!\left[\left(\\!\sum_{i=1}^{N_{\mathrm{tot}}}\gamma_{i}^{2}\\!\right)\\!-\\!\left(\\!\sum_{k\\!=\\!1}^{K}n_{k}\langle\gamma\rangle_{k}^{2}\\!\right)\right]$
(14)
$\displaystyle\\!=\\!\frac{1}{N_{\mathrm{tot}}}\\!\left[\sum_{k\\!=\\!1}^{K}\\!n_{k}\\!\left(\mathpzc{q}_{(k)}\\!-\\!\langle\gamma\rangle_{k}\right)^{2}\right]\\!+\\!\frac{1}{N_{\mathrm{tot}}}\\!\left[\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}\\!-\\!\mathcal{I}_{\theta}\right]\\!=\\!{\text{MSE}}_{\text{cal}}\\!+\\!{\text{MSE}}_{\text{sha}}\,,$

where the “calibration” ${\text{MSE}}_{\text{cal}}$ is 0 by construction in
training the DT, as $\mathpzc{q}_{(k)}$ is defined as the average sensitivity
$\langle\gamma\rangle_{k}$ of the MC events in node $k$, while the “sharpness”
${\text{MSE}}_{\text{sha}}$ is minimized when $\text{FIP}_{3}$ (or more
precisely $\text{FIP}_{\\!\text{shS}}\times\text{FIP}_{\\!\text{shB}}$, as
$N_{\mathrm{sel}}\\!\\!=\\!\\!N_{\mathrm{tot}}$) is maximised, because

$\left(1\\!-\\!\frac{N_{\mathrm{tot}}\times{\text{MSE}}_{\text{sha}}}{\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}}\right)\\!=\\!\frac{\mathcal{I}_{\theta}}{\mathcal{I}_{\theta}^{\smash{\text{(ideal})}}}\\!=\\!\text{FIP}_{3}\\!=\\!\text{FIP}_{\\!\text{shS}}\times\text{FIP}_{\\!\text{shB}}\,.$
(15)

For other algorithms, such as Neural Networks, where implementing FIP
maximization is not as easy as in a DT, minimizing MSE is probably still a
sensible training criterion.

##  3 Learning from others: probabilistic metrics in Meteorology

I now take a step backwards to consider the more general perspective of
evaluation and training metrics in different scientific domains. The reason
why metrics like FIP and MSE are relevant to HEP parameter fits is that they
capture their most characteristic feature, the simultaneous use of disjoint
event partitions to derive a measurement of $\theta$ which is effectively a
combination of the measurements performed in these individual partitions. It
should be noted in passing that most of the ideas in this paper are relevant
for both binned and unbinned fits, even if their applicability is more obvious
in the case of binned fits. In my previous study bib:chep2018av , I noted that
event partitioning is largely unaccounted for by the evaluation metrics
commonly used in Medical Diagnostics (MD), Information Retrieval (IR) and
Machine Learning (ML). Further research led me to understand two things:
first, that a key point is the categorization bib:caruana2004 ;
bib:ferriorallo2004 ; bib:wu-flach-2007 ; bib:ferriorallo2009  of performance
metrics into three distinct families, namely threshold, ranking and
probabilistic metrics; and, second, that MD, IR and ML mainly focus on binary
classification problems described by threshold and ranking metrics, whereas
HEP parameter fits require probabilistic metrics, which are widely used for
regression problems in domains such as Meteorology and Climatology, or Medical
Prognostics.

Threshold metrics are relevant in classification problems where all events are
assigned to a signal or background category by a discrete binary classifier.
This includes the case when the operating point of a scoring classifier is
chosen on its ROC bib:peterson-roc-1953 ; bib:tanner-swets-1954 ;
bib:peterson-roc-1954-pgit4 ; bib:tanner-swets-1954-pgit4 ; bib:vanmeter-
middleton-1954-pgit4 ; bib:swets-tanner-birdsall-1955 ; bib:egan-1956 ;
bib:swets-tanner-birdsall-1961 ; bib:birdsall-thesis-1973  curve (for instance
based on a cost matrix), a popular approach in MD bib:lusted1960 ;
bib:lusted1968 ; bib:lusted1971 ; bib:metz-goodenough-1973 ; bib:metz-
starr-1975 ; bib:mcneil-1975 ; bib:metz-roc-1978 ; bib:lusted1984 .
Classifiers are evaluated from the four event counts in a two-by-two confusion
matrix, namely True/False Positives/Negatives. The simplest threshold metric
is accuracy, which is widely used, but is known to have severe limitations, in
both MD bib:swets-1979-imaging ; bib:swets-1988  and ML bib:ml-spackman-1989 ;
bib:bradley ; bib:provost-1997 ; bib:provost-1998 ; bib:fawcett-roc-2006 . A
popular threshold metric in IR bib:ir-kent-1955 ; bib:ir-cleverdon-1962 ;
bib:ir-swets-1963 ; bib:ir-cleverdon-1965 ; bib:ir-rijs1974 ; bib:ir-rijs1979
; bib:ir-manning  is the F1 score: this is based on precision and recall,
which in HEP are known as purity $\varrho$ and efficiency $\epsilon_{s}$. In
HEP, threshold metrics are especially useful in counting experiments: examples
include cross section measurements by counting, where the relevant metric is
$\text{FIP}_{1}\\!=\\!\epsilon_{s}\varrho$, as discussed, but also searches
for new physics bib:punzi ; bib:cousins ; bib:cowan2011 ; bib:higgsml  that
are not based on distribution fits. An interesting way to compare different
threshold metrics is to study their symmetries and invariances bib:sokolova ;
bib:luque . A fundamental feature of HEP measurements, in particular, is the
irrelevance of the True Negatives count, i.e. of the number of rejected
background events: in this respect, HEP is more similar to IR than it is to
MD, as I briefly discussed in Ref. bib:chep2018av .

Ranking metrics are relevant in classification problems where all events are
assigned a score $\mathcal{D}$ by a scoring classifier, representing their
probability to belong to the signal category. Events can then be ranked by
their score, which is especially important if some prioritization is needed.
Ranking metrics such as precision for a fixed number of retrieved documents,
or a fixed fraction of all available documents, are often used in IR bib:ir-
trec3-taguesutcliffe-1995 ; bib:ir-trec3-appendixa-1995 ; bib:ir-
trec2-harman-1995 ; bib:ir-hull-1993 . The most commonly used ranking metric
is however the Area Under the ROC Curve (AUC), which is popular in MD
bib:green1964 ; bib:green-swets-1966 ; bib:goodenough-1972 ; bib:bamber1975 ;
bib:hanley-auc-1983  because it represents “the probability that a randomly
chosen diseased subject is correctly ranked with greater suspicion than a
randomly chosen non-diseased subject”. The AUC is however known to have severe
limitations for both MD bib:greiner2000 ; bib:zhou2002 ; bib:ray2010 ;
bib:hajian2013  and ML bib:adams-hand-1999 ; bib:drummond-2000a ;
bib:drummond-2006 ; bib:davis-goadrich-2006 ; bib:saito ; bib:he-garcia-2009 .
Ranking metrics are an active area of research in ML bib:clemencon-
vayatis-2007 ; bib:clemencon-lugosi-2008 ; bib:rudin2018 , which was also
investigated in HEP bib:higgsml . In my opinion bib:chep2018av , however,
ranking metrics, and in particular the AUC, are largely irrelevant in HEP
measurements: while threshold metrics are needed in counting experiments, for
distribution fits one should use metrics describing event partitioning, not
event ranking. In a cross section fit from the distribution of a scoring
classifier $\mathcal{D}$, for instance, a metric like $\text{FIP}_{2}$ is
relevant because it describes the fit as a combination of measurements from
subsets of events with different values of $\mathcal{D}$, independently of
which event subset has a higher score.

A related challenge in HEP distribution fits is that signal events are not all
equivalent to one another, as they have different sensitivities $\gamma_{i}$.
Research on metrics for non-dichotomous evaluation has been active on non-
binary gold standards in MD bib:pencina-2004 ; bib:obuchowski-2006 ;
bib:lambert-2008 , on graded relevance assessment in IR bib:ir-jarvelin-2000 ;
bib:ir-jarvelin-2002a ; bib:ir-jarvelin-2002b  and on cost-sensitive
classification in ML bib:turney-1994 ; bib:drummond-2000b ; bib:zadrozny-2001
; bib:elkan-2001 ; bib:zadrozny-2003 ; bib:fawcett-rociv-2006 , involving
threshold, ranking and probabilistic metrics, and even discussing the issue of
the calibration of probabilistic classifiers bib:calib-zadrozny-2001c ;
bib:guo2017 . In my opinion, however, a more appropriate solution for HEP
distribution fits may come from probabilistic metrics in other domains.

Probabilistic metrics are relevant in classification and regression problems
where the comparison of a predicted property of an event to its true value has
a probabilistic interpretation. Verification scores of forecasts in
Meteorology and Climatology bib:brier1950 ; bib:sanders1963 ; bib:murphy1973 ;
bib:licht1977 ; bib:mason1982 ; bib:murphy-winkler-1987 ; bib:wmo-svslrf-1992
, such as MSE and the closely related Brier score, are typical probabilistic
metrics. Similar metrics are also used for the evaluation of patient health
predictions in Medical Prognostics bib:spiegelhalter ; bib:harrell-lee-mark .
In both cases, the quality of forecasts is assessed by comparing a forecast
probability of a future weather event, or of a future disease, to the relative
frequency which is eventually observed for that event. Partitioning is an
essential component of this approach: for instance, ten different forecast
groups may be studied, each covering a 10% probability range, with the third
group including days (or patients), with a 20 to 30% probability of rain (or
of survival after 5 years, respectively). A good forecast is one with two
features: first, reliability or calibration, i.e. the actual fraction of rainy
days must be $\sim$25% for forecasts in the 20–30% range; second, sharpness or
resolution, i.e. it must be able to distinguish between days with a $\sim$25%
probability and days with a $\sim$75% probability of rain. As discussed in
Sec. 2, probabilistic metrics like MSE, and the concepts of sharpness and
calibration of a regressor are also relevant to describe HEP parameter fits:
the decomposition in Eq. 14 was, in fact, copied from that of the Brier score
into a calibration and a sharpness term in Meteorology bib:sanders1963 .

##  4 Outlook and conclusions

I have described a mathematical framework to evaluate HEP parameter fits, and
suggested a MC Weight Derivative Regression approach to optimize them. Data
analysis methods are similar across scientific domains, and HEP can learn a
lot from others; but different problems require different metrics, and it is
important to select from other domains the tools that make sense for us. I
pointed out in particular that ranking metrics like the AUC, a standard
practice in Medical Diagnostics, are of limited relevance for HEP, while
probabilistic metrics like the MSE and the concepts of calibration and
sharpness, commonly used in Meteorology, are directly applicable in our field.
I have not discussed systematic errors, or searches for new physics based on
distribution fits, but I hope that this work can stimulate research in that
direction. Further details on this work are available in the slides of the
CHEP2019 talk bib:avtalk  described in this paper. A more detailed article is
also planned for the future.

## References

  * (1) A. Valassi, Binary classifier metrics for optimizing HEP event selection, Proc. CHEP2018, Sofia, EPJ Web of Conf. 214 (2019) 06004. https://doi.org/10.1051/epjconf/201921406004
  * (2) G. W. Brier, Verification of forecasts expressed in terms of probability, Weather Rev. 78 (1950) 1. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2
  * (3) OPAL Collaboration, Measurement of the W boson mass and W+W- production and decay properties in e+e- collisions at $\sqrt{s}$=172 GeV, Eur. Phys. J. C 1 (1998) 395. https://doi.org/10.1007/s100520050093
  * (4) ALEPH Collaboration, Measurement of the W mass by direct reconstruction in $e^{+}e^{-}$ collisions at 172 GeV, Phys. Lett. B 422 (1998) 384. https://doi.org/%****␣chep19wdr_v2_arxiv.tex␣Line␣1725␣****10.1016/S0370-2693(98)00062-8
  * (5) V. Lemaitre and ALEPH Collaboration, Single W Production at Energies up to $\sqrt{s}$=202 GeV and Search for Anomalous Triple Gauge Boson Couplings, Proc. 30th Int. Conf. on High-Energy Physics (ICHEP2000), Osaka (2000). http://cds.cern.ch/record/531207
  * (6) J. S. Gainer, J. Lykken, K. T. Matchev, S. Mrenna, M. Park, Exploring theory space with
Monte Carlo reweighting, JHEP 2014 (2014) 78.
https://doi.org/10.1007/JHEP10(2014)078

  * (7) O. Mattelaer, On the maximal use of Monte Carlo samples: re-weighting events at NLO accuracy, Eur. Phys. J. C 76 (2016) 674. https://doi.org/10.1140/epjc/s10052-016-4533-7
  * (8) K. Kondo, Dynamical Likelihood Method for Reconstruction of Events with Missing Momentum. I. Method and Toy Models, J. Phys. Soc. Jpn. 57 (1988) 4126. https://doi.org/10.1143/JPSJ.57.4126
  * (9) R. H. Dalitz, G. R. Goldstein, Decay and polarization properties of the top quark, Phys. Rev. D 45 (1992) 1531. https://doi.org/10.1103/PhysRevD.45.1531
  * (10) D0 Collaboration, A precision measurement of the mass of the top quark, Nature 429 (2004) 638. https://doi.org/10.1038/nature02589
  * (11) K. Kondo, Dynamical Likelihood Method and Top Quark Mass Measurement at CDF, J. Phys. Conf. Series 53 (2006) 009. https://doi.org/10.1088/1742-6596/53/1/009
  * (12) CDF Collaboration, Precision measurement of the top-quark mass from dilepton events
at CDF II, Phys. Rev. D 75 (2007) 031105(R).
https://doi.org/10.1103/PhysRevD.75.031105

  * (13) O. Mattelaer, P. Artoisenet, MadWeight: automatic event reweighting with matrix elements, Proc. CHARGED2008, Uppsala (2008). https://doi.org/10.22323/1.073.0025
  * (14) A. van den Bos, Parameter Estimation for Scientists and Engineers Wiley (2007).
  * (15) D. Gelé, T. G. Shears, W. J. Stirling, A. Valassi, M. F. Watson, Measurement of $M_{W}$ from the $W^{+}W^{-}$ Threshold Cross-Section, Proc. Workshop on Physics at LEP2 vol.1, CERN-96-01-V-1 (1996). https://doi.org/10.5170/CERN-1996-001-V-1
  * (16) A. Valassi, Mesure de la masse du boson W au seuil, Doctoral thesis, Paris (1997). https://doi.org/10.17181/CERN.LT3V.WJKI
  * (17) P. Buschmann and DELPHI Collaboration, Measurement of the W-pair production cross-section and W branching ratios at $\sqrt{s}$=192-202 GeV, Proc. 30th Int. Conf. on High-Energy Physics (ICHEP2000), Osaka (2000). http://cds.cern.ch/record/2627765
  * (18) ALEPH Collaboration, Measurement of the W mass in $e^{+}e^{-}$ collisions at production threshold, Phys. Lett. B 401 (1997) 347. https://doi.org/10.1016/S0370-2693(97)00460-7
  * (19) D0 Collaboration, Evidence for production of single top quarks, Phys. Rev. D 78 (2008) 012005. https://doi.org/10.1103/PhysRevD.78.012005
  * (20) CDF Collaboration, Observation of Electroweak Single Top-Quark Production, Phys. Rev. Lett. 103 (2009) 092002. https://doi.org/10.1103/PhysRevLett.103.092002
  * (21) CMS Collaboration, Measurement of the $t$-Channel Single Top Quark Production Cross Section in $pp$ Collisions at $\sqrt{s}=7$ TeV, Phys. Rev. Lett. 107 (2011) 091802. https://doi.org/10.1103/PhysRevLett.107.091802
  * (22) D. Atwood, A. Soni, Analysis for magnetic moment and electric dipole moment form factors of the top quark via $e^{+}e^{-}\rightarrow t\bar{t}$, Phys. Rev. D 45 (1992) 2405. https://doi.org/10.1103/PhysRevD.45.2405,
  * (23) M. Davier, L. Duflot, F. LeDiberder, A. Rougé, The optimal method for the measurement of tau polarization, Phys. Lett. B 306 (1993) 411. https://doi.org/10.1016/0370-2693(93)90101-M
  * (24) M. Diehl, O. Nachtmann, Optimal observables for the measurement of three-gauge-boson couplings in $e^{+}e^{-}\\!\rightarrow\\!W^{+}W^{-}$, Z. Phys. C 62 (1994) 397. https://doi.org/10.1007/BF01555899
  * (25) O. Nachtmann, F. Nagel, Optimal observables and phase-space ambiguities, Eur. Phys. J. C 40 (2005) 497. https://doi.org/%****␣chep19wdr_v2_arxiv.tex␣Line␣1850␣****10.1140/epjc/s2005-02153-9
  * (26) L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, Classification And Regression Trees, Chapman and Hall (1984). https://doi.org/10.1201/9781315139470
  * (27) R. Caruana, A. Niculescu-Mizil, Data mining in metric space: an empirical analysis of supervised learning performance criteria, Proc. 10th Int. Conf. on Knowledge Discovery and Data Mining (KDD-04), Seattle (2004). https://doi.org/10.1145/1014052.1014063
  * (28) C. Ferri, J. HernÃ¡ndez-Orallo, R. Modroiu, An Experimental Comparison of Classification Performance Metrics, Proc. Learning 2004, Elche (2004). http://dmip.webs.upv.es/papers/Learning2004.pdf
  * (29) S. Wu, P. Flach, C. Ferri, An Improved Model Selection Heuristic for AUC, Proc. 18th Eur. Conf. Machine Learning (ECML 2007), Warsaw (2007). https://doi.org/10.1007/978-3-540-74958-5_44
  * (30) C. Ferri, J. HernÃ¡ndez-Orallo, R. Modroiu, An Experimental Comparison of Performance Measures for Classification, Pattern Recognition Letters 30 (2009) 27. https://doi.org/10.1016/j.patrec.2008.08.010
  * (31) W. W. Peterson, T. G. Birdsall, The theory of signal detectability (Part I: The general theory. Part II: Applications with Gaussian noise), Electronic Defense Group, Univ. of Michigan, Tech. Report No. 13 (1953). http://hdl.handle.net/2027.42/7068
  * (32) W. P. Tanner, J. A. Swets, A decision-making theory of visual detection, Psychological Review 61 (1954) 401. https://doi.org/10.1037/h0058700
  * (33) W. W. Peterson, T. G. Birdsall, W. C. Fox, The theory of signal detectability, Transactions of the IRE Professional Group on Information Theory (PGIT) 4 (1954) 171. https://doi.org/10.1109/TIT.1954.1057460
  * (34) W. P. Tanner, J. A. Swets, The human use of information I: Signal detection for the case of the signal known exactly, Transactions of the IRE Professional Group on Information Theory (PGIT) 4 (1954) 213. https://doi.org/10.1109/TIT.1954.1057461
  * (35) D. van Meter, D. Middleton, Modern statistical approaches to reception in communication theory, Transactions of the IRE Professional Group on Information Theory (PGIT) 4 (1954) 119. https://doi.org/10.1109/TIT.1954.1057471
  * (36) J. A. Swets, W. P. Tanner, T. G. Birdsall, The evidence for a decision-making theory of visual detection, Electronic Defense Group, Univ. of Michigan, Tech. Report No. 40 (1955). http://hdl.handle.net/2027.42/7843
  * (37) J. P. Egan, F. R. Clarke, E. C. Carterette, On the Transmission and Confirmation of Messages in Noise, J. Acoustical Soc. Am. 28 (1956) 536. https://doi.org/10.1121/1.1908387
  * (38) J. A. Swets, W. P. Tanner, T. G. Birdsall, Decision processes in perception, Psychological Review 68 (1961) 301. https://doi.org/10.1037/h0040547
  * (39) T. G. Birdsall, The theory of signal detectability : ROC curves and their character, Univ. of Michigan, Tech. Report No. 177 (1973). http://hdl.handle.net/2027.42/3618
  * (40) L. B. Lusted, Logical Analysis in Roentgen Diagnosis, Radiology 74 (1960) 178. https://doi.org/10.1148/74.2.178
  * (41) L. B. Lusted, Introduction to Medical Decision Making, Charles C. Thomas (1968).
  * (42) L. B. Lusted, Signal Detectability and Medical Decision-Making, Science 171 (1971) 1217. https://doi.org/10.1126/science.171.3977.1217
  * (43) C. E. Metz, D. J. Goodenough, K. Rossmann, Evaluation of Receiver Operating Characteristic Curve Data in Terms of Information Theory, with Applications in Radiography, Radiology 109 (1973) 297. https://doi.org/10.1148/109.2.297
  * (44) C. E. Metz, S. J. Starr, L. B. Lusted, K. Rossmann, Progress in evaluation of human observer visual detection performance using the ROC curve approach, Report CEA-CR-6, Proc. Int. Conf. on information processing in scintigraphy, Orsay (1975). https://inis.iaea.org/collection/NCLCollectionStore/_Public/07/248/7248574.pdf
  * (45) B. J. McNeil, E. Keeler, S. J. Adelstein, Primer on Certain Elements of Medical Decision Making, New England Journal of Medicine 293 (1975) 211. https://doi.org/10.1056/NEJM197507312930501
  * (46) C. E. Metz, Basic principles of ROC analysis, Seminars in Nuclear Medicine 8 (1978) 283. https://doi.org/10.1016/S0001-2998(78)80014-2
  * (47) L. B. Lusted, ROC Recollected, Medical Decision Making 4 (1984) 131. https://doi.org/10.1177/0272989X8400400201
  * (48) J. A. Swets, ROC Analysis Applied to the Evaluation of Medical Imaging Techniques, Inv. Radiology 14 (1979) 109. https://doi.org/10.1097/00004424-197903000-00002
  * (49) J. A. Swets, Measuring the accuracy of diagnostic systems, Science 240 (1988) 1285. https://doi.org/10.1126/science.3287615
  * (50) K. A. Spackman, Signal detection theory: valuable tools for evaluating inductive learning, Proc. 6th Int. Workshop on Machine Learning, Ithaca (1989). https://doi.org/10.1016/B978-1-55860-036-2.50047-3
  * (51) A. P. Bradley, The use of the area under the ROC curve in the evaluation of Machine Learning algorithms, Pattern Recognition 30 (1997) 1145. https://doi.org/10.1016/S0031-3203(96)00142-2
  * (52) F. J. Provost, T. Fawcett, Analysis and Visualization of Classifier Performance: Comparison Under Imprecise Class and Cost Distributions, Proc. KDD-97, Newport Beach (1997). https://aaai.org/Library/KDD/1997/kdd97-007.php
  * (53) F. J. Provost, T. Fawcett, R. Kohavi, The Case against Accuracy Estimation for Comparing Induction Algorithms, Proc. 15th Int. Conf. on Machine Learning (ICML ’98), Madison (1998). https://dl.acm.org/doi/abs/10.5555/645527.657469
  * (54) T. Fawcett, Introduction to ROC analysis, Pattern Recognition Letters 27 (2006) 861. https://doi.org/10.1016/j.patrec.2005.10.010
  * (55) A. Kent, M. M. Berry, F. U. Luehrs, J. W. Perry, Machine literature searching VIII - Operational criteria for designing information retrieval systems, Amer. Doc. 6 (1955) 93. https://doi.org/10.1002/asi.5090060209
  * (56) C. W. Cleverdon, ASLIB Cranfield Research Project: report on the testing and analysis of an investigation into the comparative efficiency of indexing systems (1962). http://hdl.handle.net/1826/836
  * (57) J. A. Swets, Information Retrieval Systems, Science 141 (1963) 245. https://doi.org/10.1126/science.141.3577.245
  * (58) C. W. Cleverdon, The Cranfield Hypotheses, The Library Quarterly 35 (1965) 121. https://doi.org/10.1086/619319
  * (59) C. J. van Rijsbergen, Foundation of evaluation, J. Documentation 30 (1974) 365. https://doi.org/10.1108/eb026584
  * (60) C. J. van Rijsbergen, Information retrieval, Butterworths (1979). http://www.dcs.glasgow.ac.uk/Keith/Preface.html
  * (61) C. D. Manning, P. Raghavan, H. Schütze, Introduction to Information Retrieval Cambridge University Press (2008). https://nlp.stanford.edu/IR-book
  * (62) G. Punzi, Sensitivity of searches for new signals and its optimization, Proc. PhyStat2003, Stanford (2003). https://arxiv.org/abs/physics/0308063v2
  * (63) R. D. Cousins, J. T. Linnemann, J. Tucker, Evaluation of three methods for calculating statistical significance when incorporating a systematic uncertainty into a test of the background-only hypothesis for a Poisson process, Nucl. Instr. Meth. Phys. Res. A 595 (2008) 480. https://doi.org/10.1016/j.nima.2008.07.086
  * (64) G. Cowan, K. Cranmer, E. Gross, O. Vitells, Asymptotic formulae for likelihood-based tests of new physics, Eur. Phys. J. C 71 (2011) 1554. https://doi.org/10.1140/epjc/s10052-011-1554-0
  * (65) C. Adam-Bourdarios et al., The Higgs Machine Learning Challenge, Proc. NIPS 2014 Workshop on High-Energy Physics and Machine Learning (HEPML2014), Montreal (2014). https://hal.inria.fr/hal-01208587
  * (66) M. Sokolova, G. Lapalme, A Systematic Analysis of Performance Measures for Classification Tasks, Information Processing and Management 45 (2009) 427. https://doi.org/10.1016/j.ipm.2009.03.002
  * (67) A. Luque, A Carrasco, A. Martin, J. R. Lama, Exploring Symmetry of Binary Classification Performance Metrics, Symmetry 11 (2019) 47. https://doi.org/10.3390/sym11010047.
  * (68) J. Tague-Sutcliffe, J. Blustein, A statistical analysis of the TREC-3 data, Overview of the Third Text REtrieval Conference (TREC-3), NIST Special Publication 500-226 (1995). https://trec.nist.gov/pubs/trec3/papers/T-SB.pdf
  * (69) D. Harman (editor), TREC-3 Results - Appendix A: Evaluation Techniques and Measures, Overview of the Third Text REtrieval Conference (TREC-3), NIST Special Publication 500-226 (1995). https://trec.nist.gov/pubs/trec3/t3_proceedings.html
  * (70) D. Harman, Overview of the 2nd text retrieval conference (TREC-2), Information Processing and Management 31 (1995) 271. https://doi.org/10.1016/0306-4573(94)00047-7
  * (71) D. Hull, Using statistical testing in the evaluation of retrieval experiments, Proc. 16th ACM SIGIR Conf. (SIGIR 1993), Pittsburgh (1993). https://doi.org/10.1145/160688.160758
  * (72) D. M. Green, General Prediction Relating Yes-No and Forced-Choice Results, J. Acoustical Soc. Am. 36 (1964) 1042. https://doi.org/10.1121/1.2143339
  * (73) D. M. Green, J. A. Swets, Signal detection theory and psychophysics, Wiley (1966).
  * (74) D. J. Goodenough, K. Rossmann, L. B. Lusted, Radiographic applications of signal detection theory, Radiology 105 (1972) 199. https://doi.org/10.1148/105.1.199
  * (75) D. Bamber, The area above the ordinal dominance graph and the area below the receiver operating characteristic graph, J. Math. Psych. 12 (1975) 387. https://doi.org/10.1016/0022-2496(75)90001-2
  * (76) J. A. Hanley, B. J. McNeil, The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology 143 (1982) 29. https://doi.org/10.1148/radiology.143.1.7063747
  * (77) M. Greiner, D. Pfeiffer, R. D. Smith, Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests, Preventive Veterinary Medicine 45 (2000) 23. https://doi.org/10.1016/S0167-5877(00)00115-X
  * (78) X. H. Zhou, D. K. McClish, N. A. Obuchowski, Statistical Methods in Diagnostic Medicine Wiley (2002). https://doi.org/10.1002/9780470317082
  * (79) P. Ray, Y. Le Manach, B. Riou, T. T. Houle, Statistical Evaluation of a Biomarker, Anesthesiology 112 (2010) 1023. https://doi.org/10.1097/ALN.0b013e3181d47604
  * (80) K. Hajian-Tilaki, Receiver Operating Characteristic (ROC) Curve Analysis for Medical Diagnostic Test Evaluation, Caspian Journal of Internal Medicine 4 (2013) 627. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3755824
  * (81) N. M. Adams, D. J. Hand, Comparing classifiers when the misallocation costs are uncertain, Pattern Recognition 32 (1999) 1139. https://doi.org/10.1016/S0031-3203(98)00154-X
  * (82) C. Drummond, R. C. Holte, Explicitly representing expected cost: an alternative to ROC representation, Proc. 6th Int. Conf. on Knowledge Discovery and Data Mining (KDD-00), Boston (2000). https://doi.org/10.1145/347090.347126
  * (83) C. Drummond, R. C. Holte, Cost curves: An improved method for visualizing classifier performance, Mach. Learn. 65 (2006) 95. https://doi.org/10.1007/s10994-006-8199-5
  * (84) J. Davis, M. Goadrich, The relationship between Precision-Recall and ROC curves, Proc. 23rd Int. Conf. on Machine Learning (ICML ’06), Pittsburgh (2006). https://doi.org/10.1145/1143844.1143874
  * (85) T. Saito, M. Rehmsmeier, The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets, PLoS One 10 (2015) e0118432. https://doi.org/10.1371/journal.pone.0118432
  * (86) H. He, E. A. Garcia, Learning from Imbalanced Data, IEEE Trans. Knowl. Data Eng. 21 (2009) 1263. https://doi.org/10.1109/TKDE.2008.239
  * (87) S. Clémençon, N. Vayatis, Ranking the Best Instances, J. Mach. Learn. Res. 8 (2007) 2671. http://www.jmlr.org/papers/v8/clemencon07a.html
  * (88) S. Clémençon, G. Lugosi, N. Vayatis, Ranking and Empirical Minimization of U-statistics, Ann. Statist. 36 (2008) 844. https://doi.org/10.1214/009052607000000910
  * (89) C. Rudin, Y. Wang, Direct Learning to Rank And Rerank, Proc. 21st Int. Conf. on Artificial Intelligence and Statistics (AISTATS2018), PMLR 84 (2018) 775. http://proceedings.mlr.press/v84/rudin18a.html
  * (90) M. J. Pencina, R. B. D’Agostino, Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation, Statistics in Medicine 23 (2004) 2109. https://doi.org/10.1002/sim.1802
  * (91) N. A. Obuchowski, An ROC-Type Measure of Diagnostic Accuracy When the Gold Standard is Continuous-Scale, Statistics in Medicine 25 (2006) 481. https://doi.org/10.1002/sim.2228
  * (92) J. Lambert et al., How to Measure the Diagnostic Accuracy of Noninvasive Liver Fibrosis Indices: The Area Under the ROC Curve Revisited, Clinical Chemistry 54 (2008) 1372. https://doi.org/10.1373/clinchem.2007.097923
  * (93) K. Järvelin, J. Kekäläinen, IR evaluation methods for retrieving highly relevant documents, Proc. 23rd ACM SIGIR Conf. (SIGIR 2000), Athens (2000). https://doi.org/10.1145/345508.345545
  * (94) J. Kekäläinen, K. Järvelin, Using graded relevance assessments in IR evaluation, J. Am. Soc. Inf. Sci. Tech. 53 (2002) 1120. https://doi.org/10.1002/asi.10137
  * (95) K. Järvelin, J. Kekäläinen, Cumulated gain-based evaluation of IR techniques, J. ACM Trans. on Inf. Sys. (TOIS) 20 (2002) 422. https://doi.org/10.1145/582415.582418
  * (96) P. D. Turney, Cost-sensitive classification: empirical evaluation of a hybrid genetic decision tree induction algorithm, J. Art. Intell. Res. 2 (1994) 369. https://doi.org/10.1613/jair.120
  * (97) C. Drummond, R. C. Holte, Exploiting the Cost (In)sensitivity of Decision Tree Splitting Criteria, Proc. 17th Int. Conf. on Machine Learning (ICML ’00), Stanford (2000). https://www.aaai.org/Library/Workshops/2000/ws00-05-009.php
  * (98) B. Zadrozny, C. Elkan, Learning and making decisions when costs and probabilities are both unknown, Proc. 7th Int. Conf. on Knowledge Discovery and Data Mining (KDD-01), San Francisco (2001). https://doi.org/10.1145/502512.502540
  * (99) C. Elkan, The Foundations of Cost-Sensitive Learning, Proc. 17th Int. Joint Conf. on Artificial Intelligence (IJCAI-01), Seattle (2001). https://dl.acm.org/doi/abs/10.5555/1642194.1642224
  * (100) B. Zadrozny, J. Langford, N. Abe, Cost-sensitive learning by cost-proportionate example weighting, Proc. 3rd IEEE Int. Conf. on Data Mining (ICDM-2003), Melbourne (2003). https://doi.org/10.1109/ICDM.2003.1250950
  * (101) T. Fawcett, ROC graphs with instance-varying costs, Pattern Recognition Letters 27 (2006) 882. https://doi.org/%****␣chep19wdr_v2_arxiv.tex␣Line␣2350␣****10.1016/j.patrec.2005.10.012
  * (102) B. Zadrozny, C. Elkan, Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers, Proc. 18th Int. Conf. on Machine Learning (ICML ’01), Williamstown (2001). http://cseweb.ucsd.edu/~elkan/calibrated.pdf
  * (103) C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On calibration of modern neural networks, Proc. 34th Int. Conf. on Machine Learning (ICML ’17), Sydney (2017). https://arxiv.org/abs/1706.04599
  * (104) F. Sanders, On Subjective Probability Forecasting, J. Applied Meteorology 2 (1963) 191. https://doi.org/10.1175/1520-0450(1963)002%3C0191:OSPF%3E2.0.CO;2
  * (105) A. H. Murphy, A New Vector Partition of the Probability Score, J. Applied Meteorology 12 (1973) 595. https://doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2
  * (106) S. Lichtenstein, B. Fischhoff, L. Phillips, Calibration of Probabilities: the State of the Art, DARPA Tech. Rep. DDI-3 (1976). https://apps.dtic.mil/dtic/tr/fulltext/u2/a033248.pdf
  * (107) I. Mason, A model for assessment of weather forecasts, Australian Meteorological Magazine 30 (1982) 291. http://www.bom.gov.au/jshess/docs/1982/mason.pdf
  * (108) A. H. Murphy, R. L. Winkler, A General Framework for Forecast Verification, Monthly Weather Review 115 (1987) 1330. https://doi.org/10.1175/1520-0493(1987)115%3C1330:AGFFFV%3E2.0.CO;2
  * (109) World Meteorological Organization, Standardized Verification System (SVS) for Long-Range Forecasts (LRF), Attachment II.8 to WMO Manual N. 485 (2010). https://www.wmo.int/pages/prog/www/DPFS/documents/485_Vol_I_en_colour.pdf
  * (110) D. J. Spiegelhalter, Probabilistic prediction in patient management and clinical trials, Statist. Med. 5 (1986) 421. https://doi.org/10.1002/sim.4780050506
  * (111) F. E. Harrell, K. L. Lee, D. B. Mark, Tutorial in Biostatistics – Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors, Statist. Med. 15 (1996) 361. https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4
  * (112) A. Valassi, Optimising HEP parameter fits through MC weight derivative regression, CHEP2019 presentation slides, Adelaide. https://doi.org/10.5281/zenodo.3523164