# Calibration of Machine Reading Systems at Scale

Shehzaad Dhuliawala1, Leonard Adolphs1, Rajarshi Das2, Mrinmaya Sachan1

1ETH Zürich, 2University of Massachusetts Amherst
<EMAIL_ADDRESS>
###### Abstract

In typical machine learning systems, an estimate of the probability of the
prediction is used to assess the system’s confidence in the prediction. This
confidence measure is usually uncalibrated; i.e. the system’s confidence in
the prediction does not match the true probability of the predicted output. In
this paper, we present an investigation into calibrating open setting machine
reading systems such as open-domain question answering and claim verification
systems. We show that calibrating such complex systems which contain a
discrete retrieval and deep reading components is challenging and current
calibration techniques fail to scale to these settings. We propose simple
extensions to existing calibration approaches that allow us to adapt these
calibrators to these settings.

Our experimental results reveal that the joint calibration of the retriever
and the reader outperforms the reader calibrator by a significant margin. We
also show that the calibrator can be useful for selective prediction, e.g.,
when question answering systems are posed with unanswerable or out-of-the-
training distribution questions.

##  1 Introduction

With recent advances in machine reading, there has been a surge of interest in
practical applications of the technology such as open-domain question
answering Karpukhin et al. (2020); Lee et al. (2019) and claim verification
Thorne et al. (2018b). Due to various scale limitations in practical settings,
these systems are seldom trained end-to-end. Such systems typically make use
of a Retriever alongside a Reader – the evidence is first retrieved from a
large corpus and is then used by a machine reading model to provide an answer.

As these systems are increasingly being deployed in the real world, it is
important that they are not only accurate but also trustworthy. A way to make
these systems trustworthy is to indicate when they are likely to be incorrect
by providing a calibrated confidence measure in addition to the prediction.

A naive solution for this is to use the system’s output probability as the
confidence. However, this confidence score is often uncalibrated Kuleshov and
Liang (2015); Guo et al. (2017); i.e. it is not representative of the true
correctness likelihood.111For a perfectly calibrated system, given 100 answer
predictions, each with a confidence of 0.7, we expect that 70 should be
correct.

Previous work Jiang et al. (2020); Jagannatha and Yu (2020); Desai and Durrett
(2020) has shown that large language models especially suffer from
miscalibration. Thus, several methods have been proposed to calibrate language
models based on gradient-based calibration methods such as temperature scaling
Guo et al. (2017) and feature-based forecasters Kuleshov and Liang (2015).
While gradient-based calibration is intuitive and easy to implement, feature-
based forecasters require manual feature engineering.

In this work, we contribute a simple method to calibrate practical Retriever
\- Reader machine reading pipelines. These systems typically include a hard
retrieval step which makes gradient-based calibration infeasible. Thus, we
make use of the Gumbel machinery Jang et al. (2017); Maddison et al. (2017);
specifically the Gumbel top-$K$ procedure of Vieira (2014); Xie and Ermon
(2019) to obtain a differentiable sampling routine for the retrieval step.
This sampler can then be combined with any gradient-based calibration
technique such as Platt’s scaling.

We conduct experiments on three different models – a generative and extractive
open-domain question answering model and a claim verification model. We find
that calibrating the Retriever and the Reader jointly is better than
calibrating only the Reader or the Retriever . We also show that our approach
can produce calibrated scores that can be used to selectively abstain from
answering questions that are contrived or ill-posed or questions that are out-
of-the-training distribution. Finally, we also demonstrate how the calibration
of such a system works – the calibration techniques lower the confidence of
the predicted answer when the question is unanswerable or when the retriever
is not able to retrieve any relevant evidence for answering the question. We
perform our analysis on different types of unanswerable questions and show
that incorporating the confidence of the Retriever along with the Reader can
improve the confidence estimate of the answer.

##  2 Preliminaries

###  2.1 Machine Reading at Scale

Figure 1: General architecture of the two machine reading systems considered
in this paper. a) Claim verification (top half) and b) Open-domain QA (bottom
half). The systems follow the same architecture and are composed of a
retriever and a reader. Given the query, the retriever retrieves a set of K
documents from the corpus along with scores for each of them. The reader then
takes these as input and produces the output: a veracity label for claim
verification and an answer span for the QA model. This can be seen as a
probabilistic model with latent retrieval ($D_{k}$ shown in red). The goal of
this paper is to calibrate the final output probabilities $P(a|q)$.

Practical real-world machine reading systems such as open-domain question
answering systems Chen et al. (2017) Karpukhin et al. (2020) Izacard and Grave
(2020b) or claim verification systems Hanselowski et al. (2018) rely on an
information retrieval (IR) component called a Retriever to reduce the search
space over a large corpus of documents. This smaller set of documents is then
passed to a Reader model that reasons over the text and produces an answer.
This setting, where the Reader is not given labeled documents is referred to,
in the literature, as an open-domain setting.

We now proceed to formally define the pipeline for a machine reading system in
the open-domain setting.

Let $\mathcal{D}=\\{d_{1},\dots,d_{N}\\}$ denote the given corpus of
documents. Let $q$ denote the user query (a question or a claim). We denote
the answer to the question or the veracity label of the claim as $a$. The
retreiver model takes in $q$ and scores all the documents $d\in\mathcal{D}$ to
produce a set of scores:

$\displaystyle\textsc{Retriever}(d_{1},\dots,d_{N}|q)\longrightarrow
S_{d_{1}},\dots,S_{d_{N}}$ (1)

This formulation of the Retriever is generic. This allows our method to work
with any IR model such as the traditional BM25 model Wikipedia contributors
(2004) to more modern methods such as Dense Passage Retrieval (DPR) by
Karpukhin et al. (2020).

The documents are then sorted based on the scores and the $k$ top-scoring
documents are chosen. We call this set of top-$K$ documents $D_{k}$. $D_{k}$
is then given to a Reader model which extracts the answer or predicts a
veracity label for the claim, $a$. The Reader can vary depending on the task.
For extractive QA, the Reader produces a score for each span ($s_{i}$) in the
documents provided to it.

$\displaystyle\textsc{Reader }(q,D_{k})\longrightarrow
S^{Read}(s_{i}),s_{i}\in D_{k}$ (2)

In claim verification, the Reader produces a score for each veracity label:
SUPPORTED, REFUTED or NOT ENOUGH INFO, which indicate whether the claim can be
verified by the given set of documents.

$\displaystyle\textsc{Reader }(D_{k},q)\longrightarrow$ $\displaystyle
S_{\textsf{SUPPORTED}},$ $\displaystyle S_{\textsf{REFUTED}},$ $\displaystyle
S_{\textsf{NOT ENOUGH INFO}},$

###  2.2 Calibration

We summarize below the calibration framework Kuleshov and Liang (2015) in the
context of machine reading. Given a query $q$, true output $a$, model output
$\hat{a}$, and probability $P(\hat{a}|q)$ calculated over this output, a
perfectly calibrated model satisfies the following condition:

$\displaystyle\mathbb{P}(\hat{a}=a|P(\hat{a}|q)=p)=p\hskip 5.69046pt\forall
p\in[0,1]$ (3)

In simple words, for the confidence estimate $P(\hat{a}|q)$ to be calibrated,
we require that $P(\hat{a}|q)$ follows the unknown true probability
distribution $\mathbb{P}$.

In a multi/binary class setting, a calibrator can be learned to map the output
distribution to a calibrated confidence score. However, in a machine reading
setting, the space of possible documents retrieved and answers contained in
them is usually very large. Thus, we only focus on a specific event set $I(q)$
of interest. The event set $I(q)$ can be defined using the outputs relevant to
the deployment requirements of the machine reading model. In our work, we
consider all answer candidates in the retrieved set of documents $D_{k}$:
$I(q)=\\{a|a\in\operatorname*{arg\,max}\limits_{D_{k}}P(\hat{a}|D_{k},q)\\}$

###  2.3 Measuring Calibrated-ness

Calibration can be measured by computing the difference in expectation between
confidence scores and accuracies.

$\displaystyle\mathbb{E}_{P(\hat{a}|q)}\Big{[}\mathbb{P}(\hat{a}=a|P(\hat{a}|q)=p)-p\Big{]}$
(4)

This is known as expected calibration error (ECE) Naeini et al. (2015).
Practically, ECE is estimated by partitioning the predictions in M equally
spaced bins $(B_{1}\dots B_{M})$ and taking the weighted average of the
difference between the average accuracy and average confidence of the bins.

$\displaystyle\textsc{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|\textit{acc}(B_{m})-\textit{conf}(B_{m})\right|$
(5)

#### Reliability Diagrams

Another common tool to visualize model calibration is a reliability diagram. A
reliability diagram plots sample accuracy as a function of confidence for each
bin. If a model is perfectly calibrated, the confidence and accuracy bars
should be identical.

###  2.4 Calibration methods

The general algorithm used for calibrating classification models involves
transforming the logits produced by the model. The parameters for this
transformation are trained on a held-out calibration set
$C=\\{(q_{i},a_{i})\\}_{i=1}^{N}$. This method has been shown to improve the
model’s ECE without a significant loss in accuracy. In our work, we use
negative log-likelihood (NLL) to tune a model $P(a|q)$ to be a good
probability estimate of the output answers:

$\displaystyle l_{\theta}=-\sum_{i=1}^{N}log(P(a_{i}|q_{i}))$ (6)

ML theory guarantees that NLL is minimized if and only if $P(a_{i}|q_{i})$
recovers the ground-truth conditional distribution $\mathbb{P}(a|q)$. In the
following part of this section, we describe some of these key methods.

#### Temperature Scaling

Temperature scaling Guo et al. (2017) is one of the simplest methods for
calibration and has been shown to be very effective. Temperature scaling
allows the logits of the system’s output ($Z$) to be scaled by a single
temperature value $\tau$. This scaling is done before the computation of the
softmax.

$\displaystyle Y=\texttt{softmax}(Z/\tau)$ (7)

We optimize $\tau$ by maximizing $\mathcal{L}_{\theta}$ on the dev set.

#### Temperature prediction

The temperature prediction approach Kumar and Sarawagi (2019) extends
temperature scaling to a gradient-based approach. The output logits of the
classifier are featurized and passed through an MLP which predicts a
temperature value. This temperature value is used to scale the logits. In
contrast to temperature scaling which learns one temperature parameter for
each example, in this approach, a new temperature value can be learned for
each example.

$\displaystyle\frac{1}{\tau_{i}}=\sigma(\texttt{MLP}(Z_{i}))$ $\displaystyle
Y_{i}=\texttt{softmax}(Z_{i}/\tau_{i})$

#### Forecasters

Forecasters were introduced to calibrate structured prediction models Kuleshov
and Liang (2015); Jagannatha and Yu (2020). The forecaster approach introduces
a feature-rich calibration model that uses various features of the model such
as its logits and various uncertainties estimated to predict the confidence
score. This approach generally only produces a calibrated score over a smaller
set of candidate predictions referred to as the interest set $I(.)$ Previous
work has successfully used gradient boosted decision trees (XGB) as
forecasters.

##  3 Calibration of Machine Reading Systems

Previous work has looked at calibration in the aspect of machine reading
Jagannatha and Yu (2020); Jiang et al. (2020). However, these works do not
consider the open setting in which the evidence document for each query is not
provided. We are interested in determining the calibrated probability
distribution of the system, $\mathbb{P}(a|q)$. In the first set of methods, we
do this by calibrating the confidence of the model $P(\hat{a}=a|q)$. For a
machine reading system,

$\displaystyle
P(\hat{a}=a|q)=\sum_{D_{k}\in\mathcal{D}}\underbrace{P(D_{k}|q)}_{\text{conf
of {Retriever }}}\times\underbrace{P(\hat{a}=a|q,D_{k})}_{\text{conf of
{Reader }}}$ (8)

We discuss three possible ways to calibrate $P(\hat{a}=a|q)$

##### Only reader

One way to calibrate $P(\hat{a}=a|q)$ is to assume that the Retriever is
perfectly accurate and perfectly calibrated. We refer to his approach in our
results as Only reader. In this approach, we only calibrate
$P(\hat{a}=a|q,D_{k})$. We can use all the previously mentioned calibration
approaches for this task. Indeed, this is the approach taken by Jagannatha and
Yu (2020); Jiang et al. (2020). For extractive QA, the output logits lie over
all the possible text spans, while for fact verification we have a single
logit per class. In our experiments, we show that this leads to subpar
calibration.

##### Individually calibrated

We explore another possible approach where we calibrate $P(D_{k}|q)$ and
$P(\hat{a}=a|q,D_{k})$ indivdually using the objectives of the Retriever and
Reader individually. We refer to this approach as Individually calibrated.

This happens in two steps, first the confidence of the retriever,
$P(D_{k}|q)$, is calibrated. The confidence of the retriever is then fixed and
the we calibrate the only the confidence of the reader,
$P(\hat{a}=a|q,D_{k})$. Finally, the confidence of the systems is computed as
$P(\hat{a}=a|q)$ in Equation 8.

We posit that this method results in subpar calibration owing to the Retriever
not having gold labels and is calibrated using the less accurate distance
supervision objective.

##### Jointly Calibrated

Finally, we discuss our approach to calibrate the entire system using the
final objective of the system. We refer to this approach as Jointly
Calibrated. In this approach, we treat the documents retrieved by the
retriever as a latent variable
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}D_{k}}$.

We define our calibration likelihood in eqn. 6 as:

$\displaystyle\mathcal{L}_{\theta}=\sum_{q}\sum_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}D_{k}}\in\mathcal{D}}P({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}D_{k}}|q)P(\hat{a}=a|q,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}D_{k}})$
(9)

Clearly, it is infeasible to marginalize over all possible $D_{k}$ (subsets of
the corpus of size $k$). Thus, we propose a diffentiable sampler for $D_{k}$:

$\displaystyle\mathcal{L}_{\theta}=\sum_{D_{k}\sim
P_{\theta}(D_{k}|q)}P(\hat{a}=a|q,D_{k})$ (10)

To make our calibrator differentiable, we apply the Gumbel–softmax trick
Maddison et al. (2017) and, in particular, its extension to top-K subset
selection Vieira (2014); Xie and Ermon (2019). The Gumbel-top-$K$ trick
generalizes Gumbel–softmax and essentially repeats the Gumbel trick $K$ times
until we have a set of the desired size. We describe the approach further in
Appendix A.

##  4 Experimental Details

#### Open Domain Question Answering

##### Extractive

We test the described calibration techniques on the open domain QA, using the
pretrained models from Karpukhin et al. (2020). We perform our experiments on
the Natural Questions (NQ) dataset Kwiatkowski et al. (2019). We randomly
split our validation set into two equal parts which we will call calib and
valid. We use these splits for training and tuning our calibration models
respectively. We use the test set of NQ as our test set (test). During
inference, we use the Retriever to retrieve top 10 documents which are passed
to the Reader to extract the answer.

##### Generative

We use the FiD model proposed by Izacard and Grave (2020b) for our calibration
experiments. As generative models don’t produce a confidence over multiple
answers, we use the approach described by Jiang et al. (2020) to generate an
interest set. First we calculate the probabilities of the first generated
tokens. We mask out any tokens not in the retrieved passages. Next we, select
the top $R$ tokens we find their location in the passages and calculate the
probability of all continuing spans up to a certain length (of 10 tokens). We
then keep the top-10 scoring spans in our candidate set.

#### Claim Verification

For the claim verification task, we experiment on the FEVER dataset Thorne et
al. (2018a). We use a recently published state-of-the-art model, Liu et al.
(2020), in our calibration experiments. For every test example, we retrieve 5
sentences that are provided to the claim verification model to ascertain the
veracity of the claim.

#### Temperature based methods

For the Reader -Retriever setup we require two temperature parameters t1 and
t2 for the Retriever and Reader respectively. We use gradient descent to
optimize t1 and t2 by maximizing $\mathcal{L}_{\theta}$ on the valid set. For
temperature prediction we add a 2-layer MLP that predicts t1 and t2 for each
example. Once again, the optimization is performed on valid.

#### Forecaster

For our forecaster, we use gradient boosted decision trees. We train the model
to perform binary classification with the model’s accuracy as the objective,
i.e., if the model’s prediction was correct, we assign a positive label to the
example. We do not experiment extensively with various features as previous
work has done and instead just use the raw logit scores. Similar to Jagannatha
and Yu (2020), we create the interest set of the forecaster by choosing the
top-3 predictions of the model, i.e., we choose the top-$3K$ choices of the
Retriever over which we evaluate our Reader and choose the top-3 choices.

#### Gumbel top-$K$

For the Gumbel top-$K$ approach required to train the vector scaling and
temperature prediction models, we start out with a high temperature value
$T_{0}$ which we linearly decrease to $T_{\infty}$. We treat these parameters
as hyperparameters.

##  5 Results

Figure 2: The Reader is highly confident about its prediction, but when we
incorporate the confidence of the evidence from the Retriever which can
identify that the sentences are irrelevant to the claim, the confidence of the
prediction can be better calibrated.

Figure 3: Reliability plots for uncalibrated versus Individually calibrated
versus Jointly calibrated on the Generative QA task using Temperature Scaling.
Blue bars denote bin accuracy, red bars denote bin confidence, difference
indicates miscalibration.

We now present the results of the various calibration techniques in Table 1.
We also plot the reliability diagrams in Figure 3. We compare all the
described calibration algorithms in the three settings discussed. As can be
seen, in all the cases there is a benefit to jointly calibrate the Retriever
and Reader . We give some reasons for why this setting works best in the
discussion section below.

###  5.1 Discussion

#### Calibrating only the Reader

In all our experiments we show that calibrating the Reader alone performs
worse. We believe that this is because, at train time, the Reader is only
trained on positive documents. This makes the Reader overconfident on
documents that don’t have the answer. This phenomenon has been also been
discussed in Clark and Gardner (2017). We show an example in Fig 2. We also
notice that adding the Retriever helps more in the QA task than for claim
verification. We posit that this is because in the open-domain setting, the QA
passage Retriever has a lower accuracy than the sentence Retriever for claim
verification.222QA, hits@10:0.77, CV, hits@5:0.94
We use top-10 passages for QA and top-5 sentences for claim verification.

#### Calibrating Individually

Our experimental results show that in almost all cases, it is detrimental to
individually calibrate the Reader and Retriever . We believe that this is due
to the Retriever ’s accuracy being misaligned with the final objective. In
several cases, such as in QA, supervision for the Retriever is not provided,
and instead a distant supervision objective is used where the document is
marked as positive when it contains the answer string. We show an example in
Figure 1 where, for the question "Who won the women’s worldcup in 2017", a
document saying "world cup to be held in England" would be assigned a positive
label as it contains the answer string "England". This mismatch in accuracy
for the Retriever can result in an incorrectly calibrated system. This problem
has been well discussed in the literature and more recently by Izacard and
Grave (2020a)

##### Reliability plots

As can be seen from Figure 3, miscalibration results from the model being
overconfident. This is evident with the blue bars being lower than the red –
model accuracy is less than model confidence for several bins. We also notice
that all calibration techniques address this overconfidence by rescaling the
output distribution.

Task | Setting | Uncalibrated | Temp scaling | Temp predictor | Forecaster
---|---|---|---|---|---
Generative QA | Generator |  | 47.31 | 45.22 | 5.40
Individually | 55.1 | 33.47 | 35.31 | 11.35
Jointly |  | 3.75 | 3.56 | 4.21
Extractive QA | span extractor |  | 8.56 | 8.11 | 4.68
Individually | 37.1 | 10.32 | 7.42 | 12.74
Jointly |  | 2.94 | 2.38 | 2.96
Claim Verification | Claim verifier |  | 1.42 | 1.64 | 1.66
Individually | 7.02 | 16.35 | 23.6 | 26.73
jointly |  | 1.15 | 1.30 | 0.98

Table 1: Values in % ECE, ($\downarrow$ is better). individually denotes the
retriever and reader have been calibrated separately, while jointly indiciates
that calibration on a joint objective.

##  6 Analysis

Next, we attempt to verify the following claims:

C1: The existing approach for calibrating only the reader doesn’t result in a
good calibration of the overall system. Jointly calibrating the reader and the
retriever model is better.

C2: Calibrated ODQA systems do better selective prediction when they are
allowed to not provide answers to some questions.

C3: Calibrated ODQA systems are better at handing domain shifts in questions
at test time.

C4: Calibrated ODQA systems are better at handling unanswerable questions at
test time.

###  6.1 Selective Prediction for Machine Reading

One key use of confidence estimation is selective prediction. The selective
prediction setting allows the model to decide whether it wants to make a
prediction or abstain on each given test point. Selective prediction has been
a long-standing research area in machine learning Chow (1957); El-Yaniv et al.
(2010).

We investigate how different calibration methods perform on the task of
selective prediction. There have been some recent efforts to understand
selective prediction for QA models with regard to domain shift; Kamath et al.
(2020) investigate how forecasters can be effectively used as calibrators to
predict when a model should abstain from providing an answer. We further this
investigation in the open-domain setting to see if different calibration
techniques can improve the model’s performance on the selective prediction
task. The evaluation metric used to judge a model’s effectiveness in learning
to abstain is the area under the risk-coverage curve.

Given an input $q$, the model’s prediction $\hat{a}$ along with the confidence
of the prediction $P(\hat{a}=a|q)$ and a threshold $\tau$, our model predicts
the the answer $\hat{a}$ if $P(\hat{a}=a|q)\geq\tau$. For the test set and a
value of $\tau$ there is an associated risk: the fraction of the test set that
the model answers incorrectly, and coverage: the fraction of the test set the
model makes a prediction on. As $\tau$ increases, so do the risk and coverage.
We plot risk vs coverage as $\tau$ varies and report the area under the risk-
coverage curve (AURC). Our results are shown in table 2. We can infer from the
results that all calibration methods help reduce the AURC to some extent
however the Temperature predictor is able to perform the best on extractive QA
while the forecaster is the best on claim verification and generative QA
indicating that improving model calibration can also help for the task of
selective prediction in the setting of machine reading.

Task | Setting | AURC
---|---|---
Extractive QA | Uncalibrated | 47.39
Forecaster | 44.21
Temp scaling | 43.68
Temp prediction | 42.64
Best possible | 26.71
Generative QA | Uncalibrated | 53.57
Forecaster | 39.10
Temp scaling | 44.85
Temp prediction | 43.21
Best possible | 22.25
Claim Verification | Uncalibrated | 11.04
Forecaster | 3.53
| Temp scaling | 10.96
| Temp prediction | 9.99
| Best possible | 2.76
Table 2: Area under Risk-Coverage curve. $\downarrow$ is better

##### Domain Adaptation

With the increasing use of machine reading systems in the wild, a common
problem encountered by them is that they are not resilient to inputs that do
not come from the distribution of the data they were trained on. A method of
selective prediction is often employed, where the model the model can abstain
from answering the question. (Kamath et al., 2020) show that training a
seperate model to distinguish between in- and out-of- domain helps in doing
selective prediction. We show that a well calibrated model is able to perform
better on the selective prediction setting even when the calibration step has
no access to an out of distribution dataset. In our experiments, we calibrate
a trained on NQ FiD model on the FiD dev set. We then evaluate the performance
on different splits which contain varying percentages of out-of-distribution
data (TriviaQA) Joshi et al. (2017). We plot the AURC with different splits
containing different percentages of OOD questions in figure 4. We notice that
an uncalibrated model gets significantly worse when the amount of OOD samples
are added. However calibation techniques are able to mitigate this and are
able to maintain a steady AURC with increasing OOD samples. We found that the
Forecaster (XGB) performing the best in this evaluation.

Figure 4: Area under risk coverage curves (AURC) using different calibration techniques Question |  Retrieved passage and answer |  Model confidences
---|---|---
what harry potter movie came out in 2008 |  …harry potter and the half - blood prince is a 2009 fantasy film … |  Uncalibrated: $0.99\times 0.79=0.78$
Temp Scaling:$0.98\times 0.54=0.53$
Temp Prediction:$0.85\times 0.43=0.36$
Forecaster: $0.31$
who played the joker in the dark (k)night rises |  …he was played by australian actor heath ledger … |  Uncalibrated:$0.99\times 0.57=0.56$
Temp Scaling: $0.98\times 0.40=0.39$
Temp Prediction: $0.73\times 0.33=0.24$
Forecaster: $0.28$
who do you think you are book pdf |  Book of Ryan: …comedian cedric the entertainer makes a cameo … |  Uncalibrated: $0.97\times 0.08=0.08$
Temp Scaling:$0.78\times 0.01=0.08$
Temp Prediction:$0.63\times 0.01=0.06$
Forecaster: $0.03$
when it is winters in delhi how will the weather be in chennai |  Chennai: Climate: …coolest part of the year is january … |  Uncalibrated:$0.99\times 0.43=0.43$
Temp Scaling: $0.96\times 0.21=0.20$
Temp Prediction: $0.85\times 0.21=0.18$
Forecaster: $0.10$
how are the suburbs of paris different than those of most canadian cities |  …Suburb: land use patterns in canadian suburbs are often more mixed … |  Uncalibrated:$0.98\times 0.38=0.37$
Temp Scaling: $0.96\times 0.17=0.16$
Temp Prediction: $0.81\times 0.19=0.15$
Forecaster: $0.07$
Table 3: Examples of unanswerable questions. We show how each calibration
approach is able to lower the confidence of the incorrect answer.

###  6.2 Effect of $K$ on Calibration

We analyze how the ECE values are affected by increasing the number of
documents the reader model consumes. We run this experiment on the open-domain
extractive QA task. Along with $K=10$, we evaluate on $K=20$ and $K=50$. As
can be seen in Table 4 all our methods scale well when we increase the value
of $K$. Platt’s scaling and temperature scaling are able to maintain low ECE
scores even when the Reader collates the answer over multiple documents. We
believe that owing to the simplicity of the approach, these calibration
methods are able to adapt well to different settings while in contrast, the
forecaster and temperature predictor being more complex models struggle with
this. The forecaster stands at a disadvantage here owing to the fact that the
number of features given to the forecaster scales with $K$ and we believe that
this could also have an impact on its inability to scale to larger values of
$K$.

Calibration model | $K=10$ | $K=20$ | $K=50$
---|---|---|---
Temperature Scaling | 2.94 | 2.83 | 2.97
Platt’s Scaling | 2.88 | 2.21 | 2.32
Temperature Predictor | 2.38 | 2.83 | 4.90
Forecaster | 2.96 | 4.62 | 5.10
Table 4: How ECE changes with no of documents $K$ Question and Passage |  Model confidences
---|---
how many episodes of corrie has there been
Clarkson (TV series):…The series ran for ten episodes, during a weekly airing schedule … |  Uncalibrated: $0.99\times 0.49=0.49$
Temp Scaling:$0.99\times 0.23=0.22$
Temp Prediction:$0.90\times 0.18=0.16$
Forecaster: $0.06$
what is in a pat o brien hurricane
Sucker hole: …Sucker hole is a colloquial term referring to a short spate of good weather … |  Uncalibrated: $0.99\times 0.49=0.49$
Temp Scaling:$0.99\times 0.24=0.23$
Temp Prediction:$0.95\times 0.21=0.20$
Forecaster: $0.07$
Table 5: Examples of questions where the Retriever fetches the wrong passages

##### Unanswerable Questions

Another challenge that a user facing QA system can encouter is malformed
questions. These include questions that were not probably questions, for
example a user query containing a named entity which is a question or a
question that cannot be answered because it contains a false premise. To
investigate if a calibrated model can be used to abstain from answering such
questions, we evaluate our approaches on the set of unanswerable questions
proposed by Asai and Choi (2020). We plot Risk-Coverage curves for different
calibration techniques in Figure 5. We find all calibration techniques help in
performing selective prediction when compared to an uncalibrated model.
However, the Forecaster outperforms all other methods. To exemplify how
calibration techniques can help the model abstain we provide a few examples in
table 3. These results come from running an extractive QA model on the set of
unanswerable questions. We only evaluate using the top passage for simplicity.
It can be seen that all calibration techniques are able to lower the
confidence of the predicted answer in cases when the question is unasnwerable.
The first two questions provide examples of questions containing a false
premise: No Harry Potter movie came out in 2008 and there was no Joker
character in The Dark Knight Rises. In these cases, both the Reader and
Retriever are confident about their prediction. The third question exemplifies
a query that is not a question. Here it can be seen that the Reader still
places a high confidence in its provided answer, however, the Retriever
assigns a very low score to its retrieved passage. This provides a great
insight into examples where using the retriever score can further help in
calibration. The last two questions are examples of types of questions
generally not part of the dataset. They either require more sophisticated
reasoning or are non-factoidal. Here we can observe that eventhough the Reader
is confident, the retrieved passages are assigned a much lower score. We find
that by using the forecaster on generative QA we reduce the confidence of an
incorrectly answered question by 46% while decreasing the confidence of a
correctly answered question by 38%.

Figure 5: Risk coverage curve for unanswerable questions

#####  Retriever mistakes

Another common seen scenario in an open domain setting is when the Retriever
is not able to provide any relevant passages. In such cases, because the
Reader is generally trained on only correct passages, it still produces a high
confidence for the incorrect answer. We show that calibration methods that
take into account the Retriever confidences can mitigate this by lowering the
confidence of the answer. We provide few such examples in table 5

##  7 Related Work

Obtaining calibrated confidence scores for NLP tasks has recently gained
attention. Jagannatha and Yu (2020) and Jiang et al. (2020) study how
forecasters can be used and what features can be useful to calibrate the
confidence of QA models. Kamath et al. (2020) study calibration in the context
of selective answering, i.e., learning when QA models should abstain from
answering questions. They show that training a forecaster to predict the
model’s confidence can perform well when facing a distributional shift. Su et
al. (2019) also investigate selective answering using a probe in the model to
determine the model’s confidence.

Also related to our work is uncertainity estimation Gal and Ghahramani (2016);
Lakshminarayanan et al. (2017) as model uncertainities can be seen as
confidence scores. In NLP, Xiao and Wang (2019) propose an approach to
characterize model and data uncertainties for various NLP problems. Wang et
al. (2019) use uncertainty estimation for confidence estimation in MT. Dong et
al. (2018) study confidence estimation for semantic parsing. We are the first
to study calibration of open-domain machine reading systems.

Our Gumbel-topk inspired approach of jointly calibrating the Reader and
Retriever together is interesting in the light of recent open domain QA
methods such as Lewis et al. (2020) and Sachan et al. (2021) that train the
entire system jointly. As a future work we would also want to compare our
approach with these other end-to-end training approaches for the task of
calibration.

##  8 Discussion and Conclusion

In this paper, we analyzed how various calibration techniques can be adopted
to open-domain machine reading systems which are now being used in user-facing
scenarios. We showed that in such systems that include a retriever,
calibrating the system’s confidence is not trivial and we proposed a technique
that allows calibration of the system jointly. Finally, we also provide an
analysis on how the calibration techniques can help the model abstain from
answering a question especially in settings where the model’s prediction can
be incorrect due to malformed or out-of-domain questions.

While we do not find evidence to prove that one calibration method (e.g. a
gradient-based method) is better that the other (e.g. a forecaster approach),
it would be important to investigate these questions with more nuanced human
studies.

## Ethical Considerations

In recent years, deep learning approaches have been the main models of choice
for practical machine reading systems. However, these systems are often
overconfident in their predictions. A calibrated confidence score would help
system users better understand the system’s decision making. Our work
introduces a simple and general way for calibrating these systems. While our
models are not tuned for any specific application domain, our methods could be
used in sensitive contexts such as legal or healthcare settings, and it is
also essential that any work using our method undertake additional quality
assurance and robustness testing before using it in their setting. The
datasets used in our work do not contain any sensitive information to the best
of our knowledge.

## Acknowledgments

We gratefully acknowledge feedback and helpful comments from the various
anonymous reviewers. This work is partially supported by a Responsible AI
grant by the Haslerstiftung and by an ETH Grant (ETH-19 21-1). SD is also
partially supported by the IBM PhD fellowship.

## References

  * Asai and Choi (2020) Akari Asai and Eunsol Choi. 2020.  Challenges in information seeking qa: Unanswerable questions and paragraph retrieval.  _arXiv preprint arXiv:2010.11915_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_.
  * Chow (1957) Chi-Keung Chow. 1957.  An optimum character recognition system using decision functions.  _IRE Transactions on Electronic Computers_ , (4):247–254.
  * Clark and Gardner (2017) Christopher Clark and Matt Gardner. 2017.  Simple and effective multi-paragraph reading comprehension.  _arXiv preprint arXiv:1710.10723_.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  _arXiv preprint arXiv:2003.07892_.
  * Dong et al. (2018) Li Dong, Chris Quirk, and Mirella Lapata. 2018.  Confidence modeling for neural semantic parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 743–753, Melbourne, Australia. Association for Computational Linguistics.
  * El-Yaniv et al. (2010) Ran El-Yaniv et al. 2010.  On the foundations of noise-free selective classification.  _Journal of Machine Learning Research_ , 11(5).
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1050–1059, New York, New York, USA. PMLR.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  _arXiv preprint arXiv:1706.04599_.
  * Hanselowski et al. (2018) Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018.  Ukp-athene: Multi-sentence textual entailment for claim verification.  _arXiv preprint arXiv:1809.01479_.
  * Izacard and Grave (2020a) Gautier Izacard and Edouard Grave. 2020a.  Distilling knowledge from reader to retriever for question answering.  _arXiv preprint arXiv:2012.04584_.
  * Izacard and Grave (2020b) Gautier Izacard and Edouard Grave. 2020b.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_.
  * Jagannatha and Yu (2020) Abhyuday Jagannatha and Hong Yu. 2020.  Calibrating structured output predictors for natural language processing.  _arXiv preprint arXiv:2004.04361_.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know when language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective question answering under domain shift.  _Association of Computational Linguistics_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Kuleshov and Liang (2015) Volodymyr Kuleshov and Percy S Liang. 2015.  Calibrated structured prediction.  _Advances in Neural Information Processing Systems_ , 28:3474–3482.
  * Kumar and Sarawagi (2019) Aviral Kumar and Sunita Sarawagi. 2019.  Calibration of encoder decoder models for neural machine translation.  _arXiv preprint arXiv:1903.00802_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _NeurIPS_.
  * Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020.  Fine-grained fact verification with kernel graph attention network.  In _Proceedings of ACL_.
  * Maddison et al. (2017) Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017.  The concrete distribution: A continuous relaxation of discrete random variables.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 29.
  * Plötz and Roth (2018) Tobias Plötz and Stefan Roth. 2018.  Neural nearest neighbors networks.  In _Advances in Neural Information Processing Systems_ , volume 31.
  * Sachan et al. (2021) Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. 2021.  End-to-end training of multi-document reader and retriever for open-domain question answering.  _NeurIPS_.
  * Su et al. (2019) Lixin Su, Jiafeng Guo, Yixin Fan, Yanyan Lan, and Xueqi Cheng. 2019.  Controlling risk of web question answering.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 115–124.
  * Thorne et al. (2018a) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a.  Fever: a large-scale dataset for fact extraction and verification.  _arXiv preprint arXiv:1803.05355_.
  * Thorne et al. (2018b) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b.  The fact extraction and verification (fever) shared task.  _arXiv preprint arXiv:1811.10971_.
  * Vieira (2014) Tim Vieira. 2014.  Gumbel-max trick and weighted reservoir sampling.
  * Wang et al. (2019) Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019.  Improving back-translation with uncertainty-based confidence estimation.  _CoRR_ , abs/1909.00157.
  * Wikipedia contributors (2004) Wikipedia contributors. 2004.  Okapi bm25— Wikipedia, the free encyclopedia.  [Online; accessed 22-July-2004].
  * Xiao and Wang (2019) Yijun Xiao and William Yang Wang. 2019.  Quantifying uncertainties in natural language processing tasks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):7322–7329.
  * Xie and Ermon (2019) Sang Michael Xie and Stefano Ermon. 2019.  Reparameterizable subset sampling via continuous relaxations.  _arXiv preprint arXiv:1901.10517_.

##  Appendix A Details on the Gumbel Top-K Solution

The Gumbel-top-$K$ trick is a generalization the Gumbel–softmax. It
essentially repeats the Gumbel trick $K$ times until we have a set of the
desired size.

In order to sample a subset of size $K$ according to the categorical
distribution given by $P(D_{k}|q)$, the method use the well-known two-step
process to massage categorical sampling into a differentiable sampling
procedure which includes: 1) reparameterization of the categorical using
Gumbels and 2) softening the argmax into a softmax. We formally describe the
procedure below:

##### Gumbel top-K:

We first perturb the logits $S_{d_{i}}$ with Gumbel noise
$n_{i}\sim\textit{Gumbel}(0;1)$ such that $\tilde{S}_{d_{i}}=S_{d_{i}}+n_{i}$.
Then, sampling from a categorical is equivalent to taking an argmax:

$\displaystyle d^{*}=\operatorname*{arg\,max}_{i}\tilde{S}_{d_{i}}$ (11)

In the top-K case, we start by sampling the first document using the gumbel
pertubation and taking the argmax:

$\displaystyle d_{1}^{*}=\operatorname*{arg\,max}_{i}\tilde{S}_{d_{i}}$ (12)

Then we remove $d_{1}^{*}$ from the pool of documents under consideration and
repeat the same procedure:

$\displaystyle d_{2}^{*}$
$\displaystyle=\operatorname*{arg\,max}_{i\in\mathcal{D}\setminus\\{d_{1}^{*}\\}}\tilde{S}_{d_{i}}$
(13) $\displaystyle\quad\quad\quad\vdots$ $\displaystyle d_{k}^{*}$
$\displaystyle=\\!\\!\\!\\!\\!\\!\\!\\!\\!\operatorname*{arg\,max}_{i\in\mathcal{D}\setminus\\{d_{1}^{*},\ldots,d_{k-1}^{*}\\}}\tilde{S}_{d_{i}}$
(14)

Now, we can construct a fully differentiable procedure by replacing the argmax
with a softmax (the Gumbel softmax trick Jang et al. (2017)). We begin by
relaxing the one-hot vector of the first document:

$d_{i}^{(1)}=\frac{\exp(\tilde{S}_{d_{i}}^{(1)})}{\sum_{j}\exp(\tilde{S}_{d_{j}}^{(1)})}$
(15)

Next, we continue relaxing the successive argmaxes with successive softmaxes
Plötz and Roth (2018) as follows:

$\displaystyle d_{i}^{(2)}$
$\displaystyle=\frac{\exp(\tilde{S}_{d_{i}}^{(2)})}{\sum_{j}\exp(\tilde{S}_{d_{j}}^{(2)})}$
(16) $\displaystyle\quad\quad\quad\quad\quad\vdots$ $\displaystyle
d_{i}^{(k)}$
$\displaystyle=\frac{\exp(\tilde{S}_{d_{i}}^{(k)})}{\sum_{j}\exp(\tilde{S}_{d_{j}}^{(k)})}$
(17)

where we define the $\tilde{S}_{d_{i}}^{(k)}$ recursively

$\displaystyle\tilde{S}_{d_{i}}^{(1)}$ $\displaystyle=\tilde{S}_{d_{i}}$ (18)
$\displaystyle\tilde{S}_{d_{i}}^{(k)}$
$\displaystyle=\tilde{S}_{d_{i}}^{(k-1)}+\log\left(1-d_{i}^{(k-1)}\right)$
(19)

Xie and Ermon (2019) have shown that this procedure is a reasonable relaxation
of the Gumbel-top-$K$. We refer the interested reader to their paper for more
details. Finally, we sum over all the relaxed one-hot vectors $d_{i}^{(k)}$ to
arrive at our softened $K$-hot retrieval:

$D_{K}=\sum_{k=1}^{K}d_{i}^{(k)}$ (20)

This allows us to train the calibration parameters in $P(D_{k}|q)$ using the
objective $\mathcal{L}_{\theta}$ .


* (92) B. S. Acharya, P. Kumar, K. Bobkov, G. Kane, J. Shao and S. Watson, JHEP 0806 (2008) 064 [arXiv:0804.0863 [hep-ph]].
  * (93) B. S. Acharya, G. Kane, S. Watson and P. Kumar, Phys. Rev. D 80 (2009) 083529 [arXiv:0908.2430 [astro-ph.CO]].
  * (94) E. Hardy, J. March-Russell and J. Unwin, JHEP 1210 (2012) 072 [arXiv:1207.1435 [hep-ph]].
  * (95) M. Farina, M. Perelstein and B. Shakya, JHEP 1404 (2014) 108 [arXiv:1310.0459 [hep-ph]].
  * (96) M. Drees, H. Dreiner, D. Schmeier, J. Tattersall and J. S. Kim, Comput. Phys. Commun. 187 (2014) 227 [arXiv:1312.2591 [hep-ph]]; see also https://checkmate.hepforge.org/ .
  * (97) K. Krizka, A. Kumar and D. E. Morrissey, Phys. Rev. D 87 (2013) 9, 095016 [arXiv:1212.4856 [hep-ph]].
  * (98) A. Delgado, G. F. Giudice, G. Isidori, M. Pierini and A. Strumia, Eur. Phys. J. C 73 (2013) 3, 2370 [arXiv:1212.6847 [hep-ph]].
  * (99) G. Belanger, D. Ghosh, R. Godbole, M. Guchait and D. Sengupta, Phys. Rev. D 89 (2014) 015003 [arXiv:1308.6484 [hep-ph]].
  * (100) R. Grober, M. Mühlleitner, E. Popenda and A. Wlotzka, arXiv:1408.4662 [hep-ph].
  * (101) G. Ferretti, R. Franceschini, C. Petersson and R. Torre, arXiv:1502.01721 [hep-ph].
  * (102) B. Dutta, W. Flanagan, A. Gurrola, W. Johns, T. Kamon, P. Sheldon, K. Sinha and K. Wang et al., Phys. Rev. D 90 (2014) 9, 095022 [arXiv:1312.1348 [hep-ph]].
  * (103) W. S. Cho, J. S. Gainer, D. Kim, K. T. Matchev, F. Moortgat, L. Pape and M. Park, arXiv:1411.0664 [hep-ph].
  * (104) J. Eckel, S. Su and H. Zhang, arXiv:1411.1061 [hep-ph].
  * (105) L. Covi and F. Dradi, JCAP 1410 (2014) 10, 039 [arXiv:1403.4923 [hep-ph]].
  * (106) G. Aad et al. [ATLAS Collaboration], JHEP 1501 (2015) 068 [arXiv:1411.6795 [hep-ex]].
  * (107) A. de la Puente and A. Szynkman, arXiv:1504.07293 [hep-ph].
  * (108) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H.-S. Shao and T. Stelzer et al., JHEP 1407 (2014) 079 [arXiv:1405.0301 [hep-ph]].
  * (109) A. Belyaev, N. D. Christensen and A. Pukhov, Comput. Phys. Commun. 184 (2013) 1729 [arXiv:1207.6082 [hep-ph]].
  * (110) ATLAS Collaboration, ATLAS-CONF-2014-046
  * (111) ATLAS Collaboration, ATLAS Note, ATL-PHYS-PUB-2015-001.
  * (112) J. Pumplin, D. R. Stump, J. Huston, H. L. Lai, P. M. Nadolsky and W. K. Tung, JHEP 0207 (2002) 012 [hep-ph/0201195].
  * (113) W. Beenakker, M. Kramer, T. Plehn, M. Spira and P. M. Zerwas, Nucl. Phys. B 515 (1998) 3 [hep-ph/9710451]; see also `http://pauli.uni-muenster.de/~akule_01/nllwiki/index.php/NLL-fast` .
  * (114) See also `http://www.thphys.uni-heidelberg.de/~plehn/index.php?show=prospino&visible=tools` .
  * (115) T. Sjostrand, S. Mrenna and P. Z. Skands, JHEP 0605 (2006) 026 [hep-ph/0603175].
  * (116) B. C. Allanach, C. Balazs, G. Belanger, M. Bernhardt, F. Boudjema, D. Choudhury, K. Desch and U. Ellwanger et al., Comput. Phys. Commun. 180 (2009) 8 [arXiv:0801.0045 [hep-ph]]; see also http://skands.web.cern.ch/skands/slha/ .
  * (117) M. L. Mangano, M. Moretti, F. Piccinini and M. Treccani, JHEP 0701 (2007) 013 [hep-ph/0611129]; see also M. Mangano, ‘‘The so-called MLM prescription for ME/PS matching’’, presented at the Fermilab ME/MC Tuning Workshop, October 4, 2002.
  * (118) J. Alwall, S. Hoche, F. Krauss, N. Lavesson, L. Lonnblad, F. Maltoni, M. L. Mangano and M. Moretti et al., Eur. Phys. J. C 53 (2008) 473 [arXiv:0706.2569 [hep-ph]].
  * (119) J. de Favereau et al. [DELPHES 3 Collaboration], JHEP 1402 (2014) 057 [arXiv:1307.6346 [hep-ex]]; see also https://cp3.irmp.ucl.ac.be/projects/delphes .
  * (120) M. Cacciari, G. P. Salam and G. Soyez, Eur. Phys. J. C 72 (2012) 1896 [arXiv:1111.6097 [hep-ph]]; see also the webpage fastjet.fr .
  * (121) G. Aad et al. [ATLAS Collaboration], JHEP 1406 (2014) 035 [arXiv:1404.2500 [hep-ex]].
  * (122) E. Conte, B. Fuks and G. Serret, Comput. Phys. Commun. 184 (2013) 222 [arXiv:1206.1599 [hep-ph]].
  * (123) E. Conte, B. Dumont, B. Fuks and C. Wymant, Eur. Phys. J. C 74 (2014) 10, 3103 [arXiv:1405.3982 [hep-ph]]; see also https://launchpad.net/madanalysis5 .
  * (124) See https://root.cern.ch/drupal/ .
  * (125) See http://cepa.fnal.gov/psm/stdhep/ .
  * (126) G. L. Bayatian et al. [CMS Collaboration], J. Phys. G 34 (2007) 995.
  * (127) G. Aad et al. [ATLAS Collaboration], Phys. Rev. D 87 (2013) 11, 112001 [Phys. Rev. D 88 (2013) 7, 079906] [arXiv:1210.2979 [hep-ex]].
  * (128) S. Chatrchyan et al. [CMS Collaboration], Eur. Phys. J. C 73 (2013) 10, 2610 [arXiv:1306.1126 [hep-ex]].
  * (129) S. Chatrchyan et al. [CMS Collaboration], Phys. Lett. B 721 (2013) 190 [arXiv:1301.4698 [hep-ex]].
  * (130) G. Aad et al. [ATLAS Collaboration], arXiv:1503.03290 [hep-ex].