#  Adaptive Outlier Detection for Power MOSFETs Based on Gaussian Process
Regression

Kyohei Shimozato§ Michihiro Shintani†, and Takashi Sato§

§ Graduate School of Informatics, Kyoto University Yoshida-hon-machi, Sakyo,
Kyoto 606-8501, Japan
† Graduate School of Science and Technology, Nara Institute of Science and
Technology
8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan
Phone: +81-75-753-4801 E-mail<EMAIL_ADDRESS>

###### Abstract

Outlier detection of semiconductor devices is important since manufacturing
variation is inherently inevitable. In order to properly detect outliers, it
is necessary to consider the discrepancy from underlying trend. Conventional
methods are insufficient as they cannot track spatial changes of the trend.
This study proposes an adaptive outlier detection using Gaussian process
regression (GPR) with Student-t likelihood, which captures a gradual spatial
change of characteristic variation. According to the credible interval of the
GPR posterior distribution, the devices having excessively large deviations
against the underlying trend are detected. The proposed methodology is
validated by the experiments using a commercial SiC wafer and simulation.

###### Keywords:

Outlier detection, characteristic variation, Gaussian process regression

##  I INTRODUCTION

Power devices, such as SiC MOSFETs, are the important components for building
efficient converters. Semiconductor devices are subject to characteristic
variations, and power devices are no exception. Typically, the characteristic
variation of a chip on a wafer is known to be separated into two: an
underlying trend and a random noise added to it [1]. Therefore, the proper
modeling of fundamental trends and the decomposition of these components are
important for detecting outliers and improving manufacturing process.

SiC wafers are commonly manufactured using physical vapor transport (PVT)
crystal growth. In the PVT method, wafers are grown in a heated crucible, so
the basal plane bending and crystallographic dislocations due to temperature
gradients are unavoidable. This heterogeneity is considered as the cause of
the spatial variation of the characteristics [2, 3, 4]. In addition, bulk
micro-defects can be formed randomly. Numerous studies have reported that such
random defects can pose a reliability risk [5, 6], thus chips that may contain
these defects must be judged as outliers. Chips with excessively large
deviation from the spatial trend are likely to contain such defects because
these random defects have a significant impact on the characteristic
degradation.

In practice, it is difficult to detect these anomalies using conventional
methods. Among others, the dynamic part average testing (DPAT) [7] and nearest
neighbor residual (NNR) [8, 9] are widely used testing methods. DPAT is based
on wafer-wide distribution and hence cannot capture the chips whose
characteristics deviate significantly from the spatial trend. NNR, on the
other hand, takes into account the local trend changes. It predicts the trend
based on the characteristics of neighboring chips. However, since the
prediction is carried out using the limited number of neighboring chips, the
trend may be biased by the presence of a cluster of outliers.

In this study, adaptive detection of outlier chips based on a statistical
methodology is proposed. The Gaussian process regression (GPR) [10, 11] with
Student-t likelihood [12] is utilized to define both the spatial
characteristic trend and the allowable range of the characteristic. GPR is a
non-parametric statistical model that can calculate the underlying trends from
the measured data without prior assumption of the model function. Through GPR,
the credible interval can be obtained in addition to the mean of the
prediction. Since this interval is an indicator for finding a statistically
reasonable range of the chip characteristics, we use the interval to judge the
outliers.

The proposed outlier detection method is validated using the wafer
measurements of a commercial power MOSFET and an artificially generated
dataset that we know the ground truth. The performance of the detection is
compared with that of conventional methods. The experimental results show that
the accuracy of the proposed method is significantly better than that of the
conventional methods.

Figure 1: Concept of the proposed adaptive outlier detection.

##  II GAUSSIAN PROCESS REGRESSION

In this section, the details of Gaussian process regression (GPR) are
explained. GPR is a regression method based on the Gaussian process and can be
adapted without assuming the form of the regression function beforehand. From
the known input-output pairs $\mathbf{X}\to\mathbf{Y}$ of a latent function
$f$, the output $Y^{*}$ that corresponds to arbitrary inputs $X^{*}$ can be
predicted using GPR. In GPR, the prediction $Y^{*}$ is obtained as a
probability distribution, which is useful for determining whether the
regression results are credible or not. The narrower the distribution as a
prediction, the greater the certainty, whereas the wider the distribution, the
greater the likelihood that the actual observation may contain larger
randomness.

Assume that $\mathbf{f}=(f(X_{1}),f(X_{2}),...,f(X_{n}))$ as a set of random
variables for inputs $\mathbf{X}=(X_{1},X_{2},...,X_{n})$. When $\mathbf{f}$
is a Gaussian process (GP), any finite subset follows a multivariate Gaussian
distribution $p(\mathbf{f})=\mathcal{N}(\boldsymbol{\mu},\mathbf{K})$.
Generally, the mean $\boldsymbol{\mu}$ is set to $\mathbf{0}$ and the
covariance matrix $\mathbf{K}=(k_{ij})$ is calculated by a kernel function of
$\mathbf{X}$. A typical kernel function is automatic relevance detection
radial basis function (ARD-RBF):

$\displaystyle
k_{ij}=k(X_{i},X_{j})=\theta_{1}\exp\left(-\sum_{p}\frac{|X_{i}^{(p)}-X_{j}^{(p)}|^{2}}{\theta_{2}^{(p)}}\right)$
(1)

where $p$ indicates each dimension of $X$. $\theta_{1},\theta_{2}$ are
hyperparameters. A set of observation $\mathbf{Y}$ contains random noise in
addition to $\mathbf{f}$. In case that the random noise follows the Student-t
distribution,

$\displaystyle p(Y_{i}|f_{i})$ $\displaystyle=\mathcal{ST}(f_{i},\nu,\sigma)$
$\displaystyle=\cfrac{\Gamma((\nu+1)/2)}{\Gamma(\nu/2)\sqrt{\nu\pi}\sigma}\left(1+\cfrac{(Y_{i}-f_{i})^{2}}{\nu\sigma^{2}}\right)^{-(\nu+1)/2}$
(2)

where $\nu$ is called the degrees of freedom and $\sigma$ the scale parameter.
Since the Student-t distribution has heavier tails compared to the Gaussian
distribution, it is tolerant to outliers. Specifically, GPR with Student-t
noise is more robust against the outliers compared to that with Gaussian noise
[12]. This property is important to ensure that the regression results are not
overly influenced by outliers.

In order to deal with Student-t distribution, Laplace approximation is
introduced.

$\displaystyle p(\mathbf{f}|\mathbf{Y})$
$\displaystyle\approx\mathcal{N}(\mathbf{f}|\hat{\mathbf{f}},\boldsymbol{\Sigma})$
(3) $\displaystyle\hat{\mathbf{f}}$ $\displaystyle={\mathop{\rm
argmax}\limits}_{f}p(\mathbf{f}|\mathbf{Y})$ (4)
$\displaystyle\boldsymbol{\Sigma}$ $\displaystyle=\mathbf{K}^{-1}+\mathbf{W}$
(5) $\displaystyle\mathbf{W}_{ij}$
$\displaystyle=\begin{cases}-(\nu+1)\cfrac{(Y_{i}-f_{i})^{2}-\nu\sigma^{2}}{((Y_{i}-f_{i})^{2}+\nu\sigma^{2})^{2}}&(i=j)\\\
0&(i\neq j)\end{cases}$ (6)

All hyperparameters $\boldsymbol{\theta}=(\theta_{1},\theta_{2},\nu,\sigma)$
are optimized by maximizing log likelihood function $\mathcal{L}$ [12].

$\displaystyle\mathcal{L}$ $\displaystyle=\log
p(\mathbf{Y}|\mathbf{X},\boldsymbol{\theta})$ $\displaystyle=\log
p(\mathbf{Y}|\hat{\mathbf{f}})-\frac{1}{2}\log|{\mathbf{K}}|-\frac{1}{2}\hat{\mathbf{f}}^{T}\mathbf{K}^{-1}\hat{\mathbf{f}}$
(7) $\displaystyle\quad-\frac{1}{2}\log|\mathbf{K}^{-1}+\mathbf{W}|$ (8)

In practice, gradient-based methods, such as L-BFGS-B method, can be applied
for the optimization.

The posterior distribution $p(f^{*}|X^{*})$ for an arbitrary input $X^{*}$ can
be predicted as Gaussian distribution, because the concatenated vector
$(\hat{\mathbf{f}},f^{*})$ also follows Gaussian process [12].

$\displaystyle p\left(\begin{pmatrix}\hat{\mathbf{f}}\\\
f^{*}\end{pmatrix}\right)$
$\displaystyle=\mathcal{N}\left(\mathbf{0},\,\begin{pmatrix}\mathbf{K}&\mathbf{K}_{*}\\\
\mathbf{K}_{*}^{T}&k_{**}\end{pmatrix}\right)$ (9) $\displaystyle
p(f^{*}|\mathbf{X}^{*},\mathbf{X},\hat{\mathbf{f}})$
$\displaystyle=\mathcal{N}(\mathbf{K}_{*}^{T}\mathbf{K}^{-1}\hat{\mathbf{f}},\mathbf{k_{**}-\mathbf{k}_{*}^{T}\mathbf{K}^{-1}\mathbf{k}_{*}})$
(10)

where $\mathbf{K}_{*,i}=k(X_{i},X^{*})$, $k_{**}=k(X^{*},X^{*})$. Then, the
prediction of the observation $p(Y^{*})$ can be obtained by integrating
Student-t distribution, which can be calculated by quadrature integration.

$\displaystyle p(Y^{*})=\int
p(Y^{*}|f^{*})p(f^{*}|\mathbf{X}^{*},\mathbf{X},\hat{\mathbf{f}})df^{*}$ (11)

For implementing the proposed method, GPy [13], a Python library of Gaussian
process, is used.

Figure 2: Measured wafer of a commercial SiC MOSFET.

Figure 3: Chip under measurement on an aluminum chip holder.

(a) Measured $V_{\mathrm{th}}$ distribution (b) GPR mean of $V_{\mathrm{th}}$

(c) Measured $R_{\mathrm{on}}$ distribution

(d) GPR mean of $R_{\mathrm{on}}$

Figure 4: Wafer measurement results and the underlying trend obtained as GPR
mean. The chips with a dot are the outliers judged by the proposed method.

##  III PROPOSED METHOD

In this section, a method for detecting outliers using GPR with Student-t
noise is proposed. As described in the previous section, GPR is a Bayesian
non-parametric regression method that yields a latent function with noise as a
probability density. The Student-t noise is included to alleviate the effects
of outliers, i.e., to avoid overfitting as compared to using standard Gaussian
noise. In the proposed method, the spatial underlying trend of chip
characteristics is considered as a latent function $f$, whose input $X$ is the
coordinate $(x,y)$ of a chip and whose output with noise $Y$ is the
characteristics of that chip.

The concept of the proposed method is illustrated in Fig. 1. The measured
value $\mathbf{Y}$ of chips in a wafer follows an underlying spatial trend
with some noise. GPR is applied to chip coordinate $\mathbf{X}$ and the
measured values $\mathbf{Y}$, to optimize the hyperparameters. Then, the
posterior distribution $p(\mathbf{Y}^{*})$ for each chip coordinate
$\mathbf{X}^{*}$ is predicted by GPR. Note that the mean of the distribution
$p(\mathbf{Y}^{*})$ is just one of the representations of the underlying
trend, because the underlying trend is a set of distributions
$p(\mathbf{f}^{*})$. The probability of measured value to be in that
distribution means the credibility of each chip on the basis of the underlying
characteristics trend. From the set of posterior distribution
$p(\mathbf{Y}^{*})$, the $100(1-\alpha)\%$ credible interval can be
calculated. Here, $\alpha$ is called rejection rate. In the proposed method,
the credible interval is used for the outlier detection. By comparing the
measured values with their credible intervals, each chip is classified as
either outlier or not. Fig. 1 shows an example wherein $\alpha$ is set as 0.1.
Though the performances of chip A and chip B are close, chip B should be
considered as an outlier while chip A is not, because the performance of chip
B excessively deviates from the underlying trend. Chips like B are considered
to potentially contain defects. By adaptively determining the test limit for
each chip on the basis of the credible interval, the proposed method provides
the optimal testing criteria even when the large underlying trend is expected.

More formally, the proposed outlier detection method consists of the following
five steps.

  1. 1.

Measure the performance of interest, such as on-resistance, breakdown voltage,
threshold voltage, etc., of all available chips on a wafer.

  2. 2.

Using the coordinates of chips $\mathbf{X}$ and their measured characteristics
$\mathbf{Y}$, the hyperparameters of GPR are optimized.

  3. 3.

For the $i$-th chip in the wafer, infer by GPR the probability distribution of
$p(Y_{i}^{*})$ that $Y_{i}$ should follow, with the chip coordinate $X_{i}$ as
$X_{i}^{*}$

  4. 4.

Judge the chip as an outlier if $Y_{i}$ is out of a $100(1-\alpha)\%$ credible
interval. Otherwise, the chip is considered consistent with the underlying
trend.

  5. 5.

When there exist multiple performances of interest, the rejection results are
OR-ed. In other words, a chip that is judged to be an outlier in at least one
performance should be considered as an outlier.

##  IV MEASUREMENTS AND VALIDATION

In this section, the proposed method is validated and compared with the two
conventional methods, DPAT and NNR. The results for the measurement data of a
commercial SiC wafer are presented in Sec. IV-A, and the results of an
artificially generated demo dataset are presented in Sec. IV-B.

###  IV-A Measurement of commercial wafer

The 435 chips on a commercial SiC wafer shown in Fig. 4 are measured. The
photograph of a chip during probing is shown in Fig. 4. In this study, the
threshold voltage $V_{\mathrm{th}}$ and on-resistance $R_{\mathrm{on}}$ are
chosen as the performances of interest. $V_{\mathrm{th}}$ is defined as the
gate voltage when
$V_{\mathrm{ds}}=20~{}\mathrm{V},I_{\mathrm{d}}=4.4~{}\mathrm{mA}$, and
$R_{\mathrm{on}}$ is measured at
$V_{\mathrm{gs}}=20~{}\mathrm{V},V_{\mathrm{ds}}=0.5~{}\mathrm{V}$. The
temperature of the wafer is controlled at 40$\mathrm{{}^{\circ}C}$.

Figs. 4(a) and 4(c) show the measured $V_{\mathrm{th}}$ and $R_{\mathrm{on}}$
distributions. The measured values of $V_{\mathrm{th}}$ range from 2.5 to 3.1
V, and those of $R_{\mathrm{on}}$ range from 75 to 130 $\mathrm{m\Omega}$.
Most of these values are still within the range of the official datasheet.
However, chips having the values significantly off the underlying trend
present randomly on the wafer. In other words, those devices that exhibit
significantly different characteristics from their surrounding trend can be
considered to have long-term reliability issues. The means of GPR posterior
distribution are shown in Fig. 4(b) for $V_{\mathrm{th}}$ and Fig. 4(d) for
$R_{\mathrm{on}}$. Here, the GPR posterior prediction is considered as the
underlying trend, and regard the chips with their measured value exceeding 90%
credible interval as outliers. The detected outliers for each performance are
marked with a black dot in Fig. 4. From the figure, it can be said that the
proposed method successfully detects outlier chips. It is found that
$V_{\mathrm{th}}$ follows the estimated underlying trend well, while
$R_{\mathrm{on}}$ suffers from a larger random component. As a result, there
are as many outliers in $V_{\mathrm{th}}$ as those in $R_{\mathrm{on}}$,
because the prediction of $V_{\mathrm{th}}$ has a narrower credible interval.

(a) $V_{\mathrm{th}}[\mathrm{V}]$

(b) $R_{\mathrm{on}}[\mathrm{m\Omega}]$

Figure 5: Measurement versus the mean of GPR with 90% credible interval at
$y=5$ and $y=13$. The trend given by NNR and the thresholds given by DPAT are
also drawn.

Fig. 5 shows a comparison of the measurement and the predicted distributions
of GPR for the rows $y=5$ and 13. In these figures, in addition to the mean of
the estimated distribution, the 90% credible intervals are also depicted.
Regarding $V_{\mathrm{th}}$ of $y=13$, most of the chips are located very
close to the estimated mean curve and have fallen within the credible
interval, indicating the underlying trend is dominant over random noise.
$R_{\mathrm{on}}$ of $y=5$ has a wider credible interval due to the large
random noise. Meanwhile, though there are many outliers for $V_{\mathrm{th}}$
of $y=5$ and $R_{\mathrm{on}}$ of $y=13$, the GPR posterior is not overfitted
to them.

The 90% (1.645$\sigma$) limit of DPAT and the underlying trend predicted by
NNR are also indicated in Fig. 5. Since DPAT imposes the same limit for all
the chips without taking into account the underlying trends on the wafer, a
large number of outlier chips with different characteristics from their
neighbors are classified as inliers. This may lead to inappropriate testing
results. The NNR trend is not smooth compared to the GPR mean. Since the
residual from the NNR trend is often utilized to judge outliers, the unsmooth
trend may cause wrong judgments.

In the proposed method, the fraction of chips that are judged as outliers is
highly dependent on the rejection rate $\alpha$. In order to reject potential
outliers close to the threshold, a larger $\alpha$ may be used. However, too
large $\alpha$ value will result in a lower yield, so it must be set
carefully. Fig. 6 shows the outlier chip rate as a function of $\alpha$. Here,
the outlier chip rate is defined as the ratio of chips that are determined to
be outliers out of all chips measured.

(a) $V_{\mathrm{th}}$

(b) $R_{\mathrm{on}}$

Figure 6: The relationship between $\alpha$ and outlier chip rate. The
theoretical line where $\alpha$ is equal to the outlier chip rate is also
drawn.

When $\alpha$ is small, the outlier chip rate becomes higher than what is
expected by $\alpha$ because there exist significantly bad performing chips
that will be judged outliers regardless of the value of $\alpha$. In contrast,
when $\alpha$ is larger than 0.2, the outlier chip rate is less than what is
expected by $\alpha$. In that range of $\alpha$, less obvious outliers are
found, resulting in more yield loss. Therefore, the intersection where
$\alpha$ becomes equal to the outlier rate can be used as a good candidate to
determine the appropriate $\alpha$ value. In this example, $\alpha$ is defined
at around 0.1 for both $V_{\mathrm{th}}$ and $R_{\mathrm{on}}$
characteristics.

###  IV-B Virtual measurement dataset

In order to demonstrate the effectiveness of the proposed method, a virtual
dataset consisting of 519 chips on a wafer shown in Fig. 8(a) is generated.

(a) Performance of virtual chips

(b) GPR mean

Figure 7: Visualization of a virtual measurement dataset. The chips with a dot
are the outliers determined by the proposed method.

Figure 8: The relationship between $\alpha$ and the outlier chip rate for the
virtual measurement dataset. The theoretical line where $\alpha$ is equal to
the outlier chip rate is also drawn.

The virtual performance data $p(x,y)$ has a baseline trend that consists of
two components: global variation component $p_{\mathrm{g}}(x,y)$ as a function
of the chip coordinate $(x,y)$ on the wafer and random variation component
$p_{\mathrm{r}}$ that is sampled from a normal distribution that is mutually
independent and identical for each chip. Here, the coordinates $x$ and $y$ are
both integers. Additionally, a relatively large random deviation
$p_{\mathrm{d}}$ was taken into account for the 20% of the entire chips. The
chips with this extra component added are defective and thus should be judged
as outliers. The overall performance of a chip is determined by:

$\displaystyle p(x,y)=p_{\mathrm{g}}(x,y)+p_{\mathrm{r}}+p_{\mathrm{d}},$ (12)

where

$\displaystyle p_{\mathrm{g}}(x,y)$
$\displaystyle=90+0.06((x-17)^{2}+4(y-11)^{2})$ (13) $\displaystyle
p_{\mathrm{r}}$ $\displaystyle\overset{iid}{\sim}\mathcal{N}(2.0,1.0)$ (14)
$\displaystyle p_{\mathrm{d}}$
$\displaystyle\overset{iid}{\sim}\begin{cases}0&(80\%)\\\
\mathrm{Uniform}(9,11)&(10\%)\\\ \mathrm{Uniform}(-11,-9)&(10\%)\end{cases}.$
(15)

Fig. 8(b) shows the mean of the GPR posterior distribution. The global trend
given by Eq. (13) is reproduced by the proposed method. The theoretical and
GPR-based outlier chip rates as functions of $\alpha$ are shown in Fig. 8.
From the intersection in Fig. 8, the $\alpha$ is set to 0.2.

Figure 9: Virtual measurement data versus the mean of GPR with 80% credible
interval at $y=3$ and $y=15$. The trend given by NNR and the thresholds given
by DPAT are also drawn.

In Fig. 9, the virtual data along $y=3$ and $y=15$ is presented with the mean
and the credible interval of the GPR. Even with the presence of outliers, the
proposed method extracts reasonably smooth global trend, which is very close
to the true global variation $p_{\mathrm{g}}$, and provides a reasonable
threshold for separating good chips from defective ones. The given chips with
different performance components are classified properly. In contrast, the
threshold given by the DPAT is too broad and most of the outlier chips have
fallen in the “pass” region. Similarly, the background trend extracted by the
NNR method is different from the ground truth due to the influence of
outliers.

TABLE I: Accuracy comparison of outlier detection methods. Positive and negative refer to outliers and inliers, respectively. Sensitivity is the rate of outliers detected out of all given outliers. Specificity is the rate of inliers detected out of all given inliers. Higher score is better for both sensitivity and specificity. Method | True Positive | True Negative |  | False Positive
---
(yield loss)
| False Negative
---
(test escape)
Sensitivity | Specificity
This work | 102 | 407 | 10 | 0 | 1.00 | 0.98
NNR | 89 | 395 | 22 | 13 | 0.87 | 0.95
DPAT | 44 | 384 | 33 | 58 | 0.43 | 0.92

The accuracy of each outlier detection method is evaluated for the virtual
wafer in Fig. 8 and summarized in Table I. The components of the confusion
matrix given by each method, sensitivity, and specificity are presented. In
terms of confusion matrix, the proposed method clearly outperforms the two
existing methods. Accordingly, the sensitivity and specificity of the proposed
method are better than that of NNR and DPAT. In particular, the sensitivity of
the proposed method is 1.00, meaning that all the given outliers are detected.
The specificity is 0.98, which means that the rate of misclassification of
inliers as outliers is remarkably small.

##  V CONCLUSION

In this study, an adaptive outlier detection methodology based on the
predicted posterior of GPR is proposed. The proposed method successfully
detected outliers in both experiments using the measurement data of a
commercial SiC wafer and virtual measurement dataset. In the experiments using
the virtual measurement data, the accuracy of the proposed method
significantly outperformed the two conventional methods, NNR and DPAT. Though
$V_{\mathrm{th}}$ and $R_{\mathrm{on}}$ are selected as the performance of
interests, the proposed method can be effective for the tests using other
characteristics, such as breakdown voltage or parasitic capacitances.

## Acknowledgment

The part of this work is supported by JST-OPERA Program Grant Number
JPMJOP1841, JSPS KAKENHI Grant 20H04156, and 20K21793.

## References

  * [1] B. Stine, D. Boning, and J. Chung, “Analysis and decomposition of spatial variation in integrated circuit processes and devices,” _IEEE Transactions on Semiconductor Manufacturing_ , vol. 10, no. 1, pp. 24–41, Feb. 1997.
  * [2] T. Kimoto, “Bulk and epitaxial growth of silicon carbide,” _Progress in Crystal Growth and Characterization of Materials_ , vol. 62, no. 2, pp. 329–351, Jun. 2016.
  * [3] Y. Cui, X. Hu, X. Xie, R. Wang, and X. Xu, “Spatial variation of lattice plane bending of 4H-SiC substrates,” _CrystEngComm_ , vol. 19, no. 27, pp. 3844–3849, 2017.
  * [4] N. A. Mahadik, H. Das, S. Stoupin, R. E. Stahlbush, P. L. Bonanno, X. Xu, V. Rengarajan, and G. E. Ruland, “Evolution of lattice distortions in 4H-SiC wafers with varying doping,” _Scientific Reports_ , vol. 10, no. 1, p. 10845, Jul. 2020.
  * [5] K. Hasegawa, K. Taguchi, Y. Kagawa, E. Suekawa, N. Kaguchi, Y. Ata, H. Haruguchi, Y. Nakashima, and T. Minato, “Which is harder SOA test for SiC MOSFET to do Unclamped Inductive Switching (UIS) or Unloaded Short Circuit mode switching (USCS)? Does UIS play a role of USCS?” in _Proceedings of International Symposium on Power Semiconductor Devices and ICs (ISPSD)_ , Sep. 2020, pp. 62–65.
  * [6] R. Madge, M. Rehani, K. Cota, and W. Daasch, “Statistical post-processing at wafersort — an alternative to burn-in and a manufacturable solution to test limit setting for sub-micron technologies,” in _Proceedings of IEEE VLSI Test Symposium (VTS)_ , Apr. 2002, pp. 69–74.
  * [7] _Guidlines for part average testing_ , Automotive Electronics Council (AEC), 2011. [Online]. Available: http://www.aecouncil.com/Documents/AEC_Q001_Rev_D.pdf
  * [8] W. Daasch, J. McNames, D. Bockelman, and K. Cota, “Variance reduction using wafer patterns in $I_{\mathrm{DDQ}}$ data,” in _Proceedings of International Test Conference (ITS)_ , Oct. 2000, pp. 189–198.
  * [9] S. Sabade and D. Walker, “Comparison of wafer-level spatial $I_{\mathrm{DDQ}}$ estimation methods: NNR versus NCR,” in _Proceedings of IEEE International Workshop on Current and Defect Based Testing (DBT)_ , Apr. 2004, pp. 17–22.
  * [10] C. E. Rasmussen and C. K. I. Williams, _Gaussian Processes for Machine Learning_ , ser. Adaptive Computation and Machine Learning series, F. Bach, Ed. Cambridge, MA, USA: MIT Press, Nov. 2005.
  * [11] C. Bishop, _Pattern Recognition and Machine Learning_ , ser. Information Science and Statistics. New York: Springer-Verlag, 2006.
  * [12] J. Vanhatalo, P. JylÃ¤nki, and A. Vehtari, “Gaussian process regression with Student-t likelihood,” in _Proceedings of Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 22. Curran Associates, Inc., 2009.
  * [13] GPy, “GPy: A gaussian process framework in python,” http://github.com/SheffieldML/GPy, since 2012.


# Pruning the Index Contents for Memory Efficient Open-Domain QA

Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz
Brno University of Technology
<EMAIL_ADDRESS>
# R2-D2: A Modular Baseline for Open-Domain Question Answering

Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz
Brno University of Technology
<EMAIL_ADDRESS>
###### Abstract

This work presents a novel four-stage open-domain QA pipeline R2-D2 (Rank
twice, reaD twice). The pipeline is composed of a retriever, passage reranker,
extractive reader, generative reader and a mechanism that aggregates the final
prediction from all system’s components. We demonstrate its strength across
three open-domain QA datasets: NaturalQuestions, TriviaQA and EfficientQA,
surpassing state-of-the-art on the first two. Our analysis demonstrates that:
(i) combining extractive and generative reader yields absolute improvements up
to 5 exact match and it is at least twice as effective as the posterior
averaging ensemble of the same models with different parameters, (ii) the
extractive reader with fewer parameters can match the performance of the
generative reader on extractive QA datasets111Our demo is available at
http://r2d2.fit.vutbr.cz/. Code and preprocessed data are available at
https://github.com/KNOT-FIT-BUT/R2-D2..

##  1 Introduction

Last year showed rapid progress in neural factoid open-domain question
answering based on _retriever-reader_ architecture (Open-QA). Such Open-QA
systems Chen et al. (2017) seek evidence for answering the questions inside
the knowledge source using the _retriever_ and then extract the answer from
the retrieved knowledge using the _reader_. The knowledge source is often a
large corpus of short snippets of natural language, so-called passages (e.g.,
taken from an encyclopedia).

The progress can be attributed to advances in neural retrieval methods
(Karpukhin et al., 2020; Izacard and Grave, 2020; Khattab et al., 2020; Luan
et al., 2021; Xiong et al., 2020, inter alia) that benefit from smarter
negative sampling strategies or a better trade-off between complex question-
passage interaction and its efficiency. It also can be attributed to reading
methods that enable processing large quantities of retrieved passages Izacard
and Grave (2021). They compensate for a certain amount of the retrieval error
and enable early aggregation of answer’s evidence between passages.

This work demonstrates the relative improvement of 23-32% compared to last
year’s state-of-the-art DPR system Karpukhin et al. (2020), while using the
same knowledge source and the retriever. We propose a state-of-the-art Open-QA
baseline composed of retriever, passage reranker, extractive reader,
generative reader, and a novel component fusion approach. We follow the
practice from information retrieval and show that our moderately sized
reranker allows to reduce the passage count needed at the input of large
reader models about four times. Our readers then take the best from both
worlds. The extractive reader proposes a list of salient answer spans. The
generative reader reranks these spans, seeing all the passages at once, or
generates its own answer. The proposed pipeline is heterogeneous and modular,
making it an ideal benchmark.

To sum up, our contributions are three-fold:

  1. 1.

We present a simple novel approach to aggregate scores from all system
components and show that combining extractive and generative approaches is
superior to a posterior averaging ensemble of homogeneous models.

  2. 2.

We show that the extractive reader can sometimes match the performance of the
generative approaches without taking the advantage of the fusion between
retrieved passages. This indicates that the evidence aggregation from multiple
passages in the generative approaches is either not learned or not necessary
to perform well on these datasets.

  3. 3.

We push the state-of-the-art for two large and popular datasets, demonstrating
what is achievable with the proposed approach, having the same knowledge
source and the retriever as in the previous works Karpukhin et al. (2020);
Izacard and Grave (2021).

##  2 Open-QA Pipeline

We propose the R2-D2 (Rank twice, reaD twice), 4-stage pipelined system that
can choose whether to generate or to extract an answer. The parameters of each
component in pipeline are estimated separately. It is composed of DPR passage
retriever Karpukhin et al. (2020), passage reranker (see subsection 2.1), and
two readers. Figure 1 shows the diagram of our system. The first reader
performs an extractive span-selection similar to Fajcik et al. (2020). The
second reader is based on Fusion-In-Decoder (FiD) Izacard and Grave (2021).

Formally, given a question $q\in\mathcal{Q}$ from the set of all possible
questions $\mathcal{Q}$ and the corpus
$\mathcal{C}=\\{p_{1},p_{2},...,p_{n}\\}$ composed of passages $p_{i}$, the
retriever learns a ranking function
$\operatorname{rank}:\mathcal{Q}\times\mathcal{C}\rightarrow\mathbb{R}$ that
assigns a score to each passage. We assume each passage contains its passage
title (e.g., title from the Wikipedia article).

Taking a top-$K$ scoring passages $\mathcal{C}_{r}\subset\mathcal{C}$,
reranker again rescores $\mathcal{C}_{r}$ scoring passages by learning a
reranking function
$\operatorname{rerank}:\mathcal{Q}\times\mathcal{C}_{r}\rightarrow\mathbb{R}$.
Note that while $\operatorname{rank}$ and $\operatorname{rerank}$ have similar
signatures, the computational cost of $\operatorname{rerank}$ over the same
amount of passages is drastically higher, as it computes fine-grained
interaction between tokens of question and passage.

Next, the rescored passages are passed to two readers: the extractive reader
reads top-$V$ passages $\mathcal{C}_{rr}\subset\mathcal{C}_{r}$ independently
of each other and assigns the probability
$\boldsymbol{P}_{e}(a_{e}|q,\mathcal{C}_{rr})$ to each span $a_{e}$ in the
passages (see subsection 2.2). The FiD generative reader reads top-$V_{2}$
passages $\mathcal{C}_{rr}^{\prime}\subset\mathcal{C}_{r}$ jointly and
generates an answer from probability space
$\boldsymbol{P}_{g}(a_{g}|q,\mathcal{C}_{rr}^{\prime})$ via greedy search.

Finally, R2-D2 aggregates the outputs from all components using two fusions
(described in subsection 2.3).

In which Czech city is the brewery of its largest beer exporter? top-K
passages 1\. … town of České Budějovice, known as Budweis… 2\. Czech Beer
Festival is the biggest … 3\. Plzeň, also called Pilsen is a city… Retriever
index top-K reranked passages 1\. Plzeň, also called Pilsen is a city… 2\. …
town of České Budějovice, known as Budweis… 3\. Czech Beer Festival is the
biggest … Passage reranker top-M answer spans 1\. České Budějovice 2\.
Festival 3\. Plzeň Extractive reader top generated answer 1\. Brno Abstractive
reader top-M reranked spans 1\. Plzeň 2\. Festival 3\. České Budějovice
Abstractive reader top-M aggr. spans 1\. Plzeň 2\. České Budějovice 3\.
Festival Score aggregation top answer 1\. Plzeň Binary decision Figure 1:
R2-D2 pipeline.

###  2.1 Passage Reranker

The proposed passage reranker is based on transformer cross-encoder similar to
Nogueira and Cho (2019); Luan et al. (2021). The input is the concatenation of
question $q\in\mathcal{Q}$ and passage $p\in\mathcal{C}_{r}$ with a special
SEP token between them. The passage consists of a title and context that are
prepended with special start tokens and concatenated together. We denote the
contextual representation of input token $w$ obtained by the cross-encoder as
$\operatorname{En}(p,q)[w]\in\mathbb{R}^{d}$.

Now we can define the reranking function for passage rescoring as

$\operatorname{rerank}(q,p)=\operatorname{En}(p,q)[\texttt{CLS}]^{\top}w$ (1)

where $w\in\mathbb{R}^{d}$ is a trainable vector and CLS is the special token
added at the start of an input sequence. Finally, we define the following
formula222Formal definition of softmax over a set is described in the Apendix
D.

$\boldsymbol{P}_{rr}\left(p|q,\mathcal{C}_{r}\right)=\operatorname*{softmax}\limits_{p\in\mathcal{C}_{r}}\left(\operatorname{rerank}\left(q,p\right)\right)_{p}$
(2)

to assign a probability to the case that passage $p$ contains answer to the
question $q$.

Training.

The model input for each question is exactly one positive sample supplemented
with hard negatives from the retriever. The ground truth passage, annotated
the same way as in Karpukhin et al. (2020), is primarily used as a positive
sample. If the ground truth is unknown, the positive sample is the best
retriever passage containing the answer. The hard negatives are uniformly
sampled from retriever’s top-$K$ results that do not contain the answer. The
used loss function is the cross-entropy.

###  2.2 Extractive Reader

Extractive reader estimates the probability
$\boldsymbol{P}_{e}(a_{e}|q,\mathcal{C}_{rr})$. It is the probability of a
span $a_{e}$ from top-$V$ passage $p\in\mathcal{C}_{rr}$ being an answer to a
question $q$. We decompose the $\boldsymbol{P}_{e}(a_{e}|q,\mathcal{C}_{rr})$
into four probabilities of:

  * •

token $s$ being starting token of an answer span,

  * •

token $e$ being ending token of an answer span,

  * •

tokens $s$ and $e$ being boundary tokens of an answer span Fajcik et al.
(2020),

  * •

passage $p$ containing an answer for the question $q$ (inner reranker) as in
Karpukhin et al. (2020).

To obtain the final probability used in test-time, we compute their
product333We tried decoding from the subsets of these probabilities in
Appendix E not observing significant difference.. These probabilities are
defined as:

$\boldsymbol{P}_{*}(*|q,\mathcal{C}_{rr})=\operatorname{softmax}(s_{*})_{i}\>,$
(3)

where $*$ may stand for a _start_ , _end_ , _joint_ , and a _passage_. The $i$
is an index of a given element, and the $s_{*}$ is a vector of scores for each
element among all passages in $\mathcal{C}_{rr}$. So the _softmax_
normalization sum goes through all the passages. On the other hand, the
$s_{*}$ scores are estimated by the model with just a single passage at its
input Clark and Gardner (2018). The scores are as follows:

$\displaystyle s^{i}_{start}=\operatorname{En}(p,q)[s]^{\top}w_{start}$ (4)
$\displaystyle s^{i}_{end}=\operatorname{En}(p,q)[e]^{\top}w_{end}$ (5)
$\displaystyle
s^{i}_{joint}=(W_{j}\operatorname{En}(p,q)[s]+b_{j})^{\top}\operatorname{En}(p,q)[e]$
(6) $\displaystyle
s^{i}_{passage}=\operatorname{En}(p,q)[\texttt{CLS}]^{\top}w_{p}\>.$ (7)

Where $w_{*},b_{j}\in\mathbb{R}^{h}$,
$\operatorname{En}(p,q)[\cdot]\in\mathbb{R}^{h}$, and
$W_{j}\in\mathbb{R}^{h\times h}$ are all trainable parameters.

We omit the spans of a title and question for answer span selection. Therefore
the final answer can be selected only from the context.

The following training objective with independently marginalized components is
used:

$\begin{split}-\log\sum_{s\in
starts(C_{rr})}\boldsymbol{P}_{start}(s|q,\mathcal{C}_{rr})\\\ -\log\sum_{e\in
ends(C_{rr})}\boldsymbol{P}_{end}(e|q,\mathcal{C}_{rr})\\\ -\log\sum_{j\in
boundaries(C_{rr})}\boldsymbol{P}_{joint}(j|q,\mathcal{C}_{rr})\\\
-\log\sum_{p\in
C_{rr}}\boldsymbol{P}_{passage}(p|q,\mathcal{C}_{rr})\>.\end{split}$ (8)

The sums are going through target annotations (starts, ends, etc.) obtained by
the distant supervision approach.

###  2.3 Component Fusion

To produce the final answer, R2-D2 aggregates the log-probabilities of all
system components via linear combinations tuned on validation data.

Firstly, the log-probabilities of all system components for top-$M$ answer
spans proposed by the extractive reader are aggregated. Formally, assume the
$\mathcal{A}_{q}$ is the set of top-$M$ answer spans from
$\boldsymbol{P}_{e}(a|q,\mathcal{C}_{rr})$ for question $q$. The generative
model performs the answer reranking evaluating the log-probability of the
answer spans

$\\{\log\boldsymbol{P}_{g}(a|q,\mathcal{C}_{rr}^{\prime}):a\in\mathcal{A}_{q}\\}.$
(9)

Next a logistic regression loss (11) is minimized to perform score
aggregation. It combines the scores across the R2-D2 components to maximize
the correct answer span probability over dataset $\mathcal{D}$. This dataset
is composed of the top-$M$ outputs of the extractive reader with the correct
answer.

$\displaystyle
x(a)=[\boldsymbol{P}_{e}(a)\;\boldsymbol{P}_{g}(a)\;\boldsymbol{P}_{r}(p_{a})\;\boldsymbol{P}_{rr}(p_{a})]$
(10)
$\displaystyle-\sum_{\mathclap{(\mathcal{A}_{q},gt)\in\mathcal{D}}}\log\operatorname*{softmax}\limits_{a\in\mathcal{A}_{q}}\big{(}{w^{\top}\log
x(a)+b}\big{)}_{gt}$ (11)

Here $p_{a}$ denotes the passage containing the answer span $a$,
$\mathcal{A}_{q}$ is a set of proposed answer spans, $gt$ is the correct
answer span, distribution dependencies are dropped for clarity and only the
logistic regression parameters $w,b$ are tuned in this step.

Finally, we theorized the correct answer span might not always be available in
the passage set $\mathcal{C}_{rr}$, but the generative reader might be able to
generate the answer from its parameters and the evidence given in passages. We
introduce the binary classifier, which decides whether to select the best span
answer from answer aggregation step or a free-form answer generated via FiD.
Given that $s_{agg}(q)=\max_{a\in\mathcal{A}_{q}}w^{\top}x(a)+b$ is the best
span score and
$s^{*}_{g}(q)=\log\boldsymbol{P}_{g}(a_{q}^{*}|q,\mathcal{C}_{rr}^{\prime})$
is the log-probability of the answer $a_{q}^{*}$ obtained via greedy decoding
for question $q$, a classifier is trained via binary cross-entropy $BCE(l,t)$
with log-odds ratio $l$ and target $t$ to do the binary decision

$\sum_{(e,t)\in\mathcal{D}}BCE(w^{\top}[s_{agg}(e);s^{*}_{g}(e)]+b,t).$ (12)

Here, the training dataset $\mathcal{D}$ contains only cases where either the
extractive or the abstractive prediction is correct (but not both).

##  3 Experimental Setup

Our models are implemented in PyTorch Paszke et al. (2019) using Transformers
Wolf et al. (2020). We use 12GB GPU to train the passage reranker, 48GB GPU
for the generative reader, and 16x 32GB GPUs to train the extractive reader
with $V=128$ passages at its input. The inference runs on 12GB GPU. In all
experiments, we used Adam optimizer with a decoupled weight decay Loshchilov
and Hutter (2019). Our models are evaluated by two metrics:

Exact match (EM)

measures the proportion of examples, for which the system prediction matched
at least one annotated ground-truth answer. We use the script from Lee et al.
(2019)444https://cutt.ly/rkZNIer.

Accuracy@K

measures the proportion of examples, for which the ground-truth answer string
is present in top-K retrieved passages. We match the string exactly as
Karpukhin et al. (2020)555https://cutt.ly/0luNhx4.

###  3.1 Datasets and Data Pre-processing

We evaluate our models on three datasets. Their statistics are available in
Table 1. To train the reranker we filter out examples, which do not contain
golden passage or exact match in top-$K$ retrieved passages. To train the
extractive reader, only examples with exact match in a golden passage or top-1
retrieved passage are kept. Both filtering strategies are closely described in
Appendix C.

NQ-Open

Kwiatkowski et al. (2019); Lee et al. (2019) or NaturalQuestions-Open consists
of real user queries obtained from Google search engine. The maximum length of
each answer is at most 5 tokens. Each training and development sample contains
1 annotated answer, while test data contain 5-way answer annotation.

TQ-Open

Joshi et al. (2017) or TriviaQA-Open consists of question-answer pairs from 14
different trivia quiz websites. Each question contains human annotated answer
and a set of answer aliases gathered from Wikipedia. We use the unfiltered
version.

EfficientQA

Min et al. (2021) is a dataset collected the same way as NQ-Open through 2019
and thus may contain more questions without evidence in our corpus than NQ-
Open. We use the officially released dev set for testing666The test set was
not released during our experiments. models trained on NQ-Open training data.

Additionally, we also report results according to train-test set overlaps
discovered by Lewis et al. (2021) in Appendix H.

Dataset | Train | Dev | Test
---|---|---|---
NQ-Open | 79,168 | 8,757 | 3,610
\- filt. reranker | 71,238 | - | -
\- filt. ext. reader | 61,755 | - | -
\- w/ golden passage | 58,876 | 6,515 | -
TQ-Open | 78,785 | 8,837 | 11,313
\- filt. reranker | 69,346 | - | -
\- filt. ext. reader | 62,332 | - | -
\- w/ golden passage | 60,413 | 6,760 | -
EfficientQA | - | - | 1,800

Table 1: Dataset statistics. The filt. lines report how many examples are kept
for training the reranker (filt. reranker) and extractive reader (filt. ext.
reader). The lines w/ golden passage denote how many examples from the set
contain golden passage annotation.

###  3.2 Models and Pipeline

Retriever.

We use BERT-based DPR from the official
checkpoint777https://github.com/facebookresearch/DPR. Each passage is
represented via 768-dimensional embedding. We use a multiset checkpoint for
TQ-Open, as the checkpoint for TQ directly isn’t officially released. We use
the same knowledge corpus containing 21,015,320 passages based on 12-20-2018
Wikipedia snapshot as Karpukhin et al. (2020). In inference time, the
retriever passes $K=200$ passages $\mathcal{C}_{r}$ to reranker.

Passage reranker.

We use the RoBERTa-base Liu et al. (2019) and truncate the inputs to a maximum
length of 256. The linear scheduler with 0.1 warmup proportion is used, the
number of epochs is 5 and the model is validated every 40,000 optimization
steps. We use learning rate $1.6\cdot 10^{-4}$ and batch size 8. In training,
the model reranks 24 passages per question with negatives uniformly sampled
from top-400 passages retrieved by DPR. During the inference, top-$K$
($K=200$) retriever passages are rescored and passed to readers.

Extractive reader.

The extractive reader encoder is based on pre-trained ELECTRA-large. Its
inputs are truncated if they are longer than the allowed maximum size (512
tokens). During the training phase, all spans from all
$p\in\mathcal{C}_{r}$888Note that we train on data from retriever, not
reranker. that match999Matching strategies are described in Appendix C. with
at least one of the known answers are selected as target annotations.
Therefore the annotations might appear in the wrong context. The extractive
reader reads the top-$V=128$ passages during the training phase and when it is
used without the reranker. To demonstrate the effect of reranker, the reader
reads only the top-$V=24$ passages if the reranker is used. We use a linear
scheduler with a warmup for the first 20,000 steps for all models. The maximum
number of training steps is 200,000. The model is validated every 20,000
steps, and the best checkpoint among validations is selected. The learning
rate is $2\cdot 10^{-5}$ and the optimization step was done after each
training example.

Generative reader.

We utilize T5-large Raffel et al. (2020) and use a concatenation of question,
passages and their respective titles at the Fusion-in-Decoder’s input the same
way as Izacard and Grave (2020). We truncate each passage to the length of 250
tokens for NQ. For TQ, as questions are significantly longer, we truncate
whole inputs to the same size. Following FiD for TQ, we use only human-
generated answer. In training, the golden passage always comes first, if
available, and we take the rest of passages as ranked by retriever up to
$V_{2}$ passages. Izacard and Grave (2021) trained FiD with $V_{2}=100$
passages at its input. However, such approach requires tremendous amount of
GPU memory, and thus requires employing speed-memory trade-offs such as
gradient checkpointing Chen et al. (2016). Unlike the original approach, we
use only $V_{2}=25$ passages in our FiD. We note that in practice combining
reranker with shorter-context FiD yields results similar to original
implementation with much lower memory consumption and better throughput in the
R2-D2 setting101010Due to the numerous decoder computations in answer re-
ranking.. We analyze the speed of our implementation in Appendix I. Other
hyperparameters are similar to the original work—batch size 64, learning rate
$5\cdot 10^{-5}$ but no learning rate schedule. In test time, we decode an
answer via greedy decoding.

##  4 Results and Analysis

| Method | NQ | TQ | #$\boldsymbol{\theta}$
---|---|---|---|---
Extractive | BM25+BERT Mao et al. (2020) | 37.7 | 60.1 | 110M
Hard EM Min et al. (2019a) | 28.1 | 50.9 | 110M
Path Retriever Asai et al. (2020) | 32.6 | - | 447M
Graph Retriever Min et al. (2019b) | 34.5 | 56.0 | 110M
ORQA Lee et al. (2019) | 33.3 | 45.0 | 220M
REALM Guu et al. (2020) | 40.4 | - | 660M
ProQA Xiong et al. (2021) | 34.3 | - | 220M
DPR Karpukhin et al. (2020) | 41.5 | 56.8 | 220M
RDR Yang and Seo (2020) | 42.1 | 57.0 | 110M
GAR+DPR Mao et al. (2020) | 43.8 | - | 626M
ColBERT Khattab et al. (2020) | 48.2 | 63.2- | 440M
RIDER (GAR+DPR) Mao et al. (2021) | 48.3 | - | 626M
UnitedQA-E Cheng et al. (2021) | 51.8 | 68.9 | 440M
Generative | BM25+SSG Mao et al. (2020) | 35.3 | 58.6 | 406M
T51.1+SSM Roberts et al. (2020) | 35.2 | 61.6 | 11B
RAG Lewis et al. (2020) | 44.5 | 56.8 | 516M
DPR+SSG Min et al. (2020) | 42.2 | - | 516M
FiD-base Izacard and Grave (2021) | 48.2 | 65.0 | 333M
FiD-large Izacard and Grave (2021) | 51.4 | 67.6 | 848M
FiD-large++ Izacard et al. (2020) | 54.7 | 73.3 | 848M
UnitedQA-G Cheng et al. (2021) | 52.3 | 68.6 | 880M
| UnitedQA (Ens. E+G+G) Cheng et al. (2021) | 54.7 | 70.5 | 1.87B
Ours | R1-D1 (Generative) | 49.9 | 65.4 | 848M
R1-D1 (Extractive) | 50.8 | 65.0 | 445M
R2-D2 (21M) | 55.0 | 69.9 | 1.29B
R2-D2 (21M) w/ HN-DPR | 55.9 | - | 1.29B

Table 2: Comparison with the state-of-the-art in EM. #$\theta$ denotes the
estimated amount of model parameters. Symbol - reports the result only for
smaller system with $220M$ parameters.

Readers | Fusion | NQ-Open | TQ-Open | EfficientQA
---|---|---|---|---
ret. | +rr | $\Delta$ | ret. | +rr | $\Delta$ | ret. | +rr | $\Delta$
ext | - | 50.78 | 50.72 | -0.06 | 65.01 | 65.46 | 0.45 | 47.00 | 47.56 | 0.56
gen | - | 49.92 | 50.69 | 0.77 | 65.38 |  69.14 | 3.76 | 44.83 | 47.33 | 2.50
ext+gen | naive | 51.88 | 52.44 | 0.56 | 66.17 | 68.01 | 1.84 | 47.06 | 49.11 | 2.05
ext+gen | aggr | 54.13 | 54.90 | 0.77 | 67.42 | 68.66 | 1.24 | 50.44 | 52.00 | 1.56
ext+gen | aggr+bd | 54.07 | 54.99 | 0.92 | 67.37 | 69.94 | 2.57 | 49.72 | 52.22 | 2.50

Table 3: Ablation study. We report results for extractive (ext), generative
(gen) and both readers (ext+gen) without (ret.) and with reranking (+rr). The
$\Delta$ column shows the exact match difference caused by passage reranking.

The effectiveness of our approach is compared with the state-of-the-art in
Table 2. Our system, composed of just the retriever and FiD reader R1-D1
(Generative), shows inferior performance compared to FiD-large. This is most
likely caused by 4 times fewer passages at its input, as in Izacard and Grave
(2021). In contrast, our ELECTRA based extractive reader R1-D1 (Extractive)
shows large gains compared to extractive state-of-the-art, while having the
same retriever as DPR. We hypothesize this may be caused by ELECTRA pre-
training method, which shows strong performance through variety of tasks and
we further show that it is also due to training and inference with large input
size of 128 passages and better objective (discussed in Section 4.2 and
Appendix G). Only system that matches the performance of our extractive reader
is the concurrent work on UnitedQA-E Cheng et al. (2021), which uses advanced
regularization and HardEM techniques. We note that these are orthogonal to our
approach and could potentially lead to further improvements.

Finally, we find that our R2-D2 system with 21M passages corpus is competitive
even with FiD++, which uses DPR retriever improved via knowledge distillation,
and 26M passage corpus, which also includes lists. Additionally, we evaluate
our model with a better retrieval model (HN-DPR) based on the DPR checkpoint
where hard negatives are mined using the retrieval model
itself111111https://cutt.ly/Ux5Yt4h. Note that we do not compare EfficientQA
with state-of-the-art, as the previous works didn’t reported results on dev
set we use for testing.

###  4.1 Reranker Performance

Next, we compare the performance of our retriever, reranker and reader with
Accuracy@K in Figure 2. The passage reranker improves the accuracy
consistently and we observe the same trend on other datasets (Appendix A). We
also include analysis, where we rerank each passage $p_{i}$ according its
$s^{i}_{passage}$ score from extractive reader. We observe results similar or
even better to reranker for $K<10$, indicating the extractive reader reranks
well on its own. However, in subsequent experiments we do not replace the
reranker with reader because: (i) passage reranker has fewer parameters, (ii)
extractive reading can run in parallel with reranking and generative reading
as extractive reader is not benefiting from reranking, and (iii) passage
reranking scores often improve results during score aggregation (see Section
4.4).

$1$$10$$100$$0.5$$0.6$$0.7$$0.8$$0.9$KAccuracy@Kretrievedrerankedreader Figure
2: Accuracy@K on test-data of NQ-Open.

###  4.2 Extractive Reader Performance

$8$$16$$32$$64$$128$$0.44$$0.45$$0.46$$0.47$$0.48$Test input sizeEM8163264128
Figure 3: Influence of test input size on extractive reader’s performance for
various train input sizes (different curves) on NQ-Open test dataset.

In order to investigate the influence of the number of input passages on the
extractive reader’s performance, we trained multiple ELECTRA-base models, each
with different input size. In test time, we evaluate each of them on various
input sizes. Figure 3 shows that increasing train/test input size has a
positive influence on extractive reader’s performance. However, input size 128
doesn’t seem to increase the performance anymore.

###  4.3 Ablations

The ablations are listed in Table 3. We ablate results without using passage
reranker, with separate readers and their combination and with different
stages of component fusion. Namely, performing a _naive_ answer re-ranking by
generative reader means the system chooses the most probable answer span among
the top-$M$ spans provided by the extractive reader according to generative
reader log-probabilities as shown in equation (9). Analogously, the _aggr_
fusion denotes that the system chooses the most probable answer span according
to aggregated scores, as in equation (11). Finally, the _aggr+bd_ fusion
denotes the binary decision, as shown in equation (12).

| $\boldsymbol{P}_{*}$ | $\emptyset$ | $\\{r\\}$ | $\\{rr\\}$ | $\\{r,rr\\}$
---|---|---|---|---|---
NQ-Open | $\\{e\\}$ | 50.72 | 51.41 | 51.55 | 51.69
$\\{g\\}$ | 52.44 | 52.88 | 53.35 | 53.19
$\\{e,g\\}$ | 54.63 | 55.10 | 54.82 | 54.90
TQ-Open | $\\{e\\}$ | 65.54 | 65.64 | 65.60 | 65.61
$\\{g\\}$ | 68.25 | 68.17 | 68.21 | 68.26
$\\{e,g\\}$ | 68.45 | 68.57 | 68.66 | 68.66

Table 4: Results for different pipeline components used for score aggregation
on NQ-Open a TQ-Open. See text for details.

As expected, we observe that reranker improves the results consistently for
generative model in all cases. The gains are especially large for TQ-Open
(over 3.7 EM, underscored in Table 3). In fact, the results are comparable to
Izacard and Grave (2021), suggesting that using the FiD reader with smaller
context window and reranker is a reasonable alternative to memory inefficient
FiD with large input size. Furthermore as expected, the extractive reader
without reranker already has top-128 passages at the input, and improvements
from the passage reranking are only negligible if any (less than 1 EM).

Finally, the results on NQ-Open and EfficientQA suggest applying the binary
decision does not bring large improvements over the score aggregation if any.
However, notice that this is not the case for TQ-Open, where the generative
reader performs significantly better compared to extractive reader, suggesting
both component fusions play important role in the system.

| $\boldsymbol{P}_{*}$ | $\emptyset$ | $\\{r\\}$ | $\\{rr\\}$ | $\\{r,rr\\}$
---|---|---|---|---|---
NQ-Open | $\\{e\\}$ | 52.85 | 53.30 | 53.10 | 52.94
$\\{g\\}$ | 52.44 | 52.77 | 53.21 | 53.07
$\\{e,g\\}$ | 54.35 | 55.10 | 54.46 | 54.99
TQ-Open | $\\{e\\}$ | 69.34 | 69.28 | 69.23 | 69.26
$\\{g\\}$ | 69.76 | 69.71 | 69.65 | 69.77
$\\{e,g\\}$ | 69.80 | 69.89 | 69.88 | 69.94

Table 5: Results for binary decision on NQ-Open and TQ-Open for different
aggregated pipeline components from Table 4.

###  4.4 Component Fusion

Furthermore, we analyze the performance of each component combination in the
score aggregation and its impact on the component fusion via binary decision.
Both fusions are tuned on validation data and reported on test data of the NQ-
Open and TQ-Open datasets. See Appendix B for analysis on additional datasets.
Table 4 shows all relevant combinations of ranker _r_ , reranker _rr_ ,
extractive reader _e_ and generative reader _g_ probabilities used in score
aggregation. In overall, we observe that combining retriever and reranker
scores with the reader leads to better or equal performance. On NQ-Open, we
observe minor improvements up to ~1 EM. However, there is no difference on TQ-
Open.

The impact of adding a binary decision after the score aggregation is shown in
Table 5. Interestingly, the binary decision component significantly improves
the performance only without reranked answer scores ($\\{e\\}$ rows in both
tables). However, fusing the generative and extractive reader via binary
decision performs significantly worse on NQ-Open than fusing both readers
together with score aggregation ($\\{e\\}$ row in Table 5 vs. $\\{e,g\\}$ row
in Table 4). As already noted in ablations, we find this to be quite the
opposite for TQ-Open. We hypothesize that the binary decision is strong in
cases, where generative reader performs better to extractive reader (the case
of TQ-Open). We argue that if the generative reader is better, the abstractive
answer should be used far more often, than when it’s not. We support the
hypothesis by analyzing the proportion of test samples, on which the binary
decision component activated (i.e. an abstractive prediction was selected). On
NQ-Open, the component almost never activated (only on 3.5% samples), but this
proportion is much higher (26.6%) on TQ-Open.

Readers | Ensemble | EM | $\Delta_{ext}$ | $\Delta_{gen}$
---|---|---|---|---
ext | - | 46.79 | - | -
2 models | 48.30 | 1.51 | -
3 models | 48.59 | 1.80 | -
gen | - | 45.00 | - | -
2 models | 46.30 | - | 1.30
3 models | 46.59 | - | 1.59
ext+gen | aggr | 49.92 | 3.13 | 4.92

Table 6: Comparison between ensembling via posterior averaging and score
aggregation on NQ-Open.

###  4.5 Comparison with Posterior Averaging

Finally, we compare our score aggregation with the ensemble computed via
posterior probability averaging. In particular, we train three extractive and
generative base-sized models initialized with different random seed. We do not
use reranker in this experiment, and set train/test input size of extractive
reader to 32. We assess the predictions using the averaged posterior
probabilities and compare their average performance with score aggregation in
Table 6. Concretely, we compare with average of all 2 model ensembles (2
models) and with an ensemble of all 3 checkpoints (3 models). We observe two
to three times improvement of score aggregation over the posterior probability
averaging on NQ-Open test data.

##  5 Related Work

Passage reranking.

Previous work in QA based on neural nets used Bi-LSTM encoders Wang et al.
(2018); Lee et al. (2018) that score each document independently. Over time,
Bi-LSTM were replaced by BERT-like transformer encoders Qiao et al. (2019);
Wang et al. (2019a). For document ranking, Nogueira et al. (2019) proposed a
multi-stage architecture. The first stage scores each document independently,
and the second estimates the more relevant document from all document pairs.
Another document ranking approach uses the seq2seq model to generate a true or
false answer to the document’s relevance to the query Nogueira et al. (2020).
Recent works have often focused on effective reranking. Xin et al. (2020)
achieved inference speedup using early exiting, Jang and Kim (2020) proposed a
smaller and faster model, and Mao et al. (2021) came up with a method which
uses reader’s predictions to rerank the passages. Our reranker is most similar
to Nogueira and Cho (2019); Luan et al. (2021), except that unlike in IR, we
assume there is just one correct passage and thus train our model via
categorical cross-entropy.

Multipassage Reading Comprehension

Related work considers generative and extractive approaches towards modeling
the reader. The generative reader generates an answer while conditioned on
question alone Roberts et al. (2020), or question with relevant passages Lewis
et al. (2020); Min et al. (2020). Izacard and Grave (2021) showed it suffices
to concatenate the passages in the decoder of seq2seq model, increasing the
amount of top-passages the model can depend on dramatically. The extractive
reader used in Open-QA assumes that the answer is a continuous span string in
located in retrieved paragraphs Chen et al. (2017). Clark and Gardner (2018)
proposed to aggregate the probabilities of distantly supervised answer matches
via maximum marginal likelihood (MML). Lin et al. (2018) proposed to denoise
distantly supervised answer string matches in MML via paragraph-ranker. Cheng
et al. (2020a) experimented with different assumptions for MML, showing
improvement when marginalizing over components of span probability
independently. Fajcik et al. (2020) proposed to model joint span probability
directly via compound objective, instead of modeling the probability of span’s
start and end independently. Karpukhin et al. (2020) incorporated an
independent passage classifier loss to his MML objective. Our objective is
similar to the last work, except that it uses joint component and also
optimizes MML over relevant passages’ probabilities.

Component Fusion.

Yang et al. (2019) also combined BM25 ranker and reader scores via linear
combination. Our work can be seen as an extension of this idea to combining
the scores of all pipeline’s components. Iyer et al. (2020) proposed a system
which directly learns to rerank question-passage-answer triplets proposed via
extractive model. However, reranking answers from their large extractive model
via large reranker leads to ~1 EM improvement absolute, whereas R2-D2s score
aggregation improves 4 to 5 EM w.r.t. the extractive reader. Concurrently with
our work, Cheng et al. (2021) proposed hard voting ensembling scheme to
combine the reader predictions. Firstly, each model from an ensemble produces
its best prediction, then the votes for identical predictions are combined,
omitting the scores produced by the individual models. The authors obtained
best results using two FiD readers and single extractive reader, leading to
1.6 and 2.4 EM improvement on TQ-Open and NQ-Open, compared to their best
single extractive or generative model.

##  6 Conclusion

This work proposed R2-D2, a novel state-of-the-art pipeline for open-domain QA
based on 4 components: retriever, reranker, generative reader and extractive
reader. We showed that employing a reranker is a reasonable alternative to
using large passage counts at the input of both the extractive and the
generative reader. Our results on NQ-Open and EfficientQA showed that the
extractive and the generative reader could perform equally in Open-QA,
although the generative reader is twice the size of the extractive reader. On
the other hand, we observe the extractive reader underperforms on TQ-Open. We
hypothesize, that the cause is (1) the complexity of trivia questions with
many entities, which often require combining evidence from multiple passages —
these are impossible to answer for the extractive reader by design — and (2)
the expensive hyperparameter search, as we used NQ-Open hyperparameters also
for TQ-Open. Contrary to belief based on the results on different datasets
Yang et al. (2019); Wang et al. (2019b); Izacard and Grave (2021), we found
the extractive reader can also benefit from larger input sizes, both in
training and test time. Finally, we proposed a component fusion, which allows
merging the complementary behavior of generative and extractive approaches
along with the ranking components and found it improves the results
significantly. Due to its heterogenous and modular nature, our pipeline forms
an ideal base for future research of component integration in modern Open-QA.

## Acknowledgments

We would like to thank Jan Doležal for implementing an R2-D2 demo. This work
was supported by the Czech Ministry of Education, Youth and Sports, subprogram
INTERCOST, project code: LTC18054. The computation used the infrastructure
supported by the Ministry of Education, Youth and Sports of the Czech Republic
through the e-INFRA CZ (ID:90140).

## References

  * Asai et al. (2020) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020.  Learning to retrieve reasoning paths over wikipedia graph for question answering.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016.  Training deep nets with sublinear memory cost.  _ArXiv_ , abs/1604.06174.
  * Cheng et al. (2020a) Hao Cheng, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2020a.  Probabilistic assumptions matter: Improved models for distantly-supervised document-level question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5657–5667, Online. Association for Computational Linguistics.
  * Cheng et al. (2020b) Hao Cheng, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2020b.  Probabilistic assumptions matter: Improved models for distantly-supervised document-level question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5657–5667, Online. Association for Computational Linguistics.
  * Cheng et al. (2021) Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2021.  UnitedQA: A hybrid approach for open domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3080–3090, Online. Association for Computational Linguistics.
  * Clark and Gardner (2018) Christopher Clark and Matt Gardner. 2018.  Simple and effective multi-paragraph reading comprehension.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 845–855, Melbourne, Australia. Association for Computational Linguistics.
  * Fajcik et al. (2020) Martin Fajcik, Josef Jon, Santosh Kesiraju, and Pavel Smrz. 2020.  Rethinking the objectives of extractive question answering.  _arXiv preprint arXiv:2008.12804_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Iyer et al. (2020) Srinivasan Iyer, Sewon Min, Yashar Mehdad, and Wen-tau Yih. 2020.  RECONSIDER: re-ranking using span-focused cross-attention for open domain question answering.  _CoRR_ , abs/2010.10757.
  * Izacard and Grave (2020) Gautier Izacard and Edouard Grave. 2020.  Distilling knowledge from reader to retriever for question answering.  _arXiv preprint arXiv:2012.04584_.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Izacard et al. (2020) Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. 2020.  A memory efficient baseline for open domain question answering.  _arXiv preprint arXiv:2012.15156_.
  * Jang and Kim (2020) Youngjin Jang and Harksoo Kim. 2020.  Document re-ranking model for machine-reading and comprehension.  _Applied Sciences_ , 10(21).
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab et al. (2020) Omar Khattab, Christopher Potts, and Matei Zaharia. 2020.  Relevance-guided supervision for openQA with colBERT.  _arXiv preprint arXiv:2007.00814_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2018) Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018.  Ranking paragraphs for improving answer recall in open-domain question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 565–569, Brussels, Belgium. Association for Computational Linguistics.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021.  Question and answer test-train overlap in open-domain question answering datasets.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1000–1008, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Lin et al. (2018) Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018.  Denoising distantly supervised open-domain question answering.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1736–1745, Melbourne, Australia. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.  _Transactions of the Association for Computational Linguistics_.
  * Mao et al. (2020) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020.  Generation-augmented retrieval for open-domain question answering.  _arXiv preprint arXiv:2009.08553_.
  * Mao et al. (2021) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021.  Reader-guided passage reranking for open-domain question answering.  _arXiv preprint arXiv:2101.00294_.
  * Min et al. (2021) Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, et al. 2021.  NeurIPS 2020 EfficientQA competition: Systems, analyses and lessons learned.  _arXiv preprint arXiv:2101.00133_.
  * Min et al. (2019a) Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a.  A discrete hard EM approach for weakly supervised question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2851–2864, Hong Kong, China. Association for Computational Linguistics.
  * Min et al. (2019b) Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b.  Knowledge guided text retrieval and reading for open domain question answering.  _arXiv preprint arXiv:1911.03868_.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5783–5797, Online. Association for Computational Linguistics.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with bert.  _arXiv preprint arXiv:1901.04085_.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 708–718, Online. Association for Computational Linguistics.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019.  Multi-stage document ranking with BERT.  _arXiv preprint arXiv:1910.14424_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8024–8035.
  * Qiao et al. (2019) Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019.  Understanding the behaviors of BERT in ranking.  _arXiv preprint arXiv:1904.07531_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Wang et al. (2018) Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018.  R3: Reinforced ranker-reader for open-domain question answering.  In _AAAI_ , pages 5981–5988.
  * Wang et al. (2019a) Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019a.  Multi-passage BERT: A globally normalized BERT model for open-domain question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5878–5882, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2019b) Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019b.  Multi-passage BERT: A globally normalized BERT model for open-domain question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5878–5882, Hong Kong, China. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xin et al. (2020) Ji Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin. 2020.  Early exiting BERT for efficient document ranking.  In _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_ , pages 83–88, Online. Association for Computational Linguistics.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_.
  * Xiong et al. (2021) Wenhan Xiong, Hong Wang, and William Yang Wang. 2021.  Progressively pretrained dense corpus index for open-domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2803–2815, Online. Association for Computational Linguistics.
  * Yang and Seo (2020) Sohee Yang and Minjoon Seo. 2020.  Is retriever merely an approximator of reader?  _arXiv preprint arXiv:2010.10999_.
  * Yang et al. (2019) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019.  End-to-end open-domain question answering with BERTserini.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 72–77, Minneapolis, Minnesota. Association for Computational Linguistics.

##  Appendix A Additional Accuracy@K Analysis

Analysis of Accuracy@K on NQ-Open development data in Figure 4(a), on
EfficientQA data is shown in Figure 4(b), and on TQ-Open development data in
Figure 4(c) and test data in Figure 4(d).

$10^{0}$$10^{1}$$10^{2}$$0.45$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$$0.9$KAccuracy@Kretrievedrerankedreader

(a) NQ-Open (dev).

$10^{0}$$10^{1}$$10^{2}$$0.4$$0.45$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$$0.9$KAccuracy@Kretrievedrerankedreader

(b) EfficientQA.

$10^{0}$$10^{1}$$10^{2}$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$KAccuracy@Kretrievedrerankedreader

(c) TQ-Open (dev).

$10^{0}$$10^{1}$$10^{2}$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$KAccuracy@Kretrievedrerankedreader

(d) TQ-Open (test).

Figure 4: Analysis of Accuracy@K on different datasets.

##  Appendix B Additional Component Fusion Analysis

This section includes results analogical to Tables 4, 5 on EfficientQA and
development data of NQ-Open and TQ-Open (Tables 7, 8).

| $\boldsymbol{P}_{*}$ | $\emptyset$ | $\\{r\\}$ | $\\{rr\\}$ | $\\{r,rr\\}$
---|---|---|---|---|---
NQ-Open | $\\{e\\}$ | 48.38 | 48.94 | 48.85 | 49.14
$\\{g\\}$ | 49.99 | 50.49 | 50.35 | 50.47
$\\{e,g\\}$ | 51.79 | 51.97 | 51.82 | 51.80
TQ-Open | $\\{e\\}$ | 65.07 | 65.21 | 65.16 | 65.24
$\\{g\\}$ | 67.68 | 67.72 | 67.73 | 67.76
$\\{e,g\\}$ | 68.13 | 68.19 | 68.17 | 68.12
EfficientQA | $\\{e\\}$ | 47.56 | 48.33 | 48.89 | 48.72
$\\{g\\}$ | 49.11 | 49.56 | 50.22 | 50.11
$\\{e,g\\}$ | 50.78 | 51.67 | 50.89 | 52.00

Table 7: Score aggregation on validation data of NQ-Open, TQ-Open and
EfficientQA.

| $\boldsymbol{P}_{*}$ | $\emptyset$ | $\\{r\\}$ | $\\{rr\\}$ | $\\{r,rr\\}$
---|---|---|---|---|---
NQ-Open | $\\{e\\}$ | 50.65 | 51.24 | 51.01 | 51.17
$\\{g\\}$ | 50.36 | 50.91 | 50.68 | 50.90
$\\{e,g\\}$ | 52.24 | 52.29 | 52.27 | 52.07
TQ-Open | $\\{e\\}$ | 69.03 | 69.03 | 69.01 | 68.99
$\\{g\\}$ | 69.54 | 69.46 | 69.62 | 69.70
$\\{e,g\\}$ | 69.77 | 69.79 | 69.67 | 69.61
EfficientQA | $\\{e\\}$ | 48.33 | 50.06 | 49.39 | 49.67
$\\{g\\}$ | 48.94 | 49.50 | 50.06 | 49.72
$\\{e,g\\}$ | 50.78 | 51.83 | 50.94 | 52.22

Table 8: Binary decision on NQ-Open, TQ-Open and EfficientQA.

##  Appendix C Data Pre-processing

This section describes how the training datasets for reranker and extractive
reader are filtered, and how the distant supervision labeling is generated.
Note not each example contains golden passage, as not each example can be
mapped to the used dump of Wikipedia. We use the same golden passage mapping
as Karpukhin et al. (2020).

For passage reranking, the input must contain at least one positive example.
We meet this condition either by adding a golden passage or searching for the
passage with an answer in the top-400 results retrieved by DPR. In detail
about the search, first the Simple tokenizer proposed in
DrQA121212https://github.com/facebookresearch/DrQA tokenizes each passage and
golden answer. The positive example is the best-scored tokenized passage that
contains an exact match with one of the tokenized answers. Note the search
proceeds in the same way as in DPR’s Accuracy@K
implementation131313https://github.com/facebookresearch/DPR.

The extractive reader is trained only on samples which contain exact match to
at least one of the annotated answers in the top-1 passage, or golden passage
if it is available. The exact match is performed on the subword token level
(i.e. in ELECTRA’s tokenization).

Next, the span annotations are extracted from the passages at the reader’s
input. Note each sample may contain multiple answers. The annotations for each
answer in each sample are obtained differently in retrieved passages and in
the golden passage. For retrieved passages, we search for the answer’s exact
matches in passages, and use each match as target annotation. For golden
passage, we also search for the answer’s exact matches in it. If there is
none, the answer is soft matched with single sub-sequence of golden passage,
which yields highest non-zero F1 score. The F1 soft match is also performed on
the subword token level. Therefore answers with zero highest F1 soft match
with golden passage and no exact match in any of the reader’s input passages
are discarded.

###  C.1 Upper Bound on F1 Matching

Because the brute-force computation of a span with the greatest nonzero F1
score is potentially very demanding, we found the length limit for spans that
are worth searching (see Theorem C.2).

To compare brute-force with upper bound implementation, we run an experiment
on 16,741 passages (retrieved for NQ-Open dev). The average time per passage
for brute-force approach was 121 ms while it was only 9 ms for implementation
that uses the upper bound optimization.

The soft match is described in Algorithm 1. It assumes that there is no exact
match.

Algorithm 1 Soft match

1:set of spans S and answer span a

2:function SoftMatch($\texttt{S},\texttt{a}$)

3: $\texttt{actSize}\leftarrow 1$

4: $\texttt{lenLimit}\leftarrow 2$

5: $\texttt{bestSpan}\leftarrow\texttt{None}$

6: $\texttt{bestScore}\leftarrow 0$

7: while $\texttt{actSize}<\texttt{lenLimit}$ do

8: for all $t\in S$ of size actSize do

9: $\texttt{score}\leftarrow\texttt{F1}(t,a)$

10: if $\texttt{score}>\texttt{bestScore}$ then

11: $\texttt{bestSpan}\leftarrow t$

12: $\texttt{bestScore}\leftarrow\texttt{score}$

13: $\texttt{lenLimit}\leftarrow|a|\frac{|t|+|a|-s_{ta}}{s_{ta}}$

14: $\texttt{actSize}\leftarrow\texttt{actSize}+1$

15: return bestSpan

######  Lemma C.1.

Let $t$ and $a$ be non-empty spans and $0<s_{ta}\leq|a|$ number of shared
tokens for them. Then141414—x— symbolises number of tokens in span x.

$|t|\leq|a|\frac{|t|+|a|-s_{ta}}{s_{ta}}\>.$ (13)

###### Proof.

To prove it by contradiction assume that

$|t|>|a|\frac{|t|+|a|-s_{ta}}{s_{ta}}\>,$ (14)

then

$s_{ta}|t|>|a||t|+|a||a|-|a|s_{ta}\>,$ (15)

and also $0<s_{ta}\leq|a|$, thus $|a||a|-|a|s_{ta}\geq 0$. Therefore even if
we assume that
$|a||a|-|a|s_{ta}=0$. We get

$\begin{split}s_{ta}|t|>|a||t|\\\ s_{ta}>|a|\>,\end{split}$ (16)

which is in contradiction with $0<s_{ta}\leq|a|$. ∎

######  Theorem C.2.

Let $S$ be a set of non-empty spans, $a$ an non-empty answer span, $t$ non-
empty trial span, $0<s_{ta}\leq|a|$ is number of shared tokens for $t$ and
$a$, and $S_{b}=\\{z|z\in S\land|z|\geq|a|\frac{|t|+|a|-s_{ta}}{s_{ta}}\\}$.
Then the theorem states that

$\forall x\in S_{b}(\operatorname{F1}(x,a)\leq\operatorname{F1}(t,a))\>.$ (17)

###### Proof.

To prove it by contradiction assume that

$\exists x\in S_{b}(\operatorname{F1}(x,a)>\operatorname{F1}(t,a))\>.$ (18)

F1 score can be expressed as:

$\operatorname{F1(b,c)}=\frac{2s_{bc}}{|b|+|c|}\>,$ (19)

thus

$\frac{2s_{xa}}{|x|+|a|}>\frac{2s_{ta}}{|t|+|a|}\>.$ (20)

From Lemma C.1 $|t|\leq|x|$. Therefore $s_{ta}<s_{xa}$, to satisfy the
inequality (in equation 20), and we know that $0<s_{xa}\leq|a|$. So let the
$s_{xa}=|a|$ (the maximum) then

$\begin{split}\frac{2|a|}{|x|+|a|}>\frac{2s_{ta}}{|t|+|a|}\\\
|a|(|t|+|a|)>s_{ta}|x|+s_{ta}|a|\\\
|x|<|a|\frac{|t|+|a|-s_{ta}}{s_{ta}}\>,\end{split}$ (21)

which is in contradiction with $x\in S_{b}$. ∎

##  Appendix D Softmax Notation

Usually, softmax function $\sigma:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ is
defined as:

$\sigma(v)_{i}=\frac{e^{v_{i}}}{\sum^{K}_{j=1}e^{v_{j}}}.$ (22)

However, some parts of this work used variant of softmax that is defined as
follows:

$\begin{split}\operatorname*{softmax}\limits_{x\in
D}\big{(}{f(x)}\big{)}_{y}=\frac{e^{f(y)}}{\sum\limits_{x\in
D}e^{f(x)}}\>,\end{split}$ (23)

where $D$ is the input set, $f:D\rightarrow\mathbb{R}$, $y\in D$.

##  Appendix E Decoding the Distributions from the Extractive Reader

We analyzed the subsets of joint probability space over spans obtained via
multiplication of distributions as explained in section 2.2 in Table 9. The
factors of this space are the distribution given by the outer product of
independent probability distributions
$\boldsymbol{P}_{start}(.)\boldsymbol{P}_{end}(.)^{\top}$ denoted as I, joint
probability distribution $\boldsymbol{P}_{joint}(.)$ denoted as J, and passage
distribution $\boldsymbol{P}_{passage}(.)$ denoted as C.

Factorization | NQ-dev | NQ-test | EfficientQA
---|---|---|---
I | 48.32 | 50.58 | 47.33
J | 48.53 | 51.25 | 47.83
I+J | 48.57 | 50.83 | 47.83
I+C | 48.22 | 50.55 | 47.22
J+C | 48.49 | 51.11 | 47.56
I+J+C | 48.50 | 50.86 | 47.67

Table 9: The results of the pipeline with different types of extractive reader’s distribution used for decoding. See text for details. | marginalizes
---
independently
| joint
---
comp.
| start&end
---
comp.
EM
- | - | ✓ | 45.42
- | ✓ | ✓ | 45.41
✓ | - | ✓ | 45.71
✓ | ✓ | - | 47.09
✓ | ✓ | ✓ | 47.06
Table 10: Ablation of loss components on NQ-Open test dataset using ELECTRA-
base model.

##  Appendix F Passage Reranker Revision

In preliminary experiments of this work we used a Longformer encoder Beltagy
et al. (2020) with concatenated passages at it’s input to benefit from the
early fusion between passages. In particular, the passages at the Longformer’s
input were shuffled and concatenated, and we used presoftmax score computed
from the first Longformer’s output representation of each passage as the
$rerank$ function. The passages were shuffled with a fixed seed in both,
training and test time. Therefore each passage was scored not only according
to the question but also according to other passages. However, we did not
observe any significant benefits when we used the Longformer setup over a
RoBERTa which scores each passage independently (see Table 11).

Readers | Fusion | NQ-Open (dev) | NQ-Open (test) | TQ-Open (test) | EfficientQA
---|---|---|---|---|---
Long. | RoB. | $\Delta$ | Long. | RoB. | $\Delta$ | Long. | RoB. | $\Delta$ | Long. | RoB. | $\Delta$
ext | - | 48.50 | 48.38 | -0.12 | 50.86 | 50.72 | -0.14 | 65.41 | 65.46 | -0.05 | 47.67 | 47.56 | -0.11
gen | - | 49.34 | 49.40 | 0.06 | 51.50 | 50.69 | -0.81 | 68.85 | 69.14 | -0.29 | 47.33 | 47.33 | 0.00
ext+gen | naive | 49.91 | 49.99 | 0.08 | 53.43 | 52.44 | -0.99 | 67.82 | 68.01 | -0.19 | 49.06 | 49.11 | 0.05
ext+gen | aggr | 52.05 | 51.80 | -0.25 | 54.96 | 54.90 | -0.06 | 68.49 | 68.66 | -0.17 | 51.56 | 52.00 | 0.44
ext+gen | aggr+bd | 52.36 | 52.07 | -0.29 | 55.01 | 54.99 | -0.02 | 69.62 | 69.94 | -0.32 | 51.06 | 52.22 | 1.16

Table 11: Exact match comparison of Longformer (Long.) and RoBERTa (RoB.)
based passage reranker.

##  Appendix G Ablating the Extractive Reader’s Objective

Firstly let us demonstrate that start and end components in the used loss (see
equation 8) perform summation over both inter-passage and intra-passage
combinations of starts and ends:

$\begin{split}-\log\sum_{s\in S}\boldsymbol{P}_{start}(s)-\log\sum_{e\in
E}\boldsymbol{P}_{end}(e)=\\\ =-\log\sum_{s\in
S}\boldsymbol{P}_{start}(s)\sum_{e\in E}\boldsymbol{P}_{end}(e)=\\\
=-\log\sum_{s\in S,e\in
E}\boldsymbol{P}_{start}(s)\boldsymbol{P}_{end}(e)\,.\end{split}$ (24)

Where $S=starts(C_{rr})$, $E=ends(C_{rr})$ and distribution dependencies are
dropped for clarity. Inter-passage combinations obviously do not correspond to
a real answer. Even though that this loss does not reflect the task correctly,
it achieves better results (see Table 10) than the following loss

$-\log\sum_{c\in
C_{rr}}\sum_{a_{e}\in\operatorname{answers}(c)}\boldsymbol{P}_{e}(a_{e}|q,\mathcal{C}_{rr})$
(25)

that marginalizes components jointly, and thus the summation is done only
through intra-passage start-end combinations. Such results agree with previous
work Cheng et al. (2020b).

Table 10 also shows that the joint component improves the independent loss
variant, but not the other one that marginalizes jointly. We hypothesize that
this is because the loss in equation 25 already considers only the intra-
passage start-end pairs. Lastly, Table 10 shows that using just the joint and
passage component is sufficient for NQ-Open, which agrees with Fajcik et al.
(2020).

##  Appendix H Results According to Question and Answer Test-Train Overlap

In addition to evaluation on the TQ-Open and NQ-Open shown in Table 2, we also
report results on subsets of these datasets in Table 12, as split by Lewis et
al. (2021). We compare R2-D1 (retriever, reranker and extractive or generative
reader, marked as gen and ext respectively) and R2-D2 (ext+gen) to official
results on FiD Izacard and Grave (2021).

Model | NQ-Open | TQ-Open
---|---|---
| Total |  | Question
---
Overlap
| Answer
---
Overlap
Only
| No
---
Overlap
Total |  | Question
---
Overlap
| Answer
---
Overlap
Only
| No
---
Overlap
FiD | 51.40 | 71.30 | 48.30 | 34.50 | 67.60 | 87.50 | 66.90 | 42.80
ext+gen | 54.99 | 75.00 | 48.89 | 39.91 | 69.94 | 90.18 | 71.53 | 44.83
gen | 50.69 | 70.06 | 46.98 | 34.04 | 69.14 | 87.50 | 70.32 | 44.83
ext | 50.72 | 72.53 | 45.40 | 35.11 | 65.46 | 83.63 | 66.42 | 39.46
$\Delta_{\text{gen}}$ | 4.30 | 4.94 | 1.91 | 5.87 | 0.80 | 2.68 | 1.21 | 0.00
$\Delta_{\text{ext}}$ | 4.27 | 2.47 | 3.49 | 4.80 | 4.48 | 6.55 | 5.11 | 5.37
Table 12: Results on the overlapping and non-overlapping parts of test sets
for NQ and TQ. Total column corresponds to overall result on the whole
dataset, as reported before, Question Overlap corresponds to samples with
train-test question overlap and answer overlap, Answer Overlap Only
corresponds to samples with answer overlap, but no question overlap, and No
Overlap corresponds to samples with no overlap between train and test sets.

##  Appendix I Inference Speed of Our Implementation

While optimizing the R2-D2’s inference speed was not the main focus of this
paper, we show that even our unoptimized implementation can be used in
practice in small scale. We analyze the speed of our implementation on NQ-Open
test data in Table 13. The times were measured on a workstation with Intel
Xeon Silver 4214 48-core CPU, 188GB RAM and Nvidia 2080Ti 12GB GPU. Table
columns show settings with and without passage reranker. Table rows are split
into two parts; intermediate rows show time spent by the pipeline’s single
component (e.g., row ext. reader shows what time the pipeline spent by running
just the ext. reader), and total rows show the total time taken by the whole
pipeline. The retriever and reranker infer with batch sizes 32 and 100
respectively, the readers run with batch size 1.

We note that in retrieval, we do not use any approximate K-NN algorithm to
facilitate retrieval of top-K nearest passages and instead do the dot product
with the matrix of passages directly on the CPU. Secondly, we note that we do
not parallelize the inference of generative reader and extractive reader.
Thirdly, notice the difference in extractive reader’s speed with and without
passage reranker is caused by its different input size (see details of
extractive reader’s experiments setup in subsection 3.2). Finally, we compare
the speed of our approach using FiD with 25 and 100 input passages, like in
the original FiD implementation151515We simply pass 100 input passages to the
model trained with 25 passages in the experiment.. The ratios of our
measurements are compared explicitly in Table 14.

| Modules | Rankers
---|---|---
|  | retriever | +reranker
intermediate | retriever | 0.21 | 0.21
passage reranker | - | 1.94
ext. reader | 2.21 | 0.35
gen. reader (25) | 0.55 | 0.55
answer reranker (25) | 3.11 | 3.11
gen. reader (100) | 1.85 | -
answer reranker (100) | 11.67 | -
total | ext | 2.41 | 2.19
gen (25) | 0.76 | 2.70
ext+gen (25) | 6.08 | 6.16
gen (100) | 2.06 | -
ext+gen (100) | 15.94 | -
Table 13: Inference times on NQ-Open in seconds per question. See text for details. Setup ratios | Modules
---|---
| only gen. | ans. reranker | gen pipe. | ext+gen pipe.
gen(100) / gen(25) | 3.36x | 3.75x | 2.71x | 2.62x
rr+gen(25) / gen(25) |  ∗1.00x |  ∗1.00x | 3.55x | 1.01x
gen(100) / rr+gen(25) |  ∗3.36x |  ∗3.75x | 0.76x | 2.58x
Table 14: Ratios of inference times on NQ-Open. First two columns compare the
speed in stage of generating abstractive answer (only gen.) and answer
reranking (ans. reranker). The subsequent columns compare speed of whole
pipeline just with generative reader and no component fusion (gen pipe.) and
full R2-D2 pipeline (ext+gen pipe.). Row gen(100)/gen(25) compares the speedup
of pipeline when using just 25 passages in FiD’s input (denoted as gen(25))
instead of 100 (denoted as gen(100)). Row rr+gen(25)/gen(25) shows speedup
gained from not using passage reranker (denoted as rr). Row
gen(100)/rr+gen(25) compares the speed of using rr and gen(25) instead of
gen(100) (with no passage reranking). Results marked with ∗ are not affected
by passage reranking component, as they only measure speed of pipeline’s
individual component. For instance, table shows that doing answer reranking
with generative reader with just 25 passages at its input runs 3.75x faster
than doing answer reranking with generative reader that uses 100 passages.