# MuGER2: Multi-Granularity Evidence Retrieval and Reasoning for Hybrid
Question Answering

Yingyao Wang1, Junwei Bao2, Chaoqun Duan2, Youzheng Wu2,
Xiaodong He2, Tiejun Zhao1
1Harbin Institute of Technology 2JD AI Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Work was done
during the first author’s internship at JD AI mentored by Junwei Bao:
<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Hybrid question answering (HQA) aims to answer questions over heterogeneous
data, including tables and passages linked to table cells. The heterogeneous
data can provide different granularity evidence to HQA models, e.t., column,
row, cell, and link. Conventional HQA models usually retrieve coarse- or fine-
grained evidence to reason the answer. Through comparison, we find that
coarse-grained evidence is easier to retrieve but contributes less to the
reasoner, while fine-grained evidence is the opposite. To preserve the
advantage and eliminate the disadvantage of different granularity evidence, we
propose MuGER2, a Multi-Granularity Evidence Retrieval and Reasoning approach.
In evidence retrieval, a unified retriever is designed to learn the multi-
granularity evidence from the heterogeneous data. In answer reasoning, an
evidence selector is proposed to navigate the fine-grained evidence for the
answer reader based on the learned multi-granularity evidence. Experiment
results on the HybridQA dataset show that MuGER2 significantly boosts the HQA
performance. Further ablation analysis verifies the effectiveness of both the
retrieval and reasoning designs.111Our code is publicly available in
https://github.com/JD-AI-Research-NLP/MuGER2..

##  1 Introduction

Traditional knowledge-based question answering systems derive answers to
questions based on homogeneous knowledge, including knowledge graphs Bao et
al. (2016); Gu et al. (2021), tables Jauhar et al. (2016); Zhang et al.
(2020), or passages Zhao et al. (2021); Zhou et al. (2022a) and achieve
remarkable performances. However, they neglect a more general scenario
requiring reasoning over heterogeneous data to answer a question. To
investigate this challenge, Chen et al. (2020) propose the hybrid question
answering (HQA) and publish a corresponding dataset, HybridQA. Specifically,
each question of HybridQA is aligned with a table and passages linked to table
cells (abbreviated as links) as knowledge. The answer to the question may come
from either table cells or links. Figure 1 illustrates two examples of
HybridQA. For the first one, the answer ‘Philip Mulkey’ comes from the cell,
while for the second one, its answer ‘Mississippi River’ is derived from the
link.

Figure 1: Two example questions of HQA task.

Existing HQA models usually consist of two components, a retriever to learn
evidence and a reasoner to leverage the evidence to derive the answer.
Intuitively, the heterogeneous data shown in Figure 1 can provide different
granularity evidence to the HQA models, such as coarse-grained column and row
and fine-grained cell and link. Previous models always retrieve coarse or
fine-grained evidence and directly use a span-based reading comprehension
model to reason the answer. For example, Kumar et al. (2021) choose a coarse-
grained region as the evidence, e.g., a table row. On the contrary, Chen et
al. (2020) and Sun et al. (2021) focus on the fine-grained units, table cell
and link. Intuitively, compared with fine-grained evidence, coarse-grained
evidence is easier to be accurately retrieved. However, it contributes less to
the reasoner’s performance since it contains more noisy information. The fine-
grained evidence is just the opposite for the retriever and the reasoner.

To prove this argument, we conduct experiments on the development set of
HybridQA to analyze the retriever and the reasoner performance222In these
experiments, the retriever and reader models are all initialized with BERT-
base. with different granularity evidence. The results of evidence retrieval
and answer reasoning are evaluated with the retrieval recall (R@1) and the F1
score, respectively. As shown in Figure 2, the coarse-grained evidence
achieves better retrieval recalls but lower F1 scores than the fine-grained
evidence. With the granularity changing from coarse to fine, the F1 score
increases while the R@1 declines. The results are consistent with our
hypothesis.

Figure 2: The retriever and reasoner performance with different granularity
evidence. Note that the $F1$ score in this figure is calculated based on the
oracle evidence to evaluate the answer reasoning performance. Intuitively,
$R@1\times F1$ reflects the end-to-end HQA performance.

To preserve the advantage and eliminate the disadvantage of different
granularity evidence, we propose MuGER2, a Multi-Granularity Evidence
Retrieval and Reasoning approach for HQA. In the retrieval stage, a unified
retriever is designed to learn the multi-granularity evidence from the
heterogeneous data involving columns, rows, cells, and links. Compared with
existing methods, the multi-granularity evidence preserves the retrieval
recall and provides more aspects of information to reason the answer. In the
reasoning stage, to avoid the redundant information in the multi-granularity
evidence that lowers the answer reader’s performance, an evidence selector
(E-SEL) is designed to navigate the fine-grained evidence for the reader by
fusing the learned multi-granularity evidence. Together with our multi-
granularity evidence retrieval and reasoning designs, our MuGER2 preserves
both the retrieval recall and the F1 score and further boosts the end-to-end
HQA performance.

We conduct extensive experiments on the HybridQA dataset to verify the
effectiveness of our proposed MuGER2. Experiment results show that MuGER2
achieves remarkable improvements, which outperforms a publicly available
strong baseline by $10.0\%$ EM and $12.9\%$ F1 scores on the end-to-end HQA
performance. Ablation studies verify the effectiveness of the proposed multi-
granularity evidence retrieval and the evidence selector. Moreover, results
compared with different mono-granularity baselines prove the effectiveness of
multi-granularity evidence for HQA.

Our contributions are as follows: (1) We analyze the limitation of HQA systems
which solely rely on only one certain granularity evidence in both the
retriever and reasoner and propose a multi-granularity evidence retrieval and
reasoning approach, which boosts the end-to-end HQA results. (2) We propose a
joint retrieval method that improves the evidence retrieval performance and an
evidence selector to accurately navigate the fine-grained evidence from the
multi-granularity information to preserve the reader’s performance. (3) We
conduct extensive experiments on the HybridQA dataset and show the
effectiveness of MuGER2.

Figure 3: The framework of MuGER2 which performs multi-granularity evidence
retrieval and answer reasoning. Evidence of four kinds of granularity
including
$\\{\hbox{\pagecolor{mypink}{column}},\hbox{\pagecolor{mygreen}{row}},\hbox{\pagecolor{myyellow}{cell}},\hbox{\pagecolor{myblue}{link}}\\}$
are in color.

##  2 Methodology

###  2.1 Task: Hybrid Question Answering

The hybrid question answering (HQA) aims to tackle the answer reasoning over
heterogeneous data, as shown in Figure 1. The input of this task includes a
question $\mathcal{Q}$, a table $\mathcal{T}$, and a set of links
$\mathcal{L}$. Specifically, the table $\mathcal{T}$ consists of cells
$\\{c_{i,j}\\}_{i=1,j=1}^{i=M,j=N}$, where $M$ and $N$ are the numbers of rows
and columns. Each column has a header $h_{j}$ to describe the field of the
cells $\\{c_{i,j}\\}_{i=1}^{i=M}$. A table cell $c_{i,j}$ may link to a subset
of links $\mathcal{L}_{i,j}\subseteq\mathcal{L}$. Given the input, an HQA
system aims to return the answer text $\mathcal{A}$ of the question
$\mathcal{Q}$ by reasoning on the heterogeneous information. In this work,
$\mathtt{In}\text{-}\mathtt{Table}$ and $\mathtt{In}\text{-}\mathtt{Passage}$
respectively represent that the answer is in a cell or in a link.

###  2.2 Model Overview

As described above, coarse-grained evidence is easier to be derived by the
retriever and it makes less contribution to the reasoner while the fine-
grained is just the opposite. To preserve their advantage and get rid of their
disadvantage, we propose MuGER2, a multi-granularity evidence retrieval and
reasoning approach. As shown in Figure 3, in the retrieval stage, MuGER2
adopts a unified retriever to learn the multi-granularity evidence. In the
reasoning stage, an evidence selector is designed to navigate the fine-grained
evidence for the reader based on the multi-granularity evidence. The details
of these modules are introduced in the following sections.

###  2.3 Stage-1: Multi-Granularity Evidence Retrieval

In this section, we first introduce the definition of the multi-granularity
evidence and then introduce the retrieval model of MuGER2.

####  2.3.1  Multi-Granularity Evidence

According to the characteristics of the HQA, we define the multi-granularity
evidence as $\mathcal{E}$. $\mathcal{E}$ involves four kinds of evidence with
different granularity which is denoted as
$\mathcal{E}$$=$$\langle\mathcal{E}_{\mathtt{col}}$,
$\mathcal{E}_{\mathtt{row}}$, $\mathcal{E}_{\mathtt{cell}}$,
$\mathcal{E}_{\mathtt{link}}\rangle$. Specifically,
$\mathcal{E}_{\mathtt{col}}$ indicates the column which includes the answer;
$\mathcal{E}_{\mathtt{row}}$ represents the row that contains the answer; The
evidence $\mathcal{E}_{\mathtt{cell}}$ means the cell that contains the answer
in its cell value; and $\mathcal{E}_{\mathtt{link}}$ indicates the link
containing the answer.

####  2.3.2 Evidence Retriever

To integrate the above mentioned four kinds of evidence into our method, we
adopt their joint probabilities to model the evidence retriever. Formally,
given a question $\mathcal{Q}$ and the multi-granularity evidence candidate
$\mathcal{E}$ derived from $\mathcal{T}$ and $\mathcal{L}$, the evidence
retriever can be formulated as follows:

$p_{ret}(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})=\prod_{t}p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})$
(1)

where $t\in\\{\mathtt{col},\mathtt{row},\mathtt{cell},\mathtt{link}\\}$,
$\mathcal{E}_{t}$ denotes $t$-kind element of $\mathcal{E}$, and
$p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})$ represents the
probability that $\mathcal{E}_{t}$ is the evidence. To compute
$p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})$, we further
introduce a retrieval score $s_{t}$ for $\mathcal{E}_{t}$, which weighs the
possibility that $\mathcal{E}_{t}$ contains the answer, and adopt it as
follows:

$p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})=s_{t}/Z_{t}$ (2)

where $Z_{t}$ is a normalization factor and equal to the sum of retrieval
scores of all $t$-kind evidence. Given that for different evidence $Z_{t}$ is
consistent, we omit it in our implementation, and only use the retrieval score
for evidence selection.

To compute all retrieval scores for the multi-granularity evidence
$\mathcal{E}$, a straightforward method is to train four separate models and
predict the different granularity evidence respectively. However, this method
neglects relations among different granularity evidence. To tackle this
challenge, we propose to adopt a unified framework for all kinds of evidence
and train it in a joint way.

Specifically, we first utilize BERT Devlin et al. (2019) as the encoder to
learn the representation for each evidence. Formally, given evidence
$\mathcal{E}_{t}$, we concatenate $t$, $\mathcal{Q}$, and $\mathcal{E}_{t}$
with “[SEP]” into a sequence and prepend “[CLS]” at its start. Subsequently,
we feed the sequence into BERT. Finally, we adopt the max-pooling operation
over all tokens of the BERT output to obtain the sequence representation
$\mathbf{h}_{t}$. This process is denoted as follows:

$\begin{split}\mathcal{S}_{t}=[\text{[CLS]}&;t;\text{[SEP]};\mathcal{Q};\text{[SEP]};\mathcal{E}_{t}]\\\
\mathbf{h}_{t}=&\mathtt{MaxP}(\mathtt{BERT}(\mathcal{S}_{t}))\end{split}$ (3)

where $t$ is inserted into the input sequence to indicate the kind of the
evidence. After obtaining the representation of $\mathcal{E}_{t}$, we feed it
into a linear projection followed by the sigmoid function to compute the
retrieval score as follows, where $\mathbf{W}\in\mathbb{R}^{K}$ is a
$K$-dimensional trainable parameter.

$s_{t}=\mathtt{sigmoid}{(\mathbf{h}_{t}^{\top}\mathbf{W})}$ (4)

####  2.3.3 Content of Different Evidence

For $\mathcal{E}_{t}$, it refers to different content when $t$ denotes
different kinds of evidence as follows. Notably, the granularity of certain
evidence is determined by what kinds of candidates are ranked, e.g., cells or
rows, but NOT what content a candidate uses.

##### Column

Column evidence aims to take the meaning of a specific column header in the
table into account. The column header usually describes the fields of an
answer. Therefore, we utilize the table header as column evidence. Namely,
$\mathcal{E}_{\mathtt{col}}\\!=\\!h_{j}$.

##### Link

Link evidence indicates a passage that a table cell links to. It is another
form of knowledge in the HQA task and we assign each
$\mathcal{E}_{\mathtt{link}}$ candidate a specific passage.

##### Cell

We formalize cell evidence $\mathcal{E}_{\mathtt{cell}}$ as a concatenation of
its header $h_{j}$, its neighbors in the same row
$\\{h_{j},c_{i,j}\\}_{j=1}^{j=N}$, and its links $\mathcal{L}_{i,j}$. Namely,
$\mathcal{E}_{\mathtt{cell}}\\!=\\![h_{j};$ $\text{[SEP]};$
$\\{h_{j},c_{i,j}\\}_{j=1}^{j=N};$ $\text{[SEP]};$ $\mathcal{L}_{i,j}]$, where
“[SEP]” is the separator. In cell encoding, the above neighbor information
refers to the values of other cells in the same row without their linked
passages. Neighbor information and the corresponding header of a cell are used
as its positional information (row, column) which is essential to distinguish
cells, especially for those with the same value but in different positions.
This process is also applied to the modeling of row evidence.

##### Row

Traditional methods usually concatenate all row cells into an input sequence
to score a table row evidence. However, in the HQA task, a table row may link
to several passages, and directly concatenating all these contents will result
in an overlength of input sequences. Therefore, we score a row based on its
cells. Therefore, to encode and score a row candidate, we first encode all its
cells separately and then aggregate the cell scores by maximization operation
to obtain the row score. Namely, $\mathcal{E}_{\mathtt{row}}$ includes a set
of $\mathcal{E}_{\mathtt{cell}}$.

####  2.3.4 Training

##### Joint Training for Multi-Granularity

As mentioned above, we only adopt the retrieval score for the evidence
selection. Therefore, the core of training of evidence retriever is to learn a
model to produce the retrieval score. Since the retrieval score $s_{t}$ means
the probability that the evidence $\mathcal{E}_{t}$ contains the answer, we
treat the retrieval score prediction as a binary classification and use the
binary cross entropy (BCE) loss as the training objective to learn the model
as follows:

$\mathtt{Loss}_{t}^{bce}={\mathtt{BCE}(s_{t},y_{t})}$ (5)

where $y_{t}\in\\{0,1\\}$ is the label of $\mathcal{E}_{t}$ indicating whether
the evidence $\mathcal{E}_{t}$ contains the answer. Note that, since only the
answer texts are annotated in the HybridQA dataset, we obtain the evidence
labels by distant supervision method like Chen et al. (2020) which basically
follows the principle that the positive evidence contains the answer.

Furthermore, to consider the relative information among different granularity
evidence, we propose to leverage a unified retriever to model them and learn
it in a joint way:

$\mathtt{Loss}^{bce}=\sum_{t}{\mathtt{Loss}^{bce}_{t}}$ (6)

##### Contrastive Learning within Each Granularity

The above joint training method is capable of capturing the relation among
different granularity evidence. However, it neglects the relations among
different candidate evidence within the same granularity. Because it depends
on the BCE loss, which regards each candidate evidence as an individual
instance and does not require any involvement of other instances besides the
given evidence. To alleviate this problem, we propose to leverage contrastive
learning (CL) Hadsell et al. (2006) within candidate evidence with the same
granularity to model their relative similarities and further enhance the
model’s representation and scoring capability.

To construct the training objective, we first define a function
$\mathtt{sim}(\cdot)$ to denote the similarity between different evidence as
follows:

$\displaystyle\mathtt{sim}({\mathbf{h}_{t}},{\mathbf{h}_{t}^{{}^{\prime}}})=\mathbf{h}_{t}\mathbf{W}+\mathbf{h}_{t}^{{}^{\prime}}\mathbf{W}$
(7)

It also requires building positive and negative instances for a given evidence
candidate. For positive instance, following Gao et al. (2021), we set
$\mathcal{E}_{t}^{+}\\!=\\!\mathcal{E}_{t}$. Since there are dropout masks
placed on fully-connected layers as well as attention probabilities during
training of BERT, feeding the same input to the encoder twice we get two
different representations $\mathbf{h}_{t}$ and $\mathbf{h}_{t}^{+}$ for it. To
obtain the negative samples $\mathcal{E}_{t}^{-}$ for the given evidence, like
most existing work with contrastive learning Gao et al. (2021), we regard
other evidence in the same batch as negative ones and the representation of
$\mathcal{E}_{t}^{-}$ is denoted as $\mathbf{h}_{t}^{-}$.

After that, we take a cross-entropy objective with in-batch negatives.
Formally, given a batch, we split it into four groups and each one corresponds
to a granularity. Let $D$ denote all instances in a group, the training
objective is:

$\displaystyle\mathtt{Loss}_{t}^{cl}=-\mathtt{log}\frac{e^{\mathtt{sim}(\mathbf{h}_{t},\mathbf{h}_{t}^{+})/\tau}}{\sum_{\mathbf{h}_{t}^{-}\in
D}e^{\mathtt{sim}(\mathbf{h}_{t},\mathbf{h}_{t}^{-})/\tau}}$ (8)

where $\tau$ is a temperature hyperparameter. Actually, we compute four loss
as Eq. (8) based on four groups and jointly learn them as follows:

$\displaystyle\mathtt{Loss}^{cl}=\sum_{t}\mathtt{Loss}_{t}^{cl}$ (9)

Finally, we combine $\mathtt{Loss}^{bce}$ and $\mathtt{Loss}^{cl}$ to produce
the final objective function:

$\displaystyle\mathtt{Loss}=\mathtt{Loss}^{bce}+\mathtt{Loss}^{cl}$ (10)

###  2.4 Stage-2: Answer Reasoning with the Multi-Granularity Evidence

As shown in Figure 3, answer reasoning module
$p_{rea}(\mathcal{A}|\mathcal{E})$ is responsible for deriving answer based on
the output of the evidence retriever. This module consists of two essential
components, evidence selector and reader. Algorithm 1 shows the whole workflow
of this module.

####  2.4.1 Evidence Selector

As analysed above, multi-granularity evidence is easier to be accurately
scored by the retriever. However, they contain too much redundant information
and brings burden to the reader. To alleviate this issue, the evidence
selector (E-SEL) is proposed to navigate the fine-grained evidence for the
reader based on the multi-granularity evidence.

Specifically, we define two scores $s_{tab}^{i,j}$ and $s_{pass}^{i,j}$ to
respectively indicate the probabilities that the answer is in the cell
$c_{i,j}$ and the links $\mathcal{L}_{i,j}$. When calculating the scores, we
take the multi-granularity evidence into account. Suppose that the link set
$\mathcal{L}_{i,j}$ has $X$ links denoted as $\\{l_{x},x\in[1,X]\\}$. The
$s_{tab}^{i,j}$ and $s_{pass}^{i,j}$ are calculated as follows:

$\displaystyle\begin{split}s_{tab}^{i,j}&=s_{\mathtt{col}}^{j}+s_{\mathtt{row}}^{i}+s_{\mathtt{cell}}^{i,j}\\\
s_{pass}^{i,j}&=s_{\mathtt{col}}^{j}+s_{\mathtt{row}}^{i}+\mathtt{Max}_{x=1}^{x=X}{(s_{\mathtt{link}}^{x})}\end{split}$
(11)

where $s_{col}^{j}$, $s_{row}^{i}$, $s_{cell}^{i,j}$, and $s_{link}^{x}$ are
retrieval scores. After that, we can obtain two global maximum scores
$s_{tab}$ and $s_{pass}$ as follows:

$\displaystyle\begin{split}s_{tab}&=\mathtt{Max}_{i=1,j=1}^{i=M,j=N}(s_{tab}^{i,j})\\\
s_{pass}&=\mathtt{Max}_{i=1,j=1}^{i=M,j=N}(s_{pass}^{i,j})\end{split}$ (12)

These two scores, $s_{tab}$ and $s_{pass}$, indicate that the answer is in the
table or passages by simply comparing their values. Meanwhile, the cell and
link corresponding to them are considered as the navigated evidence for the
reader. Based on $s_{tab}$ and $s_{pass}$, E-SEL is able to judge the answer
type of the current question as follows:

$\mathtt{E\text{-}SEL}(p_{ret}(\cdot))=\left\\{\begin{aligned}
&\mathtt{In}\text{-}\mathtt{Table},&s_{tab}>s_{pass}\\\
&\mathtt{In}\text{-}\mathtt{Passage},&s_{tab}\leq
s_{pass}\end{aligned}\right.$ (13)

where $\mathtt{In}\text{-}\mathtt{Table}$ denotes that the answer to the
question is in the cell, while $\mathtt{In}\text{-}\mathtt{Passage}$ indicates
that the answer is in the link.

After obtaining the answer type, we can navigate the cell or link as the final
fine-grained evidence to derive the answer. Formally, if the type is
$\mathtt{In}\text{-}\mathtt{Table}$, the cell with score $s_{tab}$ will be
selected and its value will be returned as the answer. Otherwise, the link
with the score $s_{pass}$ will be selected and it will be feed to the reader
to extract answer.

####  2.4.2 Reader

For the reader $\mathtt{RC}(\mathcal{Q},{l})$, following existing works Chen
et al. (2020); Sun et al. (2021), we implement it with a span-based reading
comprehension model which adopt BERT as backbone.

Input: question $\mathcal{Q}$, table $\mathcal{T}$, links $\mathcal{L}$

Output: answer text $\mathcal{A}$

1 calculate $p_{ret}(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})$;

2 if  _$\mathtt{E\text{-}SEL}(p_{ret}(\cdot))$
==$\mathtt{In}\text{-}\mathtt{Table}$_ then

3 get the cell $c_{tab}$ with score $s_{tab}$;

4 return cell value of $c_{tab}$ as answer $\mathcal{A}$

5 end if

6else if  _$\mathtt{E\text{-}SEL}(p_{ret}(\cdot))$
==$\mathtt{In}\text{-}\mathtt{Passage}$_ then

7 get the link $l_{pass}$ with score $s_{pass}$;

8 return $\mathtt{RC}(\mathcal{Q},l)$ as answer $\mathcal{A}$

9 end if

Algorithm 1 Answer Reasoning Model | Dev | Test
---|---|---
In-Table | In-Passage | Total | In-Table | In-Passage | Total
EM / F1 | EM / F1 | EM / F1 | EM / F1 | EM / F1 | EM / F1
MQA-QG (unsupervised) | – / – | – / – | – / – | 36.2 / 40.6 | 19.8 / 25.0 | 25.7 / 30.5
Table-Only | 14.7 / 19.1 | 2.4 / 4.5 | 8.4 / 12.1 | 14.2 / 18.8 | 2.6 / 4.7 | 8.3 / 11.7
Passage-Only | 9.2 / 13.5 | 26.1 / 32.4 | 19.5 / 25.1 | 8.9 / 13.8 | 25.5 / 32.0 | 19.1 / 25.0
Hybrider (BERT-base, $\tau$=0.9) | 51.5 / 58.6 | 40.5 / 47.9 | 43.7 / 50.9 | 52.1 / 59.3 | 38.1 / 46.3 | 42.5 / 50.2
Hybrider (BERT-large, $\tau$=0.8) | 54.3 / 61.4 | 39.1 / 45.7 | 44.0 / 50.7 | 56.2 / 63.3 | 37.5 / 44.4 | 43.8 / 50.6
Dochopper | – / – | – / – | 47.7 / 55.0 | – / – | – / – | 46.3 / 53.3
MuGER2-base | 58.2 / 66.1 | 52.9 / 64.6 | 53.7 / 63.6 | 56.7 / 64.0 | 52.3 / 63.9 | 52.8 / 62.5
MuGER2-large | 60.9 / 69.2 | 56.9 / 68.9 | 57.1 / 67.3 | 58.7 / 66.6 | 57.1 / 68.6 |  56.3 / 66.2
Table 1: The EM and F1 scores on HybridQA of different models. Split | Train | Dev | Test | Total
---|---|---|---|---
In Passage | 35,215 | 2,025 | 2,045 | 39,285
In Table | 26,803 | 1,349 | 1,346 | 29,498
Compute | 664 | 92 | 72 | 864
Total | 62,682 | 3,466 | 3,463 | 69,611
Table 2: Statistics of HybridQA dataset.

##  3 Experiments

###  3.1 Implementation Details

####  3.1.1 Dataset

To verify our proposed model, we conduct experiments on
HybridQA333https://github.com/wenhuchen/HybridQA Chen et al. (2020), a dataset
of multi-hop question answering over tabular and textual data. The basic
statistics of HybridQA are listed in Table 2. In the data split, ‘In-Table’
means the answer is a table cell value, and ‘In-Passage’ means the answer
exists in a linked passage. ‘Compute’ means the answer is computed by
performing numerical operations. In this paper, we mainly focus on the first
two types.

####  3.1.2 Settings

Two variants of BERT Devlin et al. (2019) are utilized in MuGER2, namely
MuGER2-base and MuGER2-large. The learning rate is set to $5$e-$5$ and the
training batch size is set to $30$. A batch is divided into 4 groups
corresponding to the 4 kinds of evidence and each group contains 6 instances.

###  3.2 Comparison with Previous Methods

####  3.2.1 Baselines

MQA-QG Pan et al. (2021) is an unsupervised framework that generates multi-hop
questions from the hybrid evidence and uses the generated questions to train
the QA model. Table-Only Chen et al. (2020) only relies on the tabular
information to find the answer by parsing the question into a symbolic form
and executes it. Passage-Only Chen et al. (2020) only uses the hyperlinked
passages to find the answer by retrieving related passages to perform the RC
process. Hybrider Chen et al. (2020) solves the HybridQA using a two-stage
pipeline framework to retrieve a table cell and extract the answer in its
value or linked passages. Dochopper Sun et al. (2021) builds a retrieval index
for each cell and passage sentence, and perform RC on the retrieved evidence.

####  3.2.2 Results and Analysis

We use the EM and F1 as the evaluation metrics to compare the performance of
our MuGER2 and that of baselines. As shown in Table 1, MuGER2-base outperforms
strong baselines of HybridQA task and achieves significant improvements on
both development and test sets. Specifically, its performance exceeds the
Hybrider-larger whose parameter scale is larger than it. This indicates that
our proposed multi-granularity evidence is beneficial for this task. In
addition, compared with the base model, MuGER2-large further obtains gains.

###  3.3 Ablation Study

####  3.3.1 Stage-1: Retrieval

##### Effect of Contrastive Learning

Without the contrastive learning (CL) within each granularity of evidence, the
unified retriever is only trained by the binary cross entropy objective in the
joint way. The results in Table 3 demonstrate that CL brings $0.8\%$ EM and
$1.1\%$ F1 improvements to our model. These improvements are from the ability
of CL to enhance the evidence representation learning by pushing away the
positive and negative evidence representation in the semantic space.

##### Effect of Joint Training

In our method, we adopt the joint training (JT) method to train the unified
retriever to capture the relative information among different granularity. To
prove the effectiveness of this design, we compare it with a straightforward
way which trains four separate retrievers for different granularity evidence.
For fairness, we turn off the contrastive learning and only use the binary
cross entropy objective to train the four separate retrievers and the unified
retriever. The results in Table 3 show that JT brings $1.6\%$ EM and $1.3\%$
F1 improvements to the HQA performance.

Besides the end-to-end HQA performance, we further give the retrieval recall
of the four different granularity evidence and the complete multi-granularity
evidence in Figure 4. The retrieval recall (R@1) indicates whether the
retrieved top-$1$ candidate of the corresponding evidence contains the answer.
The results show that JT improves the retrieval performance on all these
retrieval tasks. It is proved that sharing knowledge from different
granularity brings benefit to the evidence retrieval.

Model | In-Table | In-Passage | Total
---|---|---|---
EM / F1 | EM / F1 | EM / F1
MuGER2-base | 58.2 / 66.1 | 52.9 / 64.6 | 53.7 / 63.6
w/o CL | 59.1 / 66.9 | 50.6 / 62.2 | 52.9 / 62.5
w/o JT | 54.2 / 61.9 | 51.5 / 63.2 | 51.3 / 61.2
w/o E-SEL | 38.6 / 47.0 | 52.7 / 63.8 | 45.9 / 55.7
Table 3: Ablation results based on MuGER2-base. Figure 4: R@1 of different
granularity evidence.

####  3.3.2 Stage-2: Reasoning

##### Effect of Evidence Selector

To show the effectiveness of the evidence selector (E-SEL), we compare it with
a straightforward answer reasoning method without any fine-grained evidence
navigation. This method simply flattens the retrieved top-1 multi-granularity
evidence and feeding the sequence into the RC process. Results in Table 3 show
that the evidence selector increases the results by $7.8\%$ EM and $7.9\%$ F1
compared with the above reasoning method. This is due to its ability of
accurately navigating the cell or link based on the multi-granularity
evidence. The evidence selector reduces the burden of the RC model. In
contrast, the above straightforward method brings redundant and noisy
information which leads to weak RC performance.

Granularity | In-Table | In-Passage | Total
---|---|---|---
EM / F1 | EM / F1 | EM / F1
Column | 10.4 / 16.9 | 17.1 / 22.9 | 14.1 / 20.1
Row | 52.0 / 59.3 | 49.6 / 60.1 | 49.3 / 58.3
Cell | 21.6 / 29.6 | 0.0 / 1.4 | 8.4 / 12.3
Link | 13.0 / 18.3 | 40.6 / 50.1 | 28.8 / 36.5
Multi (ours) | 58.2 / 66.1 | 52.9 / 64.6 | 53.7 / 63.6
Table 4: HQA results of different granularity evidence.

###  3.4 Different Granularity Evidence

####  3.4.1 Baselines

To compare the HQA performance based on different granularity evidence, we
conduct baselines for the four kinds of evidence including column, row, cell,
and link. Specifically, the baselines of column, row and link granularity
first retrieve the top-1 evidence of the selected granularity, and then
flatten the retrieved evidence into a sequence to derive the answer by the RC
model. While for the cell granularity, the value of the retrieved top-1 cell
is directly returned as the answer. Note that, all the retrievers and the RC
models of these baselines are initialized by the pre-trained BERT-base model.

Figure 5: An example of evidence retrieval and answer reasoning results based
on different granularity evidence.

####  3.4.2 Results and Analysis

The end-to-end HQA results based on different granularity evidence are shown
in Table 4 and Figure 2. The coarse-grained Column evidence achieves the best
retrieval recall. This is because the question domains are easily mapped to a
certain table header. However, reasoning the answer in a column is difficult
because cells in a column are nearly indistinguishable in the absence of other
semantic information, which leads to the lowest HQA performance. In contrast,
the fine-grained Cell and Link evidence reduces the burden of answer reasoning
but lowers the retrieval recall due to its huge candidates and limited
semantic information. Moreover, only adopting the cell or link evidence cannot
solve all questions which further causes low HQA performance. Row granularity
achieves the best performance among these baselines since it balances the
retrieval recall and RC performance to a certain extent. Compared with these
baselines, our Multi granularity evidence brings remarkable improvements to
the end-to-end HQA performance by preserving the evidence retrieval recall and
the RC performance at the same time.

##  4 Case Study

Through statistics on the development set of HybridQA, we find there are 135
questions whose answers are only recalled by our multi-granularity evidence,
but failed to be recalled by the coarse- and fine- grained evidence. An
example of such a question is given in Figure 5. The table along with linked
passages on the left is the given knowledge to answer the question. The answer
facebook watch is contained in link1 of cell(2,0) where (2,0) represents
(row2, column0).

The table on the right lists the retrieved different granularity evidence. It
is observed that the methods based on row, cell and link granularity evidence
all retrieve incorrect evidence, including <row4, cell(4,0), link2>. The
reason is that the wrong evidence has high semantic similarity with the
question which causes a higher ranking score. Although the column granularity
based method retrieves the evidence <column0> correctly, the reading
comprehension model fails to derive the answer text accurately from the
column.

In contrast, our method retrieves evidence including <column, row, cell,
link>. The evidence is correctly retrieved except for the cell since the
question is an in-passage question. The result demonstrates that the multi-
granularity evidence improves the retrieval recall which benefits from
learning the interactive information. The evidence selector further help to
navigate the fine-grained cell or link based on the multi-granularity
evidence.

##  5 Error Analysis

Figure 6: Error of the three modules of MuGER2-large.

To further analyze our method, we also make statistics on error cases in the
prediction of MuGER2-large. The error statistics are based on the development
set of HybridQA. As shown in Figure 6, these cases can be divided into three
categories according to the causes. The first category is caused by evidence
retrieval errors, which means the retrieved evidences overlook the answer.
According to our statistics, multi-granularity evidence has an advantage in
improving coverage of answers. Therefore, the proportion of this category is
very low, only 3%. The second category is caused by the evidence selector,
which accounts for about 40% in the whole results. There are two reasons for
the evidence selector to make the error. Firstly, although multi-granularity
evidence improves the coverage, it also introduces noise. As a result, it may
also lead to a mis-navigation of the fine-grained cell or link. In our
results, it accounts for about 33%. In addition, for those questions which are
offered correct cells or links, they also may be answered incorrectly due to
the wrong judgment of the answer type. It accounts for about 7% in our
results. The last category is caused by the reading comprehension module.
Limited by the performance of the RC model, even if feed correct fine-grained
evidence, the RC model also may produce the wrong answer. According to our
analysis, this category has the maximum proportion, accounting for about 57%
of the whole error cases.

##  6 Discussion

More recently, two novel approaches, MATE Eisenschlos et al. (2021) and MITQA
Kumar et al. (2021), are proposed and achieve significant results,
respectively $70.0$ and $71.9$ F1 scores on the HybridQA test set.
Specifically, MATE mainly focuses on strengthening the ability to encode large
tables with multi-view attention. While MITQA designs a multi-instance
training method based on distant supervision to filter the noise from multiple
answer spans. In contrast, our MuGER2 puts more effort into integrating multi-
granularity evidence for the HQA problem to enhance the reasoning ability. We
think that the two novel approaches and our model pay attention to different
perspectives for solving the HQA problems, and it’s probably to combine them
to achieve better performance.

##  7 Related Work

Recently, many researchers attempt to tackle the question answering task using
heterogeneous knowledge Sun et al. (2019); Sawant et al. (2019); Zayats et al.
(2021); Li et al. (2021); Xiong et al. (2019); Zhou et al. (2022b). However,
traditional researches only combine the information of heterogeneous
knowledge, but fail to answer questions that require reasoning over
heterogeneous data. To fill in the gap, the hybrid question answering (HQA)
task together with the HybridQA dataset is proposed by Chen et al. (2020).
HybridQA provides a WiKiTable Pasupat and Liang (2015) along with its
hyperlinked Wikipedia passages Rajpurkar et al. (2016) as evidence for each
question. Based on the HybridQA, Chen et al. (2021a) further propose the OTT-
QA dataset, which requires the system to retrieve relevant tables and text for
the given questions. Zhu et al. (2021) and and Chen et al. (2021b) propose
TAT-QA and FinQA requiring numerical reasoning over the heterogeneous data.

Existing works on HybridQA usually retrieve mono-granularity evidence from the
heterogeneous data to derive the answer. Hybrider Chen et al. (2020) proposes
a two-phase pipeline framework to retrieve a table cell as the evidence and
feeds its value and linked passages into an RC model to extract the answer.
Dochopper Sun et al. (2021) propose an end-to-end multi-hop retrieval method
to directly retrieve a passage sentence or a cell value as the evidence. In
addition, Pan et al. (2021) explore an unsupervised multi-hop QA model MQA-QG,
which can generate human-like multi-hop training data from heterogeneous data
resources.

##  8 Conclusion

We propose MuGER2, a multi-granularity evidence retrieval and reasoning
approach for hybrid question answering (HQA). In our method, a unified
retriever is designed to learn multi-granularity evidence and an evidence
selector is proposed to navigate fine-grained evidence for the reader.
Experiment results show that the MuGER2 boosts the end-to-end HQA performance
and outperforms the strong baselines on the HybridQA benchmark.

## Limitations

In this paper, we focus on the hybrid question answering task, in which the
answers to most questions can be extracted from the cell values or linked
passages using the reading comprehension model. Although our MuGER2 performs
well on this task, one limitation is that it cannot answer the questions
needing numerical operations, such as count, compare and etc. To enable our
model to answer more complex questions, in future work we will develop the
numerical reasoning capabilities of our model.

## Acknowledgment

The authors would like to thank the anonymous reviewers for their thoughtful
comments. The work of this paper is funded by the projects of National key
research and development program of China (No.2020YFB1406902) and the National
Key Research and Development Program of China (No. 2020AAA0108600).

## References

  * Bao et al. (2016) Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016.  Constraint-based question answering with knowledge graph.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 2503–2514, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Chen et al. (2021a) Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021a.  Open question answering over tables and text.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Chen et al. (2020) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1026–1036, Online. Association for Computational Linguistics.
  * Chen et al. (2021b) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021b.  FinQA: A dataset of numerical reasoning over financial data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eisenschlos et al. (2021) Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William Cohen. 2021.  MATE: Multi-view attention for table transformer efficiency.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7606–7619, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gu et al. (2021) Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021.  Beyond iid: three levels of generalization for question answering on knowledge bases.  In _Proceedings of the Web Conference 2021_ , pages 3477–3488.
  * Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2, pages 1735–1742.
  * Jauhar et al. (2016) Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016.  Tables as semi-structured knowledge for question answering.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 474–483, Berlin, Germany. Association for Computational Linguistics.
  * Kumar et al. (2021) Vishwajeet Kumar, Saneem Chemmengath, Yash Gupta, Jaydeep Sen, Samarth Bharadwaj, and Soumen Chakrabarti. 2021.  Multi-instance training for question answering across table and linked text.  _ArXiv preprint_ , abs/2112.07337.
  * Li et al. (2021) Alexander Hanbo Li, Patrick Ng, Peng Xu, Henghui Zhu, Zhiguo Wang, and Bing Xiang. 2021.  Dual reader-parser on hybrid textual and tabular evidence for open domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4078–4088, Online. Association for Computational Linguistics.
  * Pan et al. (2021) Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, and William Yang Wang. 2021\.  Unsupervised multi-hop question answering by question generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5866–5880, Online. Association for Computational Linguistics.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1470–1480, Beijing, China. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Sawant et al. (2019) Uma Sawant, Saurabh Garg, Soumen Chakrabarti, and Ganesh Ramakrishnan. 2019.  Neural architecture for question answering using a knowledge graph and web corpus.  _Information Retrieval Journal_ , 22(3):324–349.
  * Sun et al. (2019) Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019.  PullNet: Open domain question answering with iterative retrieval on knowledge bases and text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2380–2390, Hong Kong, China. Association for Computational Linguistics.
  * Sun et al. (2021) Haitian Sun, William W Cohen, and Ruslan Salakhutdinov. 2021.  End-to-end multihop retrieval for compositional question answering over long documents.  _ArXiv preprint_ , abs/2106.00200.
  * Xiong et al. (2019) Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019.  Improving question answering over incomplete KBs with knowledge-aware reader.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4258–4264, Florence, Italy. Association for Computational Linguistics.
  * Zayats et al. (2021) Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021.  Representations for question answering from documents with tables and text.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2895–2906, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020.  Table fact verification with structure-aware transformer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1624–1629, Online. Association for Computational Linguistics.
  * Zhao et al. (2021) Jing Zhao, Junwei Bao, Yifan Wang, Yongwei Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2021.  RoR: Read-over-read for long document machine reading comprehension.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1862–1872, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhou et al. (2022a) Yongwei Zhou, Junwei Bao, Chaoqun Duan, Haipeng Sun, Jiahui Liang, Yifan Wang, Jing Zhao, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022a.  OPERA: Operation-pivoted discrete reasoning over text.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1655–1666, Seattle, United States. Association for Computational Linguistics.
  * Zhou et al. (2022b) Yongwei Zhou, Junwei Bao, chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022b.  UniRPG: Unified discrete reasoning over table and text as program generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3277–3287, Online. Association for Computational Linguistics.