# Enhance Information Propagation for Graph Neural Network by Heterogeneous
Aggregations

Dawei Leng , Jinjiang Guo, Lurong Pan, Jie Li, Xinyu Wang
_AIDD Group_
Global Health Drug Discovery Institute, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
The first and corresponding author

###### Abstract

Graph neural networks are emerging as continuation of deep learning success
w.r.t. graph data. Tens of different graph neural network variants have been
proposed, most following a neighborhood aggregation scheme, where the node
features are updated via aggregating features of its neighboring nodes from
layer to layer. Though related research surges, the power of GNNs are still
not on-par-with their counterpart CNNs in computer vision and RNNs in natural
language processing. We rethink this problem from the perspective of
information propagation, and propose to enhance information propagation among
GNN layers by combining heterogeneous aggregations. We argue that as richer
information are propagated from shallow to deep layers, the discriminative
capability of features formulated by GNN can benefit from it. As our first
attempt in this direction, a new generic GNN layer formulation and upon this a
new GNN variant referred as HAG-Net is proposed. We empirically validate the
effectiveness of HAG-Net on a number of graph classification benchmarks, and
elaborate all the design options and criterions along with.

##  1 Introduction

Success of deep learning in computer vision and natural language processing
has recently boosted flood of research on applying neural networks to graph
data (Wu et al., 2020). Graph is a simple yet versatile data structure jointly
described by sets of nodes and edges. Aside from image and text data we’re
familiar, lots of real world data are better described as graph and thus
processed by graph neural networks, such as social networks (Fan et al.,
2019), financial fraud detection (Wang et al., 2020), knowledge graph (Zhang
et al., 2020), biology interaction network (Higham et al., 2008), small
molecule in drug discovery (Hu et al., 2019), to name a few.

Since the seminal works (Kipf and Welling, 2016; Hamilton et al., 2017), tens
of different graph neural network variants have been proposed, emphasizing
different graph properties and design options. GNN research routes can be
roughly divided into two categories: spectral based and spatial based.
Spectral based GNNs try to approximate CNN’s convolution by defining Fourier
transform on graph (Kipf and Welling, 2016) and thus where the name graph
convolution network comes from. The major limitation of spectral based GNNs is
that graph convolution is defined on a fixed global graph, thus not suitable
for tasks where graph structure is changing from sample to sample. Meanwhile,
most recent works take the spacial-based direction (Xu et al., 2018; Hu et
al., 2019; Gao and Ji, 2019; Veličković et al., 2017; Gilmer et al., 2017),
i.e., GNNs process the graph data following a neighborhood aggregation scheme,
where the node features at layer $k$ are updated via aggregating features of
its neighboring nodes from layer $k-1$.

GNN’s neighborhood aggregation is a direct imitation of CNN’s convolution in
spatial dimensions. However, unlike image in computer vision where data reside
on regular grid, graph data are intrinsically orderless. For an undirected
graph $G=(V,E)$ there’re only nodes $V$ and edges $E$ defined. By "orderless",
it means from the perspective of any center node $v\in V$, there’s no way to
tell which node $u\in\mathcal{N}(v)$ is its $n$-th neighbor, where
$\mathcal{N}(v)$ stands for the neighborhood of node $v$. This special
property of graph impedes GNNs from replicating CNN’s success on image data
and RNN’s success on sequential data.

Most work in spatial-based GNNs can be formally summarized by the following
two-stage architecture (Wu et al., 2020; Hu et al., 2019): first propagate
node information among each other by neighborhood aggregation, then form the
whole graph representation by a read-out function. The $k$-th layer of a
spatial-base graph neural network is like

$h_{v}^{(k)}=\mathbf{C}^{(k)}(h_{v}^{(k-1)},\mathbf{A}^{(k)}(\\{(h_{v}^{(k-1)},h_{u}^{(k-1)},e_{uv}),u\in\mathcal{N}(v)\\}))$
(1)

in which $h_{v}^{(k)}$ is the feature vector of node $v$ at the $k$-th layer,
$e_{uv}$ is the edge feature vector between node $u$ and $v$, and
$\mathcal{N}(v)$ is usually the one-hop neighbors. Function
$\mathbf{A(\cdot)}$ aggregates node features in the neighborhood of $v$, and
$\mathbf{C(\cdot)}$ combines features both from center node and aggregated
from its neighborhood by $\mathbf{A(\cdot)}$. To obtain the entire graph’s
representation $h_{G}$, the READOUT function pools node features from the
final iteration $K$ as

$h_{G}=\mathbf{R}(\\{h_{v}^{(K)}|v\in V\\})$ (2)

Normally a spatial-base graph neural network consists of a stack of multiple
aggregation layers and finally one readout layer. Published works differ in
either $\mathbf{A(\cdot)}$, $\mathbf{C(\cdot)}$ or $\mathbf{R(\cdot)}$, among
which aggregation function $\mathbf{A(\cdot)}$ is the most important part
because it determines how information propagates among nodes. Due to the
orderless property of graph data, function $\mathbf{A(\cdot)}$ must be
permutation-free, i.e., for any given center node, all its neighbor nodes must
be treated equally. This drastically restricts possible choices for the
aggregation function, commonly $\\{max,min,mean,sum,mul,att\\}$ in which $att$
is an attention operator as in (Veličković et al., 2017). The permutation-free
restriction is also true for read-out function $\mathbf{R(\cdot)}$.

Usually one certain neighborhood aggregation operator is chosen for GNN
design. For example, the seminal work GraphSAGE (Hamilton et al., 2017) takes
the layer formulation as

$h_{v}^{(k)}=\phi_{1}(h_{v}^{(k-1)})+\phi_{2}(mean({h_{u}^{(k-1)},u\in\mathcal{N}(v)}))$
(3)

where $\mathbf{A(\cdot)}=mean({h_{u}^{(k-1)},u\in\mathcal{N}(v)})$,
$\mathbf{C(\cdot)}=sum(\phi_{1},\phi_{2})$ and neighborhood aggregation takes
the $mean$ operator. GIN (Xu et al., 2018) argues that $mean$ operator loses
neighborhood size information, and proposes the layer formulation as

$h_{v}^{(k)}=\phi((1+\epsilon)h_{v}^{(k-1)}+\sum_{u\in\mathcal{N}(v)}h_{u}^{(k-1)})$
(4)

where $\mathbf{A(\cdot)}=\sum_{u\in\mathcal{N}(v)}h_{u}^{(k-1)})$,
$\mathbf{C(\cdot)}=\phi(sum(1+\epsilon,1)$ and neighborhood aggregation takes
the $sum$ operator. As further improvement, (Hu et al., 2019) incorporates
edge features $e_{u,v}$ by taking the layer formulation as

$h_{v}^{(k)}=\phi((1+\epsilon)h_{v}^{(k-1)}+\sum_{u\in\mathcal{N}(v)}ReLU(h_{u}^{(k-1)}+e_{u,v})$
(5)

where
$\mathbf{A(\cdot)}=\sum_{u\in\mathcal{N}(v)}ReLU(h_{u}^{(k-1)}+e_{u,v})$,
$\mathbf{C(\cdot)}=\phi(sum(1+\epsilon,1)$ and neighborhood aggregation also
takes the $sum$ operator. (Gilmer et al., 2017) takes the layer formulation as

$h_{v}^{(k)}=\phi_{1}(h_{v}^{(k-1)})+aggr_{u\in\mathcal{N}(v)}h_{u}^{(k-1)}\phi_{2}(e_{u,v})$
(6)

in which neighborhood aggregation $aggr$ takes either $mean$, $max$ or $sum$
operator.

From the perspective of information propagation, when only permutation-free
operators are allowed, information loss after neighborhood aggregation is
always inevitable, since there’s no way to differentiate among neighbor nodes,
and consequently, it won’t be possible to recover the input graph $G$ by
aggregation result $G^{\prime}$. Consider the commonly used $mean$ operator,
after each aggregation layer, features from different nodes are averaged, the
result graph $G^{\prime}$ will be a blurred version of the input graph $G$.
Deeper the graph neural network, more blurred the result is. With such a lossy
intermediate representation, it’ll be hard for the neural network to fulfill
the downstream learning task effectively.

In this manuscript we try to improve GNN’s performance from the perspective of
enhancing information propagation from shallow to deep layers. We argue that
as richer information are propagated, the discriminative capability of
features formulated by GNN can benefit from there. As our first attempt in
this direction, we propose to enhance information propagation by combining
heterogeneous aggregations in function $\mathbf{A}(\cdot)$. The underlining
philosophy is straightforward: each aggregation operator extracts/describes
different aspect of the input graph $G$, by combining different aggregation
operators, the information propagation loss can be mitigated, thus allowing
more effective features for downstream task to propagate to deep layers. With
this in mind, a new generic GNN layer formulation and upon this a new GNN
variant referred as HAG-Net is proposed. We empirically validate the
effectiveness of HAG-Net on a number of graph classification benchmarks, and
elaborate all the design options and criterions along with. We focus on graph-
level tasks here whereas the same technique can be also applied to node-level
tasks without any difficulty.

Figure 1: Network structure of HAG-Net. The proposed network is designed to be
generic, the pyramid feature stacking by multiple READOUT layers and dense
connection among neighborhood aggregation layers are optional. The
neighborhood aggregation layer is formulated by eq.7 and READOUT layer is
formulated by eq.8

##  2 Method

We begin by reformulating eq.1. For graph $G=(V,E)$ with $v,u\in V$ and
$e_{uv}\in E$, the generic layer formulation with heterogeneous aggregations
is as

$h_{v}=\psi(\mathbf{C}(h_{v},\bigoplus_{i}^{M-1}\phi_{i}(\mathbf{A}_{i,u\in\mathcal{N}(v)}(\\{(h_{v},h_{u},e_{uv})\\}))))$
(7)

In which $\mathbf{A}_{i}(\cdot),i=0\cdots M-1$ are $M$ different aggregation
operators, $\bigoplus$ is the merge operator for $M$ neighborhood aggregation
results, and $\mathbf{C}(\cdot)$ updates center node $v$’s feature with the
merged aggregation result. $\psi$ and $\phi_{i}$ are linear/non-linear
transform functions, and layer index $k$ is omitted here for conciseness, the
left-hand computation is always about layer $k-1$ if not specified otherwise.

For node-level tasks, the node representation $h_{v}$ from the final
aggregation layer is usually used for prediction. For graph-level tasks, the
READOUT function aggregates node features from the final aggregation layer to
form the entire graph’s representation as

$h_{G}=\psi(\bigoplus_{i}^{M-1}\phi_{i}(\mathbf{A}_{i,v\in V}(\\{h_{v}\\})))$
(8)

Compared to eq.2, the READOUT function here also benefits from the enhanced
information propagation by heterogeneous aggregations. Note for READOUT
function the aggregations are performed over all the graph nodes instead of
local neighborhood.

###  2.1 Design Options

#### Aggregation operators

Commonly used aggregation operators include $\\{max,sum,mean,att\\}$.
Aforementioned $\\{mul,min\\}$ operators though satisfying the permutation-
free restriction, could easily cause computation instability issues, thus
seldom used for neighborhood aggregation in practice. $sum$ and $mean$
operators act similarly, though $sum$ is theoretically preferred over $mean$
(Xu et al., 2018) because it keeps the neighborhood size information, there’s
latent risk that computation overflow could happen for large graph, where
$mean$ is a much safer choice. $mean(\mathcal{N}(v))$ is actually the 0-order
statistical moment of $\mathcal{N}(v)$, higher order moments such as
$variance$, $skewness$ and $kurtosis$ also meet the permutation-free
restriction. $max$ operator is widely used in neural networks in fields such
as computer vision and natural language processing, where it usually acts as
pooling function. $att$ from (Veličković et al., 2017) is a transformer like
multi-head self-attention for $\mathcal{N}(v)$ following by a $sum$ operator,
thus it acts more like $sum(\varphi(\mathcal{N}(v)))$. In this manuscript we
focus on operator set $\\{max,sum,mean,att\\}$, and leave other operators for
future investigation.

#### Merge heterogeneous aggregations

For $\bigoplus$ operator in eq.7 and 8, there’re two commonly used options
$\\{cat,sum\\}$ in which $cat$ means concatenation. The transform function
$\phi_{i}$ after each aggregation result is commonly implemented by a stack of
dense layers.

#### Update center node features

In eq.7, center node features $h_{v}$ will be updated with information
aggregated from neighborhood by function $\mathbf{C}(\cdot)$. Possible choices
include $\\{sum,max,cat,rnn\\}$, in which $rnn$ (Li et al., 2015) is an RNN
cell for example `LSTMCell` or `GRUCell` in Pytorch with merged neighborhood
aggregation result as hidden state and center node feature as input. Note this
sequential setup is totally artificial, their roles for the RNN cell can be
exchanged at will. We leave $mean$ method out here because it acts quite
similar with $sum$, possibly due to the following transform $\psi$ and batch
normalization built-in. Like $\phi_{i}$, The transform function $\psi$ is
commonly implemented by a stack of dense layers.

###  2.2 Model Structure

With the generic neighborhood aggregation layer as eq.7 and READOUT layer as
eq.8, we build our graph neural network HAG-Net for graph-level tasks by
stacking multiple neighborhood aggregation and READOUT layers. The simplest
structure is just a sequential stacking a multiple neighborhood aggregation
layers and one READOUT layer, the output of the READOUT layer will be used as
the representation for the whole graph. Former works such as (Xu et al., 2018;
Gao and Ji, 2019) implement complex structures by using features from
intermediate layers. We follow this idea, and design optional pyramid feature
structure and dense connection among intermediate layers. The complete network
structure is illustrated in Figure 1.

For READOUT layers in Figure 1, we empirically find that downstream task
performance can benefit marginally from restricting their weights tied if
pyramid structure is enabled. The downstream task classifier is implemented as
a stack of 3 dense layers.

With heterogeneous aggregations the neighborhood aggregation layer and READOUT
layer could be implemented as numerous variants with different options. Plus
there’re variation options within the model structure of HAG-Net itself. To
determine these options, we treat them as model hyper-parameters and tune them
by human expert as well as automatic grid search, the details are described in
section 3.3.

##  3 Experiments

In this section we will try to answer the following questions through
experiments:

  * •

Q1. Whether combining heterogeneous aggregations would improve GNN’s
performance?

  * •

Q2. What aggregation operator combination is the best, will larger combination
always gear up model performance?

  * •

Q3. How to choose proper aggregation operator combination in practice?

For real-world case study, in the following we’ll conduct experiments on drug
discovery datasets, where GNN is used to predict whether an organic small
molecule is active w.r.t. certain biology target. A molecule is converted to
graph representation by treating each atom as node and covalent bond between
atoms as edge, see Figure 2. We didn’t choose benchmark datasets used
previously in (Xu et al., 2018; Gao and Ji, 2019; Hamilton et al., 2017; Liao
et al., 2019) due to their limited sample size, usually less than 1K. All GNN
models we’ve tested present large performance variation with different data
splitting and weights initialization on them. Instead, we collect 5 datasets
from drug discovery industry with sample size >5K. All these datasets are
binary, vary both in size (from 7K to 76K) and class distribution (from
balanced to highly biased). The details are given below.

Figure 2: A molecule is converted to graph representation by treating each
atom as node and covalent bond between atoms as edge, then a GNN with
classification head is used to predict its activity. For general purpose
different covalent bonds are all modeled into the same typed edges. With this
we can do comparison among different GNN models, though it’s not the most
accurate way to model a molecule.

###  3.1 Dataset

#### Antivirus7k

This is the smallest among all the 5 datasets, with 7,305 samples in total,
and P/N ratio = 0.49 / 0.51. These’re phenotypic records of antiviral
bioactivity from various species and in vitro assays, collected from
commercial database. We choose EC50 <= 100nM as cutoff threshold.

#### CYP2C9V12k

This dataset is from recently published TDC project (Huang et al., 2020), with
12,092 samples in total and P/N ratio = 0.67 / 0.33. The CYP P450 genes are
involved in the formation and breakdown (metabolism) of various molecules and
chemicals within cells. Specifically, the CYP P450 2C9 plays a major role in
the oxidation of both xenobiotic and endogenous compounds (Veith et al.,
2009).

#### Malaria16k

This dataset contains 16,933 samples in total with P/N ratio = 0.93 / 0.07.
It’s curated from public dataset of Malaria sensitivities assays (Kato et al.,
2016) by removing molecule chirality and merging consequent duplicates by `OR`
operation.

#### Mtb18k

This is a public dataset with 18,886 samples in total, and P/N ratio = 0.88 /
0.12. These’re phenotypic records of in vitro assays against Mycobacterium
tuberculosis (Lane et al., 2018). We choose activity cutoff of 10 $\mu$M.

#### Globalvirus76k

This is the biggest among all the 5 datasets, with 76,247 samples in total and
P/N ratio = 0.57 / 0.43. These’re combination of target based and phenotype
based records of antiviral bioactivity from various species and in vitro
assays, collected from commercial database. We choose EC50 <= 100nM as cutoff
threshold.

###  3.2 Metrics

With all the datasets being about binary classification task, commonly ER and
AuROC metrics are used to evaluate model performance. Whereas for dataset with
highly biased class distribution, AuROC will be dominated by the major class,
thus we propose to use AuPR metric instead for model evaluation, specifically,
the harmonic average of AuPRs of both positive and negative classes are
reported in the following experiments.

There’s a less studied topic in GNN research but we find important in
production: convergence stability. During our evaluation of various GNN
models, we notice that some models present much higher variance than others
along the convergence progress, see Figure 3 for an illustration. Model with
large convergence variance is not reliable in production environment since
evaluation data is always limited. To measure the model convergence variance
quantitatively, we propose to use median filtering method. For any convergence
metric curve $x$, we first smooth the curve by median filtering and then
compute the standard variance of their difference, i.e.,

$mstd=std(x-f_{median}^{w}(x))$ (9)

in which $f_{median}^{w}$ is the median filter with window size = $2*w+1$. We
set $w=5$ for all the experiments.

###  3.3 Hyper-parater Determination

The proposed HAG-Net is a generic structure supporting numerous variants with
different combinations of design options. With this versatility, we explore
answers to aforementioned questions Q1, Q2 and Q3. We tune hyper-parameters on
Antivirus7k dataset because its suitable size and balanced class distribution,
then results are reported on all the 5 datasets studied.

For answer to Q3 first, we tune hyper-parameters by human expert at the
initial stage. We choose to copy GIN (Xu et al., 2018) structure for HAG-Net
as baseline. Better hyper-parameter combinations are then explored by human
expert heuristically. This process converges to configuration set cfg1 in
Table 1 after dozens of trials. Evaluation results of HAG-Net with
configuration set cfg1 on all the 5 datasets are given in Table 2.

We then further tune the hyper-parameters by automatic grid search. We use
cfg1 as start point, and resort to Optuna (Akiba et al., 2019) package for the
searching. This process runs for approximately one week on a cluster with 8
P100 GPUs. The hyper-parameter set with the best AuPR value is selected as the
final decision, refer to configuration set cfg2 in Table 1 for details. And
the whole process answers question Q3.

For questions Q1 & Q2, by comparing cfg2 with cfg1 and evaluation results in
Table 2, we can see that GNN’s performance can surely benefit from combining
heterogeneous aggregations, but this benefit is not always consistent with the
combination size. In the human expert optimized cfg1, both
$\\{\mathbf{A}_{aggr}\\}$ and $\\{\mathbf{A}_{RO}\\}$ are combinations of 2
different aggregation operators; whereas in grid search optimized cfg2, for
$\\{\mathbf{A}_{RO}\\}$, single operator $max$ is chosen over previous
combination of $\\{max,sum\\}$. And during the hyper-parameter tuning process,
we notice that for both $\\{\mathbf{A}_{aggr}\\}$ and $\\{\mathbf{A}_{RO}\\}$,
combination with $\geq 3$ aggregation operators never wins out. One possible
reason might be that the investigated operator set $\\{max,sum,mean,att\\}$
here is quite limited, thus only marginal complementary effect could be
obtained from larger combination.

Table 1: Hyper-parameter configuration set for HAG-Net∗

Name | # ALayer | $\\{\mathbf{A}_{aggr}\\}$ | $\bigoplus_{aggr}$ | $\mathbf{C}$ | $\\{\mathbf{A}_{RO}\\}$ | $\bigoplus_{RO}$ | Pyramid | RO Tied | DC
---|---|---|---|---|---|---|---|---|---
cfg1 | 5 | $max,mean$ | $sum$ | $rnn$ | $max,sum$ | $sum$ | True | True | True
cfg2 | 5 | $max,sum$ | $sum$ | $cat$ | $max$ | - | True | True | False

  * *

"# ALayer" is the number of aggregation layers, $\\{\mathbf{A}_{aggr}\\}$ is
the set of aggregation operators in eq.7 and $\\{\mathbf{A}_{RO}\\}$ is the
set of aggregation operators in eq.8. $\bigoplus_{aggr}$ is the merge operator
in eq.7 and $\bigoplus_{RO}$ is the merge operator in eq. 8. When "Pyramid" is
True, there will be READOUT layer attached to each aggregation layer as in
Figure 1, othewise it will be a total sequential structure. "RO Tied"
indicates whether the weights of READOUT layers are tied. "DC" indicates
whether there are dense connections among aggregation layers.

###  3.4 Performance Comparison

In this subsection we benchmark the performance of the proposed HAG-Net with
different hyper-parameter configurations as in Table 1. Results from multiple
state-of-the-art models such as GIN (Xu et al., 2018), GUNet (Gao and Ji,
2019) and model from DeepChem (Ramsundar et al., 2019) are also reported on
the 5 datasets studied. GIN is a very popular model according to the
leaderboard of OGB project (Hu et al., 2020). For GUNet we use
$\\{\mathbf{A}_{RO}\\}=\\{sum\\}$ and the same classifier as in GIN. The model
from DeepChem is specially designed for small moleclue tasks, both in network
structure and input node features.

All the models are implemented with Pytorch 1.6.0. For model from DeepChem,
node/atom features specially customized from DeepChem itself is used, which is
of dimension $d=75$. For GIN, GUNet and proposed HAG-Net, a node embedding
layer is utilized to learn features from training data automatically, with
also dimension $d=75$ for comparison consistency. For GIN, GUNet and DeepChem
models, default values are kept for their hyper-parameters, and Adam (Kingma
and Ba, 2014) optimizer with learning rate 1e-3 is used. For HAG-Net with
different hyper-parameter configurations, SGD optimizer with learning rate
1e-2 is used. The training batch size and epoch number is fixed to 256 and
1,000 for all experiments. Evaluation results are given in Table 2, reported
as 5-fold average.

Table 2: Evaluation Results: 5-Fold Average∗

Model | Antivirus7k | CYP2C9V12k | Mtb18k | Malaria16k | Globalvirus76k
---|---|---|---|---|---
GUNet | $80.6\pm 0.5$ | $24.8\pm 1.0$ | $82.3\pm 2.0$ | $19.9\pm 1.0$ | $50.2\pm 7.1$ | $11.9\pm 0.2$ | $\bm{69.0}\pm 0.1$ | $6.9\pm 0.1$ | $76.3\pm 1.5$ | $27.7\pm 0.6$
GIN | $91.1\pm 0.6$ | $14.6\pm 0.5$ | $84.6\pm 0.9$ | $18.1\pm 0.7$ | $71.1\pm 1.8$ | $10.0\pm 0.2$ | $64.2\pm 2.5$ | $6.3\pm 0.2$ | $92.5\pm 0.7$ | $13.3\pm 0.4$
DeepChem | $91.2\pm 0.8$ | $13.9\pm 1.3$ | $85.7\pm 0.7$ | $17.2\pm 0.5$ | $73.6\pm 1.9$ | $\ \ 9.5\pm 0.4$ | $67.3\pm 1.4$ | $\bm{6.1}\pm 0.1$ | $93.7\pm 0.3$ | $12.8\pm 0.1$
HAG-Net cfg1 | $92.2\pm 0.8$ | $12.7\pm 0.4$ | $86.6\pm 1.3$ | $\bm{16.4}\pm 1.0$ | $75.9\pm 0.7$ | $\ \ 9.3\pm 0.2$ | $66.1\pm 3.4$ | $6.7\pm 0.3$ | $94.9\pm 0.2$ | $11.7\pm 0.4$
HAG-Net cfg2 | $\bm{92.5}\pm 0.5$ | $\bm{12.0}\pm 0.4$ | $\bm{87.1}\pm 0.4$ | $\bm{16.4}\pm 0.1$ | $\bm{77.1}\pm 1.2$ | $\ \ \bm{9.0}\pm 0.3$ | $67.8\pm 3.3$ | $6.4\pm 0.3$ | $\bm{95.6}\pm 0.2$ | $\bm{10.8}\pm 0.3$

  * *

For each dataset AuPR and ER results are reported, both in range
$[0.0,100.0]$.

###  3.5 Discussion and Future Work

#### Convergence stability

As mentioned in section 3.2, GNN models exhibit strikingly different stability
during the convergence process. For GIN, GUNet and DeepChem models we studied,
for different dataset and different weights initialization, frequent spikes
and ditches can be observed in metric curves (with different patterns, see
Figure 3 for illustration). This problem is not observable with the single
point metrics such as ER, AuPR as reported in Table 2 since model with large
convergence variance can still achieve high AuPR score. We use the $mstd$
metric defined in eq.9 to measure the model convergence variance
quantitatively, results are reported in Table 3 for ER curve, also as 5-fold
average. Note the $mstd$ metric is still not perfect for evaluation of
convergence stability, for example as illustrated by Figure 3.(a), DeepChem
model exhibits spurious fluctuation meanwhile achieves the smallest $mstd$
value.

#### Future work

Results reported in Table 2 and 3 validate our conjecture that GNN’s
performance can benefit from enhanced information propagation from shallow to
deep layers. Our first attempt by combining heterogeneous aggregations
succeeds, but not significantly. One possible future work is to investigate
more neighborhood aggregation operators.

Another possible method for enhancing information propagation is to use multi-
channelling mechanism as in CNN. The channel dimension is essential for the
success of CNN, where each channel encodes different part of information from
the input samples. Combining heterogeneous aggregations can be considered as
primitive imitation of this multi-channelling mechanism.

Figure 3: Convergence curves of P/R/F1 for class 1 on Mtb18k dataset. (a)
DeepChem model (b) GIN model (c) GUNet model (d) HAG-Net cfg1 (e) HAG-Net cfg2
Table 3: Convergence Variance for ER Curves

Name | Antivirus7k | CYP2C9V12k | Mtb18k | Malaria16k | Globalvirus76k
---|---|---|---|---|---
GUNet | $165.7\pm 40.1$ | $215.4\pm 39.7$ | $219.5\pm 91.8$ | $342.4\pm 229.6$ | $207.2\pm 8.7$
GIN | $225.2\pm 15.0$ | $156.7\pm 21.2$ | $181.1\pm 34.6$ | $225.2\pm 38.1$ | $157.0\pm 3.9$
DeepChem | $\ \ 62.0\pm 15.0$ | $\ \ 36.1\pm 4.7$ | $\ \ 28.9\pm 5.4$ | $\ \ 16.1\pm 1.9$ | $\ \ 15.6\pm 1.6$
HAG-Net cfg1 | $\ \ 66.6\pm 14.4$ | $\ \ 65.6\pm 4.2$ | $101.7\pm 42.3$ | $\ \ 91.4\pm 11.1$ | $\ \ 31.4\pm 12.6$
HAG-Net cfg2 | $\ \ 76.1\pm 41.1$ | $\ \ 62.9\pm 24.3$ | $\ \ 45.2\pm 3.6$ | $\ \ 42.5\pm 3.7$ | $\ \ 24.5\pm 7.3$

##  4 Conclusion

In this manuscript we improve GNN’s performance from the perspective of
enhancing information propagation from shallow to deep layers. As our first
attempt in this direction, we propose to enhance information propagation by
combining heterogeneous aggregation operators in GNN’s neighborhood
aggregation layers. By combining different aggregation operators, the
information propagation loss can be mitigated, thus allowing more effective
features for downstream task to propagate to deep layers. A new generic GNN
layer formulation and upon this a new GNN variant referred as HAG-Net is
proposed. We empirically validate the effectiveness of HAG-Net on a number of
graph classification datasets. Our future work will investigate enhancing
information propagation for GNN in the perspective of channelling as done in
CNN.

## References

  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  _IEEE transactions on neural networks and learning systems_ , 2020\.
  * Fan et al. (2019) Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin.  Graph neural networks for social recommendation.  In _The World Wide Web Conference_ , pages 417–426, 2019.
  * Wang et al. (2020) Daixin Wang, Jianbin Lin, Peng Cui, Quanhui Jia, Zhen Wang, Yanming Fang, Quan Yu, Jun Zhou, Shuang Yang, and Yuan Qi.  A semi-supervised graph attentive network for financial fraud detection.  _arXiv preprint arXiv:2003.01171_ , 2020.
  * Zhang et al. (2020) Zhao Zhang, Fuzhen Zhuang, Hengshu Zhu, Zhiping Shi, Hui Xiong, and Qing He.  Relational graph neural network with hierarchical attention for knowledge graph completion.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9612–9619, 2020.
  * Higham et al. (2008) Desmond J Higham, Marija Rašajski, and Nataša Pržulj.  Fitting a geometric graph to a protein–protein interaction network.  _Bioinformatics_ , 24(8):1093–1099, 2008.
  * Hu et al. (2019) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec.  Strategies for pre-training graph neural networks.  _arXiv preprint arXiv:1905.12265_ , 2019.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016.
  * Hamilton et al. (2017) William L Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  _arXiv preprint arXiv:1706.02216_ , 2017.
  * Xu et al. (2018) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  _arXiv preprint arXiv:1810.00826_ , 2018.
  * Gao and Ji (2019) Hongyang Gao and Shuiwang Ji.  Graph u-nets.  In _international conference on machine learning_ , pages 2083–2092. PMLR, 2019.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ , 2017.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In _International Conference on Machine Learning_ , pages 1263–1272. PMLR, 2017.
  * Li et al. (2015) Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel.  Gated graph sequence neural networks.  _arXiv preprint arXiv:1511.05493_ , 2015.
  * Liao et al. (2019) Renjie Liao, Zhizhen Zhao, Raquel Urtasun, and Richard S Zemel.  Lanczosnet: Multi-scale deep graph convolutional networks.  _arXiv preprint arXiv:1901.01484_ , 2019.
  * Huang et al. (2020) Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik.  Therapeutics data commons: Machine learning datasets for therapeutics.  https://tdcommons.ai, November 2020.
  * Veith et al. (2009) Henrike Veith, Noel Southall, Ruili Huang, Tim James, Darren Fayne, Natalia Artemenko, Min Shen, James Inglese, Christopher P Austin, David G Lloyd, et al.  Comprehensive characterization of cytochrome p450 isozyme selectivity across chemical libraries.  _Nature biotechnology_ , 27(11):1050–1055, 2009\.
  * Kato et al. (2016) Nobutaka Kato, Eamon Comer, Tomoyo Sakata-Kato, Arvind Sharma, Manmohan Sharma, Micah Maetani, Jessica Bastien, Nicolas M Brancucci, Joshua A Bittker, Victoria Corey, et al.  Diversity-oriented synthesis yields novel multistage antimalarial inhibitors.  _Nature_ , 538(7625):344–349, 2016.
  * Lane et al. (2018) Thomas Lane, Daniel P Russo, Kimberley M Zorn, Alex M Clark, Alexandru Korotcov, Valery Tkachenko, Robert C Reynolds, Alexander L Perryman, Joel S Freundlich, and Sean Ekins.  Comparing and validating machine learning models for mycobacterium tuberculosis drug discovery.  _Molecular pharmaceutics_ , 15(10):4346–4360, 2018.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 2623–2631, 2019.
  * Ramsundar et al. (2019) Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu.  _Deep Learning for the Life Sciences_.  O’Reilly Media, 2019.  https://www.amazon.com/Deep-Learning-Life-Sciences-Microscopy/dp/1492039837.
  * Hu et al. (2020) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  _arXiv preprint arXiv:2005.00687_ , 2020.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.


# Augment before You Try: Knowledge-Enhanced Table Question
Answering via Table Expansion

Yujian Liu Jiabao Ji Tong Yu<EMAIL_ADDRESS>Ryan
Rossi<EMAIL_ADDRESS>
Sungchul Kim<EMAIL_ADDRESS>Handong Zhao
<EMAIL_ADDRESS>Ritwik Sinha
<EMAIL_ADDRESS>Yang Zhang Shiyu Chang

###### Abstract

Table question answering is a popular task that assesses a model’s ability to
understand and interact with structured data. However, the given table often
does not contain sufficient information for answering the question,
necessitating the integration of external knowledge. Existing methods either
convert both the table and external knowledge into text, which neglects the
structured nature of the table; or they embed queries for external sources in
the interaction with the table, which complicates the process. In this paper,
we propose a simple yet effective method to integrate external information in
a given table. Our method first constructs an augmenting table containing the
missing information and then generates a SQL query over the two tables to
answer the question. Experiments show that our method outperforms strong
baselines on three table QA benchmarks. Our code is publicly available at
https://github.com/UCSB-NLP-Chang/Augment_tableQA.

kattributesattributestyleattributestyleld

Augment before You Try: Knowledge-Enhanced Table Question
Answering via Table Expansion

##  1 Introduction

Tables are ubiquitous types of information sources that have attracted
significant attention in the NLP community. Researchers have developed models
that understand tabular data and perform various tasks, including table
question answering (QA) (Pasupat and Liang, 2015; Chen et al., 2020c; Nan et
al., 2022; Zhu et al., 2021a; Chen et al., 2021), table fact verification
(Chen et al., 2020b; Aly et al., 2021), table-to-text generation (Parikh et
al., 2020; Chen et al., 2020a; Nan et al., 2021), _etc._ A critical challenge
in these tasks is that tables often lack sufficient information for the task
at hand, which necessitates the integration of additional knowledge. For
example, in Figure 1, to answer the question ‘How many chords have a root not
based on a sharp or flat note?’, a model needs to have the knowledge of
whether each root is based on a sharp or flat note, which is not provided in
the table and can only be obtained from external sources.

Existing methods for integrating information from tables and external sources
can be mainly categorized into two groups. The first method, exemplified by
Program-of-Thought (Chen et al., 2023), linearizes the table into text and
combines it with external knowledge in textual format (Xie et al., 2022; Chen,
2023). However, the linearized table no longer has the structured format,
making it difficult to retrieve required values from the table and perform
comparisons and calculations.

An alternative approach, represented by Binder (Cheng et al., 2023), combines
the symbolic language execution with large language models (LLMs). In this
method, the model interacts with the table through symbolic language like SQL,
which maintains the structured format. Part of the SQL query is replaced with
an LLM query that extracts knowledge from the LLM and returns the results for
further SQL execution. For instance, in Figure 1 (b), the method queries LLMs
for whether each root is sharp or flat and uses the results as a filtering
criterion in a SQL statement. However, it requires the model to learn to embed
LLM queries in the standard SQL language, which differs substantially from the
SQL statements the model has been trained on. As a result, it is more likely
to generate syntactically wrong statements that lead to execution errors.

Figure 1: Comparison between Program-of-Thought, Binder, and our method.

In this paper, we propose a simple yet effective method for combining external
knowledge with a given table. As shown in Figure 1 (c), our method starts by
analyzing the additional information required for answering the question. It
then queries a knowledge source for the information and organizes the results
in a tabular format. This newly created table augments the original table with
additional information, and a SQL query is generated to obtain the answer from
the two tables. Such an augment-then-generate pipeline eliminates the need to
embed LLM queries in SQL statements while preserving the structured format of
the table.

We evaluate our method on three table QA datasets that require different types
of external knowledge (Chen et al., 2021; Zhu et al., 2021b; Pasupat and
Liang, 2015). Our method outperforms or matches strong baselines on all
datasets. Particularly, it demonstrates significant improvements over Program-
of-Thought in questions with large tables or require complex tabular
operations, and compared to Binder, it exhibits fewer execution errors and
achieves better performance (8.32$\%$).

##  2 Related work

Table question answering task requires both the ability to reason over
structured data and to understand textual contents in the table. Traditional
methods utilize semantic parsing to convert the natural language question into
executable commands, which retrieve and process data in the table to obtain
answers (Zettlemoyer and Collins, 2012; Berant et al., 2013; Yin and Neubig,
2017; Zhong et al., 2017; Shaw et al., 2020; Yu et al., 2018; Rajkumar et al.,
2022). However, these methods require all question-related information to
present in the table in a rigorous format, which is limited when applied to
web tables that often do not have a clean schema. Recent works pre-train
neural models on large-scale tabular data, and directly encode tables and
generate answers in an end-to-end fashion (Liu et al., 2022; Xie et al., 2022;
Herzig et al., 2020; Yin et al., 2020; Zhao et al., 2022; Deng et al., 2020).
To reduce the training cost, some works further leverage LLMs to read and
reason over tables (Chen, 2023; Pourreza and Rafiei, 2023).

Although end-to-end methods achieve impressive performance on table QA
benchmarks, their predictions lack interpretability and are not robust to
input perturbations (Yang et al., 2022). For this reason, recent works propose
to combine LLMs with symbolic language execution. Particularly, Cheng et al.
(2023) expands the SQL language by incorporating function calls to LLMs in SQL
statements. Ye et al. (2023) utilizes LLMs to decompose the question and table
into sub-problems that can be solved with SQL queries. Chen et al. (2023)
prompts LLMs to generate the reasoning process as Python programs. Similar to
our method, a concurrent work (Wang et al., 2024) proposes to dynamically
update the table content in the reasoning process. They employ LLMs to
iteratively generate operations such as selecting a subset of rows or adding a
new column, and the final resulting table is fed to LLMs to generate the
answer. However, their chain of operations is prone to error propagation,
while our method retains the original table content and augments it with
required information.

##  3 Methodology

###  3.1 Problem Formulation

Given a natural language question $Q$, a table $T$, and a knowledge source
$S$, the task is to generate a correct answer for the question. Crucially, $T$
might not contain all the necessary information to answer the question, which
necessitates the use of $S$ to obtain additional information. In this paper,
we consider $S$ to be either a relevant text document or an LLM that we can
query.

###  3.2 Overall Framework

Our method contains three steps, as illustrated in Figure 1 (c). The detailed
instructions and examples for each step are listed in Appendix A.

Step 1: Analyze question. An LLM is instructed to analyze the given question
and table to determine what additional information is needed to answer the
question. We instruct the LLM to first list out all the necessary information
for answering $Q$. For each piece of information, it then determines if the
information is present in $T$ or not. The output of this step is a list of
queries that can be later used to obtain additional information from $S$, or
empty if no additional information is needed. For example, in Figure 1 (c),
the model outputs ‘Is the root based on a sharp or flat note?’. Additionally,
for information that needs to be obtained based on the table, the LLM will
also specify which columns are needed, _e.g.,_ the model specifies that the
query needs to be answered for each row in the ‘Root’ column.

Step 2: Construct augmenting table. Using output queries from step 1, the LLM
is used to obtain corresponding information from the source $S$. Specifically,
when $S$ is a text document, this step is similar to the reading comprehension
task where the LLM needs to extract answers to the queries from the document.
When $S$ is an LLM, this step resembles a QA task where the LLM needs to
directly answer the query. Finally, the obtained information is organized into
a separate table that can complement the existing table $T$. Figure 1 (c)
shows an example where a new table of two columns is constructed. It is worth
mentioning that this step is flexible and can be easily extended to other
types of sources $S$.

Step 3: Generate SQL query. With the original and newly constructed tables,
the LLM then generates a SQL query that can be executed to obtain the answer
to the question. Importantly, the two tables contain sufficient information
for answering $Q$, and the LLM can generate a standard SQL query, which is
easier and more similar to its pre-training data.

##  4 Experiments

In this section, we empirically evaluate our method on table QA benchmarks,
focusing on two types of questions that might require external knowledge from
different sources.

  * •

Open-domain knowledge where external information comes from an open domain. We
use the embedded knowledge in LLMs as the source.

  * •

Closed-domain knowledge where all information is within a given table and a
text document. In this case, the document is the external source.

We will discuss the common experiment settings in Section 4.1 and individual
experiments for each type in Sections 4.2 and 4.3 respectively.

###  4.1 Experiments Setup

#### Implementation details.

We prompt an LLM with detailed instructions and several in-context examples to
complete all three steps in our method. To feed the table to the LLM for
question analysis (Step 1) and generating SQL queries (Step 3), we linearize
the table by concatenating columns with special tokens (_e.g.,_ ‘|’) following
previous works (Chen et al., 2023). In all experiments, we use
GPT-3.5-turbo-1106 through the official API as the backbone LLM for our method
and all baselines. We use greedy decoding (_i.e.,_ temperature equals 0) for
our method and all baselines. To have a fair comparison, we use the same
number of in-context examples as baselines (details in Appendix A).

#### Baselines.

We compare with five LLM-based baselines. ❶ End-to-End that prompts the LLM to
directly output the answer given the table, question, and optionally the text
document. ❷ Table-CoT (Chen, 2023) that uses the chain-of-thought prompting
(Wei et al., 2022) to ask the model to additionally output the reasoning
chain. ❸ Dater (Ye et al., 2023), ❹ Binder (Cheng et al., 2023), and ❺
Program-of-Thought (PoT) (Chen et al., 2023) that combine LLMs with symbolic
language execution. Please refer to Section 2 for details. Particularly, since
Binder does not generate the reasoning chain, we include an improved variant
where we add chain-of-thought prompting (Binder+CoT).

#### Evaluations.

We adopt the exact match rate (EM) between the model-predicted answer and the
ground-truth answer as the metric. We use the same evaluation code for all
methods for fair comparison.

###  4.2 Open-Domain Knowledge

#### Datasets.

For open-domain knowledge, we evaluate on WikiTQ dataset (Pasupat and Liang,
2015), which requires complex table reasoning to answer the question.
According to Shi et al. (2020), around 20% of questions in WikiTQ are not
answerable by pure SQL queries, which are likely to require additional
knowledge not present in the table. We evaluate all methods on a subset of
1000 samples in the test set due to the cost of full evaluation.

#### Results.

Table 1 presents the EM. There are two observations from the table. First,
methods that involve program execution are generally better than those that do
not, highlighting the value of accurate data retrieval or processing. Second,
our method achieves the best performance, showing its effectiveness. To
further evaluate scalability across table sizes, Figure 2 plots the
performance breakdown by the number of tokens in the table. As can be
observed, our method and Binder+CoT are the only methods that maintain
performance on large tables, whereas methods that rely on LLMs to extract
information from linearized tables such as Table-CoT and PoT suffer
significant performance degradation on large tables. This illustrates the
advantage of SQL queries when interacting with the table.

| Test EM
---|---
End-to-End | 46.80
Table-CoT (Chen, 2023) | 49.40
PoT (Chen et al., 2023) | 53.60
Dater (Ye et al., 2023) | 48.40
Binder (Cheng et al., 2023) | 34.80
Binder+CoT | 54.10
Ours | 55.80

Table 1: Exact match on WikiTQ test set. Methods in the bottom panel involve
program execution. Figure 2: Performance grouped by table length.

#### Comparison with Binder+CoT.

To further verify whether our augment-then-generate pipeline leads to easier
and more accurate SQL generation over the best-performing baseline Binder+CoT
(hereafter Binder), we compare the two methods on the subset of questions that
are not solvable by pure SQL (Shi et al., 2020), which rely more on the
integration of external knowledge. Figure 3 shows the EM and percentage of
execution errors, where our method demonstrates a more pronounced improvement.
To better pinpoint the cause of performance difference, we add a post-
processing step for Binder, where we extract the LLM queries from the SQL
statement generated by Binder, query LLMs for desired information and add it
as a new column in the original table, and re-generate a standard SQL (without
LLM queries) based on the augmented table. This variant (dubbed Binder-
separate) improves the EM and reduces execution errors over Binder, which
validates our hypothesis that combining LLM queries with SQL complicates the
generation, leading to more syntax errors in generated programs. Notably, our
method still incurs fewer execution errors than Binder-separate, which is
likely due to the fact that our method generates more augmentations for the
table, thus reducing the complexity of required SQL (see Appendix C.1 for
details and examples).

In Appendix B, we also compare our method with the concurrent work Chain-of-
Table (Wang et al., 2024). Results show that our method achieves 1.85 higher
EM when using GPT3.5 as the backbone LLM, demonstrating its effectiveness
despite being simpler and not requiring sequential operations. Please refer to
Appendix B for details.

Figure 3: Comparison between our method and Binder.

###  4.3 Closed-Domain Knowledge

#### Datasets.

For closed-domain setting, we experiment on TATQA (Zhu et al., 2021b) and
FinQA (Chen et al., 2021). Questions in these two datasets are about a table
and a financial report, and the answer often requires arithmetic operations in
addition to table understanding ability. Since some questions are answerable
with only the table or report, we filter the datasets to only include those
that require both table and report (details in Table 5).

#### Results.

Table 2 presents the results. Binder and Dater are not included because the
original paper did not evaluate on these datasets and extension to this
setting requires substantial modification. There are two observations. First,
our method and PoT significantly outperform the other two baselines that do
not involve program executions, which shows the benefits of leveraging
programs when questions require arithmetic calculations. Second, although the
input tables are much smaller, which is beneficial for PoT, our method is on
par with PoT on FinQA and outperforms it by 2 EM on TATQA. A further
performance breakdown by the number of table cells required to answer a
question in Figure 4 shows that our method is more effective on questions that
require information from multiple cells, indicating that our method is more
likely to generalize to complex questions. Furthermore, it is easier to locate
and correct errors made by our method as it only requires inspection of the
generated SQL queries, whereas PoT requires checking the whole table contents
(see examples in Appendix C.2).

| TATQA | FinQA
---|---|---
End-to-End | 35.50 | 34.18
Table-CoT (Chen, 2023) | 34.91 | 39.87
PoT(Chen et al., 2023) | 61.14 | 54.43
Ours | 63.12 | 53.80

Table 2: Exact match on TATQA and FinQA. Figure 4: Performance decomposition
by the number of table cells needed to answer the question.

##  5 Conclusion

In this paper, we propose a simple method that augments a given table by
creating a new table that contains the information from external sources. The
LLM then generates a SQL query over the two tables to answer the given
question. Experiments show that our method outperforms or matches strong
baselines on three table QA benchmarks.

##  6 Limitation

There are several limitations in this work that need to be further improved.
First, our framework relies on the LLM’s ability to generate correct SQL
statements. If the LLM has limited SQL generation ability, such as Llama2 in
Appendix B, the performance of our method will be affected. In addition, we
only evaluate our method on integrating external knowledge from two different
sources. The generalizability of our method to other knowledge sources remains
to be assessed.

## References

  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task.  In _Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)_.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1533–1544.
  * Chen (2023) Wenhu Chen. 2023.  Large language models are few(1)-shot table reasoners.  In _Findings of the Association for Computational Linguistics: EACL 2023_.
  * Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a.  Logical natural language generation from open-domain tables.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Chen et al. (2023) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _Transactions on Machine Learning Research_.
  * Chen et al. (2020b) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b.  Tabfact: A large-scale dataset for table-based fact verification.  In _International Conference on Learning Representations_.
  * Chen et al. (2020c) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Chen et al. (2021) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021.  FinQA: A dataset of numerical reasoning over financial data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Cheng et al. (2023) Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023.  Binding language models in symbolic languages.  In _The Eleventh International Conference on Learning Representations_.
  * Deng et al. (2020) Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020.  Turl: Table understanding through representation learning.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020.  TaPas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2022) Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022.  TAPEX: Table pre-training via learning a neural SQL executor.  In _International Conference on Learning Representations_.
  * Nan et al. (2022) Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. 2022.  FeTaQA: Free-form table question answering.  _Transactions of the Association for Computational Linguistics_.
  * Nan et al. (2021) Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021.  DART: Open-domain structured data record to text generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
  * Pourreza and Rafiei (2023) Mohammadreza Pourreza and Davood Rafiei. 2023.  Din-sql: Decomposed in-context learning of text-to-sql with self-correction.  _arXiv preprint arXiv: 2304.11015_.
  * Rajkumar et al. (2022) Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022.  Evaluating the text-to-sql capabilities of large language models.
  * Shaw et al. (2020) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2020.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both?  _arXiv preprint arXiv:2010.12725_.
  * Shi et al. (2020) Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. 2020.  On the potential of lexico-logical alignments for semantic parsing to SQL queries.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2024) Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024.  Chain-of-table: Evolving tables in the reasoning chain for table understanding.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_.
  * Yang et al. (2022) Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022.  TableFormer: Robust transformer modeling for table-text encoding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Ye et al. (2023) Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023.  Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning.
  * Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017.  A syntactic neural model for general-purpose code generation.  _arXiv preprint arXiv:1704.01696_.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020.  TaBERT: Pretraining for joint understanding of textual and tabular data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Zettlemoyer and Collins (2012) Luke S. Zettlemoyer and Michael Collins. 2012.  Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars.
  * Zhao et al. (2022) Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir R. Radev. 2022.  Reastap: Injecting table reasoning skills during pre-training via synthetic reasoning examples.  _Conference on Empirical Methods in Natural Language Processing_.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2sql: Generating structured queries from natural language using reinforcement learning.
  * Zhu et al. (2021a) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021a.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
  * Zhu et al. (2021b) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021b.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.

##  Appendix A Implementation Details

| WikiTQ | TATQA | FinQA
---|---|---|---
top_p | 1.0 | 1.0 | 1.0
max_output_tokens | 512 | 512 | 512
num_shots | 8 | 8 | 4

Table 3: Parameters for our greedy generation (sections 4.2 and 4.3).

| GPT3.5 | Llama2
---|---|---
| Augmentation | SQL | Augmentation | SQL
| generation | generation | generation | generation
temperature | 0.6 | 0.4 | 0.8 | 0.4
top_p | 1.0 | 1.0 | 1.0 | 1.0
sampling_n | 3 | 2 or 4 | 4 | 3 or 4
max_output_tokens | 512 | 512 | 256 | 256
num_shots | 8 | 8 | 8 | 8

Table 4: Generation parameters for our ensemble model on WikiTQ (Appendix B).
Augmentation generation and SQL generation correspond to the step 1 and 3 in
our method.

| WikiTQ | WikiTQ SQL unsolvable | TATQA | FinQA
---|---|---|---|---
# questions | 1000 | 625 | 507 | 158
Split | Test | Dev | Dev | Test
# table rows | 26.1 | 28.0 | 9.7 | 6.8
# table tokens | 571.7 | 685.7 | 119.1 | 86.2
Knowledge source $S$ | LLMs | LLMs | Document | Document

Table 5: Summary of the datasets used in this paper.

For all methods on all three datasets, we use the greedy decoding for
generation, _i.e.,_ temperature equals 0. Table 3 lists other generation
parameters of our method. Table 5 shows the statistics of datasets used in
this paper.

###  A.1 Open-domain Knowledge

For the open-domain knowledge setting on WikiTQ, since our method generates
queries that will be asked for every single row in one or more columns, the
constructed augmenting table will always have the same number of rows as the
original table. For simplicity, we directly join the two tables based on the
row index before feeding them to LLMs to generate the SQL statement in step 3.
In other words, the newly constructed table is joined on the original table as
additional columns, and the SQL statement will be generated based on the
joined table. Figures 11 and 12 show the detailed instruction used for this
step and a demonstration of the in-context example. For step 2, we use the
same instruction and in-context examples as Cheng et al. (2023) to query LLMs
for required information. An example is shown in Figure 13. For step 3, we
provide LLMs with in-context examples along with a one-sentence instruction,
as illustrated in Figure 14.

To select a subset of 1000 questions for evaluation, we select one question in
every four questions of the first 4000 samples in WikiTQ, _i.e.,_ dataset =
dataset.select(range(0, 4000, 4)). We use the evaluation code in Cheng et al.
(2023) to calculate the EM for all methods.

###  A.2 Closed-domain Knowledge

For the closed-domain setting on TATQA and FinQA, we feed both the text
document and the table to the LLM to provide enough context. To save the
inference cost, we merge steps 1 and 2 together such that the model analyzes
the required additional information and then extracts them from the document
in a single run. We instruct the model to extract information in a JSON format
that can be easily organized into a table. Figures 15 and 16 show the detailed
instruction used and a demonstration of the in-context example on TATQA, and
Figures 18 and 19 show the same for FinQA. For step 3, we provide the original
table and the newly constructed table if available to LLMs. Figures 17 and 20
show a demonstration of the in-context examples on TATQA and FinQA
respectively.

To select questions for evaluation, we only use those that require both the
table and the document. Specifically, for TATQA, we select questions that have
answer_from$=$table-text, and for FinQA, we select those whose ground truth
evidence contains at least one table row and one document sentence. We follow
Chen et al. (2023) to calculate the EM.

##  Appendix B Comparison with Chain-of-Table

| # generated samples | EM
---|---|---
GPT3.5
Binder | 50 | 56.74
Chain-of-Table | $\leq 25$ | 59.94
Ours (6 SQLs) | 11.4 | 61.05
Ours (12 SQLs) | 17.4 | 61.79
Llama2
Binder | 50 | 30.92
Chain-of-Table | $\leq 25$ | 42.61
Ours (12 SQLs) | 19.82 | 34.00
Ours (16 SQLs) | 23.82 | 35.34

Table 6: Exact match on full WikiTQ test set. $\\#$ generated samples denotes
the total number of generated samples to answer one question.

We additionally compare with Chain-of-Table (Wang et al., 2024) on WikiTQ.
Since their implementation is not available at the submission time of this
paper, we use the same dataset and backbone LLMs as theirs and directly
compare with the numbers reported in their paper. Specifically, we use
GPT-3.5-turbo-16k-0613 and Llama2-13b-chat (Touvron et al., 2023) as backbone
LLMs and evaluate on the full test set of WikiTQ. Since their sequential
operations require multiple queries for LLMs, we consider the majority vote of
execution results from $N$ SQL queries as our final prediction. To generate
these SQL queries, we sample $m$ different outputs for step 1 (_i.e.,_ $m$
different augmentations), and for each augmentation, we sample $k$ SQL
queries. The total number of generated samples for each question is $m+\alpha
m+mk$, where $\alpha$ is the percentage of step 1 outputs that actually need
additional information. Table 4 lists the parameters for generation.

The results are shown in Table 6. As can be observed, our method outperforms
Chain-of-Table and Binder when using GPT3.5 as the backbone LLM, despite using
fewer LLM queries. When using Llama2, Chain-of-Table achieves better
performance than Binder and our method. We hypothesize that the performance
difference is due to the limited SQL generation ability of Llama2. An
important difference is that Chain-of-Table feeds the final table to LLMs and
directly asks LLMs to generate the answer, whereas Binder and our method
prompt LLMs to generate SQL queries and execute to get the answer, which is
affected more when the LLM has limited SQL generation ability. In fact, the
generated SQL of our method contains $32.9\%$ of execution errors when using
Llama2 as the LLM, compared to that of $8.7\%$ when using GPT3.5. However, our
method still outperforms Binder on Llama2, demonstrating the benefits of our
augment-then-generate pipeline.

##  Appendix C Additional Examples

###  C.1 Comparison with Binder

In this section, we elaborate on the comparison between our method, Binder,
and Binder-separate. In Figure 3, it can be observed that our method achieves
better performance and exhibits fewer execution errors than Binder. Moreover,
Binder-separate, which separates the SQL generation and LLM queries in Binder,
reduces its execution errors, validating our hypothesis that integrating LLM
queries in SQL generation could lead to more syntax errors. Figures 5 and 6
show two examples where Binder encounters execution errors when trying to
generate a SQL statement with LLM queries, whereas our method and Binder-
separate correctly generate SQL statements to answer the question.

Our method also incurs fewer execution errors than Binder-separate, which can
be ascribed to the fact that our method generates more augmentations for the
table, which significantly reduces the complexity of required SQL statements.
Figure 7 illustrates one such example, where Binder-separate gets errors
because the required information is missing from the table, whereas our method
correctly answers the question based on the augmented table. In fact, our
method generates augmentations for $72.3\%$ of the questions, while Binder
only includes LLM queries for $6.1\%$ of the questions, showing that our
method also benefits the augmentation of additional information.

###  C.2 Comparison with PoT

We now provide more examples for the comparison between our method and PoT.
Based on Figure 4, our method is more effective on questions that require
multiple table cells for the answer. Figures 8 and 9 show two such examples,
where our method selects the correct values from the table to perform
calculations, but PoT retrieves wrong values from the table, despite
generating programs with correct logic. According to Chen et al. (2023), this
type of value grounding errors take up 47$\%$ of the errors made by PoT.
Moreover, correcting these errors requires manual efforts to look into the
contents of the table, which is time-consuming when the table is large.

On the contrary, Figure 10 shows an example question that only requires a
single cell from the table. PoT correctly selects the answer but our method
selects the value in the wrong column. However, correcting this error requires
only manual inspection of the generated SQL statement, which is much more
efficient than checking the whole table contents.

Figure 5: An example question in WikiTQ. Binder generates a SQL statement that queries LLMs for unsolvable parts. However, the statement leads to an execution error. Our method augments the table with an additional column and correctly generates a SQL statement to answer the question. Figure 6: An example question in WikiTQ. Binder generates a SQL statement that queries LLMs. However, the statement leads to an execution error. Our method and Binder-separate correctly generate a pure SQL statement to answer the question. Figure 7: An example question in WikiTQ. Binder generates a SQL statement without syntax error, but it does not query LLMs for additional knowledge, resulting in a wrong answer. Since the original SQL statement generated by Binder does not augment the table with additional information, the question is not answerable using pure SQL, leading to an execution error in Binder-separate. Our method augments the table and correctly answers the question. Figure 8: An example question in FinQA that requires two table cells to answer. PoT retrieves the wrong value (highlighted in red) from the table, despite generating a program with correct logic. Identifying the error requires looking into the table contents manually. Our method correctly selects the values and answers the question. Figure 9: An example question in TATQA that requires two table cells to answer. PoT retrieves the wrong value (highlighted in red) from the table, despite generating a program with correct logic. Our method correctly selects the values and answers the question. Figure 10: An example question in TATQA that requires a single table cell to answer. PoT correctly retrieves the value from the table. Our method mistakenly selects the value. However, the error is easy to be spotted and corrected by inspecting the SQL statement. ⬇ Task Description: Your task is to prepare a table for SQL query generation in order to answer a specific question. This may require modifying the table by adding extra columns. These new columns are created based on natural language questions, with each question applied individually to every row in the existing columns. The goal is to transform existing data into a format that’s suitable for SQL operations, or to incorporate additional information into the table. Procedure: 1. Evaluate the Table and Question: Assess if the table, in its current format, is suitable for generating a SQL query to answer the given question. 2. Determine Additional Columns: - If the table is already suitable for the SQL query, simply output "None" - If the table requires modifications, identify and define the necessary changes. Specifically, add new columns where each row’s value is derived from a natural language question applied to the relevant columns. Use the format: ‘new_column‘ = @("question"; [relevant_columns]), where ‘question‘ is the question asked for each row to derive the new column’s contents, and ‘relevant_columns‘ are the existing columns that provide the information needed for the question. Response Format: Begin your response with "Transformation:" and include: - Solution outline: Describe a step-by-step reasoning chain of how to answer the question. - Further analysis: Determine if modifications are required for each step. - Final output: List each required additional column in the specified format, each on a new line. If no modifications are needed, output "None". Figure 11: System prompt used for augmentation generation (Step 1) on WikiTQ. ⬇ Title: 2007 New Orleans Saints season CREATE TABLE t1( row_id int, date text, game site text, result/score text) /* 3 example rows: SELECT * FROM t1 LIMIT 3; row_id date game site result/score 0 2007-9-6 rca dome l 41-10 1 2007-9-16 raymond james stadium l 31-14 2 2007-9-24 louisiana superdome l 31-14 */ Q: what number of games were lost at home? Transformation: Solution outline: 1. Find the losing games. 2. Find the games at home. 3. Count the number of games that satisfy both conditions. Further analysis: For step 1, we need information in ‘result/score‘ column. We need to parse if it’s a win or loss. We will add a column called ‘is_loss‘. For step 2, we need information in ‘game site‘ column. We need additional information on whether it’s a home game or not. We will add a column called ‘is_home_game‘. Step 3 can be done with a SQL query. Final output: ‘is_loss‘ = @("Is it a loss?"; [result/score]) ‘is_home_game‘ = @("Is it the home court of New Orleans Saints?"; [game site]) Figure 12: A demonstration of the in-context example used for augmentation generation (Step 1) on WikiTQ. ⬇ Give a database as shown below: Table: 1963 International Gold Cup /* row_id driver 0 jim clark 1 richie ginther 2 graham hill 3 jack brabham 4 tony maggs */ Q: Answer question "What is his/her country?" row by row. Output: /* row_id driver 0 jim clark scotland 1 richie ginther united states 2 graham hill england 3 jack brabham australia 4 tony maggs south africa */ Figure 13: A demonstration of the in-context example used for querying additional information (Step 2) from LLMs on WikiTQ. ⬇ Read the following table and write a SQL query to answer the question: Title: 2007 New Orleans Saints season CREATE TABLE t1( row_id int, date text, game site text, result/score text, is_loss text, is_home_game text) /* 3 example rows: SELECT * FROM t1 LIMIT 3; row_id date game site result/score is_loss is_home_game 0 2007-9-6 rca dome l 41-10 yes no 1 2007-9-16 raymond james stadium l 31-14 yes no 2 2007-9-24 louisiana superdome l 31-14 yes yes */ Q: what number of games were lost at home? SQL: To answer the question, we need following steps: 1. Find the losing games by ‘is_loss‘ column. 2. Find the games at home by ‘is_home_game‘ column. 3. Count the number of games that satisfy both conditions. Final SQL query: ‘‘‘ SELECT COUNT(*) FROM t1 WHERE ‘is_loss‘ = ’yes’ AND ‘is_home_game‘ = ’yes’ ‘‘‘ Figure 14: A demonstration of the in-context example used for SQL generation (Step 3) on WikiTQ. ⬇ Task Description: You are tasked with analyzing a provided table and an accompanying report to answer a specific question. This involves assessing whether the table contains all necessary information for answering the question. If additional information is needed, you must extract this from the report and create a supplementary table. Your primary focus is on the analysis and information extraction process, which will facilitate in forming a SQL query to answer the question. Procedure: 1. Assess the Given Table and Question: Determine whether the provided table contains all the required information to answer the question. 2. Extract Information for Additional Table Creation: - If the existing table is sufficient, simply output "None" - If the existing table lacks essential information, extract the required data from the report in the following JSON format: ‘{"column_name": [value1, ...], ...}‘ Each example is given in the following structure: - Report: Contents of the report that may contain additional information. - Tables: Contents of the table, with columns separated by " | " and rows by "\n". - Question: The specific question that needs to be answered. Response Format: Begin your response with "Analysis:" and include: - Solution outline: Describe the step-by-step outline for answering the question. - Further analysis: Determine whether each step’s information is available in the existing table or needs to be extracted from the report. - Final output: Extract necessary information from the report in JSON format as described above; if no additional information is needed, output "None". Notes: - You may extract information with any number of columns and rows. However, all columns should have the same number of values. - Make the JSON self-explanatory. Use descriptive column names, add context where needed, and include units in column names to prevent ambiguity. - Avoid creating columns with empty or NaN values. Figure 15: System prompt used for constructing augmenting table (Steps 1 and 2) on TATQA. ⬇ Report: NOTE 5 - PROPERTY AND EQUIPMENT The Company owned equipment recorded at cost, which consisted of the following as of December 31, 2019 and 2018: Depreciation expense was $80,206 and $58,423 for the years ended December 31, 2019 and 2018, respectively Tables: row_id | filledcolumnname | 2019 | 2018 0 | computer equipment | 137763 | 94384 1 | furniture and fixtures | 187167 | 159648 2 | subtotal | 324930 | 254032 3 | less accumulated depreciation | 148916 | 104702 4 | property and equipment, net | 176014 | 149330 Question: What is the ratio of depreciation expense to accumulated depreciation of property and equipment in 2019? Analysis: Solution outline: 1. Find the amount of depreciation expense and accumulated depreciation of property and equipment in 2019. 2. Calculate the ratio. Further analysis: For step 1, the accumulated depreciation is mentioned in the table in row 3. But the depreciation expense is missing from the table. So we need to extract it from the report. Step 2 can be done with a SQL query. Final output: {"depreciation_expense_2019": ["$80,206"]} Figure 16: A demonstration of the in-context example used for constructing augmenting table (Steps 1 and 2) on TATQA. ⬇ Report: NOTE 5 - PROPERTY AND EQUIPMENT The Company owned equipment recorded at cost, which consisted of the following as of December 31, 2019 and 2018: Depreciation expense was $80,206 and $58,423 for the years ended December 31, 2019 and 2018, respectively Tables: CREATE TABLE t1( row_id int, filledcolumnname text, 2019 int, 2018 int) /* All rows of the table: SELECT * FROM t1; row_id filledcolumnname 2019 2018 0 computer equipment 137763 94384 1 furniture and fixtures 187167 159648 2 subtotal 324930 254032 3 less accumulated depreciation 148916 104702 4 property and equipment, net 176014 149330 */ CREATE TABLE t2( row_id int, depreciation_expense_2019 int) /* All rows of the table: SELECT * FROM t2; row_id depreciation_expense_2019 0 80206 */ Q: What is the ratio of depreciation expense to accumulated depreciation of property and equipment in 2019? SQL: Reasoning process: We need following steps to answer the question: 1. Get the depreciation expense in 2019 from t2. 2. Get the accumulated depreciation in 2019 from t1, which is in row 3. 3. Calculate the ratio. Final SQL query: ‘‘‘ SELECT (SELECT ‘depreciation_expense_2019‘ FROM t2 WHERE ‘row_id‘ = 0) / CAST((SELECT ‘2019‘ FROM t1 WHERE ‘row_id‘ = 3) AS REAL) AS depreciation_ratio FROM t1 LIMIT 1 ‘‘‘ Units: "" Figure 17: A demonstration of the in-context example used for SQL generation (Step 3) on TATQA. ⬇ Task Procedure: 1. Assess the Given Table and Question: Determine whether the provided table contains all the required information to answer the question. 2. Extract Missing Information from Report: - If the existing table is sufficient, simply output "None" - If the existing table lacks essential information, extract the required data from the report in the following JSON format: ‘{"column_name": [value1, ...], ...}‘ Each example is given in the following structure: - Report: Contents of the report that may contain additional information. - Tables: Contents of the table, with columns separated by " | " and rows by "\n". - Question: The specific question that needs to be answered. Response Format: Begin your response with "Analysis:" and include: - Solution formula: Write a formula to calculate the answer. - Further analysis: Determine for each variable in the formula whether it is available in the table or needs to be extracted from the report. - Final output: For variables not in the table, extract them from report in JSON format as described above; if all variables are in the table, output "None". Notes: - Make the JSON self-explanatory. Use descriptive column names and include units in column names to prevent ambiguity. Figure 18: System prompt used for constructing augmenting table (Steps 1 and 2) on FinQA. ⬇ Report: purchases of equity securities 2013 during 2014 , we repurchased 33035204 shares of our common stock at an average price of $ 100.24 . [b] effective january 1 , 2014 , our board of directors authorized the repurchase of up to 120 million shares of our common stock by december 31 , 2017 . Tables: row_id | period | total number ofsharespurchased[a] | averageprice paidpershare | total number of sharespurchased as part of apublicly announcedplan or program [b] | maximum number ofshares that may yetbe purchased under the planor program [b] 0 | oct . 1 through oct . 31 | 3087549 | 107.59 | 3075000 | 92618000 1 | nov . 1 through nov . 30 | 1877330 | 119.84 | 1875000 | 90743000 2 | dec . 1 through dec . 31 | 2787108 | 116.54 | 2786400 | 87956600 3 | total | 7751987 | 113.77 | 7736400 | n/a Question: what percent of the share repurchases were in the fourth quarter? Analysis: Solution formula: share_repurchase_fourth_quarter / share_repurchase_whole_year Further analysis: share_repurchase_fourth_quarter is in row 3 of the table share_repurchase_whole_year is not in the table, so we need to extract it from the report Final output: {"share_repurchase_whole_year": [33035204]} Figure 19: A demonstration of the in-context example used for constructing augmenting table (Steps 1 and 2) on FinQA. ⬇ Report: purchases of equity securities 2013 during 2014 , we repurchased 33035204 shares of our common stock at an average price of $ 100.24 . [b] effective january 1 , 2014 , our board of directors authorized the repurchase of up to 120 million shares of our common stock by december 31 , 2017 . Tables: CREATE TABLE t1( row_id int, period text, total number ofsharespurchased[a] int, averageprice paidpershare real, total number of sharespurchased as part of apublicly announcedplan or program [b] int, maximum number ofshares that may yetbe purchased under the planor program [b] text) /* All rows of the table: SELECT * FROM t1; row_id period total number ofsharespurchased[a] averageprice paidpershare total number of sharespurchased as part of apublicly announcedplan or program [b] maximum number ofshares that may yetbe purchased under the planor program [b] 0 oct . 1 through oct . 31 3087549 107.59 3075000 92618000 1 nov . 1 through nov . 30 1877330 119.84 1875000 90743000 2 dec . 1 through dec . 31 2787108 116.54 2786400 87956600 3 total 7751987 113.77 7736400 n/a */ CREATE TABLE t2( row_id int, share_repurchase_whole_year int) /* All rows of the table: SELECT * FROM t2; row_id share_repurchase_whole_year 0 33035204 */ Q: what percent of the share repurchases were in the fourth quarter? SQL: Solution formula: share_repurchase_fourth_quarter / share_repurchase_whole_year Further analysis: share_repurchase_fourth_quarter is in row 3, column ‘total number ofsharespurchased[a]‘ of t1 share_repurchase_whole_year is in row 0, column ‘share_repurchase_whole_year‘ of t2 Final SQL query: ‘‘‘ SELECT CAST((SELECT ‘total number ofsharespurchased[a]‘ FROM t1 WHERE ‘row_id‘ = 3) AS REAL) / (SELECT ‘share_repurchase_whole_year‘ FROM t2 WHERE ‘row_id‘ = 0) * 100 ‘‘‘ Figure 20: A demonstration of the in-context example used for SQL generation (Step 3) on FinQA.