# Doc2SoarGraph: Discrete Reasoning over Visually-Rich Table-Text Documents
with Semantic-Oriented Hierarchical Graphs

Fengbin Zhu1, Chao Wang2, Fuli Feng 3, Zifeng Ren1, Moxin Li1, Tat-Seng Chua1
1National University of Singapore, 26Estates Pte Ltd, 3University of Science
and Technology of China
{zhfengbin<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Discrete reasoning over table-text documents (e.g., financial reports) gains
increasing attention in recent two years. Existing works mostly simplify this
challenge by manually selecting and transforming document pages to structured
tables and paragraphs, hindering their practical application. In this work, we
explore a more realistic problem setting in the form of TAT-DQA, i.e. to
answer the question over a visually-rich table-text document. Specifically, we
propose a novel Doc2SoarGraph framework with enhanced discrete reasoning
capability by harnessing the differences and correlations among different
elements (e.g., quantities, dates) of the given question and document with
Semantic-oriented hierarchical Graph structures. We conduct extensive
experiments on TAT-DQA dataset, and the results show that our proposed
framework outperforms the best baseline model by 17.73% and 16.91% in terms of
Exact Match (EM) and F1 score respectively on the test set, achieving the new
state-of-the-art.

##  1 Introduction

Figure 1:  An example from TAT-DQA. We leverage four types of semantic
elements from the question and document to facilitate discrete reasoning,
i.e., _Date_ in red rectangle, _Quantity_ in purple rectangle, _Question_ in
yellow rectangle and _Block_ in blue rectangle. The quantities with yellow
background are the supporting evidence to the question. The “million” with
green background is the scale of the answer.

Discrete reasoning is a fundamental part of human intelligence Lu et al.
(2022), and therefore also an indispensable capability we pursue in artificial
intelligence that enables machines to reason over multiple parts of natural
language content for comprehension and inference. In the past two years, there
has been a surge of works addressing discrete reasoning over a hybrid of
table-text content like TAT-QA Zhu et al. (2021a), FinQA Chen et al. (2021)
and MultiHiertt Zhao et al. (2022). However, these works focus on the well-
maintained tables and manually selected paragraphs from the original
documents, which is not in line with reality.

Recently, research on discrete reasoning over real-world table-text documents
directly has been activated with the release of TAT-DQA Zhu et al. (2022), a
Document Visual Question Answering (DocVQA) dataset over financial documents
(See Figure 1). To address this challenge, Zhu et al. (2022) also proposed an
MHST model, which adopts the multi-modal pre-trained model LayoutLMv2 Xu et
al. (2021) to encode the question and one document page, and then applies
sequence tagging on each token to select the relevant tokens to the question,
followed by answer inference over the selected tokens. Though effective, the
performance of MHST is still not optimal. One reason is that the tokens only
carry part of the semantics of the original data, resulting in sub-optimal
evidence extracted from the document. For example, as shown in Figure 1, the
spectrum license fee in $2019$ is $1,731$ million. The quantity $1,731$
corresponds to four tokens, i.e., “$\underline{1}$”, “$\underline{,}$”,
“$\underline{73}$”, and “$\underline{\textrm{\\#\\#}1}$” after tokenization.
The model can hardly infer the meaning of the original number from every
single token unless they are all combined.

Instead of only relying on token-level semantics, we propose to exploit the
element-level semantics to facilitate discrete reasoning. In particular, we
consider four types of elements in the input, including _Question_ , _Block_ ,
_Quantity_ and _Date_ , as shown in Figure 1. Specifically, _Question_ refers
to the question; _Block_ refers to each document block generated by the
OCR/PDF converter; _Quantity_ , _Date_ respectively refer to each quantity and
each date in the question and the document block. Each of these elements
carries more complete semantics than single tokens that can be leveraged by
the model. The differences and correlations among them can provide rich and
crucial clues for the model to conduct reasoning to derive the answer. For
example, though $2019$ and $1,731$ in Figure 1 are both numerical values,
actually the former refers to “year 2019” (date), while the latter is “the
spectrum license fee” (quantity), so they cannot be compared. Obviously, it
would be more appropriate to model the different types of elements separately.
On the other hand, to understand the numerical value $1,731$ in the document,
the model needs to consider the text information of the corresponding document
block. Thus, the correlations of different elements should also be leveraged
to facilitate the model reasoning.

In this work, we extend SoarGraph Zhu et al. (2023) to tackle the challenge
over visually-rich table-text documents in the form of TAT-DQA Zhu et al.
(2022). Specifically, we propose a novel Doc2SoarGraph framework for question
answering over these documents with semantic-oriented hierarchical graphs.
Doc2SoarGraph models the differences and correlations of the elements (i.e.,
quantities, dates, question and document blocks) in the input data with graph
structures taking each element as one node, gaining a more powerful discrete
reasoning capability. In TAT-DQA, about $20\%$ of the documents in test set
have multiple pages. Hence, for a multi-page document, our Doc2SoarGraph first
transforms it to a single image of the model preferred dimension with a simple
yet effective method. Then, given a question and a document, Doc2SoarGraph
adopts LayoutLMv2 to take in the question, document text and the corresponding
layout and document image, and initializes the representations of all semantic
elements with the output. After that, we build a Quantity Comparison (QC)
graph to model the magnitude and comparison among all the _Quantity_ nodes.
Similarly, we build a Date Comparison (DC) graph to model the time sequence
among all the _Date_ nodes. We also build a Text Relation (TR) graph with the
_Question_ node and _Block_ nodes as these nodes usually contain rich text
information. On top of these three graphs, a Semantic Dependency (SD) graph is
built with all types of nodes to model the semantic relationships and
dependencies among them. Then, the framework selects the most question-
relevant nodes from the SD graph and applies different reasoning strategies
over the selected nodes to derive the final answer based on the answer type.

To sum up, we make three-fold contributions. 1) We propose to exploit element-
level semantics to facilitate discrete reasoning over visually-rich table-text
documents. 2) We develop a novel Doc2SoarGraph framework to model the
differences and correlations among various elements with semantic-oriented
hierarchical graph structures, which owns greatly enhanced evidence extraction
and discrete reasoning capability. 3) We conduct extensive experiments on TAT-
DQA dataset, and the results show that our Doc2SoarGraph framework outperforms
the MHST model by 17.73% and 16.91% respectively in terms of Exact Match and
F1 score on the test set, demonstrating remarkable effectiveness.

##  2 Doc2SoarGraph Framework

We first formally define our problem. Consider a natural language question
denoted as $Q$, and a visually-rich table-text document denoted as $D$ with
several pages $P=(P_{1},P_{2},...,P_{|P|}$), where $|P|$ is the number of
pages. In the document $D$, the page $p$ has a list of blocks
$B^{p}=(B^{p}_{1},B^{p}_{2},...,B^{p}_{|B|})$ that are generated by the
OCR/PDF converter, where $|B|$ is the number of blocks on the page $p$. Our
goal is to generate the answer to the question $Q$ that usually requires
discrete reasoning based on the document $D$. To solve the problem, we develop
a Doc2SoarGraph framework. An overall architecture is illustrated in Figure 2.
We now elaborate the details of each component below.

Figure 2:  An overview of proposed Doc2SoarGraph framework. Take the sample in
Figure 1 as an example.

###  2.1 Multi-page Document Transformation

In TAT-DQA, about 20% of the documents in test set consist of multiple pages.
we first pre-process these multi-page documents into one-page ones with a
simple yet effective method. In particular, we first transform each page to a
single image with the same dimension, and then combine the corresponding
multiple images of the pages vertically following the original page order.
After that, we resize the combined image to the model preferred dimension as
the final document image, and obtain its initial visual embeddings by applying
the same CNN-based encoder as LayoutLMv2 Xu et al. (2021). Since the document
text and layout information are available in TAT-DQA, we further adjust the
layout information according to the dimension of the final document image.

###  2.2 Node Initialization

Instead of only relying on token-level semantics, our method also exploits
element-level semantics to facilitate discrete reasoning with graph
structures. In particular, we harness four types of elements, namely, the
question, each document block generated by the OCR/PDF converter, each
quantity and each date in the question and the document block, which are named
_Question_ , _Block_ , _Quantity_ and _Date_ , respectively. We take each type
of element as a kind of node, and get four types of nodes to build the graphs,
i.e., _Question_ node, _Block_ node, _Quantity_ node and _Date_ node. We then
employ LayoutLMv2${}_{\text{LARGE}}$ Xu et al. (2021) to take as input the
question, the document text and layout information, and the final document
image, and output the token-level hidden representations. Then, we compute the
mean of the corresponding tokens for each node as its initial representation.

###  2.3 Node Selection

Based on the four types of nodes as explained above, we construct hierarchical
graphs to model their relationships so as to select those most relevant nodes
as the supporting evidence to the question and facilitate discrete reasoning
of the model.

####  2.3.1 Hierarchical Graphs Construction

We construct four graphs, which form a two-level hierarchy, to model the
element-level semantics. Formally, a graph G is represented by an adjacency
matrix $\textsc{A}\in R^{N\times N}$, where $N$ is the number of nodes. If
there is an edge connecting the $i^{\textrm{th}}$ and $j^{\textrm{th}}$ nodes,
we assign value 1 to the corresponding position $(i,j)$ in the matrix $A$, and
otherwise 0.

Quantity Comparison (QC) Graph (denoted as $G_{QC}$): It is dedicated to
retaining the numerical magnitude and comparison between every two quantities.
For two _Quantity_ nodes $q_{i}$, $q_{j}$, if $q_{i}$ $\geq$ $q_{j}$, a
directed edge $e_{ij}$ = ($q_{i}$, $q_{j}$) pointing from $q_{i}$ to $q_{j}$
is added following NumNet Ran et al. (2019).

Date Comparison (DC) Graph (denoted as $G_{DC}$): It is dedicated to retaining
the time sequence and comparison between every two dates. For two _Date_ nodes
$d_{i}$, $d_{j}$, a directed edge $e_{ij}$ = ($d_{i}$, $d_{j}$) pointing from
$d_{i}$ to $d_{j}$ is added if $d_{i}$ $\geq$ $d_{j}$ ($d_{i}$ later than
$d_{j}$).

Text Relation (TR) Graph (denoted as $G_{TR}$): It is dedicated to associating
the informative descriptions among the question and the document blocks. The
_Question_ node and a _Block_ node or every two _Block_ nodes will have an
undirected edge between them, forming a fully-connected graph.

Semantic Dependency (SD) Graph (denoted as $G_{SD}$): It is built with all the
four types of nodes to model the semantic dependencies of the _Quantity_ or
_Date_ node upon the _Question_ or _Block_ node, besides attaining all the
correlations in the above three graphs. 1) Edges for two _Quantity_ nodes, two
_Date_ nodes, a _Question_ node and a block node, or two _Block_ nodes will be
added in $G_{SD}$ following the construction rules for $G_{QC}$, $G_{DC}$, and
$G_{TR}$, respectively. 2) Between one _Quantity_ node and one _Question_ or
_Block_ node, a directed edge pointing from the _Quantity_ node to the
_Question_ node or the _Block_ node will be added to the graph $G_{SD}$ if the
quantity is part of the question or block; edges between one _Date_ node and
one _Question_ or _Block_ node are added in the same way.

####  2.3.2 Node Classifier

After constructing the hierarchical graphs, a dedicated graph convolution
network (GCN) Kipf and Welling (2017) is applied for each graph to learn node
representations respectively. As illustrated in Figure 2, the GCN (QC), GCN
(DC) and GCN (TR) are applied respectively on the QC graph, DC graph and TR
graph to learn corresponding node representations, which are then used to
initialize the node representations of the SD graph. The GCN (SD) is applied
on the SD graph to learn the final representation of each node $h_{node}$. A
binary node classifier is then applied on each node in the SD graph to predict
whether the node is relevant to the question or not. The probability on node
classification is computed as

$\displaystyle\textrm{P}_{\text{node}}$
$\displaystyle=\textrm{softmax}(\textrm{FFN}(h_{\textrm{node}}))$ (1)

where FFN is a feed-forward network with two layers. All the nodes that are
classified as relevant to the question are collected. Note that the graph
representation of the SD graph $h_{SD}$ is obtained by computing the mean of
all the node representations in the SD graph.

###  2.4 Answer Generation

We generate the final answer with the selected nodes. We first mask the tokens
not included in the selected _Block_ nodes, and then apply corresponding
reasoning strategies for different answer types.

####  2.4.1 Token Masking

Based on the selected nodes, we mask the tokens that are not included in the
selected _Block_ nodes to reduce the search space for answer prediction and
update the token representations with their corresponding block node
representations. Particularly, we obtain the token-level representations from
the output of the LayoutLMv2 encoder first. Then, we mask the tokens not
covered by any selected block nodes. For the tokens that are included in the
selected block nodes, we update the representation of each token by
concatenating its token representation with the corresponding block
representation,

$\displaystyle h_{token}^{{}^{\prime}}$
$\displaystyle=\textrm{concat}(h_{token},h_{node})$ (2)

where $h_{token}$ is the token representation output from the encoder;
$h_{node}$ is the representation of the token’s corresponding block node
obtained from the SD graph; $concat$ denotes concatenation;
$h_{token}^{{}^{\prime}}$ is the updated token representation. For tokens that
are masked in the sequence, we pad their representations with zero. Finally,
we obtain a sequence of updated token representations
$h^{{}^{\prime}}_{\text{[}t_{1},t_{2},...,t_{s}\text{]}}$ and $s$ is the
maximum sequence length.

####  2.4.2 Answer Type Classifier

TAT-DQA offers four different answer types, i.e., Span, Spans, Counting,
Arithmetic. The Answer Type Classifier is adopted to predict the answer type
of a question, which is essentially a multi-class classifier taking the SD
graph representation $h_{SD}$ as input. The probability of each answer type is
computed as

$\displaystyle\textrm{P}_{\text{type}}$
$\displaystyle=\textrm{softmax}(\textrm{FFN}(h_{\textrm{SD}}))$ (3)

where FFN is a feed-forward network with two layers.

####  2.4.3 Span Classifier

For the Span question, the answer is a sub-sequence of the input sequence,
which is achieved by the Span Classifier. It takes the token representations
obtained in Section 2.4.1 as the input and predicts the start and end indices
of the sub-sequence. Formally, the probability distribution of the start
position over the sequence is obtained by

$\displaystyle\textrm{P}_{\text{start}}$
$\displaystyle=\textrm{softmax}(\textrm{FFN}(h^{{}^{\prime}}_{\text{[}t_{1},t_{2},...,t_{s}\text{]}}))$
(4)

where FFN is a feed-forward network with two layers. Similarly, we can obtain
the probability of the end position $\textrm{P}_{\text{end}}$.

####  2.4.4 Token Classifier

For the Spans and Counting questions, Token Classifier is employed to infer
the final answer. In particular, for each valid token obtained in Section
2.4.1, Token Classifier assigns a B, I or O label and takes those tagged with
B and I to generate the final answer. Formally, it takes in the updated
representation $h_{token}^{{}^{\prime}}$ of each valid token and computes the
probability of the label as

$\displaystyle\textrm{P}_{\textrm{token}}$
$\displaystyle=\textrm{softmax}(\textrm{FFN}({h_{token}^{{}^{\prime}}}))$ (5)

where FFN is a feed-forward network with two layers. After obtaining the
tokens, the final answer for _Spans_ and _Counting_ questions is generated
heuristically following MHST Zhu et al. (2022).

####  2.4.5 Tree Generator

For the Arithmetic question, Tree Generator is adopted to generate an
expression tree with the selected _Quantity_ and _Date_ nodes, which can be
executed to infer the answer. Following MHST Zhu et al. (2022), the Tree
Generator is implemented with GTS Xie and Sun (2019), which generates
expression trees in a goal-driven manner. To adapt GTS in our framework, we
make two major modifications. First, instead of feeding all the numbers and
dates in the input into GTS, we only feed the selected most relevant
_Quantity_ and _Date_ nodes, which significantly reduces the number of
candidates for GTS to predict each leaf node and alleviates the difficulties.
Second, when GTS predicts each node in the expression tree, we revise the
generation of the context vector by attending to all the nodes in the SD graph
instead of the tokens in the sequence, which can offer enhanced comprehensive
semantic representations of the document.

The expression tree generated by the Tree Generator includes three kinds of
nodes: the arithmetical operators $V_{op}$ (i.e., +,-,*,/), the constant
numbers $V_{con}$ (i.e., $1$,$2$,$3$, .., $100$ ), and the quantity and date
nodes $V_{node}$ that are selected in Section 2.3. The target vocabulary for
tree generation is denoted as $V=V_{op}\cup V_{con}\cup V_{node}$ and its
length is denoted as $L$. Following the typical construction method of GTS Xie
and Sun (2019), the expression tree is constructed starting from producing the
topmost operator and then the left and right child nodes.

####  2.4.6 Scale Classifier

Scale is vital for a numerical answer in TAT-DQA, including five possible
values: Thousand, Million, Billion, Percent and None. Scale Classifier is
developed to predict the scale of the final answer. In particular, it takes as
input the SD graph representation $h_{SD}$ and computes the probability of
each scale as

$\displaystyle\textrm{P}_{\textrm{scale}}$
$\displaystyle=\textrm{softmax}(\textrm{FFN}(h_{\textrm{SD}}))$ (6)

where FFN is a feed-forward network with two layers. After obtaining the
scale, we generate the final answer by multiplying or concatenating the answer
value with the scale following the practice in MHST Zhu et al. (2022).

##  3 Experiments

We conduct extensive experiments to validate the effectiveness of our proposed
framework and present comprehensive analyses.

###  3.1 Dataset, Baselines and Evaluation Metrics

We conduct all the experiments on TAT-DQA Zhu et al. (2022), which is built
with visually-rich table-text documents in the finance domain. It contains
totally $16,558$ QA pairs and most questions require discrete reasoning to
generate the answers while the answers to other questions can be extracted
directly from the documents.

We compare our framework with three baseline models. 1) NumNet+ V2 Ran et al.
(2019) is a text QA model with impressive capability of discrete reasoning
over textual data. It constructs a numerically-aware graph neural network,
which takes all numbers in the given question and document as nodes and builds
edges via numerical comparison, and then performs discrete reasoning over the
graph to infer the final answer. 2) TagOp Zhu et al. (2021a) is a table-text
QA model which first applies sequence tagging on each token to extract
question-relevant ones and then applies a set of pre-defined aggregation
operators (e.g. addition, subtraction, counting) over extracted tokens to
generate the final answer. 3) MHST Zhu et al. (2022) is a multi-modal QA model
which employs LayoutLMv2 Xu et al. (2021) as the encoder to take the question,
document text, layout and visual information as input, extracts supporting
evidence using sequence tagging, and applies a tree-based decoder Xie and Sun
(2019) to generate an expression tree with the evidence.

Following Zhu et al. (2022), Exact Match (EM) and numeracy-focused (macro-
averaged) F1 are used to measure the performance of all models, taking into
account the scale of the answer.

Table 1: Performance of our model and baselines on TAT-DQA dev and test. Best results are marked in bold. Model | Dev | Test
---|---|---
EM |  F1 | EM |  F1
NumNet+ V2 | 28.10 | 36.60 | 30.60 | 40.10
TagOp | 32.30 | 39.60 | 33.70 | 42.50
MHST | 39.10 | 47.40 | 41.50 | 50.70
Our Model | 57.97 | 65.38 | 59.23 | 67.61
18.87$\uparrow$ |  17.98$\uparrow$ |  17.73$\uparrow$ |  16.91$\uparrow$
Figure 3:  Performance comparison in F1 score on one-page and multi-page documents on TAT-DQA test set.  Table 2: Performance comparison between our model and MHST for different answer types on TAT-DQA test set. Best results are marked in bold. Model | EM |  F1
---|---|---
MHST | Ours | MHST | Ours
Span | 41.10 | 50.00 | 58.30 | 62.88
Spans | 25.70 | 41.43 | 43.30 | 71.19
Counting | 43.20 | 40.00 | 43.20 | 40.00
Arithmetic | 42.70 | 73.96 | 42.70 | 73.96

###  3.2 Implementation Details

We implement our framework in PyTorch and train it on one NVIDIA DGX-1 with
eight V100 GPUs. We adopt LayoutLMv2large as the encoder. We use Adam
optimizer with a learning rate of $5e-4$ and warmup over the first $6\%$ steps
to train. The maximum number of epochs is set to $50$, while the batch size is
set to $8$ and the gradient accumulation steps is $8$. The dropout
probabilities for the GCNs and GTS are $0.6$ and $0.5$ respectively while
$0.1$ for others. Beam search is applied during inference to select the best
expression tree and the beam size is $5$ by default. We set $12$ as the
maximum number of selected nodes in the node selection.

###  3.3 Comparison with Baselines

We first compare our Doc2SoarGraph framework with all baseline models. The
experimental results are summarized in Table 1. We can observe that our
framework significantly outperforms all the baseline models, achieving new
state-of-the-art. In particular, our framework reaches $59.23\%$ and $67.61\%$
on the test set in terms of Exact Match and F1 metrics respectively, i.e., an
increase of $17.73\%$ and $16.91\%$ over the previous state-of-the-art model
MHST Zhu et al. (2022), demonstrating the great effectiveness of our method.

Table 3: Performance analysis for different parameter sharing strategies among GCNs on the test set. Note that, two GCNs are sharing parameters if the same symbol is used for them. Best results are marked in bold.  GCN (QC) |  GCN (DC) |  GCN (TR) |  GCN (SD) | EM | F1
---|---|---|---|---|---
♣ | ♠ | ♥ | ✤ | 59.23 | 67.61
♣ | ♣ | ♥ | ✤ | 58.62 | 66.99
♣ | ♣ | ♥ | ♥ | 58.45 | 66.94
♣ | ♣ | ♣ | ✤ | 58.62 | 66.94
♣ | ♣ | ♣ | ♣ | 58.39 | 66.60

###  3.4 Analysis on Single- and Multi-page Documents

We compare our model with three baseline models over single-page documents and
multi-page documents respectively. Figure 3 shows the comparison results in
terms of F1 score on TAT-DQA test set. We can observe that our Doc2SoarGraph
significantly outperforms all baselines on both single-page and multi-page
documents, up by $25.41$% and $7.06$% respectively compared to previous best
models. Moreover, the superiority of our framework is much more obvious on
single-page documents ($70.81$%) than multi-page documents ($45.06$%). This
implies that addressing multi-page documents is still more challenging than
single-page ones.

###  3.5 Analysis on Answer Types

TAT-DQA provides four different answer types, and here we analyze the
performance of our framework on each answer type compared to the MHST model.
The results are summarized in Table 2. Compared with MHST, our framework gains
the largest increase (i.e., $31.26$% in terms of EM metric) on Arithmetic
questions, demonstrating impressive discrete reasoning capability. This
enhancement is possibly due to the effective modeling of the differences and
correlations among the quantities, dates and blocks from the documents. For
Spans and Counting questions, they share almost all techniques in the proposed
framework. Comparably, the framework gains a $15.72$% increase on Spans
questions but has a $3.2$% decrease on Counting (i.e., failing one more case).
This is probably due to the data bias on Counting questions because the number
of Counting questions (<$2.0$%) is much less than Spans (>$12.0$%) on test
set. The framework obtains an increase of $8.9$% in terms of EM on Span
questions, indicating our design also benefits the capability of answer
extraction from the document.

Table 4: Analysis on effects of the components in Doc2SoarGraph on TAT-DQA test set. Best results are marked in bold.  Model |  EM ($\uparrow$) |  F1 ($\uparrow$)
---|---|---
MHST | 41.50 (-) | 50.70 (-)
\+ Node Initialization | 54.30 (12.80) | 61.59 (10.89)
\+ Doc Transformation | 56.58 (2.28) | 64.06 (2.47)
\+ Hierarchical Graphs | 58.80 (2.22) | 66.60 (2.54)
\+ Token Masking (Full) | 59.23 (0.43) | 67.61 (1.06)

###  3.6 Analysis on Parameter Sharing

Each of the four graphs in our framework adopts a dedicated graph convolution
network (GCN) to learn node representations, i.e., GCN (QC), GCN (DC), GCN
(TR), and GCN (SD). Here we analyze the performance by adjusting the parameter
sharing strategies among the four GCNs, as shown in Table 3. We can observe
that it achieves the best performance when the four GCNs have independent
parameters. This may be because the four graphs are different and each has its
own role.

###  3.7 Ablation Study

Our Doc2SoarGraph contains four steps, i.e., Multi-page Document
Transformation, Node Initialization according to various semantic elements,
Node Selection via hierarchical graphs, and Answer Generation powered by token
masking. Here we investigate the contributions of each component to its final
performance. As shown in Table 4, we add the four steps in sequence to MHST,
and find that all added components can benefit the model performance.
Furthermore, we find node initialization makes surprisingly greater
contributions to the framework, indicating the importance of modeling the
differences and correlations among the various elements in table-text
documents.

Also, we achieve node selection with hierarchical graphs, i.e. the QC graph,
DC graph, TR graph and SD graph. To test the necessity of each graph, we
remove them one by one from the full model to see the performance. The results
are summarized in Table 5. Continuous performance drop can be observed as we
remove each graph, indicating that each graph contributes to the good
performance of our framework.

Table 5: Ablation study of the hierarchical graphs in our framework on the test set. Model | Dev | Test
---|---|---
EM |  F1 | EM |  F1
Full Graphs | 57.97 | 65.38 | 59.23 | 67.61
\- QC Graph | 56.69 | 65.18 | 57.73 | 66.89
\- DC Graph | 56.14 | 64.83 | 57.73 | 66.82
\- TR Graph | 55.23 | 63.57 | 55.86 | 65.09
\- SD Graph | 56.27 | 64.77 | 56.95 | 66.54

###  3.8 Error Analysis

We randomly sample $100$ error instances of our method on the dev set and
analyze the reasons. We find that the errors occur to all the six modules
(Col. 1 in Table 6), i.e. Span Classifier (SPC), Token Classifier (TC), Node
Classifier (NC), Tree Generator (TG), Scale Classifier (SC) and Answer Type
Classifier (ATC), listed in a descending order based on the error percentage.
These errors are classified into nine categories (Col. 2 in Table 6). See a
more comprehensive table with examples in Appendix. We can see that, 1) $32$%
errors are caused by the SPC module predicting inaccurate predictions of
starting and ending positions for Span questions, i.e., $21$% predictions
overlapping but not exactly matching ground truth, and $11$% predictions
having zero overlap with ground truth; 2) $24$% errors are caused by the NC
module failing to select the relevant nodes; 3) $19$% errors are due to the TC
module predicting less or more tokens than it needs to derive the answer; 4)
$16$% errors are caused by the TG module generating a wrong expression tree,
among which $4$% are wrong number signs (i.e., positive/negative) and $12$%
are other wrong expressions; 5) $6$% and $3$% errors are caused by the SC
module and the ATC module predicting wrong scale and answer types.

Table 6:  Types and statistics of errors in each module.  Module | Error | %
---|---|---
Span Classifier | Offset Error | 21%
No Overlap | 11%
Node Classifier | Missing Nodes | 24%
Token Classifier | Missing Tokens | 11%
Redundant Tokens | 8%
Tree Generator | Wrong Expression | 12%
Wrong Sign | 4%
Scale Classifier | Wrong Scale | 6%
Answer Type Classifier | Wrong Answer Type | 3%

##  4 Related Work

We review related work on discrete reasoning, document visual question
answering (VQA) and graph-based document representation.

###  4.1 Discrete Reasoning

Discrete reasoning has been explored in many natural language processing (NLP)
tasks since 1960s Feigenbaum et al. (1963); Dua et al. (2019); Zhu et al.
(2021b). Recent works focus on a hybrid of annotated (semi-)structured table
and a list of associated paragraphs Chen et al. (2021); Zhao et al. (2022);
Zhu et al. (2021a); Li et al. (2022b), retrieving or extracting evidences from
the given table and paragraphs and then reasoning over these evidences to
generate the answerLei et al. (2022); Zhou et al. (2022); Li et al. (2022a);
Nararatwong et al. (2022); Yarullin and Isaev (2023). Most recently, a
document VQA dataset TAT-DQA Zhu et al. (2022) is released, which triggers the
increasing interest in discrete reasoning over real-world complex financial
documents with both tables and text. To tackle this challenging task, Zhu et
al. (2022) proposed the MHST model, which extracts relevant tokens from the
document using sequence tagging and applies heuristic or “seq2tree” method to
generate the answer according to the answer type. In this work, we also
address the TAT-DQA challenge but with a more powerful framework, where the
differences and correlations of various semantic elements in the question and
document are leveraged with semantic-oriented hierarchical graph structures.

###  4.2 Document VQA

Document VQA is a high-order document understanding task to answer a question
in natural language based on a visually-rich document Cui et al. (2021);
Mathew et al. (2020); Tanaka et al. (2021); Zhu et al. (2022). This task is
mostly tackled by pre-trained language models, like LAMBERT Garncarek et al.
(2020) and StructuralLM Li et al. (2021) which exploit both textual and layout
information of the documents. Recently, some works develop multi-modal pre-
trained language models, e.g., LayoutLMv2 Xu et al. (2021) and DocFormer
Appalaraju et al. (2021) Additionally, some DocVQA models are developed by
fine-tuning the pre-trained language models, e.g. TILT Powalski et al. (2021)
and MHST Zhu et al. (2022). In this work, we build our Doc2SoarGraph framework
based on the pre-trained LayoutLMv2 Xu et al. (2021).

###  4.3 Graph-based Document Representation

Early works use grid-based methods to represent visually-rich documents, such
as representing each document page as a grid of characters Katti et al. (2018)
or a grid of contextualized word piece embedding vectors Denk and Reisswig
(2019). Later, many works Riba et al. (2019); Hwang et al. (2021); Wei et al.
(2020); Yu et al. (2020); Cheng et al. (2020) represent documents with more
powerful graphs to facilitate information extraction from visually-rich
documents. For example, Riba et al. (2019) adopts a GNN-based model to extract
structured tables from invoices; Hwang et al. (2021) constructs a directed
graph to model the spatial dependency among text tokens in the documents. In
this work, we build hierarchical graphs with different semantic elements in
the document (i.e., quantities, dates, and document blocks) to facilitate
evidence extraction and discrete reasoning.

##  5 Conclusion

In this work, we propose a novel Doc2SoarGraph framework with a strong
discrete reasoning capability to tackle QA challenge over visually-rich table-
text documents in the form of TAT-DQA. We model the differences and
correlations of various elements (i.e., quantities, dates, question, and
document blocks) in the input with hierarchical graphs. We validate its
effectiveness with extensive experiments and analyses, which show our
framework can beat the previous state-of-the-art model by large margins. Our
work significantly pushes the frontier of the research on intelligent document
understanding towards real-world application.

## Limitations

Despite the impressive performance on TAT-DQA Zhu et al. (2022), our framework
still has much room for future improvement, as shown in error analysis in
Section 3.8. Also, the Multi-page Document Transformation technique we have
developed for pre-processing multi-page documents is simple, and more
effective methods are desired. For example, our model may not be applicable to
the documents with a large number of pages (e.g., >50 pages). In addition, our
framework is designed for the documents that contain different kinds of
elements, such as numerical values and dates. This means it may have limited
advantages over those with unique elements like pure textual documents.
Furthermore, it may take 3-4 days to train our framework.

## Ethics Statement

In this work, we present a new framework to boost the performance of discrete
reasoning over visually-rich table-text documents. Our framework is developed
on open-source tools and datasets to assist human-being in process . Thus, we
do not anticipate any negative social impacts.

## References

  * Appalaraju et al. (2021) Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021.  Docformer: End-to-end transformer for document understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 993–1003.
  * Chen et al. (2021) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021.  FinQA: A dataset of numerical reasoning over financial data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cheng et al. (2020) Mengli Cheng, Minghui Qiu, Xing Shi, Jun Huang, and Wei Lin. 2020.  One-shot text field labeling using attention and belief propagation for structure information extraction.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , MM ’20, page 340–348. Association for Computing Machinery.
  * Cui et al. (2021) Lei Cui, Yiheng Xu, Tengchao Lv, and Furu Wei. 2021.  Document AI: benchmarks, models and applications.  _CoRR_ , abs/2111.08609.
  * Denk and Reisswig (2019) Timo I. Denk and Christian Reisswig. 2019.  {BERT}grid: Contextualized embedding for 2d document representation and understanding.  In _Workshop on Document Intelligence at NeurIPS 2019_.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proc. of NAACL_.
  * Feigenbaum et al. (1963) Edward A Feigenbaum, Julian Feldman, et al. 1963.  _Computers and thought_.  New York McGraw-Hill.
  * Garncarek et al. (2020) Lukasz Garncarek, Rafal Powalski, Tomasz Stanislawek, Bartosz Topolski, Piotr Halama, and Filip Gralinski. 2020.  LAMBERT: layout-aware language modeling using BERT for information extraction.  _CoRR_ , abs/2002.08087.
  * Hwang et al. (2021) Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang, and Minjoon Seo. 2021.  Spatial dependency parsing for semi-structured document information extraction.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 330–343. Association for Computational Linguistics.
  * Katti et al. (2018) Anoop R Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018.  Chargrid: Towards understanding 2D documents.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4459–4469. Association for Computational Linguistics.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations_.
  * Lei et al. (2022) Fangyu Lei, Shizhu He, Xiang Li, Jun Zhao, and Kang Liu. 2022.  Answering numerical reasoning questions in table-text hybrid contents with graph-based encoder and tree-based decoder.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1379–1390, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Li et al. (2021) Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021.  StructuralLM: Structural pre-training for form understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6309–6318. Association for Computational Linguistics.
  * Li et al. (2022a) Chenying Li, Wenbo Ye, and Yilun Zhao. 2022a.  Finmath: Injecting a tree-structured solver for question answering over financial reports.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6147–6152.
  * Li et al. (2022b) Moxin Li, Fuli Feng, Hanwang Zhang, Xiangnan He, Fengbin Zhu, and Tat-Seng Chua. 2022b.  Learning to imagine: Integrating counterfactual thinking in neural discrete reasoning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 57–69. Association for Computational Linguistics.
  * Lu et al. (2022) Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022.  A survey of deep learning for mathematical reasoning.  _arXiv preprint arXiv:2212.10535_.
  * Mathew et al. (2020) Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. 2020.  Docvqa: A dataset for VQA on document images.  _CoRR_ , abs/2007.00398.
  * Nararatwong et al. (2022) Rungsiman Nararatwong, Natthawut Kertkeidkachorn, and Ryutaro Ichise. 2022.  Enhancing financial table and text question answering with tabular graph and numerical reasoning.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 991–1000. Association for Computational Linguistics.
  * Powalski et al. (2021) Rafal Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021.  Going full-tilt boogie on document understanding with text-image-layout transformer.  _CoRR_ , abs/2102.09550.
  * Ran et al. (2019) Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019.  NumNet: Machine reading comprehension with numerical reasoning.  In _EMNLP-IJCNLP_ , pages 2474–2484.
  * Riba et al. (2019) Pau Riba, Anjan Dutta, Lutz Goldmann, Alicia Fornés, Oriol Ramos, and Josep Lladós. 2019.  Table detection in invoice documents by graph neural networks.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 122–127.
  * Tanaka et al. (2021) Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021.  Visualmrc: Machine reading comprehension on document images.  In _AAAI_.
  * Wei et al. (2020) Mengxi Wei, YIfan He, and Qiong Zhang. 2020.  Robust layout-aware ie for visually rich documents with pre-trained language models.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 2367–2376. Association for Computing Machinery.
  * Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019.  A goal-driven tree-structured neural model for math word problems.  In _IJCAI_ , pages 5299–5305.
  * Xu et al. (2021) Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021.  LayoutLMv2: Multi-modal pre-training for visually-rich document understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2579–2591. Association for Computational Linguistics.
  * Yarullin and Isaev (2023) Ramil Yarullin and Sergei Isaev. 2023.  Numerical embeddings for reasoning over text and tables.
  * Yu et al. (2020) Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, and Rong Xiao. 2020.  PICK: Processing key information extraction from documents using improved graph learning-convolutional networks.  In _2020 25th International Conference on Pattern Recognition (ICPR)_.
  * Zhao et al. (2022) Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022.  MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6588–6600. Association for Computational Linguistics.
  * Zhou et al. (2022) Yongwei Zhou, Junwei Bao, Chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022.  Unirpg: Unified discrete reasoning over table and text as program generation.  _arXiv preprint arXiv:2210.08249_.
  * Zhu et al. (2022) Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. 2022.  Towards complex document understanding by discrete reasoning.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 4857–4866.
  * Zhu et al. (2021a) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021a.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3277–3287. Association for Computational Linguistics.
  * Zhu et al. (2021b) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021b.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _CoRR_ , abs/2101.00774.
  * Zhu et al. (2023) Fengbin Zhu, Moxin Li, Junbin Xiao, Fuli Feng, Chao Wang, and Tat Seng Chua. 2023\.  Soargraph: Numerical reasoning over financial table-text data via semantic-oriented hierarchical graphs.  In _Companion Proceedings of the ACM Web Conference 2023_ , page 1236–1244. Association for Computing Machinery.

##  Appendix A Appendix

###  A.1 Training and Inference

To optimize the proposed framework, the objective is to minimize the sum of
the losses of the all classification tasks. Formally, the overall loss for
each sample can be computed as

$\displaystyle\begin{split}\mathcal{L}=&\mathcal{L}_{node}\textrm{+}\mathcal{L}_{tree}\textrm{+}\mathcal{L}_{start}\textrm{+}\mathcal{L}_{end}\textrm{+}\mathcal{L}_{type}\textrm{+}\mathcal{L}_{token}\textrm{+}\\\
&\mathcal{L}_{scale}\\\
\textrm{$\mathcal{L}_{node}$}&=~{}\frac{1}{\mathopen{|}\mathrm{N}\mathclose{|}}\sum_{\mathrm{n}\in\mathrm{N}}\textrm{CE}(\textrm{P}_{\textrm{node}}^{\textrm{n}},\textrm{g}_{\textrm{node}}^{\textrm{n}})~{}\\\
\textrm{$\mathcal{L}_{tree}$}&=~{}\frac{1}{\mathopen{|}\mathrm{S}\mathclose{|}}\sum_{\mathrm{s}\in\mathrm{S}}\textrm{CE}(\textrm{P}_{(}v^{s}|v^{1},...,v^{s-1},Q,G),\textrm{g}^{s}_{v})~{}\\\
\textrm{$\mathcal{L}_{start}$}&=~{}\textrm{CE}(\textrm{P}_{start},\textrm{g}_{start})\\\
\textrm{$\mathcal{L}_{end}$}&=~{}\textrm{CE}(\textrm{P}_{end},\textrm{g}_{end})\\\
\textrm{$\mathcal{L}_{type}$}&=~{}\textrm{CE}(\textrm{P}_{type},\textrm{g}_{type})\\\
\textrm{$\mathcal{L}_{token}$}&=~{}\frac{1}{\mathopen{|}\mathrm{T}\mathclose{|}}\sum_{\mathrm{t}\in\mathrm{T}}\textrm{CE}(\textrm{P}_{\textrm{token}}^{\textrm{t}},\textrm{g}_{\textrm{token}}^{\textrm{t}})~{}\\\
\textrm{$\mathcal{L}_{scale}$}&=~{}\textrm{CE}(\textrm{P}_{scale},\textrm{g}_{scale})\\\
\end{split}$ (7)

where $N$ is a set of nodes; $g_{node}^{n}$ is the ground-truth label if the
node $n$ is selected; $\mathrm{CE}(\cdot)$ refers to the cross-entropy loss;
$S$ is the number of decoding steps during the expression tree generation;
$g_{v}^{s}$ is the ground-truth node in the step $s$; $g_{start}$ and
$g_{end}$ are the ground-truth starting and ending positions of the span
answer; $g_{type}$ is the ground-truth answer type; $T$ refers to all the
valid tokens after applying the token masking in Section 2.4.1;
$g_{token}^{t}$ is the ground-truth label of the token $t$; $g_{scale}$ is the
ground-truth scale value. When the nodes in the ground-truths in Node
Selection are not selected, we will add them manually in order to better train
the tree-based decoder when training.

During inference, we select a maximum number of nodes in the stage of Node
Selection. Beam search is applied during inference to select the best
expression tree.

Figure 4:  Comparison of evidence extraction power between our Doc2SoarGraph
and MHST on Arithmetic questions on TAT-DQA dev set.

###  A.2 Analysis on Evidence Extraction

A general pipeline for discrete reasoning over table-text documents is to
extract supporting evidence first and then perform reasoning over the
evidence. Our framework exploits the semantic relationships and dependencies
among different elements in the table-text document via hierarchical graphs,
to facilitate the model in supporting evidence extraction and discrete
reasoning. Here we conduct experiments to validate whether the evidence
extraction power is indeed enhanced with our framework. In particular, we
compute the average recall, precision and F1 score of the extracted evidence
with our method and MHST on the dev set. For fair comparison, we only use the
Arithmetic questions that only depend on quantity and date nodes. Given one
question, assume the number of quantities/dates its answer actually requires
is $n$, the number of predicted quantities/dates by the model is $m$ and the
number of correct quantities/dates in prediction is $c$. The recall is
computed using $c$ divided by $n$ (i.e., $c/n$) while the precision is
computed using $c$ divided by $m$ (i.e., $c/m$). Then we can compute the F1
with the precision and recall. After getting the metrics of each question in
the dev set, we further obtain the average recall, precision and F1 by simply
computing the corresponding average value across all questions. The results
are shown in Figure 4. From the figure, we can see that our model demonstrates
significant improvements over the MHST. Specifically, our method has an
increase of $23.76\%$ and $24.43\%$ in terms of average precision and average
recall compared with MHST, significantly improving the evidence extraction.

###  A.3 Error Analysis with Examples

To further investigate our Doc2SoarGraph framework, we randomly sample $100$
error instances on the dev set and analyze the reasons. We find the errors
occur to all the six modules. We provide a comprehensive summary of all the
errors and corresponding percentages in Table 7, each with an example, so as
to facilitate the readers’ understandings of the discussions in our main text.

Table 7:  Examples of errors and corresponding percentage in each module. Q, G and P denote question, ground truth and prediction. Module |  Error(%) |  Example
---|---|---
Span Classifier |  Offset Error (21%) |  Q:What is the change in the projected benefit obligation of the benefit plans recognized as?
G: part of the net actuarial loss and … comprehensive loss … participants
P: part of the net actuarial loss
No Overlap (11%) |  Q: What does the table show?
G:Aggregate emoluments of the Directors of the Company
P: total amounts earned by the Company’s Directors and members of the
Executive Committee
Node Classifier |  Missing Nodes (24%) |  Q: What was the percentage change in the amount Outstanding at 31 March in 2019 from 2018?
G: 320,872, 690,791
P: 320,872, 5,416
Token Classifier |  Missing Tokens (11%) |  Q: What was the stock-based compensation expense in 2019, 2018 and 2017 respectively?
G: “1.9 million”, “2.1 million”, 2.3 million
P: “1.9 million”, “2.1 million”
Redundant Tokens (8%) |  Q: In which year was Net cash provided by operating activities less than 800 million?
G: “2017”, “2018”
P: “2017”, “2018”, “2020”
Tree Generator |  Wrong Expression (12%) |  Q: What is the difference between 1-90 and 91-180 days past due in low risk country?
G: 1,347 - 125
P: 1,347 - 313
Wrong Sign (4%) |  Q: What is the nominal difference for ROFE between F19 and F18?
G: 17.1 - 15.2 = 1.9
P: 15.2 - 17.1 = -1.9
Scale Classifier |  Wrong Scale (6%) |  Q: What was the change in the number of shares granted in 2019 from 2018?
G: 93.8 thousands
P: 93.8
Answer Type Classifier |  Wrong Answer Type (3%) |  Q: In which years was Teekay LNG revenue less than 600,000 thousands?
G: 2018 (Span)
P: 2018,2019 (Spans)