# SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural
Networks Training

Pengcheng Dai1, Jianlei Yang2, Xucheng Ye2, Xingzhou Cheng2, Junyu Luo2,
Linghao Song3,
Yiran Chen3 and Weisheng Zhao1
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>This work is supported in
part by the National Natural Science Foundation of China (61602022), State Key
Laboratory of Software Development Environment (SKLSDE-2018ZX-07), CCF-Tencent
IAGR20180101 and the 111 Talent Program B16001. 1School of Microelectronics,
BDBC, Beihang University, Beijing, China  2School of Computer Science and
Engineering, BDBC, Beihang University, Beijing, China  3Department of
Electrical and Computer Engineering, Duke University, Duram, NC, USA

###### Abstract

Training Convolutional Neural Networks (CNNs) usually requires a large number
of computational resources. In this paper, SparseTrain is proposed to
accelerate CNN training by fully exploiting the sparsity. It mainly involves
three levels of innovations: activation gradients pruning algorithm, sparse
training dataflow, and accelerator architecture. By applying a stochastic
pruning algorithm on each layer, the sparsity of back-propagation gradients
can be increased dramatically without degrading training accuracy and
convergence rate. Moreover, to utilize both natural sparsity (resulted from
ReLU or Pooling layers) and artificial sparsity (brought by pruning
algorithm), a sparse-aware architecture is proposed for training acceleration.
This architecture supports forward and back-propagation of CNN by adopting
1-Dimensional convolution dataflow. We have built a cycle-accurate
architecture simulator to evaluate the performance and efficiency based on the
synthesized design with $14nm$ FinFET technologies. Evaluation results on
AlexNet/ResNet show that SparseTrain could achieve about $2.7\times$ speedup
and $2.2\times$ energy efficiency improvement on average compared with the
original training process.

###### Index Terms:

Convolutional Neural Networks, Training, Sparse, Pruning, Accelerator,
Architecture

##  I Introduction

Recent years, Convolutional Neural Networks (CNNs) have been widely used in
computer vision tasks such as image classification [1], object detection [2],
face recognition [3], and object tracking [4]. Modern CNN models could achieve
state-of-the-art accuracy by extracting rich features and significantly
outperform traditional methods [5]. However, running large scale CNN models
often requires large memory space and consumes huge computational resources.

Many CNN accelerator architectures have been invented to improve the CNN
inference throughput and energy efficiency in both industrial [6, 7] and
academic [8, 9, 10, 11, 12, 13, 14, 15, 16] communities. Among them,
exploiting network sparsity has been observed as one of the most efficient
approaches. For example, by applying weight pruning, the model size of AlexNet
and VGG-16 could be reduced by $35\times$ and $49\times$, respectively [17],
which provides a huge potential for accelerating CNN inference.

(a) Inference without sparsity.

(b) Inference with weight sparsity.

(c) Inference with activation sparsity.

(d) Training with weight gradient sparsity.

(e) Ours: training with activation gradient sparsity.

Figure 1: CNN sparsity exploration.

In fact, by using weight sparsity, a sparse-aware computing architecture — EIE
[18] achieved $13\times$ speedups and $3400\times$ energy efficiency when
compared to GPU implementation of the same DNN without compression. Fig. 1(a)
and Fig. 1(b) show the original CNN inference process and the one with weight
pruning and sparsity utilization, respectively. One step further, SCNN [9]
achieved significant speedup in CNNs inference by utilizing both weight
sparsity and natural sparsity of activations. Fig. 1(c) shows the basic idea
of this procedure.

Most works focused on the inference of CNN. However, the sparsity of the
training process was less studied. Compared with inference, CNN training
demands much more computational resources. Usually CNN training introduces
about $3\times$ computational cost and consumes $10\times$ to $100\times$
memory space compared with the inference.

To utilize gradient sparsity in CNNs training, [19][20] proposed a method that
uses only three numerical levels {-1,0,1} for weight gradients to reduce
communication in distributed training, which can be considered as a mixture of
pruning and quantization. This scheme is shown in Fig. 1(d). However, it can
hardly reduce the computation since the main training process (forward and
back-propagation) is still calculated in a dense data format. There are also
many training accelerators try to utilizing sparsity, which can be found in
[21]. But neither of them exploit gradient sparsity.

Since activation gradients is an operand for both backward process and weight
gradients generation process, the sparsity of activation gradients provides a
significant reduction on the computation of training. To overcome the
limitation of previous works and fully exploit the sparsity of CNN training,
we proposed a layer-wise gradients pruning method that can improve the
training performance remarkably by generating artificial sparsity on
activation gradients data. Our scheme is shown in Fig. 1(e). The key
contributions of this paper are:

  * •

We present a layer-wise gradients pruning algorithm that can generate high
artificial sparsity on activation gradients data with negligible overhead. Our
experiments show that this algorithm will hardly degrade the accuracy and
convergence rate of CNNs.

  * •

We propose a 1-Dimensional Convolution based dataflow. Combined with our
pruning algorithm, it can fully utilize all kinds of sparsity in training.

  * •

To support our sparse dataflow, an accelerator architecture is designed for
CNNs training that can benefit from both activation and gradient sparsity.

The proposed SparseTrain scheme is evaluated by AlexNet [1] and ResNet [22] on
CIFAR/ ImageNet. Evaluation results on AlexNet with natural sparsity show
SparseTrain achieves $2.7\times$ speedup and $2.2\times$ energy efficiency on
average, compared with the dense baseline.

The remaining of this paper is organized as follows. Section II introduces CNN
training basics. Section III provides a gradients pruning algorithm, as well
as the threshold determination and prediction method. The details of our
sparse training dataflow are presented in Section IV. Section V demonstrates
the sparse-aware architecture designed for the proposed training dataflow.
Evaluation results are discussed in Section VI. Section VII concludes this
paper.

##  II Preliminary

Figure 2: CNN training demonstration.

A typical CNN training procedure is shown in Fig. 2. It contains three stages:
Forward, Backward and Weight Update.

Forward stage starts from input layer until final layer, including
convolutional (CONV), ReLU and MaxPool layer. Input activations $\mathbf{I}$
of $n$-th CONV layer are formulated as a 3-D tensor. The size of $\mathbf{I}$
is determined by the number of input channels $C$, height $H$ and width $W$,
and we denote the input activations of $i$-th channel as $\mathbf{I}_{i}$.
Output activations $\mathbf{O}$ are also formulated as a 3-D tensor with $F$
output channels. Weights are formulated as a 4-D tensor with size of $K\times
K\times C\times F$, where each convolution kernel $\mathbf{W}_{i,j}$ is a 2-D
tensor with the size of $K\times K$. Vector $\mathbf{b}$ is the bias applied
on $\mathbf{O}$ after convolution. If 2-D convolution is denoted by “$\ast$”,
the CONV layer can be represented as

$\mathbf{O}_{i}=\sum_{j=0}^{C}{\mathbf{W}_{i,j}}\ast\mathbf{I}_{j}+\mathbf{b}_{i},\quad
i=0,\cdots,F.$

ReLU and MaxPool are non-linear operation layers. ReLU layer applies point-
wise function $f(x)=max(0,x)$ on all activations. MaxPool layer select the
maximum value from each window of activations as output. Hence, the
activations usually become sparse after ReLU and MaxPool layers. And the non-
zero patterns generated by ReLU and MaxPool layers are recorded as mask and
will be adopted in backward stage.

Backward stage includes two steps:

  * •

Gradient To Activations (GTA): it calculates the activation gradients
(derivatives to certain layer’s activations). According to chain rule, the
gradients are calculated from loss function back to input layer. The CONV
layer in GTA step is represented as

${\mathrm{d}\mathbf{I}}_{j}=\sum_{i=0}^{F}{\mathrm{d}\mathbf{O}_{i}\ast{\mathbf{W}^{+}}_{i,j}},\quad
j=0,\cdots,C,$

where $\mathrm{d}\mathbf{I}_{j}$ indicates the input activation gradients
(derivatives of input activations) for the $j$-th channel,
$\mathrm{d}\mathbf{O}_{i}$ represents the output activation gradients for the
$i$-th channel, $\mathbf{W}_{i,j}$ is the $i$-th filter of $j$-th channel,
$\mathbf{W}^{+}_{i,j}$ is sequentially reversed of $\mathbf{W}_{i,j}$, or in
other words, $\mathbf{W}_{i,j}$ rotated by $180$ degrees. ReLU and MaxPool
layers in GTA step directly adopt the pre-stored mask in forward stage.

  * •

Gradient To Weights (GTW): it calculates the weight gradients
${\mathrm{d}\mathbf{W}}$ (derivatives of loss function to layers’ weights).
These weight gradients are utilized to update weights by Stochastic Gradient
Descent (SGD) method. Weight gradients are calculated by

${\mathrm{d}\mathbf{W}}_{i,j}={\mathrm{d}\mathbf{O}}_{i}\ast\mathbf{I}_{j},\quad
i=0,\cdots,F,~{}j=0,\cdots,C,$

where ${\mathrm{d}\mathbf{W}}_{i,j}$ is the weight gradients of $j$-th channel
in $i$-th filter.

Weight Update stage is to update the weights of each layer with the calculated
weight gradients by using SGD method. For modern CNNs, batch training is a
popular way to update the weights by sending a batch of inputs (e.g. 32
images) into the network. The weight gradients are computed by averaging the
batch of gradients. Finally, the weights are updated according to a pre-set
learning rate $\alpha$. Generally, weight update stage is not a performance
bottleneck for CNN training. Thus, only the Forward, GTA and GTW procedures
are taken into considerations for acceleration.

##  III Algorithm

###  III-A Stochastic Pruning

Figure 3: Stochastic gradients pruning algorithm. $\tau$ is the threshold that
determines whether a value should be pruned. $p$ is the probability of setting
a value to $0$ or $\pm\tau$.

From the experiments, we found that there are many gradients data whose
absolute value is very small. Intuitively, a single small gradient value has
little impact on the updating of weight, thus it can be pruned (set to zero)
directly. However, if many values are pruned, the distribution of gradients is
changed remarkably, which causes accuracy loss. Thus, we adopt a stochastic
pruning algorithm proposed by [23] to solve this problem.

This algorithm treats the gradients that to be pruned (denoted as $g$) as a
$n$-dimensional vector, and prunes the vector component whose absolute value
is smaller than the threshold ${\tau}$. Fig. 3 shows the stochastic pruning
method. By setting values to zero or $\pm\tau$ stochastically, the expectation
of each component remains unchanged, which improves the convergence and
accuracy. Detailed analysis can be found in [23].

Input: $\left[G_{1},G_{2},...,G_{N}\right]$: $N$ batch original activation
gradients, $N_{F}$: depth of FIFO, $p$: target sparsity.

Output: sparse activation gradients
$\left[\hat{G}_{1},\hat{G}_{2},...,\hat{G}_{N}\right]$

1 $F$ $\triangleq$ FIFO with depth $N_{F}$.

2for  _$i=1;i\leq N;i\leftarrow i+1$_ do

3 $g\triangleq G_{i}$ ; $\hat{g}\triangleq\hat{G}_{i}$ ;

4 $n=$ length of $g$ ; $A=0$ ;

5 for  _$j=1;j\leq n;j\leftarrow j+1$_ do

6

7 $A\leftarrow A+\left|g_{i}\right|$ ;

8

9 if  _$i >N_{F}$_ then

10 $\hat{\tau}$ = mean$\left(F\right)$ ;

11 if  _${|g_{i}| <\hat{\tau}}$_ then

12 Generate a random number $r\in\left[{0,1}\right]$ ;

13 if  _$|g_{i}| >\hat{\tau}r$_ then

14 ${\hat{g}_{i}}=({g_{i}}>0)$ ? ${\hat{\tau}}$ : $({-\hat{\tau}})$ ;

15

16 else

17 ${\hat{g}_{i}}=0$ ;

18

19

20 else

21 ${\hat{g}_{i}}={g_{i}}$

22

23

24
$\tau={\Phi^{-1}}\left({\frac{1-p}{2}}\right)\frac{1}{n}\sqrt{\frac{2}{\pi}}A$
;

25 $F.$push $\left(\tau\right)$ ;

26

Algorithm 1 Overall Pruning Scheme

###  III-B Threshold Determination and Prediction

Clearly, it’s unfeasible to find the threshold by sorting, due to its huge
memory and time consumption. Thus, we propose a hardware friendly threshold
selection scheme with less overhead. This selection method contains 2 steps:
Determination and Prediction, where the Determination step refers to [23].

Figure 4: Pruning positions of two typical structures.

Figure 5: The threshold prediction scheme for each CONV layer using FIFO.

Threshold Determination. Modern CNN models have two typical structures, as
shown in Fig. 5. For the CONV-ReLU structure, where a CONV layer is followed
by a ReLU layer, output activation gradients $\mathrm{d}\mathbf{O}$ are
sparse, but subject to a irregular distribution. On the other hand, the input
activation gradients $\mathrm{d}\mathbf{I}$, which going to be propagated to
the previous layer, is full of non-zero values. Statistics show that the
distribution of $\mathrm{d}\mathbf{I}$ is symmetric with 0 and its density
decreases with the increment of absolute value. For the CONV-BN-ReLU
structure, $\mathrm{d}\mathbf{O}$ subjects to the same distribution of
$\mathrm{d}\mathbf{I}$. Simply, we assume these gradients all subject to a
normal distribution with mean $0$ and variance ${{\sigma^{2}}}$ according to
[23].

In the first case, $\mathrm{d}\mathbf{O}$ can inherit the sparsity of
$\mathrm{d}\mathbf{I}$ because ReLU layer won’t reduce sparsity. Thus
$\mathrm{d}\mathbf{I}$ can be treated as pruning target $g$ in CONV-ReLU
structure. Besides, in CONV-BN-ReLU structure, $\mathrm{d}\mathbf{O}$ is
considered as pruning target $g$. In this way, the distribution of $g$ in both
situations could be unified to normal distribution.

Suppose the length of $g$ is $n$, the standard deviation $\sigma$ is estimated
in a unbiased way [23] using

$\hat{\sigma}=\frac{1}{n}\sqrt{\frac{2}{\pi}}\sum\limits_{i=1}^{n}{\left|{{g_{i}}}\right|}\
,\quad{g_{i}}\in g.$

Then we can compute the threshold ${\tau}$ using the cumulative distribution
function ${\Phi}$ of the standard normal distribution, target pruning rate $p$
and ${\hat{\sigma}}$:

$\tau{\rm{=}}{\Phi^{-1}}\left({\frac{1-p}{2}}\right)\hat{\sigma}.$

Threshold Prediction. The stochastic pruning scheme mentioned above needs to
access all gradients two times: for computing $A$ and for gradients pruning.
What is worse, gradients need to be stored in memory temporarily before
pruned, which brings overhead of memory access. The best way is to prune
gradients before they are sent back to memory. To accomplish this, we improve
the algorithm by predicting the threshold before calculating. Denoting the
number of batches as $N$, the prediction method keeps a FIFO with a length of
$N_{F}$ for each CONV layer, where $N_{F}$ is a hyper-parameter that satisfies
$N_{F}<<N$.

Fig. 5 shows the pruning algorithm with threshold prediction for each CONV
layer. Activation gradients of each batch will be pruned under the predicted
threshold ${\tau^{\prime}}$ as soon as they are calculated, where
${\tau^{\prime}}$ is the average of all the thresholds stored in the FIFO. At
the end of each batch, The determined threshold of this batch is calculated
and pushed into the FIFO. Gradients will not be pruned before the FIFO is
filled up. Details of the whole gradients pruning algorithm is shown in
Algorithm 1.

The arithmetic complexity of our algorithm is $\mathcal{O}\left(n\right)$,
while complexity of sorting is at least $\mathcal{O}\left(n\log n\right)$.
Besides, all the gradients will be accessed just one time in our algorithm, so
that almost no extra storage is required, which saves time and energy
consumption.

##  IV Dataflow

(a) The Forward step.

(b) The GTA step.

(c) The GTW step.

Figure 6: Demonstration of our dataflow. Operations in CONV layers are
disassembled into channel level and row level. Row level operations are picked
as the basic operations of the dataflow.

To gain more benefits from the above algorithm, it’s essential to design an
accelerator that can utilize both activation sparsity and gradient sparsity.
Prior works have shown that the utilized dataflow usually affects the
architecture’s performance significantly [8]. Thus, we first introduce the
dataflow used in the accelerator.

We propose a sparse training dataflow by dividing all computation into
1-Dimensional convolutions. This dataflow supports all kinds of sparsity in
training and provides opportunities for exploiting all types of data reuse.
This section will introduce the sparsity in training and the dataflow in
detail.

###  IV-A Data Sparsity in Training

The sparsity of involved six data types during training have been summarized
in Table I. Input activations $\mathbf{I}$ for each CONV layer are usually
sparse because the previous ReLU layer sets negative activation values into
zeros. Weights $\mathbf{W}$ are also dense for all steps of training. The
output activation gradients $\mathrm{d}\mathbf{O}$ are usually sparse. But for
networks with BN layers, $\mathrm{d}\mathbf{O}$ becomes dense after passing
through BN layers. However, this issue can be resolved by our gradients
pruning algorithm. Thus, we can regard output activation gradients of all CONV
layers as sparse data.

There is an additional optimization opportunity in the GTA step. The gradients
$\mathrm{d}\mathbf{I}$ are usually sent to a ReLU layer after generated from a
CONV layer, leading to certain values be set to zero forcefully. Actually, we
could predict the positions of these zeros and skip the corresponding
calculations, according to the mask generated in the Forward step. Besides,
both operands (input activations $\mathbf{I}$ and gradients to output
activation $\mathrm{d}\mathbf{O}$) in the GTW step are sparse. Theoretically,
it could significantly reduce the computation cost in obtaining the weight
gradients $\mathrm{d}\mathbf{W}$.

###  IV-B Sparse Training Dataflow

Aiming to utilize all the sparsity in the training process, we divide one 2-D
convolution into a series of 1-D convolutions and treat the 1-D convolutions
as basic operations for scheduling and running. In this subsection, we will
show the way to disassemble three basic steps of CNN training into 1-D
convolutions in detail.

Forward step. As shown in Fig. 6(a), for one particular 2-D convolution, a
small kernel moves through the input activations and perform multiplications
and accumulations on each position. More detailed, one output row is the
addition of $K$ 1-D convolution results, where $K$ is the kernel size. For a
1-D convolution in the Forward step, one operand is a certain row of the
kernel, which is a short dense vector. Another operand is a row of the input
activations, which is a long sparse vector. This basic operation is denoted as
Sparse Row Convolution (SRC).

TABLE I: Sparsity summary for involved data in training. Data Type | Symbol | Sparsity
---|---|---
Weights | $\mathbf{W}$ | dense
Weight Gradients | $\mathrm{d}\mathbf{W}$ | dense
Input Activations | $\mathbf{I}$ | sparse
Gradients to Input Activations | $\mathrm{d}\mathbf{I}$ | dense
Output Activations | $\mathbf{O}$ | dense
Gradients to Output Activations | $\mathrm{d}\mathbf{O}$ | sparse

GTA step. Fig. 6(b) shows the decomposition of the GTA step. Similar to the
Forward step, the GTA step can also be regarded as the summation of multiple
1-D convolutions. However, as mentioned in Section IV-A, certain gradients are
set to zero forcefully after sent to ReLU layers. Thus, the calculation of
these masked values are entirely unnecessary and can be skipped safely to save
computation cost. Considering this optimization, we define a different basic
operation named as Masked Sparse Row Convolution (MSRC).

GTW step. The weight gradients computation in the GTW step is shown in Fig.
6(c). The GTW step is significantly different from the Forward and the GTA
step in two ways. The first is that operands of extracted 1-D convolutions are
two sparse long vectors. The second lies in the range of results: traditional
convolutions need to slide one vector from the start of the other vector to
its end, and calculating multiplications and accumulations in each sliding
position. However, in the GTW step, only the results in several positions are
needed, and it is not necessary to fully calculate all the convolution
results. The resulting vector is usually short (as the kernel size $K$) and we
can store it in a small scratchpad during the whole convolution. Thus, this
1-D convolution is named Output Store Row Convolution (OSRC) and is considered
as the basic operations of the GTW step.

Gradients to bias in CONV layers are also required in the GTW step. The
calculation of bias gradients for each channel is just the summation of the
output activation gradients from the corresponding channel. It is simple to
calculate them by accumulating gradients during the GTA step.

##  V Architecture

(a) Overview architecture

(b) PPU module

(c) PE module

Figure 7: The architecture design of SparseTrain. TABLE II: Evaluation results for the gradients pruning algorithm, where acc% means the training accuracy and $\rho_{nnz}$ means the density of non-zeros. Model | Dataset | Baseline | $p=70\%$ | $p=80\%$ | $p=90\%$ | $p=99\%$
---|---|---|---|---|---|---
acc% | $\rho_{nnz}$ |  acc% | $\rho_{nnz}$ |  acc% | $\rho_{nnz}$ |  acc% | $\rho_{nnz}$ |  acc% | $\rho_{nnz}$
AlexNet | CIFAR-10 | 90.50 | 0.09 | 90.34 | 0.01 | 90.55 | 0.01 | 90.31 | 0.01 | 89.66 | 0.01
ResNet-18 | CIFAR-10 | 95.04 | 1 | 95.23 | 0.36 | 95.04 | 0.35 | 94.91 | 0.34 | 94.86 | 0.31
ResNet-34 | CIFAR-10 | 94.90 | 1 | 95.13 | 0.34 | 95.09 | 0.32 | 95.16 | 0.31 | 95.02 | 0.28
ResNet-152 | CIFAR-10 | 95.70 | 1 | 95.13 | 0.18 | 95.58 | 0.18 | 95.45 | 0.16 | 93.84 | 0.08
AlexNet | CIFAR-100 | 67.61 | 0.10 | 67.49 | 0.03 | 68.13 | 0.03 | 67.99 | 0.03 | 67.93 | 0.02
ResNet-18 | CIFAR-100 | 76.47 | 1 | 76.89 | 0.40 | 77.16 | 0.39 | 76.44 | 0.37 | 76.66 | 0.34
ResNet-34 | CIFAR-100 | 77.51 | 1 | 77.72 | 0.36 | 78.04 | 0.35 | 77.84 | 0.33 | 77.40 | 0.31
ResNet-152 | CIFAR-100 | 79.25 | 1 | 80.51 | 0.22 | 79.42 | 0.19 | 79.76 | 0.18 | 76.40 | 0.10
AlexNet | ImageNet | 56.38 | 0.07 | 57.10 | 0.05 | 56.84 | 0.04 | 55.38 | 0.04 | 39.58 | 0.02
ResNet-18 | ImageNet | 68.73 | 1 | 69.02 | 0.41 | 68.85 | 0.40 | 68.66 | 0.38 | 68.74 | 0.36
ResNet-34 | ImageNet | 72.93 | 1 | 72.92 | 0.39 | 72.86 | 0.38 | 72.74 | 0.37 | 72.42 | 0.34

Aiming to evaluate the dataflow sparsity in SparseTrain, we design an
architecture of which overview is shown in Fig. 7(a). The architecture
consists of Buffer, Controller, and PE groups. Each PE group contains $3$ PEs
and a Post Processing Unit (PPU). PE is designed to calculate 1-D convolution
and PPU is adopted for point-wise operations.

PE Architecture: As demonstrated in Section IV, there are three kinds of basic
operations in our dataflow: SRC, MSRC and OSRC. PE module is designed to
support all of these operations and the architecture of PE module is shown in
Fig. 7(c).

Our PE is designed to perform a complete 1-D convolution, instead of just one
multiplication. Each time an input value is loaded from Port-1, PE multiplies
it by $K$ values in Reg-1, providing $K$ product results stored in Reg-2.

When performing SRC operations, A PE first loads weight vector from Port-2 and
saves them in Reg-1. Then activation values are loaded from Port-1 and
multiplied by the weights in Reg-1. Results are accumulated to Reg-2. This
process repeats until the activation vector gets to the end.

For MSRC operations, the offset vector of input activations ($\mathbf{I}$) is
loaded from Port-3 and saved to Reg-2. Values in the offset vector is used to
indicate the results that should be calculated. In other words, values that
are not encoded in the offsets of $\mathbf{I}$ can be predicted as zero and
skipped safely. During calculation, PE will look ahead for the next non-skip
operand from Port-1, so that it can start computation as soon as possible
without extra cycles waiting for input data.

For OSRC operations, PE stores weight gradients ($\mathrm{d}\mathbf{W}$) in
Reg-2 during computing. $\mathbf{I}$ and $\mathrm{d}\mathbf{O}$ vector are
loaded from Port-1 and Port-2, respectively. During computing, $K$ values of
$\mathrm{d}\mathbf{O}$ will be cached in Reg-1 and multiplied with each value
in $\mathbf{I}$. The results will be accumulated to Reg-2.

PPU Architecture: In general, PPU is responsible for all point-wise
operations. Fig. 7(b) shows the internal architecture of PPU. In PPU,
resulting vector will be converted into a compressed format and sent back to
buffer. PPU will also perform ReLU operation before format conversion if
needed. During GTA step, all of the gradients through PPU modules and their
absolute value will be accumulated into two registers respectively.
Accumulation results are used for generating bias gradients and determining
pruning threshold.

##  VI Evaluation

In this section, several experiments are conducted to demonstrate that the
proposed approach could reduce the training complexity significantly with a
negligible model accuracy loss. In the following experiments, AlexNet [1] and
ResNet [22] series are evaluated on three datasets including CIFAR-10,
CIFAR-100 and ImageNet. All the models are trained for $300$ epochs on
CIFAR-{10, 100} and $180$ epochs on ImageNet, due to our limited computing
resources. PyTorch framework is adopted for the algorithm verification.

To verify the performance of our dataflow and architecture, a custom cycle-
accurate C++ simulator is implemented based on SystemC model for the proposed
architecture, and a simple compiler is designed with Python to convert CNN
models in PyTorch into our internal instructions for driving our simulator. We
also have the RTL implementation for PE, PPU and controller, which is
synthesized by Synopsys Design Compiler with Global Foundries $14nm$ FinFET
technology for area estimation and then simulated by Synopsys PrimeTime for
power estimation. The area and energy consumption of buffer (SRAM) is
estimated by PCACTI [24]. To demonstrate the advantages of our proposed sparse
training dataflow, we compare it with the architecture from Eyeriss [8]. Since
Eyeriss is designed for CNN inference rather than training, we modify the
architecture of Eyeriss to support the dense training process. We adopt $168$
PEs in both the proposed architecture and the baseline architecture. For
convenience, $386$KB SRAM is utilized as the global buffer for intermediate
data, which is sufficient for storing data used in each iteration. A larger
buffer is beneficial to improving data-reuse and energy efficiency, but it is
beyond the considerations of this work.

###  VI-A Sparsity and Accuracy

From Table II, there is no accuracy lost for most situations and even $1\%$
accuracy improvement for ResNet-50 on CIFAR-100. The only significant accuracy
lost exists in AlexNet on ImageNet, when using a very aggressive pruning
policy like $p=99.5\%$. That proves the accuracy loss caused by our layer-wise
gradients pruning algorithm is almost negligible.

The gradient density illustrated in Table II shows that our method could
reduce the gradient density by $3\times\sim 10\times$. In addition, the deeper
networks could obtain a relatively lower gradient density with our
sparsification, which means that our method works better for larger networks.

Figure 8: Average training latency per sample for different models and
datasets. The speedup is the training latency of SparseTrain compared with the
baseline. Figure 9: Average energy consumption per sample for different models
and datasets. “Reg” represents register, and “Comb” represents combinational
logic. The energy efficiency is the improvement of SparseTrain compared with
the baseline.

###  VI-B Convergency

The training loss of AlexNet/ResNet-18 on CIFAR-10 and ImageNet is
demonstrated in [23]. In general, ResNet-18 is very robust for gradients
pruning. For AlexNet, the gradients pruning could still be robust on CIFAR-10
but will degrade convergency speed a little on ImageNet with aggressive
pruning policy. These results prove that with reasonable hyper-parameter $p$,
our pruning algorithm basically has the same convergency property compared
with the original training scheme.

###  VI-C Latency and Energy

Fig. 8 shows the training latency reduction brought by sparsity exploration.
The proposed SparseTrain scheme achieves $4.5\times$ speedup at most for
AlexNet on CIFAR-10. On average, it achieves about $2.7\times$ speedup
compared with the baseline.

Fig. 9 shows the average energy consumption per data sample. Overall,
SparseTrain has $1.5\times$ to $2.8\times$ (on average $2.2\times$) energy
efficiency improvement than baseline. For the baseline, $62\%\sim 71\%$ of the
energy consumption comes from SRAM accesses. SparseTrain reduces the global
buffer accesses by utilizing sparse dataflow and reduces the energy cost by
$30\%\sim 59\%$. The energy consumption of combinational logic in SparseTrain
could be reduced by $53\%\sim 88\%$, which is more significant than SRAM and
register accesses. This also contributes much to the total energy saved.

##  VII Conclusion

As the model size and datasets scale larger, CNN training becomes more and
more expensive. In response, we propose a novel CNN training scheme called
SparseTrain for acceleration by fully exploiting sparsity in the training
process. By performing stochastic gradients pruning on each layer, SparseTrain
achieves high sparsity with negligible accuracy loss. Additionally, with a 1-D
CONV based sparse training dataflow and a sparse-aware architecture,
SparseTrain can improve CNN training speed and efficiency significantly by
exploiting different types of sparsity. With the threshold prediction method,
gradients pruning can be performed on our architecture with almost no
overhead. Experiments show that the gradients pruning algorithm could achieve
$3\times$ to $10\times$ sparsity improved with negligible accuracy loss.
Compared with the baseline, SparseTrain achieves about $2.7\times$ speedup and
$2.2\times$ energy efficiency improvement on average.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in _Proc. NIPS_ , 2012, pp. 1097–1105.
  * [2] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in _Proc. NIPS_ , 2015, pp. 91–99.
  * [3] Y. Sun, D. Liang, X. Wang, and X. Tang, “Deepid3: Face recognition with very deep neural networks,” _arXiv preprint:1502.00873_ , 2015.
  * [4] L. Bertinetto, J. Valmadre _et al._ , “Fully-convolutional siamese networks for object tracking,” in _Proc. ECCV_ , 2016, pp. 850–865.
  * [5] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , 2015.
  * [6] N. P. Jouppi, C. Young, N. Patil _et al._ , “In-datacenter performance analysis of a tensor processing unit,” in _Proc. ISCA_ , 2017, pp. 1–12.
  * [7] N. Jouppi, C. Young, N. Patil _et al._ , “Motivation for and evaluation of the first tensor processing unit,” in _Proc. MICRO_ , 2018, pp. 10–19.
  * [8] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” in _Proc. ISCA_ , 2016, pp. 367–379.
  * [9] A. Parashar, M. Rhu _et al._ , “SCNN: An accelerator for compressed-sparse convolutional neural networks,” in _Proc. ISCA_ , 2017, pp. 27–40.
  * [10] M. Imani, S. Gupta, Y. Kim, and T. Rosing, “FloatPIM: In-memory acceleration of deep neural network training with high precision,” in _Proc. ISCA_ , 2019, pp. 802–815.
  * [11] X. He, W. Lu, G. Yan, and X. Zhang, “Joint design of training and hardware towards efficient and accuracy-scalable neural network inference,” _JETCAS_ , vol. 8, no. 4, pp. 810–821, 2018.
  * [12] Y. Li, J. Park, M. Alian _et al._ , “A network-centric hardware/algorithm co-design to accelerate distributed training of deep neural networks,” in _Proc. MICRO_ , 2018, pp. 175–188.
  * [13] A. Ren, T. Zhang, S. Ye _et al._ , “ADMM-NN: An algorithm-hardware co-design framework of dnns using alternating direction methods of multipliers,” in _Proc. ASPLOS_ , 2019, pp. 925–938.
  * [14] W. Wen, C. Xu, C. Wu _et al._ , “Coordinating filters for faster deep neural networks,” in _Proc. ICCV_ , 2017, pp. 658–666.
  * [15] W. Wen, Y. He _et al._ , “Learning intrinsic sparse structures within long short-term memory,” _arXiv preprint:1709.05027_ , 2017.
  * [16] B. Chen, T. Medini, J. Farwell _et al._ , “SLIDE: In defense of smart algorithms over hardware acceleration for large-scale deep learning systems,” _arXiv preprint arXiv:1903.03129_ , 2019.
  * [17] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,” _arXiv preprint:1510.00149_ , 2015.
  * [18] S. Han, X. Liu, H. Mao _et al._ , “EIE: efficient inference engine on compressed deep neural network,” _ACM SIGARCH Computer Architecture News_ , vol. 44, no. 3, pp. 243–254, 2016.
  * [19] W. Wen, C. Xu, F. Yan _et al._ , “TernGrad: Ternary gradients to reduce communication in distributed deep learning,” in _Proc. NIPS_ , 2017, pp. 1509–1519.
  * [20] Z. He and D. Fan, “Simultaneously optimizing weight and quantizer of ternary neural network using truncated Gaussian approximation,” in _Proc. CVPR_ , 2019, pp. 11 438–11 446.
  * [21] “DNN training accelerator comparison,” http://www.ci-lab.net/research/dnn-training-accelerator-comparison/, 2019\.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. CVPR_ , 2016, pp. 770–778.
  * [23] X. Ye, J. Yang, P. Dai _et al._ , “Accelerating CNN training by pruning activation gradients,” _arXiv preprint:1908.00173_ , 2019.
  * [24] A. Shafaei, Y. Wang, X. Lin, and M. Pedram, “FinCACTI: Architectural analysis and modeling of caches with deeply-scaled FinFET devices,” in _Proc. ISVLSI_ , 2014, pp. 290–295.


# DyRRen: A Dynamic Retriever-Reranker-Generator Model for
Numerical Reasoning over Tabular and Textual Data

Xiao Li, Yin Zhu, Sichen Liu, Jiangzhou Ju, Yuzhong Qu, Gong Cheng Contact
Author

###### Abstract

Numerical reasoning over hybrid data containing tables and long texts has
recently received research attention from the AI community. To generate an
executable reasoning program consisting of math and table operations to answer
a question, state-of-the-art methods use a retriever-generator pipeline.
However, their retrieval results are static, while different generation steps
may rely on different sentences. To attend to the retrieved information that
is relevant to each generation step, in this paper, we propose DyRRen, an
extended retriever-reranker-generator framework where each generation step is
enhanced by a dynamic reranking of retrieved sentences. It outperforms
existing baselines on the FinQA dataset.

##  1 Introduction

Numerical reasoning has drawn much attention in machine reading comprehension
tasks. Previous datasets on numerical reasoning like DROP (Dua et al. 2019)
mainly focus on simple calculations and comparisons. Recently in this field,
more complicated arithmetic expression generation takes an important part,
especially in the context of financial reports. Recognized datasets like FinQA
(Chen et al. 2021b) and TAT-QA (Zhu et al. 2021) involve hybrid question
answering (QA), i.e., QA over information represented in heterogenous forms
including tables and texts.

To acquire an answer by numerical reasoning, a standard method is to generate
an arithmetic expression from structured tables and unstructured sentences.
For instance, Figure 1 shows an example in FinQA. The context includes a table
describing sales records in different years, and several sentences stating
relevant financial information. According to the question, one needs to
determine arithmetic operators $\mathtt{MINUS}$ and $\mathtt{DIVIDE}$, and
select their arguments from either tables or sentences to generate the correct
arithmetic expression and calculate the answer. In some cases, an argument of
an arithmetic expression (e.g., #0 in Figure 1) could be the result of some
former expression.

Figure 1: A question sampled from the FinQA dataset.

###  1.1 Existing Methods and Limitations

Intuitively, the generation of an arithmetic expression can be divided into
two modules: Retriever and Generator. The retriever filters the most relevant
information as the input to the generator which then predicts expressions
using operators predefined in a domain specific language (DSL) and numbers
extracted from the input. FinQANet (Chen et al. 2021b) is one such state-of-
the-art method. Its retriever converts a table into sentences. Then a BERT
(Devlin et al. 2019) binary classifier is applied to identify relevant
sentences, based on which an encoder-decoder model is implemented to generate
an arithmetic expression. Encoders can be BERT or RoBERTa (Liu et al. 2019)
which are both encoder-only models. The hidden layer output of an LSTM
(Hochreiter and Schmidhuber 1997) is used to predict expression tokens.

We observe the following limitations of FinQANet.

First, in FinQANet’s generator, retrieved sentences are concatenated into a
long sequence and inputted into the encoder, from which different numbers are
extracted at different generation steps. Existing models have difficulty in
locating the correct number from a long sequence. For example, when generating
the argument “7%” in Figure 1, the generator should pay exclusive attention to
the unique retrieved sentence $D_{2}$ which contains “7%”. Therefore, it would
be helpful to design a mechanism that can dynamically adjust the attention of
the generator at generation time.

Second, there is plenty of room to improve FinQANet’s retriever which is a
simple binary classifier. Indeed, it could not fully exploit the relationships
between question and context. For example, in the question in Figure 1, the
retriever should focus on particular entities such as “2015”, “2016”, and
“third-party sales”, which requires intensive interaction at the token level.
Moreover, it could not directly capture the differences between positive and
negative sentences.

###  1.2 Our Approach

We overcome the above limitations with our novel framework DyRRen, short for
Dynamic Retriever-Reranker-generator. Figure 2 sketches our framework. DyRRen
switches its attention to different sentences at different generation steps
via a novel dynamic reranking mechanism.

Figure 2: Overview of DyRRen.

#### Generator

In the generator, we focus on scoring the input tokens and looking for the
next token most likely to appear in the generated expression. This module is
implemented by an LSTM with the attention mechanism. To improve the accuracy
of the generator, we incorporate a reranker to dynamically pick relevant
sentences.

#### Reranker

At each step of generation, we score and rerank retrieved sentences in order
to find one that contains the target number. Specifically, we take advantage
of the relationships among the decoder output of the generator, formerly
generated expression, and retrieved sentences to dynamically rerank sentences
at each step to enhance the generator.

#### Retriever

Dense retrieval is used to replace the simple BERT binary classifier. To
improve retrieval performance, we implement meticulous interaction between
each question-sentence pair and employ a set of useful mechanisms such as
token-level late interaction. The loss function is also tailored to the
scenario in which the retrieval space contains multiple positive and negative
sentences.

To summarize, our contributions include

  * •

A novel retriever-reranker-generator framework for solving numerical reasoning
over tabular and textual data, especially using a reranker which can
dynamically locate target information at different generation steps, and

  * •

A thoughtfully designed retriever in which we strengthen token-level and
pairwise interaction.

###  1.3 Code

Our code is available on GitHub: https://github.com/nju-websoft/DyRRen.

###  1.4 Outline

We elaborate our approach in Section 2, present experiments in Section 3,
discuss related work in Section 4 and conclude our work in Section 5.

##  2 Approach

In the task of numerical reasoning over tabular and textual data, a question
$Q$ is given together with a structured table $T$ which consists of $r$ rows
$\\{T_{1},T_{2},\cdots,T_{r}\\}$ and a set of $n$ unstructured sentences
$\mathbb{D}=\\{D_{1},D_{2},\cdots,D_{n}\\}$. The goal is to generate an
arithmetic expression
$G=(\text{op}_{1}(\text{args}_{11},\text{args}_{12}),\text{op}_{2}(\text{args}_{21},\text{args}_{22}),...)$,
where $\text{op}_{i}$ is a binary operator such as $\mathtt{DIVIDE}$ defined
in a domain specific language (DSL) and $\text{args}_{ij}$ is either a
constant token, a memory token indicating the result of some previous
operation, or a span111It could be a number or a row header. For clarity, we
will refer to it as a numeric argument/token in the following. that appears in
$Q$, $T$, or $\mathbb{D}$.

We extend the existing retriever-generator framework to a retriever-ranker-
generator framework. We first use the retriever to retrieve the most relevant
sentences to the question (Section 2.1), and then use the retrieved sentences
to generate arithmetic expressions (Section 2.2) which is enhanced by a
dynamic reranker of sentences (Section 2.3).

Figure 3: Implementation of DyRRen.

###  2.1 Retriever

Given the question $Q$, table $T$ and sentences $\mathbb{D}$, our retriever
aims to retrieve supporting facts from $T$ and $\mathbb{D}$ to answer $Q$. The
retriever is shown bottom left in Figure 3.

#### Encoder

In the retriever, we process tabular data in the same way as FinQANet (Chen et
al. 2021b), converting each row of the tabular data into a sentence using
templates.Specifically, we convert each cell into “the column name of row name
is cell value ;” and then we concatenate the converted cell sentences of a row
into a row sentence. For example, the second row of the table in Figure 1 will
be converted to “the third-party sales of 2016 is $ 1802; the third-party
sales of 2015 is $ 1882; …”.

Recall that the retriever of FinQANet uses BERT encoder to obtain the
representations of sentences (including sentences converted from tables). Then
it uses a binary classifier to determine whether each sentence is relevant.
Different from FinQANet’s retriever, we use dense retrieval to improve the
performance of the retriever.

Specifically, by using templates to convert each row of $T$ into a sentence,
we obtain a set of sentences

$\mathbb{D}^{\prime}=\\{D_{1},D_{2},\cdots,D_{n}\\}\cup\\{D_{n+1},D_{n+2},\cdots,D_{n+r}\\}$
(1)

where $\\{D_{n+1},D_{n+2},\cdots,D_{n+r}\\}$ are converted from rows.

We use BERT to encode question $Q$ and each sentence $D\in\mathbb{D}^{\prime}$
into $\mathbf{H}^{Q}$ and $\mathbf{H}^{D}$ respectively. More details about
our encoder will be introduced in Section 2.2. Inspired by ColBERT (Khattab
and Zaharia 2020), we also make the following optimizations to the encoder.

  * •

Prepend the special token [Q] and [D] to the question tokens and sentence
tokens respectively. This makes it easier for BERT encoder to identify the
type of text.

  * •

We pad question tokens with BERT’s special [mask] token up to a fixed length.
The token [mask] was used to mask the words to be predicted while pre-
training. It allows BERT encoder to generate representations that match the
given question’s semantics by filling [mask] tokens to optimize the question
representation.

  * •

Concatenate the question tokens after sentence tokens. It allows the question
and sentences to interact with each other inside the BERT encoder.

#### Relevance Scorer

Using $\mathbf{H}^{Q}$ and $\mathbf{H}^{D}$, the retriever computes the
relevance score of $D$ to $Q$ through a late interaction mechanism to allow
token-level interaction by calculating token similarity as in ColBERT:

$\mathtt{sim}(\mathbf{H}^{Q},\mathbf{H}^{D})=\sum_{\mathbf{H}^{Q}_{i}\in\mathbf{H}^{Q}}\max_{\mathbf{H}^{D}_{j}\in\mathbf{H}^{D}}\frac{\mathbf{H}^{Q}_{i}\cdot{\mathbf{H}^{D}_{j}}^{T}}{\|\mathbf{H}^{Q}_{i}\|\|{\mathbf{H}^{D}_{j}}^{T}\|}$
(2)

where $\|\cdot\|$ represents vector modulus operator. We take top-$k$
sentences with the highest scores as the retrieval results.

Our retriever adopts RankNet pairwise loss (Burges 2010; Zhan et al. 2021).
Given the question $Q$, let $\mathbb{D}^{+}$ and $\mathbb{D}^{-}$ be the sets
of all positive sentences and all negative sentences respectively. Our loss
function is formulated as follows:

$\displaystyle L(Q,\mathbb{D}^{+},\mathbb{D}^{-})$ $\displaystyle=$ (3)
$\displaystyle\sum_{D^{+}_{i}\in\mathbb{D}^{+}}$
$\displaystyle\sum_{D^{-}_{j}\in\mathbb{D}^{-}}\log(1+\frac{\exp(\mathtt{sim}(\mathbf{H}^{Q},\mathbf{H}^{D^{-}_{j}}))}{\exp(\mathtt{sim}(\mathbf{H}^{Q},{\mathbf{H}^{D^{+}_{i}}}))}).$

With this loss function, we compare the scores of all positives and all
negatives pairwisely to allow pairwise interaction between sentences. It aims
to rank all positives ahead of all negatives in the retrieval results.

###  2.2 Generator

Our generator extends FinQANet’s generator by incorporating a dynamic
reranker.

Given the question $Q$ and the retrieved sentences
$\mathbb{D}^{\text{R}}=\\{D^{\text{R}}_{1},\cdots,D^{\text{R}}_{k}\\}\subseteq\mathbb{D}^{\prime}$,
our generator aims to generate the arithmetic expression that the question
implies. Figure 3 gives an overview of our generator (top left and center).

#### Encoder

For question $Q=(u^{Q}_{1},u^{Q}_{2},\cdots,u^{Q}_{\lvert Q\rvert})$, we
directly feed it into BERT to obtain its representation:

$\displaystyle\mathbf{H}^{Q}$
$\displaystyle=[\mathbf{h}^{Q}_{1};\mathbf{h}^{Q}_{2};\cdots;\mathbf{h}^{Q}_{\lvert
Q\rvert+2}]$ (4)
$\displaystyle=\mathtt{BERT}(\texttt{[CLS]},u^{Q}_{1},u^{Q}_{2},\cdots,u^{Q}_{\lvert
Q\rvert},\texttt{[SEP]})$

and for each retrieved sentence
$D^{\text{R}}_{i}=(u^{D^{\text{R}}_{i}}_{1},u^{D^{\text{R}}_{i}}_{2},\cdots,u^{D^{\text{R}}_{i}}_{\lvert
D^{\text{R}}_{i}\rvert})$, in order to let it fully interact with $Q$ when
encoding, we concatenate it with $Q$:

$\displaystyle\mathbf{H}^{D^{\text{R}}_{i}}$
$\displaystyle=[\mathbf{h}^{D^{\text{R}}_{i}}_{1};\mathbf{h}^{D^{\text{R}}_{i}}_{2};\cdots;\mathbf{h}^{D^{\text{R}}_{i}}_{\lvert{D^{\text{R}}_{i}}\rvert+\lvert
Q\rvert+3}]$ (5)
$\displaystyle=\mathtt{BERT}(\texttt{[CLS]},u^{D^{\text{R}}_{i}}_{1},u^{D^{\text{R}}_{i}}_{2},\cdots,u^{D^{\text{R}}_{i}}_{\lvert
D^{\text{R}}_{i}\rvert},$
$\displaystyle\indent\indent\indent\indent\texttt{[SEP]},u^{Q}_{1},u^{Q}_{2},\cdots,u^{Q}_{\lvert
Q\rvert},\texttt{[SEP]}).$

We then concatenate the representation of the question and the representations
of all the retrieved sentences:

$\displaystyle\mathbf{H}=\mathbf{H}^{Q}\|\mathbf{H}^{D^{\text{R}}_{1}}\|\cdots\|\mathbf{H}^{D^{\text{R}}_{k}}$
(6)

where $\|$ represents concatenation.

DSL contains predefined binary operators with numeric arguments
($\mathtt{ADD}$, $\mathtt{DIVIDE}$, etc.), operators with table row headers as
arguments ($\mathtt{TABLE\\_SUM}$, $\mathtt{TABLE\\_MAX}$, etc.) which operate
on a row, constant tokens ($\mathtt{CONST\\_1}$, $\mathtt{CONST\\_100}$,
$\mathtt{CONST\\_1000}$, etc.) and memory tokens representing the results of
previous operations ($\mathtt{\\#0}$, $\mathtt{\\#1}$, etc.). For all these
tokens
$U^{\text{DSL}}=\\{u^{\text{DSL}}_{1},u^{\text{DSL}}_{2},\cdots,u^{\text{DSL}}_{m}\\}$
in the DSL, we randomly initialise their representations
$\mathbf{H}^{\text{DSL}}=[\mathbf{h}^{\text{DSL}}_{1};\mathbf{h}^{\text{DSL}}_{2};\cdots;\mathbf{h}^{\text{DSL}}_{m}]$
in training.

#### Decoder

We begin by introducing the attention mechanism, which will be used frequently
in the following:

$\displaystyle\mathtt{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathtt{Softmax}(\mathbf{W}^{Q}\mathbf{Q}\mathbf{K}^{T})\mathbf{V}$
(7)

where $\mathbf{W}^{Q}$ is a matrix of learnable parameters.

At step $t$ of generation, the token representation sequence that has been
generated is
$\mathbf{Y}_{t}=[\mathbf{y}_{1};\mathbf{y}_{2};\cdots;\mathbf{y}_{t-1}]$ and
the decoder output representation, denoted by $\hat{\mathbf{y}}_{t}$, is
obtained by a LSTM network:

$\mathbf{\hat{y}}_{t}=\mathtt{LSTM}(\mathbf{y}_{t-1}).$ (8)

We randomly initialize $\mathbf{y}_{0}$ in training. Our goal is to determine
which token in the question, sentences, or DSL should be selected. In the
following, unless otherwise stated, all variables are for step $t$ and we may
omit the subscript $t$.

We calculate the attention representation of $\hat{\mathbf{y}}_{t}$ over
$\mathbf{Y}_{t}$ and $\mathbf{H}^{\text{DSL}}\|\mathbf{H}$ to get a decoder
history representation and a sequence representation, respectively:

$\displaystyle\mathbf{a}^{\text{Y}}=\mathtt{Attention}(\mathbf{Q}=\mathbf{\hat{y}}_{t},\mathbf{K}$
$\displaystyle=\mathbf{Y}_{t},\mathbf{V}=\mathbf{Y}_{t})$ (9)
$\displaystyle\mathbf{a}^{\text{seq}}=\mathtt{Attention}(\mathbf{Q}=\mathbf{\hat{y}}_{t},\mathbf{K}$
$\displaystyle=\mathbf{H}^{\text{DSL}}\|\mathbf{H},$ $\displaystyle\mathbf{V}$
$\displaystyle=\mathbf{H}^{\text{DSL}}\|\mathbf{H}).$

Note that $\mathbf{\hat{y}}_{t}$, as the query of attention, is not included
in $\mathbf{a}^{\text{Y}}$ and $\mathbf{a}^{\text{seq}}$, so we fuse them to
get a context representation:

$\displaystyle\mathbf{h}^{\text{c}}=\mathtt{Layernorm}(\mathbf{W}^{\text{c}}(\mathbf{a}^{\text{Y}}\|\mathbf{a}^{\text{seq}}\|\mathbf{\hat{y}}_{t}))$
(10)

where $\mathbf{W}^{\text{c}}$ is a matrix of learnable parameters. For now, we
have obtained the representations of all tokens in DSL, i.e.,
$\mathbf{H}^{\text{DSL}}$, and all tokens in $Q$ and $\mathbb{D}^{\text{R}}$,
i.e., $\mathbf{H}$. We fuse them and compute the scores of all tokens by
calculating the inner product of token representation and context
representation:

$\displaystyle\mathbf{H}^{\text{S}}$
$\displaystyle=\mathbf{W}^{\text{S}}((\mathbf{H}^{\text{DSL}}\|\mathbf{H})\|((\mathbf{H}^{\text{DSL}}\|\mathbf{H})\circ\mathbf{a}^{\text{seq}}))$
(11) $\displaystyle\mathbf{s}^{\text{ctx}}$
$\displaystyle=\mathbf{H}^{\text{S}}\mathbf{h}^{\text{c}}$

in which $\circ$ is element-wise product and $\mathbf{W}^{\text{S}}$ is a
matrix of learnable parameters. Note that in Equation (11) we duplicate the
vector $\mathbf{a}^{\text{seq}}$ to form a matrix to be concatenated with
other matrices, which for clarity is omitted in the equation.

We get the reranker scores $\mathbf{s}^{\text{r}}$ from our reranker which
will be described in detail in Section 2.3. The final scores of all tokens are
represented by the element-wise product of context scores and reranker scores:

$\mathbf{s}=\mathbf{s}^{\text{ctx}}\circ\mathbf{s}^{\text{r}}.$ (12)

We optimize the cross-entropy loss during training:

$\displaystyle\mathcal{L}=-\log\frac{\exp(s^{\text{gold}})}{\sum_{tok\in
U^{\text{DSL}}\cup U^{\text{num}}}\exp(s_{tok})}$ (13)

where $U^{\text{num}}$ is the set of all numeric tokens, $s_{tok}$ is the
final score of a token computed by Equation (12), and $s^{\text{gold}}$ is the
final score of the ground truth token.

We finally select the predicted token by $\mathbf{s}$:

$\mathbf{y}_{t}=\mathbf{H}^{\text{S}}_{\arg\max\mathbf{s}}$ (14)

i.e., the representation of the ($\arg\max\mathbf{s}$)-th token in
$\mathbf{H}^{\text{S}}$. We use teacher forcing mechanism during training,
i.e., we use the representation of the ground truth token as $\mathbf{y}_{t}$.

###  2.3 Reranker

In this module, we aim to find the sentence that is of interest to the
generator at step $t$, i.e., the one in which the next numeric argument of the
arithmetic expression is located.

#### Reranker Space

The reranker space, i.e., the objects that need to be reranked, consists of
representations of the question and retrieved sentences.

We construct the reranker space as follows. We use dual attention (Liu et al.
2020) to enhance the interaction between the question representation
$\mathbf{H}^{Q}$ and each sentence representation
$\mathbf{H}^{D_{i}^{\text{R}}}$ computed by Equation (5):

$\displaystyle\hat{\mathbf{H}}^{Q_{i}},\hat{\mathbf{H}}^{D_{i}^{\text{R}}}=\mathtt{DualAttention}(\mathbf{H}^{Q}$
$\displaystyle,\mathbf{H}^{D_{i}^{\text{R}}}).$ (15)

$\hat{\mathbf{H}}^{Q_{i}}$ is the token sequence representations of $Q$ after
interacting with $D_{i}^{\text{R}}\in\mathbb{D}^{\text{R}}$. Then mean-pooling
in token dimension is applied to obtain the representations of $Q$ and
$D_{i}^{\text{R}}$:

$\displaystyle\mathbf{h}^{Q_{i}}$
$\displaystyle=\mathtt{MeanPooling}(\hat{\mathbf{H}}^{Q_{i}})$ (16)
$\displaystyle\mathbf{h}^{D_{i}^{\text{R}}}$
$\displaystyle=\mathtt{MeanPooling}(\hat{\mathbf{H}}^{D_{i}^{\text{R}}})$

We combine the $k$ representations of $Q$ over $k$ sentences to obtain a
representation of $Q$ by attention mechanism:

$\displaystyle\mathbf{h}^{Q}=\mathtt{Attention}(\mathbf{Q}=\hat{\mathbf{y}}_{t},\mathbf{K}$
$\displaystyle=[\mathbf{h}^{Q_{1}};\mathbf{h}^{Q_{2}};\cdots,\mathbf{h}^{Q_{k}}],$
(17) $\displaystyle\mathbf{V}$
$\displaystyle=[\mathbf{h}^{Q_{1}};\mathbf{h}^{Q_{2}};\cdots,\mathbf{h}^{Q_{k}}]).$

The reranker space contains
$\mathbf{h}^{Q},\mathbf{h}^{D_{1}^{\text{R}}},\ldots,\mathbf{h}^{D_{k}^{\text{R}}}$.

#### Reranker Query

The query used to rerank is dynamically generated, not only in relation to the
expression that has been generated, i.e., $\mathbf{Y}_{t}$, but also in
relation to the source sentences from which the tokens in the expression are
extracted. We first locate the source sentence of each token in the generated
expression. We take the representation of its source sentence if the generated
token is numeric, or keep the representation of the token if it is from the
DSL, formally:

$\mathtt{source}(\mathbf{y}_{i})=\\\ \left\\{\begin{aligned}
\mathbf{y}_{i},\indent&\text{if }\mathbf{y}_{i}\in\mathbf{H}^{\text{DSL}},\\\
\mathbf{h}^{D},\indent&\text{if }\mathbf{y}_{i}\text{ comes from
}D\in\mathbb{D}^{\text{QR}}\end{aligned}\right.$ (18)

where $\mathbb{D}^{\text{QR}}=\\{Q\\}\cup\mathbb{D}^{\text{R}}$ and
$\mathbf{h}^{D}$ refers to $\mathbf{h}^{Q}$ computed by Equation (17) or
$\mathbf{h}^{D^{\text{R}}_{i}}$ computed by Equation (16). We obtain the
source sentence representations of $\mathbf{Y}_{t}$ by

$\displaystyle\mathbf{H}^{\text{src}}_{t}=\mathbf{H}^{\text{src}}_{t-1};\mathtt{source}(\mathbf{y}_{t-1})$
(19)

which refers to appending the source sentence representation of
$\mathbf{y}_{t-1}$ to the source sentence representations of
$\mathbf{Y}_{t-1}$ which were calculated at the previous step.

We use the source sentence representations of generated tokens to construct a
query to rerank all the sentences at the current step. We compute the
attention representation of $\hat{\mathbf{y}}_{t}$ over
$\mathbf{H}^{\text{src}}_{t}$ to obtain the reranker query representation
$\mathbf{Q}^{\text{s}}$:

$\displaystyle\mathbf{Q}^{\text{s}}=\mathtt{Attention}(\mathbf{Q}=\hat{\mathbf{y}}_{t},\mathbf{K}=\mathbf{H}^{\text{src}}_{t},\mathbf{V}=\mathbf{H}^{\text{src}}_{t}).$
(20)

#### Reranker Scorer

With the reranker query $\mathbf{Q}^{\text{s}}$ and the reranker space
$\\{\mathbf{h}^{Q},\mathbf{h}^{D_{1}},\cdots,\mathbf{h}^{D_{k}}\\}$, we re-
compute the relevance score of each sentence by:

$\displaystyle s^{Q}$
$\displaystyle=\mathtt{Linear}(\mathtt{Tanh}(\mathtt{Linear}([\mathbf{Q}^{\text{s}}\|\mathbf{h}^{Q}])))$
(21) $\displaystyle s^{D_{i}^{\text{R}}}$
$\displaystyle=\mathtt{Linear}(\mathtt{Tanh}(\mathtt{Linear}([\mathbf{Q}^{\text{s}}\|\mathbf{h}^{D_{i}^{\text{R}}}]))),i\in\\{1,\cdots,k\\}$

The reranker scores in Equation (12) are given by

$\mathbf{s}^{\text{r}}=\mathtt{Expand}(\mathtt{Softmax}([s^{Q},s^{D_{1}^{\text{R}}},\cdots,s^{D_{k}^{\text{R}}}])).$
(22)

Note that $s^{Q},s^{D_{1}^{\text{R}}},\cdots,s^{D_{k}^{\text{R}}}$ are for
scoring sentences while we need to obtain reranker scores for tokens, so an
$\mathtt{Expand}$ operation is applied which assigns the score of each
sentence to every token in that sentence. The scores of tokens in the DSL are
fixed to 1.0.

##  3 Experiments

###  3.1 Datasets

To the best of our knowledge, there were two datasets requiring generating
arithmetic expressions for numerical reasoning over tabular and textual data:
FinQA (Chen et al. 2021b) and MultiHiertt (Zhao et al. 2022). We used FinQA in
our experiments. We did not use MultiHiertt because it was new and yet to be
further polished after contacting its authors.

FinQA contains 8,281 financial questions which were collected from financial
reports of S&P 500 companies. Questions were divided into 6,251 (75%) for
training, 883 (10%) for development, and 1,147 (15%) for testing. Each
question contains a table where the average number of rows is 6.36. The
average number of sentences accompanying a question is 24.32. The total number
of tokens in the table and sentences accompanying a question averages 687.53,
with a maximum value of 2,679.

###  3.2 Implementation Details

We experimented on NVIDIA V100 (32GB) GPUs and Ubuntu 18.04. Our
implementations were based on PyTorch 1.11. We selected models on the
development set.

For the encoder in our retriever, we used BERT-base-uncased with $\text{hidden
layer}=12$, $\text{hidden units}=768$, and $\text{attention heads}=12$. We set
$\text{epoch}=8$ and $\text{seed}=8$. We used the Adam optimizer with
$\text{learning rate}=2e-5$ selected from $\\{1.5e-5,2e-5\\}$ with
$\text{batch size}=16$. We set $k=3$ and $\text{max sequence length}=256$.

For the encoder in our generator, we used two pretrained models: BERT-base-
uncased and RoBERTa-large. RoBERTa-large is with $\text{hidden layer}=24$,
$\text{hidden units}=1\text{,}024$, and $\text{attention heads}=16$. In our
reranker and generator, we set $\text{epoch}=300$ and $\text{seed}=8$. We used
the Adam optimizer with $\text{learning rate}=2e-5$ for BERT-base-uncased and
$\text{learning rate}=1.5e-5$ for RoBERTa-large, both selected from
$\\{7e-6,1.5e-5,2e-5\\}$. We set $\text{batch size}=16$ for BERT-base-uncased
and $\text{batch size}=24$ for RoBERTa-large, both selected from
$\\{16,24,32\\}$. We set $\text{max sequence length}=256$.

###  3.3 Baselines

We compared our proposed model DyRRen with known methods and results reported
in the literature.

We mainly compared DyRRen with FinQANet, which is a state-of-the-art
retriever-generator model specifically designed for numerical reasoning over
tabular and textual data. We compared with FinQANet’s versions on FinBERT
(Araci 2019), BERT-base-uncased, and RoBERTa-large. Chen et al. (2021b)
provided the results of all the three versions on FinQA.

Longformer (Beltagy, Peters, and Cohan 2020) is specifically designed to
handle long documents, and was used in our experiments to verify the necessity
to include a retriever in our framework. NeRd (Chen et al. 2020c) is an
expression generator based on a pointer network and has achieved competitive
results on MathQA (Amini et al. 2019) and DROP (Dua et al. 2019). We used the
results of Longformer and NeRd reported in Chen et al. (2021b).

###  3.4 Evaluation Metrics

Following the literature, we measured program accuracy (PA), i.e., proportion
of correctly generated arithmetic expressions, and execution accuracy (EA),
i.e., proportion of correct final answers. Note that sometimes a generated
expression is different from the corresponding ground truth but their answers
are equivalent, so execution accuracy is always at least as high as program
accuracy.

###  3.5 Main Results

| Dev | Test
---|---|---
Methods | EA | PA | EA | PA
Longformer${}_{\texttt{base}}$ | 23.83 | 22.56 | 21.90 | 20.48
NeRd | 47.53 | 45.37 | 48.57 | 46.76
FinQANet${}_{\texttt{FinBERT}}$ | 46.64 | 44.11 | 50.10 | 47.52
FinQANet${}_{\texttt{BERT}}$ | 49.91 | 47.15 | 50.00 | 48.00
FinQANet${}_{\texttt{RoBERTa}}$ | 61.22 | 58.05 | 61.24 | 58.86
DyRRen${}_{\texttt{BERT}}$ | 61.16 | 58.32 | 59.37 | 57.54
DyRRen${}_{\texttt{RoBERTa}}$ | 66.82 | 63.87 | 63.30 | 61.29
Human Expert | – | – | 91.16 | 87.49
General Crowd | – | – | 50.68 | 48.17
Table 1: Comparison with baselines on FinQA. BERT refers to BERT-base-uncased,
RoBERTa refers to RoBERTa-large and FinBERT is a BERT-like model pretrained in
the financial domain.

On the test set, as shown in Table 1, DyRRen outperformed all the baselines by
at least 9.37% of EA and 9.54% of PA with BERT and at least 2.06% of EA and
2.43% of PA with RoBERTa. The differences were statistically significant under
$p<0.01$. Besides, both DyRRen${}_{\texttt{BERT}}$ and
DyRRen${}_{\texttt{RoBERTa}}$ exceeded general crowd (50.68% of EA and 48.17%
of PA), although they were still not comparable with human expert (91.16% of
EA and 87.49% of PA).

It is notable that DyRRen${}_{\texttt{BERT}}$ achieved similar performance to
FinQANet${}_{\texttt{RoBERTa}}$, while the number of parameters in
FinQANet${}_{\texttt{RoBERTa}}$ ($\sim$380M) is even 2.7 times that of
DyRRen${}_{\texttt{BERT}}$ ($\sim$139M).

###  3.6 Case Study

Figure 4: Case study sampled from FinQA.

As shown in Figure 4, we sampled a question from FinQA for which the
retrievers of both DyRRen and FinQANet (BERT versions) successfully retrieved
the sentences relevant to the question, i.e., $D_{2}^{\text{R}}$ and
$D_{3}^{\text{R}}$. It can be seen that at the beginning, both DyRRen and
FinQANet succeeded in generating the correct operator DIVIDE and extracting
the first argument “259” from $D^{\text{R}}_{2}$, and our reranker in DyRRen
exhibited the highest confidence to $D^{\text{R}}_{2}$ (0.658). When
generating the second argument, our reranker assigned a higher score (0.745)
to $D^{\text{R}}_{3}$ containing the correct argument “412”, successfully
switching the generator’s attention to the correct sentence, while FinQANet
still stayed in $D^{\text{R}}_{2}$ and selects “210”, thus failing to answer
this question.

###  3.7 Ablation Study

We compared DyRRen with two variants:

  * •

DyRRen${}_{\texttt{no-reranker}}$, which removes the reranker module from our
retrieval-ranker-generator framework.

  * •

DyRRen${}_{\texttt{vanilla-retriever}}$, which replaces our retriever with
FinQANet’s retriever to verify the effectiveness of the optimizations we have
made to our retriever.

On FinQA, as shown in Table 2, DyRRen${}_{\texttt{no-reranker}}$ was 1.48%
lower in EA and 2.00% lower in PA than DyRRen on the basis of BERT on the test
set, demonstrating the usefulness of our reranker. The performance of
DyRRen${}_{\texttt{no-reranker}}$ (RoBERTa version) also noticeably dropped on
the dev set (by 0.98% in EA and 1.02% in PA), while the differences on the
test set were neglectable. We thought it was the more powerful RoBERTa which
counterbalanced some gain from our reranker.

The performance of DyRRen${}_{\texttt{vanilla-retriever}}$ decreased on both
BERT and RoBERTa based experiments. For example, DyRRen${}_{\texttt{vanilla-
retriever}}$ (BERT) was 3.22% lower in EA and 3.75% lower in PA than DyRRen on
the test set, demonstrating the effectiveness of our retriever.

We also tried to incorporate models pre-trained on tables such as TaPas
(Herzig et al. 2020) into our encoder, but did not obtain better results.

| Dev | Test
---|---|---
Methods | EA | PA | EA | PA
BERT |  |  |  |
DyRRen | 61.16 | 58.32 | 59.37 | 57.54
DyRRen${}_{\texttt{no-reranker}}$ | 60.82 | 58.66 | 57.89 | 55.54
DyRRen${}_{\texttt{vanilla-retriever}}$ | 58.78 | 56.17 | 56.15 | 53.79
RoBERTa |  |  |  |
DyRRen | 66.82 | 63.87 | 63.30 | 61.29
DyRRen${}_{\texttt{no-reranker}}$ | 65.80 | 62.85 | 62.86 | 61.46
DyRRen${}_{\texttt{vanilla-retriever}}$ | 62.85 | 60.48 | 60.33 | 57.89
Table 2: Ablation study on FinQA.

###  3.8 Fine-Grained Results

We conducted fine-grained comparison between DyRRen${}_{\texttt{BERT}}$ and
FinQANet${}_{\texttt{BERT}}$, as shown in Table 3.

#### Questions categorized by question type

Our model made notable improvements over FinQANet on all the three types of
questions. We also observed that both models performed the best on table-only
questions, i.e., those that can be answered only based on tables. This may be
due to the fact that row sentences have good structure as they are converted
by using templates, so they are relatively easy to be understood. Both models
performed the worst on table-sentence questions, indicating that there is
still room for improving hybrid data understanding.

#### Questions categorized by expression length

Our reranker helped our model focus on the unique supporting sentence for the
current step rather than on all the retrieved sentences, and it was
consistently superior on questions of all expression lengths. We also found
that both models performed poorly when expression length exceeded two. These
“long-distance reasoning” questions are very challenging for existing methods
and await further study.

| Dev | Test
---|---|---
| EA | PA | EA | PA
DyRRen${}_{\texttt{BERT}}$
Question type |  |  |  |
table-only | 72.51 | 69.37 | 68.98 | 66.71
sentence-only | 46.46 | 44.44 | 49.47 | 48.76
table-sentence | 38.46 | 35.66 | 34.18 | 32.28
Expression length |  |  |  |
$1$ | 66.16 | 63.67 | 64.37 | 63.00
$2$ | 60.98 | 57.49 | 57.46 | 54.77
$>2$ | 26.03 | 23.29 | 29.76 | 28.57
FinQANet${}_{\texttt{BERT}}$
Question type |  |  |  |
table-only | 61.62 | 58.86 | 58.22 | 55.67
sentence-only | 37.37 | 35.86 | 40.64 | 39.93
table-sentence | 28.67 | 27.27 | 31.01 | 30.38
Expression length |  |  |  |
$1$ | 52.77 | 51.05 | 52.14 | 50.76
$2$ | 54.01 | 50.52 | 53.55 | 51.10
$>2$ | 24.66 | 23.29 | 17.86 | 15.48
Table 3: Fine-grained comparison with FinQANet on FinQA by question type and
expression length.

###  3.9 Error Analysis

We randomly sampled 50 questions for which our model failed to generate the
correct arithmetic expression. The errors caused by our retriever constituted
about 20% of all errors, e.g., the retriever failed to retrieve all the
supporting sentences. About 14% of the errors were caused by the reranker
misleading the generator, i.e., the reranker assigned improper scores to
sentences in some generation step. The rest of the errors (66%) were due to
the generator. There were many reasons for these generation errors, one of the
main ones being that the generator failed to actively switch attention between
sentences, and the failure was too strong to be counterbalanced by our
reranker. Other reasons include incorrect prediction of operators, or the
generator became lost on questions that require multi-step numerical reasoning
or require external financial knowledge.

##  4 Related Work

###  4.1 Question Answering over Hybrid Data

QA that requires reasoning over a hybrid context, i.e., tabular and textual
data, is a task that has emerged in the last two years. Two of the first
datasets focusing on this task are HybridQA (Chen et al. 2020b) and TSQA (Li,
Sun, and Cheng 2021). The former is an extractive QA dataset focusing more on
multi-hop reasoning while the latter is a multiple-choice QA dataset more
focusing on numerical reasoning. Eisenschlos et al. (2021) proposes a
Transformer-based model with row-wise and column-wise attention and (Kumar et
al. 2021) applies multiple-instance training to solve HybridQA. Li, Sun, and
Cheng (2021) designs a number of templates containing numerical operations to
convert tables to sentences, and then uses a knowledge-injected pretrained
language model to solve TSQA.

Following these efforts, a growing number of more challenging hybrid QA
datasets have emerged. OTT-QA (Chen et al. 2021a) and NQ-Tables (Herzig et al.
2021) place hybrid QA in an open-domain setting where tables are not given but
need to be retrieved from a table corpus. Herzig et al. (2021) solves this
problem by dense retrieval. Later, it was found that tables in the financial
domain carry a lot of numerical content and are suitable for QA tasks that
require numerical reasoning over tabular and textual data. TAT-QA (Zhu et al.
2021), FinQA (Chen et al. 2021b), and MultiHiertt (Zhao et al. 2022) are such
datasets containing QA pairs from financial reports. TAT-QA requires the
answer to a question to be extracted or calculated from given sentences and
tables. FinQA and MultiHiertt require the generation of an arithmetic
expression, making the question answering process more interpretable.

So far, FinQANet (Chen et al. 2021b), which adopts a retrieval-generator
architecture, remains state of the art for solving hybrid QA that requires the
generation of arithmetic expressions. The major architectural difference
between FinQANet and our DyRRen is that we fuse the idea of reranking into
each generation step, i.e., our dynamic reranker, to improve the effectiveness
of our generator.

###  4.2 Numerical Reasoning in QA

While current pretrained language models (Lin et al. 2020) could not perform
well on numerical reasoning, there is a growing interest in numerical
reasoning based QA. DROP (Dua et al. 2019) requires mathematical operations
such as adding, subtracting, counting, and sorting. Later work shows that it
can be solved by semantic parsing (Gupta et al. 2020), graph-base method (Chen
et al. 2020a), neural module network (Zhou et al. 2022), or enhancing number-
related context understanding (Kim et al. 2022).

Math word problem (MWP) (Hosseini et al. 2014; Huang et al. 2016; Amini et al.
2019) is a fundamental and challenging task which has been introduced for
years. The task is to, given a short textual question, calculate the answer by
generating an arithmetic expression. There are two major differences between
MWP and the task considered in this paper: 1) MWP contains only a short
textual question without tables and supporting sentences, so solving MWPs does
not require to handle tabular data nor the retrieval problem. 2) Questions in
MWPs are described in a controlled manner, exhibiting strong regularity so
some of the early work could use statistical learning (Hosseini et al. 2014;
Kushman et al. 2014) or rule/template-based methods (Shi et al. 2015; Wang et
al. 2019) to achieve fairly good results. Huang et al. (2021) proposes a
recall learning framework for solving MWPs, a framework that recalls problems
with similar expression structures, benefiting from the regularity of MWPs.
There are also graph/tree based (Wu, Zhang, and Wei 2021; Jie, Li, and Lu
2022) or multiple-task learning (Qin et al. 2021; Shen et al. 2021) methods.
However, due to the above-mentioned differences between two tasks, these
solutions to MWPs are unlikely to obtain promising results on our task.

##  5 Conclusion

We present DyRRen, a dynamic retriever-reranker-generator model for numerical
reasoning over tabular and textual data. We believe that the idea of
incorporating a reranker to enhance generation has the potential to be
applicable to other generation tasks where the input contains multiple
sentences, including machine translation, machine writing, summarization,
etc., which we will explore as future work.

Our error analysis also reveals some limitations of our model. Our reranker is
able to help correct the selection of numerical arguments of operators, but is
unable to handle incorrect prediction of operators. We will extend our
reranker to handle operators in future work. We are also ready to experiment
on more datasets to evaluate the generalizability of our model, for instance,
when the emerging MultiHiertt dataset becomes stable.

## Acknowledgments

This work was supported in part by the NSFC (62072224) and in part by the CCF-
Baidu Open Fund.

## References

  * Amini et al. (2019) Amini, A.; Gabriel, S.; Lin, S.; Koncel-Kedziorski, R.; Choi, Y.; and Hajishirzi, H. 2019.  MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 2357–2367.
  * Araci (2019) Araci, D. 2019.  FinBERT: Financial Sentiment Analysis with Pre-trained Language Models.  _CoRR_ , abs/1908.10063.
  * Beltagy, Peters, and Cohan (2020) Beltagy, I.; Peters, M. E.; and Cohan, A. 2020.  Longformer: The Long-Document Transformer.  _CoRR_ , abs/2004.05150.
  * Burges (2010) Burges, C. J. 2010.  From RankNet to LambdaRank to LambdaMART: An Overview.  Technical Report MSR-TR-2010-82, Microsoft Research.
  * Chen et al. (2020a) Chen, K.; Xu, W.; Cheng, X.; Xiaochuan, Z.; Zhang, Y.; Song, L.; Wang, T.; Qi, Y.; and Chu, W. 2020a.  Question Directed Graph Attention Network for Numerical Reasoning over Text.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , 6759–6768.
  * Chen et al. (2021a) Chen, W.; Chang, M.; Schlinger, E.; Wang, W. Y.; and Cohen, W. W. 2021a.  Open Question Answering over Tables and Text.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
  * Chen et al. (2020b) Chen, W.; Zha, H.; Chen, Z.; Xiong, W.; Wang, H.; and Wang, W. Y. 2020b.  HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020, 1026–1036.
  * Chen et al. (2020c) Chen, X.; Liang, C.; Yu, A. W.; Zhou, D.; Song, D.; and Le, Q. V. 2020c.  Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Chen et al. (2021b) Chen, Z.; Chen, W.; Smiley, C.; Shah, S.; Borova, I.; Langdon, D.; Moussa, R.; Beane, M.; Huang, T.; Routledge, B. R.; and Wang, W. Y. 2021b.  FinQA: A Dataset of Numerical Reasoning over Financial Data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , 3697–3711.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * Dua et al. (2019) Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 2368–2378.
  * Eisenschlos et al. (2021) Eisenschlos, J.; Gor, M.; Müller, T.; and Cohen, W. W. 2021.  MATE: Multi-view Attention for Table Transformer Efficiency.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , 7606–7619.
  * Gupta et al. (2020) Gupta, N.; Lin, K.; Roth, D.; Singh, S.; and Gardner, M. 2020.  Neural Module Networks for Reasoning over Text.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Herzig et al. (2021) Herzig, J.; Müller, T.; Krichene, S.; and Eisenschlos, J. 2021.  Open Domain Question Answering over Tables via Dense Retrieval.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , 512–519.
  * Herzig et al. (2020) Herzig, J.; Nowak, P. K.; Müller, T.; Piccinno, F.; and Eisenschlos, J. M. 2020.  TaPas: Weakly Supervised Table Parsing via Pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , 4320–4333.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-Term Memory.  _Neural Comput._ , 9(8): 1735–1780.
  * Hosseini et al. (2014) Hosseini, M. J.; Hajishirzi, H.; Etzioni, O.; and Kushman, N. 2014.  Learning to Solve Arithmetic Word Problems with Verb Categorization.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_ , 523–533.
  * Huang et al. (2016) Huang, D.; Shi, S.; Lin, C.; Yin, J.; and Ma, W. 2016.  How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_.
  * Huang et al. (2021) Huang, S.; Wang, J.; Xu, J.; Cao, D.; and Yang, M. 2021.  Recall and Learn: A Memory-augmented Solver for Math Word Problems.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , 786–796.
  * Jie, Li, and Lu (2022) Jie, Z.; Li, J.; and Lu, W. 2022.  Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , 5944–5955.
  * Khattab and Zaharia (2020) Khattab, O.; and Zaharia, M. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , 39–48.
  * Kim et al. (2022) Kim, J.; Kang, J.; Kim, K.; Hong, G.; and Myaeng, S. 2022.  Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering.  In _Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , 1811–1821.
  * Kumar et al. (2021) Kumar, V.; Chemmengath, S. A.; Gupta, Y.; Sen, J.; Bharadwaj, S.; and Chakrabarti, S. 2021.  Multi-Instance Training for Question Answering Across Table and Linked Text.  _CoRR_ , abs/2112.07337.
  * Kushman et al. (2014) Kushman, N.; Zettlemoyer, L.; Barzilay, R.; and Artzi, Y. 2014.  Learning to Automatically Solve Algebra Word Problems.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers_ , 271–281.
  * Li, Sun, and Cheng (2021) Li, X.; Sun, Y.; and Cheng, G. 2021.  TSQA: Tabular Scenario Based Question Answering.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , 13297–13305.
  * Lin et al. (2020) Lin, B. Y.; Lee, S.; Khanna, R.; and Ren, X. 2020.  Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , 6862–6868.
  * Liu et al. (2020) Liu, D.; Gong, Y.; Fu, J.; Yan, Y.; Chen, J.; Jiang, D.; Lv, J.; and Duan, N. 2020\.  RikiNet: Reading Wikipedia Pages for Natural Question Answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , 6762–6771.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Qin et al. (2021) Qin, J.; Liang, X.; Hong, Y.; Tang, J.; and Lin, L. 2021.  Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , 5870–5881.
  * Shen et al. (2021) Shen, J.; Yin, Y.; Li, L.; Shang, L.; Jiang, X.; Zhang, M.; and Liu, Q. 2021.  Generate & Rank: A Multi-task Framework for Math Word Problems.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , 2269–2279.
  * Shi et al. (2015) Shi, S.; Wang, Y.; Lin, C.; Liu, X.; and Rui, Y. 2015.  Automatically Solving Number Word Problems by Semantic Parsing and Reasoning.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_ , 1132–1142.
  * Wang et al. (2019) Wang, L.; Zhang, D.; Zhang, J.; Xu, X.; Gao, L.; Dai, B. T.; and Shen, H. T. 2019\.  Template-Based Math Word Problem Solvers with Recursive Neural Networks.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , 7144–7151.
  * Wu, Zhang, and Wei (2021) Wu, Q.; Zhang, Q.; and Wei, Z. 2021.  An Edge-Enhanced Hierarchical Graph-to-Tree Network for Math Word Problem Solving.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , 1473–1482.
  * Zhan et al. (2021) Zhan, J.; Mao, J.; Liu, Y.; Guo, J.; Zhang, M.; and Ma, S. 2021.  Optimizing Dense Retrieval Model Training with Hard Negatives.  In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_ , 1503–1512.
  * Zhao et al. (2022) Zhao, Y.; Li, Y.; Li, C.; and Zhang, R. 2022.  MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , 6588–6600.
  * Zhou et al. (2022) Zhou, Y.; Bao, J.; Duan, C.; Sun, H.; Liang, J.; Wang, Y.; Zhao, J.; Wu, Y.; He, X.; and Zhao, T. 2022.  OPERA: Operation-Pivoted Discrete Reasoning over Text.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , 1655–1666.
  * Zhu et al. (2021) Zhu, F.; Lei, W.; Huang, Y.; Wang, C.; Zhang, S.; Lv, J.; Feng, F.; and Chua, T. 2021.  TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , 3277–3287.