# Improving Compositional Generalization in Math Word Problem Solving

Yunshi Lan, Lei Wang, Jing Jiang, Ee-Peng Lim Yunshi Lan and Lei Wang
contribute equally to this work.Yunshi Lan is with School of Data Science and
Engineering, East China Normal University. E-mail<EMAIL_ADDRESS>Wang (corresponding author), Jing Jiang and Ee-Peng Lim are with School of
Computing and Information System, Singapore Management University. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Compositional generalization refers to a model’s capability to generalize to
newly composed input data based on the data components observed during
training. It has triggered a series of compositional generalization analysis
on different tasks as generalization is an important aspect of language and
problem solving skills. However, the similar discussion on math word problems
(MWPs) is limited. In this manuscript, we study compositional generalization
in MWP solving. Specifically, we first introduce a data splitting method to
create compositional splits from existing MWP datasets. Meanwhile, we
synthesize data to isolate the effect of compositions. To improve the
compositional generalization in MWP solving, we propose an iterative data
augmentation method that includes diverse compositional variation into
training data and could collaborate with MWP methods. During the evaluation,
we examine a set of methods and find all of them encounter severe performance
loss on the evaluated datasets. We also find our data augmentation method
could significantly improve the compositional generalization of general MWP
methods. Code is available at https://github.com/demoleiwang/CGMWP.

###### Index Terms:

Math word problem solving, compositional generation, data augmentation.

##  I Introduction

_Math Word Problem_ (MWP) solving can be formulated as a generation task whose
goal is to generate an abstract expression to solve a given MWP. For example,
to solve MWP (a) in Figure 1, we want to be able to generate the correct
expression “$12+3$”. Among the methods proposed to solve MWPs [1, 2, 3, 4],
many leveraged advanced neural networks and have achieved promising results.
Like many other tasks such as Question Answering [5, 6], MWP solving methods
are expected to exhibit _Compositional Generalization_ , an important
capability to handle novel compositions of known components after learning the
“rules of composition” from the training data. Prior work has shown that
models often fail to capture the underlying compositional structure and suffer
from big loss of the performance on the data splits with a large compositional
gap [7, 8].

Figure 1: MWP examples with compositional challenges. Corresponding
expressions are annotated with blue color.

Without exception, an ideal MWP model is expected to correctly infer novel
compositions of seen component expressions to answer more new math problems in
the text. Assume _Constituents_ , which are the concepts described as
background scenario in MWPs (e.g. “apples”, “number”, “price”), are seen in
both training and test sets. If MWP (a) and (b) in Figure 1 are contained in
training set, we might encounter following two types of novel Compositions
during prediction: (1) Test set contains more complex combinations of
expression templates that are not observed. For example, constituents in (c)
have occurred in the training set, but (c) queries about an unseen expression
template, which is a compound version of (a). (2) Test set contains unseen
combinations of expression templates and constituents. In Figure 1, (d)
describes the same scenario as (a) but queries about an expression template
that has not occurred in this scenario but occurred in a different scenario
like (b) during training. In the perspective of semantic interpretation, these
two types of compositions require structural and lexical generalization of
models [9], respectively. While it is of utmost importance to investigate
compositional generalization on MWP solving, past research has largely
overlooked this issue. There are only few researches research works that
studied the out-of-distribution (OOD) phenomenon issues on MWPs [10, 11, 12],
their discussion is limited to length divergence or text perturbation of data,
which is only tangentially related to compositional generalization.

This manuscript aims to study compositional generation on MWP solving.
Specifically, we investigate the problem with the following steps: First, we
setup the problems by creating compositional challenges on MWPs. Next, we
propose an effective data augmentation method that could be collaborated with
any MWP methods to improve their compositional generalization capabilities.
Last, we examine a set of existing methods as well as our data augmentation
method and analyze how the mechanisms make them robust or fragile to the
compositional challenges.

In more detail, when creating the compositional challenges, on the one hand,
we adopt a data splitting method, which is built upon DBCA [5] and holds the
objective of maximizing the compositional gap with control of constituent
difference between training and test. This method creates desirable
compositional splits from large-scale Math23K and MAWPS datasets for
evaluation. On the other hand, we synthesize data via rules to isolate the
effect of each type of novel compositions, which results in a synthetic data
set with a clear partition of compositions. To improve the compositional
generalization in MWP solving, inspired by GECA [13], we develop a data
augmentation protocol aiming to discover the alternative text fragments from
MWP corpus and perform a substitution to include more compositional variation
into the training data. Furthermore, the data augmentation procedure is
conducted in an iterative manner, where a well-trained MWP solver is leveraged
to select the high-qualified augmented data and involve them into MWP corpus
for the next round of data augmentation. During experiments, we compare
general encoder-decoder models, strong MWP solving methods, pre-trained models
and MWP solving methods combined with our data augmentation method. We find
all tested methods suffer from performance loss when they are exploited under
the compositional gap. By comparison, the structured output decoder is
relatively robust. We also observe common contributions of our data
augmentation method on compositional challenges with different effects on
different compositions.

To summarize, our contributions include the following aspects: (1) We
introduce two ways to create three compositional MWP datasets to support
future research on this topic. (2) We propose an iterative data augmentation
method to improve the compositional generalization capability of general MWP
methods. (3) The experiments show that our data augmentation could
significantly improve the compositional generalization in MWP solving with the
largest increase of $15.6$ percentage points. (4) The evaluation on a wide
range of existing methods and further analysis sheds light on what mechanism
is potential in bridging the compositional gap. Our data and code will be
publicly available after the acceptance of this manuscript.

##  II Related Work

###  II-A Math Word Problem Solving

Early work on math word problem solving task can be categorized into
statistical machine learning based methods [1, 14, 2, 15] and semantic parsing
based methods [16, 17, 18]. The hand-crafted features are collected to
represent MWPs and pre-defined templates are leveraged to generate answers.
However, they cannot be applied to large-scale datasets. Recently, neural
networks have been widely applied to solve MWPs. Seq2seq models were first
leveraged to directly transform MWP text sequence to expression sequence [19],
which are still the mainstream MWP solvers until now. Li _et al._ [20]
borrowed the idea from Transformer and proposed multi-head attention to model
different types of features of MWPs. Many methods managed to encode richer
information, they proposed different ways to pre-process MWPs, such as number
mapping [19] and graph construction [21]. To capture the structural
information of expressions, more advanced work [22, 23, 24, 4] proposed to
decode an expression with implicit or explicit tree structure. Besides, there
are some other studies focusing on knowledge distillation [25], weak
supervision [10] and data augmentation [26] for MWP solving.

###  II-B Compositional Generationation

Compositional generalization gains much attention from researchers recently.
People have shown that neural network-based methods always encounter
tremendous loss when facing the compositional challenges [7, 8, 27]. Different
datasets have therefore been introduced to support the compositional
generalization research on Language-driven Navigation [28, 29], Question
Answering [30, 6, 31], Emergent Languages [32] and text2SQL [33]. However,
there is a lack of study of compositional generalization on MWP solving task.
Hong _et al._ [10] briefly discussed the OOD challenge by splitting the MWP
data based on the length of MWPs. Kumar _et al._ [11] discussed the robustness
of MWP solvers under the adversarial attack of question reordering and
sentence paragraphing. Patel _et al._ [12] included carefully chosen text
variations to MWPs for more robust evaluation of methods. The focus of them is
more about length divergence and text perturbation rather than compositional
generalization.

##  III Problem Setup

A MWP is a natural language narrative associated with a question towards the
narrative whose answer is a numerical value. It consists of multiple sentences
$q=\\{s_{1},...,s_{i},...,s_{k}\\}$ where $s_{i}$ is a sequence of tokens
$\\{w_{1},...,w_{m}\\}$. Each token could be either a word token or a
quantity. The goal of MWP solving is to generate an expression $e$ based on
$q$, which is formed by quantities and pre-defined mathematical operators and
which, when evaluated, gives the numerical answer to $q$. Assume we have a set
of training data
$\mathcal{D}^{p}=\\{(q^{p}_{1},e^{p}_{1}),...,(q^{p}_{|\mathcal{D}^{p}|},e^{p}_{|\mathcal{D}^{p}|})\\}$,
where each data point is a MWP-expression pair. We are supposed to leverage
$\mathcal{D}^{p}$ to train a MWP model that can make prediction on a test
dataset $\mathcal{D}^{q}$. We define _a challenging dataset for evaluating
compositional generalization to be a dataset where there is a potentially
large compositional difference but small constituent difference between
$\mathcal{D}^{p}$ and $\mathcal{D}^{q}$_. And we expect a good MWP model to
have the capability to maximize the performance on $\mathcal{D}^{q}$ despite
the compositional gap of $\mathcal{D}^{p}$.

To investigate the compositional generalization in MWP solving, we need to
first construct challenging datasets as described above. To this end, we
present a data splitting method and a data synthesizing method, which aim at
generating large-scale compositional splits and isolating compositional
effect, respectively.

###  III-A Split Data with DBCA

As introduced in the Sec. I, existing work [6, 5] has proposed their method
based on distribution-based compositional assessment (DBCA) to create
compositional splits for KBQA task, which could be summarized as follows:

  * •

Each data is represented using a graph, where the nodes are considered as
Atoms and the rule applications on the graph are treated as Compounds.

  * •

A distribution-based compositional assessment [6] is utilized to measure the
divergence:
$\text{DBCA}(\mathcal{D}^{p},\mathcal{D}^{q})=1-\mathcal{C}_{\alpha}(\mathcal{P}\lVert\mathcal{Q})$,
where
$\mathcal{C}_{\alpha}(\mathcal{P}\lVert\mathcal{Q})=\sum_{i}p^{\alpha}_{i}q^{1-\alpha}_{i}\in[0,1]$
is the Chernoff coefficient [34], $\mathcal{P}$ and $\mathcal{Q}$ are
distributions deriving from training and test sets, respectively. In more
detail, they obtain atom and compound distributions based on their
frequencies, and then compute Atom Divergence $\text{DBCA}^{a}$ and Compound
Divergence $\text{DBCA}^{c}$, respectively. As we can see, the atom divergence
measures the constituent difference and compound divergence measures the
compositional difference of a dataset.

  * •

Starting from a data pool $\mathcal{D}$, they apply a greedy algorithm to
assign each data to form $\mathcal{D}^{p}$ and $\mathcal{D}^{q}$, the
objective of which is to maximize $\text{DBCA}^{c}$ with the control of an
upper bound of $\text{DBCA}^{a}$.

The detailed algorithm could be found in original work [5].

Figure 2: Three examples of data. MWPs and expressions are shown in the left
column. The original expression trees and our trees are shown in the middle
and right columns, respectively. The cluster indexes and their corresponding
contexts are shown in the same colors.

When it comes to MWPs, each expression of MWPs could be represented using an
expression tree [21], which can be treated as a specific graph. The leaf nodes
are quantities and they are connected via mathematical operators. However, the
lack of semantics of the quantities leads to inapplicability of the above
divergence measurement. As shown in Figure 2, (a) and (b) describe the
different scenarios but their graphs are exactly the same. (a) and (c)
describe the similair scenarios but with different quantities, which leads to
different graphs. Therefore, we enrich the representation of quantity in the
expression tree by its contexts. Specifically, we extract tokens within a
window slide centered on a target quantity in a MWP to represent the quantity.
Then, we represent the quantities via TF-IDF vectors and perform $k$-means
clustering algorithm based on the vector-based representations, which assigns
a cluster index to each quantity111We have also tried other methods such as
Word2vec, BERT to represent the context. They cannot outperform TF-IDF vectors
even though they have more expressive architectures. It could be explained
that TF-IDF vectors are derived from the MWPs domain so it contains more
domain knowledge while other models involve more general knowledge.. As a
result, the quantities within similar contexts are labeled with the same
cluster indexes and we treat such quantities annotated with cluster indexes as
atoms. In Figure 2, after representing the trees featured with context
information, (a) and (c) will have the same graphs. Next, $\text{DBCA}^{a}$ is
derived from the frequency of cluster indexes after traversing the expression
tree. To obtain $\text{DBCA}^{c}$, we exhaustively search sub-expression trees
in a complete expression tree. Each sub-expression tree consists of a left
component, an operator node and a right component, where a component is either
a sub-expression tree or an atom in the expression tree. Afterwards, we follow
the traditional distribution-based assessment and the greedy algorithm to
generate realistic compositional splits.

###  III-B Synthesize Data to Isolate Effect

Figure 3: One example of generating synthetic MWPs with a seed MWP via a
decomposition rule. The selected MWP in the corpus is shown in bold line and
the instantiate sentence of $\Delta$ is annotated with red color.

Data splitting method results in datasets differing in compositions with a
fuzzy boundary. Since our goal is to understand how MWP solvers respond to
different compositions precisely, to disentangle compositional gaps arising
from new expression templates or new combinations of templates and
constituents, we synthetically construct data via rules.

As the preliminary step, we pre-define a Rule Table 222The complete table and
more implementation details are displayed in Appendix, which contains the
rules that we follow to synthesize data. Each rule states that given a MWP
with an expression template, it could be transformed into new MWP(s) with
certain expression template(s). For example, the MWP (d) in Figure 1 could be
represented by $\\{s_{1},s_{2},s_{3},s_{4}\\}$ with expression template
$n_{1}\times n_{2}/n_{3}$, where the quantity in the expression is replaced by
the position index [3]. Following one rule, we could decompose it into
$\\{s_{1},s_{2},\Delta,s_{4}\\}$ with template $n_{1}\times n_{2}$ and
$\\{\Delta,s_{3},s_{4}\\}$ with template $n_{1}/n_{2}$, where $\Delta$
indicates a placeholder to be instantiated. We further generate an Indexed
Corpus by indexing a set of initial MWPs with their templates as keys and MWPs
as values, which could be utilized to instantiate the placeholders. Moreover,
we define a Similarity Scoring function to measure the semantic similarity
between MWPs. Specifically, we extract a collection of nouns, verbs and
measure units from the MWPs using existing toolkits. When we measure the
similarity of two MWPs, we compare the token-level overlap. A large similarity
value indicates the MWPs share similar constituents. Next, we introduce how we
synthesize data with the two types of compositions:

  * •

Decomposition. We treat each decomposable $q\in\mathcal{D}$ as the seed MWP,
where $\mathcal{D}$ is a data pool. We employ applicable decomposition rules
from the rule table to generate two uninstantiated MPWs. We retrieve all MWPs
from the indexed corpus with their corresponding templates, select the MWP
which has the highest similarity score to the seed MWP and instantiate
$\Delta$ with corresponding sentence of the selected MWP. This step will
generate two synthetic MWPs $\\{q^{{}^{\prime}},q^{{}^{\prime\prime}}\\}$
specific to $q$. To illustrate the above procedure better, we display an
example in Figure 3. Eventually, we allocate $(q,e)$ into test set and
$\\{(q^{{}^{\prime}},e^{{}^{\prime}}),(q^{{}^{\prime\prime}},e^{{}^{\prime\prime}})\\}$
into training set to form $\mathcal{D}^{q}$ and $\mathcal{D}^{p}$,
respectively. It is worth noting that sometimes the selected MWP
$q^{{}^{\prime}}$ or $q^{{}^{\prime\prime}}$ has similarity score value to $q$
as $0$, in this case, we abandon it. This is to ensure the similar constituent
distribution between $\mathcal{D}^{p}$ and $\mathcal{D}^{q}$.

  * •

Reformulation. For each applicable $q\in\mathcal{D}$, we employ reformulation
rules to generate one uninstantiated MWP and instantiate placeholder $\Delta$
as illustrated as above to obtain a synthetic MWP $q^{\prime}$. If for any
existing $q^{\prime\prime}\in\mathcal{D}^{p}$ which has
$e^{\prime\prime}=e^{\prime}$ and the similarity score to $q^{\prime}$ is $0$,
we include $(q,e)$ into $\mathcal{D}^{p}$ and $(q^{\prime},e^{\prime})$ into
$\mathcal{D}^{q}$. Otherwise, we abandon it. This is because we would like to
only include the MWP of which the combination of templates and constituents
have not been seen during training into the test set.

After above procedure, we obtain the synthetic data sets $\mathcal{D}^{p}$ for
training and $\mathcal{D}^{q}$ for testing. $\mathcal{D}^{p}$ includes all the
MWPs having simple templates as their expressions, $\mathcal{D}^{q}$ contains
all the MWPs which are either new compounds of the simple expression templates
or new combinations of existing constituents and simple expression templates.
As synthesized MWPs always share constituents with seed MWP, the constituent
difference between training and test sets is well-controlled and the only
variation is the compositionality.

##  IV Data Augmentation for MWP Solvers

Figure 4: Two examples of data augmentation protocol on MWPs via operation 1
and operation 2 in sub-figure (a) and (b), respectively. Fragments are labeled
with underline and substituent parts in the synthetic MWPs are in red color.
Expressions are annotated with blue color.

Previous studies [35, 13] have conducted discussions on data augmentation,
which has proven to be effective in improving the robustness of models. In
this manuscript, we propose an iterative data augmentation method that aims to
generate more compositional variation without involving external data and
human annotation. Inspired by GECA [13], we discover the alternative text
fragments and substituent the text fragments in existing MWPs to synthesize
new MWPs. We further improve the quality of the generated data by measuring
the confidence of new MWPs via a well-trained MWP solver. The entire procedure
is displayed in Algorithm 1. As we can see, this algorithm is designed with an
iterative framework, where a Data Augmentation Protocol takes charge of
generating new MWPs and a Data Ranker plays the role of judging the quality of
the generated MWPs. After $T$ rounds of iteration, the augmented data will
fill the compositional gap which is leveraged to train a robust MWP model.

Input: initial training data set $\mathcal{D}^{p}$, max iteration $T$

for  _i in T_ do

$\mathcal{D}^{*}$ = $\emptyset$

for  _q in $\mathcal{D}^{p}$_ do

$\mathcal{D}_{q}^{p}$ = DataAugmentationProtocol($q$)

$\mathcal{D}^{*}=\mathcal{D}^{*}\cup\mathcal{D}_{q}^{p}$

end for

$\mathcal{D}^{*}$ = DataRanker($\mathcal{D}^{*}$)

$\mathcal{D}^{p}$ = $\mathcal{D}^{p}\cup\mathcal{D}^{*}$

end for

Return: augmented training data set $\mathcal{D}^{p}$

Algorithm 1 Data Augmentation Algorithm

We first describe our data augmentation protocol which is an improved version
of GECA. Two text fragment is defined as Alternative if they occur in some
common environments. With the alternative fragments, we define the following
operations to involve new compositions in a MWP: (1) If the first fragment is
from the last sentence of a MWP, which is the question sentence, and the
second fragment is not from the last sentence of another MWP, we use the
sentences excluding the last sentence of the first MWP to substituent the
corresponding sentence of the second MWP. In the example of Figure 4 (a), “how
many” and “there are” are alternative fragments as they appear in the same
environment “peaches”. Given a MWP querying “how many”, we could leverage its
narrative description to substituent the sentence “there are 36 peaches” in
another MWP to form a new MWP. (2) If both the fragments are not from the last
sentences of MWPs, we replace all the first fragment in a MWP with the second
fragment. In the example of Figure 4 (b), “peach” and “notebook” is a pair of
alternative fragments. When there is a MWP with fragment “peach”, we can
generate a new MWP by replacing all the “peach” with “notebook”. As we can
see, the first operation involves more diversity to the expression structure
and generates MWPs with novel expression templates. The second operation helps
to generate MWPs with novel combinations of expression templates and
constituents.

We now formally describe the above operations. Recall each MWP consists of
multiple sentences $q=\\{s_{1},...,s_{k}\\}$. A fragment is a set of non-
overlapping spans of $s$. An environment is $s$ with a text fragment removed.
We denote fragment and environment as $f$ and $s/f$, respectively. If there
are environments $x=s_{1}/f_{1}$ and $y=s_{2}/f_{2}$ with $x=y$,
$(f_{1},f_{2})$ is a pair of alternative fragments:

  * •

Operation 1. Given $q=\\{s_{1},...,s_{k}\\}$ and
$q^{\prime}=\\{s^{\prime}_{1},...,s^{\prime}_{l}\\}$ with $e$ and
$e^{\prime}$, respectively. If $f_{1}\in s_{k}$ and $f_{2}\in s^{\prime}_{i}$,
where $i\in[1,...,l-1]$, we could form a new MWP
$q^{\prime}=\\{s^{\prime}_{1},...,s^{\prime}_{i-1},s_{1},...,s_{k-1},s^{\prime}_{i+1},...,s^{\prime}_{l}\\}$.
The expression of this MWP is $e^{\prime}$ modified by replacing the quantity
in $s^{\prime}_{i}$ with $e$.

  * •

Operation 2. Given $q=\\{s_{1},...,s_{k}\\}$ with $e$, if $f_{1}\in s_{i}$
where $i\in[1,...,k-1]$, we could form a MWP
$q^{\prime}=\\{s^{\prime}_{1},..,s^{\prime}_{i},...,s^{\prime}_{k}\\}$ where
$s^{\prime}_{i}$ is obtained via replacing all the $f_{1}$ in $s_{i}$ with
$f_{2}$. The expression of this MWP is still $e$.

For each $q\in\mathcal{D}^{p}$, we generate a set of new MWPs, via either
operation 1 or operation 2. These newly generated MWPs will form a set
$\mathcal{D}^{*}$.

As the data augmentation protocol follows simple patterns, incorrect fragment
discovery and illogical fragment substitution would be easily involved and
deemed as noisy data. To improve the quality of the augmented data, we
leverage a MWP solver to perform as the data ranker to filter out the sub-
optimal augmented data. We assume that a well-trained MWP solver could learn
the capability of understanding general narrative of MWPs. Ambiguous and
illogical description would be considered as uncertain data by a well-trained
MWP solver, which results in the generation of an expression with low
confidence [36, 37]. Therefore, for $q\in\mathcal{D}^{*}$, we employ a well-
trained MWP solver to obtain its expression with a sequence of probabilities
and sort them based on the accumulative product of the probabilities. The MWPs
with low probabilities are filtered out from $\mathcal{D}^{*}$ and the rest
are merged with the initial training data set which are either used to
generate more data for the next iteration or returned as the augmented data
set. It is worth noting that there is no data leakage happening as the test
data is hidden during the entire augmentation procedure.

Discussion: Compared with the original protocol proposed by Andreas _et al._
[13], we improve their algorithm by developing novel synthetic rules to fit
the nature of compositional MWPs and collaborating with an iterative framework
to continuously improve the quality of the augmented data. Compared with the
synthesizing procedure described in Sec. III-B which heavily relies on manual
rules of expression annotations, this data augmentation method generates data
based on the patterns of MWPs and does not depend on any hand-crafted rules.

##  V Experiments

In this section, we first introduce the compositional data sets that we
created in Sec. V-A. Next, we describe our experimental setup in Sec. V-B.
Then, we show the evaluation of different methods on compositional challenges
and analysis in Sec. V-C and Sec. V-D, respectively.

###  V-A Compositional Data Sets

We create compositional challenges deriving from two datasets: Math23K dataset
from Wang _et al._ [19] and MAWPS dataset from Kedziorski _et al._ [38]. These
two datasets are both large-scale MWP datasets that are commonly used in
existing work. We utilize NLTK333https://www.nltk.org/ as the toolkit to
extract contextual features through tokenization, POS tagging and dependency
parsing. For data splitting, scikit-learn tool444https://scikit-
learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
is employed to implement TF-IDF. We choose $k$ value from range
$\\{5,10,20\\}$ based on the highest $\text{DBCA}^{c}$ it could achieve after
splitting. Following previous implementation555https://github.com/google-
research/language/tree/master/language/nqg, $\alpha$ in DBCA is set as $0.1$
and $0.5$ for $\text{DBCA}^{a}$ and $\text{DBCA}^{c}$, respectively. As a
result, we obtain the following three compositional data sets:

|  Split |  NR |  LR |  $\text{DBCA}^{a}$ |  $\text{DBCA}^{c}$
---|---|---|---|---|---
Math23K |  I.I.D. |  18543/4619 |  0.99 |  0.00 |  0.20
|  ComDiv |  18543/4619 |  1.01 |  0.00 |  0.69
MAWPS |  I.I.D. |  1899/474 |  1.01 |  0.00 |  0.09
|  ComDiv |  1899/474 |  0.91 |  0.01 |  0.77
SD |  ComDiv |  1078/745 |  0.76 |  0.01 |  0.82
TABLE I: Statistics of datasets. NR, and LR denote Number Ratio and text
Length Ratio between training and test sets, respectively. $\text{DBCA}^{a}$
and $\text{DBCA}^{c}$ are atom and compound divergences, respectively.

| Math23K | MAWPS | SD
---|---|---|---
Methods | I.I.D. | ComDiv | Rel. Gap | I.I.D. | ComDiv | Rel. Gap | Valid | Test | Rel. Gap
LSTM | 67.4 | 47.4 | 0.42 | 74.5 | 36.5 | 1.04 | 93.7 | 7.8 | 11.0
Transformer | 58.0 | 40.3 | 0.44 | 72.6 | 40.1 | 0.81 | 91.0 | 6.6 | 12.8
BERTGen | 68.1 | 43.0 | 0.58 | 70.7 | 43.0 | 0.64 | 96.9 | 9.6 | 9.09
GPT-2 | 70.4 | 49.7 | 0.42 | 62.2 | 31.9 | 0.95 | 95.1 | 8.3 | 10.5
MathEN | 66.5 | 46.9 | 0.42 | 76.4 | 40.9 | 0.87 | 94.6 | 7.5 | 11.7
GTS | 71.5 | 51.2 | 0.39 | 76.9 | 48.4 | 0.59 | 96.9 | 7.9 | 11.3
MathEN+$\mathrm{DA}$ |  67.3(+0.8) |  47.5(+0.6) | 0.42 |  77.6 (+1.2) |  42.4(+1.5) | 0.83 |  95.5 (+0.9) |  17.4(+9.9) | 4.5
GTS+$\mathrm{DA}$ |  72.1(+0.6) |  52.0(+0.8) | 0.38 |  78.1(+1.2) |  49.8(+1.4) | 0.57 |  96.4(-0.5) |  23.5 (+15.6) | 3.1

TABLE II: Answer accuracy of four groups of methods on different data sets. We
show the accuracy of test data for different splits of Math23K and MAWPS, the
accuracy of validation and test data for SD data set. To show the performance
gap between normal and compositional splits, we compare the results of Math23K
and MAWPS with I.I.D. and ComDiv splits, results of SD with validation and
test sets, respectively. Rel. Gap denotes
$\text{Acc}_{\text{I.I.D.}}/\text{Acc}_{\text{ComDiv}}-1$. The numbers in red
font denote improvement brought by data augmentation.

.

  * •

Math23K-ComDiv: This is a dataset created by performing data splitting as
described in Sec. III-A on Math23K. It contains Chinese MWPs associated with
corresponding expressions.

  * •

MAWPS-ComDiv: We conduct data splitting as described in Sec. III-A on MAWPS
and obtain this compositional splits. It contains English MWPs which query a
variety of arithmetic math expressions.

  * •

SD: This is a synthetic dataset created by conducting data synthesis as
described in Sec. III-B with Math23K performing as its data pool. After the
automatic synthesis procedure, we further do manual proofreading to ensure the
validity of this data set.

We employ above compositional challenges to evaluate compositional generation
of a series of methods. As a comparison, we also include original I.I.D.
splits of Math23K and MAWPS. Table I displays the statistics of all datasets.
As we can see, most datasets have LR value approximately to $1$, which
indicates that the questions with different length are evenly distributed over
training and test sets. The compositional data sets all have high
$\text{DBCA}^{c}$ but low $\text{DBCA}^{a}$, which indicates the datasets
indeed follow our expected objective of maximizing the compound divergence
with great control of small constituent divergence.

###  V-B Experimental Setup

Following the standard training and prediction diagrams of MWP solving, we
treat it as a generation task with the supervision of ground truth
expressions. During prediction, we report the answer accuracy by comparing the
execution results of generated expressions and the correct answers.

#### Comparable Methods.

The methods tested on compositional challenges of MWPs are mainly categorized
into four families: (1) General encoder-decoder models, i.e., LSTM [39] and
Transformer [40]. (2) Strong MWP solving methods, i.e., MathEN [22] and GTS
[4]. (3) Pretrained models i.e., BERTGen [41] and GPT-2 [42], which have been
suggested to improve general compositional generalization [27]. (4) MathEN and
GTS collaborating with our proposed data augmentation method (refer to as
MathEN+DA and GTS+DA), which aims to improve compositional generalization.

#### Implementation Details.

All the experiments are implemented by PyTorch on Nvidia V440.64.00-32GB GPU
cards. We implement the existing work by leveraging their official code or
following their methodology description. Regarding general encoder-decoder
models, we set the same embedding size and hidden size as MathEN. Regarding
pretrained baselines, we adopt base BERT and base GPT-2. Regarding MWP
baselines and MWP baselines with data augmentation, we keep hyperparameters
the same as original manuscripts666We show detailed hyperparameters in
Appendix.. For each type of data splits, we randomly allocate 20% data from
training to do validation. In terms of the data augmentation method, the MWP
solvers well trained with the augmented data in the last round of iteration
are used to rank the data. We set the maximum iteration time $T$ as 7 and keep
the top $20\%$ augmented MWPs for each iteration.

###  V-C Results

Table II displays the results of all tested methods. Based on the table, we
have the following observations: (1) For SD data set, we observe a huge
performance gap between validation set and test set. A similar performance gap
appears between I.I.D. split and ComDiv split on both Math23K and MAWPS. This
indicates that all families of methods are vulnerable to compositional
challenges MWP solving. Among these three datasets with compositional
challenges, SD dataset causes the biggest loss. This could be explained by the
nature of big compound divergence of SD data. (2) Comparing the first three
sections, in terms of the most robust method to compositional challenges,
there is no absolute agreement on all the data sets. Nevertheless, GTS shows
relatively good compositional generation as it has the lowest Rel.Gap on
Math23K and MAWPS. This may because the tree-structured neural module which
outputs an expression tree is more powerful when generalizing to unseen
compositions. (3) Our data augmentation could provide positive effect to
MathEN and GTS methods. This indicates that data augmentation is effective in
improving the compositional generalization capability of general MWP methods.
Meanwhile, data augmentation shows the significant contribution on SD dataset
with $9.9$ and $15.6$ percentage points improvement to MathEN and GTS,
respectively. This implies it indeed plays a key role in bridging the gap of
compositions. It’s worth noting that data augmentation provides the largest
performance gain 15.6% to GTS model on SD test set. It may because that data
augmentation includes more lexical and structural variations for training and
tree-structured neural network has a better inductive bias to capture them,
which is also verified in prior work [43].

###  V-D Further Analysis

#### Accuracy with Increasing Compound Divergence

Figure 5: (a) Performance changes of MathEN with the increase of
$\text{DBCA}^{c}$ on Math23K and MAWPS. (b) Evolution of answer accuracy
(blue) and average training time per epoch (red) of GTS on SD dataset with the
increase of iteration time.

To show the influence of compound divergence described in Sec. III-A, we
display Figure 5 (a). As we can see, the performance of MathEN on both Math23K
and MAWPS datasets decreases dramatically with the increase of compound
divergence $\text{DBCA}^{c}$. This indicates the performance is sensitive to
the change of compound divergence, which verifies that the compound divergence
is indeed an effective indicator for measuring the compositional challenge of
MWPs. Therefore, it is reasonable for us to split MWPs data and obtain the
compositional data sets with $\text{DBCA}^{c}$ performing as the measurement
criteria.

#### Effect of Different Composition Types

| SD
---|---
| Structural | Lexical
MathEN | 3.0 | 46.3
GTS | 2.9 | 51.3
MathEN+DA | 14.2 | 47.5
GTS+DA | 16.8 | 63.8
TABLE III: Results on partition of SD dataset with Structural and Lexical
generalization separately.

SD dataset that we introduced in Sec. III-B is created for isolating the
effect of different types of compositions. Decomposition and reformulation
rules create data requiring structural and lexical generalization,
respectively. We display Table III to discuss the effect of different rules.
From the table, we observe that new compound expression templates bring much
more difficulties to models than new combinations of expression templates and
constituents. It is easy to understand as template inferring requires a deep
understanding towards the expression. We also observe that data augmentation
benefits more to MWPs with new compound expression templates.

#### Ablation Study of Data Augmentation

| Math23K-ComDiv | MAWPS-ComDiv
---|---|---
MathEn+DA | 47.5 | 42.4
\- DataRanker | 46.6 | 41.4
\- Operation 1 | 47.3 | 41.8
\- Operation 2 | 47.2 | 41.6
TABLE IV: Ablation study of data augmentation with MathEn on Math23K-ComDiv
and MAWPS-ComDiv datasets.

To further investigate the mechanism of the proposed data augmentation method,
we remove the data ranker, operation 1 and operation 2 in turn and test
MathEn+DA on the Math23K-ComDiv and MAWPS-ComDiv datasets. We show the results
in Table IV. As we can see, removing any of them causes the decrease of
results. This indicates that data ranker, operation 1 and operation 2 all
contribute to the good effect of data augmentation method. Operation 1 and
operation 2 generate different compositions to fill the gap. Data ranker
prevents the model from the hurt of low-qualified augmented data.

#### Effect of Iteration Time.

Figure 6: Three augmented examples produced via our method. The original
training examples of Math23K is shown at the left column, corresponding
augmented data is shown at the middle column and the confidence score for the
augmented data is shown at the right column. Each MWP is associated with its
expression tree. The context of a quantity in MWPs is highlighted with colors
and the partial expression trees forming the augmented data is highlighted
with yellow boxes.

We depict the curve of data augmentation effect with the influence of
iteration time in Figure 5 (b). As we can see, with the increase of iteration
time, the answer accuracy of GTS on SD dataset increases gradually. It is
because more iteration indicates more augmented data is generated to fill the
compositional gap. But when iteration comes to $6$, the increase becomes less
significant. Meanwhile, we notice that with the increase of iteration, the
average time for each training epoch becomes larger. It is because that the
number of augmented data increases exponentially, which leads to a large time
cost for training the model. Therefore, it is a trade-off to select a suitable
iteration time with the consideration of both performance gain and time cost.

#### Human Evaluation of Augmented Data

In order to examine the quality of augmented data and understand how the
augmented data improve general MWP methods, we invite two people to evaluate
the quality of augmented MWPs. An augmented MWP will be labeled as qualified
by human evaluators if it satisfies grammatical correctness, fluency, and
logical consistency. Otherwise, it will be labeled as unqualified. For
simplicity, we randomly sample 100 augmented MWPs and ask human evaluators to
judge the quality of each MWP. We then calculate the agreement between
evaluators using Cohen’s k coefficient. Finally, the percentage of qualified
augmented examples from two evaluators are 83% and 84% respectively, and their
Cohen’s k coefficient is 0.71, which indicates the data augmentation indeed
generates a decent number of high-qualified data examples to strength the
training procedure.

#### Case Study of Augmented Data

We display some augmented data in Figure 6. As we can see, the expression tree
of augmented data is a novel combination of the expression trees of training
data. The math word problems are the corresponding narratives of the
expression trees. Example (a) shows that data augmentation method generates a
novel combination of two familiar structures. The training data contains
example of the structure $n_{1}\times n_{2}$. But the combination $n_{1}\times
n_{2}\times n_{3}$ has never been observed in training. The augmented data
fills this gap. After adding the augmented data into the training set, such
combination rules will be learned by models. Example (b) shows that data
augmentation generates a novel combination of a familiar primitive and a
familiar structure. The training data contains example of the structure
$n_{1}-n_{2}$ but the generalization concerns the constituents ”XiaoMing”,
”apples” has never been observed in the first position of $n_{1}-n_{2}$. The
augmented data fills this gap by including more lexical variations into the
training data and thus improve the lexical generalization of models. Example
(c) is another augmented example of structural generalization. Even though it
is semantically correct, it is logically wrong. When a data ranker generates
its math expression, and the confidence score is very low. Therefore, this is
not a good augmented example and our method excludes it from the training set.

##  VI Conclusion

In this manuscript, we investigated compositional generalization in MWP
solving task. We created three compositional challenges on MWPs and evaluated
a set of methods. We observed most of the methods suffered from a big loss of
performance. We further proposed an iterative data augmentation method and it
has proven to be effective in improving the compositional generalization of
general methods. While this problem is still open to be explored, our study
provides some insights for proposing new solutions in the future research.

##  VII Appendix

###  VII-A Rule Table for Synthesizing Data

We manually defined 13 rules in total to generate synthetic data (Sec. III-B)
which are displayed in Table V.

| ID | Seed MWP | Synthetic MWP(s)
---|---|---|---
Decomposition | 1 | $\\{s_{1},s_{2},s_{3}\\}$ | $\\{s1,\Delta,s_{3}\\}$ | $\\{s_{2},\Delta\\}$
$n_{1}\times(1-n_{2})$ | $n_{1}\times n_{2}$ | $1-n_{1}$
2 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$n_{1}\times n_{2}/n_{3}$ | $n_{1}\times n_{2}$ | $n_{1}/n_{2}$
3 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$n_{1}\times n_{2}+n_{3}$ | $n_{1}\times n_{2}$ | $n_{1}+n_{2}$
4 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$n_{1}\times n_{2}\times n_{3}$ | $n_{1}\times n_{2}$ | $n_{1}\times n_{2}$
5 | $\\{s_{1},s_{2},s_{3}\\}$ | $\\{s_{1},\Delta,s_{3}\\}$ | $\\{s_{2},\Delta\\}$
$n_{1}/(1-n_{2})$ | $n_{1}/n_{2}$ | $1-n_{1}$
6 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$(n_{1}-n_{2})/n_{3}$ | $n_{1}-n_{2}$ | $n_{1}/n_{2}$
7 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$(n_{1}+n_{2})\times n_{3}$ | $n_{1}+n_{2}$ | $n_{1}\times n_{2}$
8 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$n_{1}/n_{2}\times n_{3}$ | $n_{1}/n_{2}$ | $n_{1}\times n_{2}$
9 | $\\{s_{1},s_{2},s_{3},s_{4}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ | $\\{\Delta,s_{3},s_{4}\\}$
$n_{1}\times n_{2}-n_{3}$ | $n_{1}\times n_{2}$ | $n_{1}-n_{2}$
Reformulation | 10 | $\\{s_{1},s_{2},s_{3}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ |
$n_{1}+n_{2}$ | $n_{1}-n_{2}$ |
11 | $\\{s_{1},s_{2},s_{3}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ |
$n_{1}-n_{2}$ | $n_{1}+n_{2}$ |
12 | $\\{s_{1},s_{2},s_{3}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ |
$n_{1}\times n_{2}$ | $n_{1}/n_{2}$ |
13 | $\\{s_{1},s_{2},s_{3}\\}$ | $\\{s_{1},s_{2},\Delta\\}$ |
$n_{1}/n_{2}$ | $n_{1}\times n_{2}$ |
TABLE V: The rule table used in our work. The middle section contains all the
rules for decomposition and the third section contains all the rules for
reformulation.

###  VII-B Hyperparameters Choice

We show the hyperparameters of all the tested methods in Table VI. We set
hyperparameters of any model to be the same on all datasets. For BERTGen and
GPT-2, we use original default hyperparameters of BERT and GPT-2,
respectively. We tune hyper-parameters, i.e., learning rate, batch size,
dimension of latent vector (including embedding size and hidden size), dropout
ratio, number of layers, and number of heads, within the range of {0.0001,
0.0003, 0.0005, 0.001, 0.003, 0.01}, {16, 32, 64, 128}, {128, 256, 512},
{0.05, 0.1, 0.15, 0.2}, and {1,2,4,8}, and {2,4,8} respectively and choose the
best group of hyperparameters by validation accurarcy.

| LSTM | Transformer | MathEN | GTS
---|---|---|---|---
train epochs | 100 | 100 | 100 | 100
batch size | 32 | 64 | 32 | 64
embedding size | 128 | 512 | 128 | 128
hidden size | 512 | 512 | 512 | 512
# layers | 2 | 4 | 2 | 2
# heads | – | 8 | – | –
learning rate | 0.001 | 0.003 | 0.001 | 0.001
warmup steps | – | 1500 | – | –
dropout | 0.5 | – | 0.5 | 0.5

TABLE VI: Summary of hyperparameters that deviate from the defaults.

## References

  * [1] N. Kushman, L. Zettlemoyer, R. Barzilay, and Y. Artzi, “Learning to automatically solve algebra word problems,” in _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_ , 2014, pp. 271–281.
  * [2] S. Roy and D. Roth, “Unit dependency graph and its application to arithmetic word problem solving,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , 2017, p. 3082–3088.
  * [3] L. Wang, D. Zhang, L. Gao, J. Song, L. Guo, and H. T. Shen, “Mathdqn: Solving arithmetic word problems via deep reinforcement learning,” 2018, pp. 5545–5552.
  * [4] Z. Xie and S. Sun, “A goal-driven tree-structured neural model for math word problems,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_ , 2019, pp. 5299–5305.
  * [5] P. Shaw, M.-W. Chang, P. Pasupat, and K. Toutanova, “Compositional generalization and natural language variation: Can a semantic parsing approach handle both?” _arXiv preprint arXiv:2010.12725_ , 2020.
  * [6] D. Keysers, N. Schärli, N. Scales, H. Buisman, D. Furrer, S. Kashubin, N. Momchev, D. Sinopalnikov, L. Stafiniak, T. Tihon, D. Tsarkov, X. Wang, M. van Zee, and O. Bousquet, “Measuring compositional generalization: A comprehensive method on realistic data,” in _8th International Conference on Learning Representations_ , 2020.
  * [7] M. Gardner, Y. Artzi, V. Basmova, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, N. Gupta, H. Hajishirzi, G. Ilharco, D. Khashabi, K. Lin, J. Liu, N. F. Liu, P. Mulcaire, Q. Ning, S. Singh, N. A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, A. Zhang, and B. Zhou, “Evaluating nlp models via contrast sets,” _arXiv preprint_ , 2020.
  * [8] D. Kaushik, E. Hovy, and Z. Lipton, “Learning the difference that makes a difference with counterfactually-augmented data,” in _International Conference on Learning Representations_ , 2020.
  * [9] N. Kim and T. Linzen, “COGS: A compositional generalization challenge based on semantic interpretation,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 9087–9105.
  * [10] Y. Hong, Q. Li, R. Gong, D. Ciao, S. Huang, and S.-C. Zhu, “Smart: A situation model for algebra story problems via attributed grammar,” in _arXiv preprint arXiv:2012.14011_ , 2020.
  * [11] V. Kumar, R. Maheshwary, and V. Pudi, “Adversarial examples for evaluating math word problem solvers,” in _Findings of the Association for Computational Linguistics: EMNLP 2021_. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 2705–2712.
  * [12] A. Patel, S. Bhattamishra, and N. Goyal, “Are NLP models really able to solve simple math word problems?” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 2080–2094.
  * [13] J. Andreas, “Good-enough compositional data augmentation,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 7556–7566.
  * [14] A. Mitra and C. Baral, “Learning to use formulas to solve simple arithmetic problems,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 2144–2153.
  * [15] Y. Zou and W. Lu, “Quantity tagger: A latent-variable sequence labeling approach to solving addition-subtraction word problems,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 5246–5251.
  * [16] S. Shi, Y. Wang, C.-Y. Lin, X. Liu, and Y. Rui, “Automatically solving number word problems by semantic parsing and reasoning,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015, pp. 1132–1142.
  * [17] S. Roy and D. Roth, “Solving general arithmetic word problems,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015, pp. 1743–1752.
  * [18] D. Huang, S. Shi, C.-Y. Lin, and J. Yin, “Learning fine-grained expressions to solve math word problems,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017, pp. 805–814.
  * [19] Y. Wang, X. Liu, and S. Shi, “Deep neural solver for math word problems,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017, pp. 845–854.
  * [20] J. Li, L. Wang, J. Zhang, Y. Wang, B. T. Dai, and D. Zhang, “Modeling intra-relation in math word problems with different functional multi-head attentions,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 6162–6167.
  * [21] J. Zhang, L. Wang, R. K.-W. Lee, Y. Bin, Y. Wang, J. Shao, and E.-P. Lim, “Graph-to-tree learning for solving math word problems,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 3928–3937.
  * [22] L. Wang, Y. Wang, D. Cai, D. Zhang, and X. Liu, “Translating math word problem to expression tree,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 1064–1069.
  * [23] T.-R. Chiang and Y.-N. Chen, “Semantically-aligned equation generation for solving and reasoning math word problems,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 2656–2668.
  * [24] Q. Liu, W. Guan, S. Li, and D. Kawahara, “Tree-structured decoding for solving math word problems,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , 2019, pp. 2370–2379.
  * [25] J. Zhang, R. K.-W. Lee, E.-P. Lim, W. Qin, L. Wang, J. Shao, and Q. Sun, “Teacher-student networks with multiple decoders for solving math word problem,” in _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , 2020.
  * [26] Q. Liu, W. Guan, S. Li, F. Cheng, D. Kawahara, and S. Kurohashi, “Reverse operation based data augmentation for solving math word problems,” 2021.
  * [27] I. Oren, J. Herzig, N. Gupta, M. Gardner, and J. Berant, “Improving compositional generalization in semantic parsing,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020, pp. 2482–2495.
  * [28] B. Lake and M. Baroni, “Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks,” in _35th International Conference on Machine Learning_ , 2018, pp. 4487–4499.
  * [29] L. Ruis, J. Andreas, M. Baroni, D. Bouchacourt, and B. M. Lake, “A benchmark for systematic generalization in grounded language understanding,” in _Advances in Neural Information Processing Systems_ , 2020, pp. 19 861–19 872.
  * [30] D. Bahdanau, H. d. Vries, T. J. O’Donnell, S. Murty, P. Beaudoin, Y. Bengio, and A. Courville, “Closure: Assessing systematic generalization of clevr models,” _arXiv preprint arXiv:1912.05783_ , 2019.
  * [31] Y. Gu, S. Kase, M. T. Vanni, B. M. Sadler, P. Liang, X. Yan, and Y. Su, “Beyond i.i.d.: Three levels of generalization for question answering on knowledge bases,” in _Proceedings of The Web Conference 2021_ , 2021.
  * [32] R. Chaabouni, E. Kharitonov, D. Bouchacourt, E. Dupoux, and M. Baroni, “Compositionality and generalization in emergent languages,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, 2020, pp. 4427–4442.
  * [33] C. Finegan-Dollak, J. K. Kummerfeld, L. Zhang, K. Ramanathan, S. Sadasivam, R. Zhang, and D. Radev, “Improving text-to-SQL evaluation methodology,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 351–360.
  * [34] J. Chung, P. Kannappan, C. T. Ng, and P. Sahoo, “Measures of distance between probability distributions,” _Journal of Mathematical Analysis and Applications_ , vol. 138, pp. 280–292, 1989.
  * [35] R. Jia and P. Liang, “Data recombination for neural semantic parsing,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 12–22.
  * [36] J. Adebayo, M. Muelly, I. Liccardi, and B. Kim, “Debugging tests for model explanations,” in _Advances in Neural Information Processing Systems_ , 2020, pp. 700–712.
  * [37] J. Antoran, U. Bhatt, T. Adel, A. Weller, and J. M. Hernández-Lobato, “Getting a {clue}: A method for explaining uncertainty estimates,” in _International Conference on Learning Representations_ , 2021.
  * [38] R. Koncel-Kedziorski, H. Hajishirzi, A. Sabharwal, O. Etzioni, and S. D. Ang, “Parsing algebraic word problems into equations,” _Transactions of the Association for Computational Linguistics_ , vol. 3, pp. 585–597, 2015.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5998–6008.
  * [40] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [41] W. Chen, J. Chen, Y. Su, Z. Chen, and W. Y. Wang, “Logical natural language generation from open-domain tables,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 7929–7942.
  * [42] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” _OpenAI blog_ , 2019.
  * [43] T. McCoy, E. Pavlick, and T. Linzen, “Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 3428–3448.