#  Towards Tractable Mathematical Reasoning: Challenges, Strategies, and
Opportunities

Keyur Faldu
Embibe
&Amit Sheth
University of
South Carolina
&Prashant Kikani
Embibe
&Manas Gaur
University of
South Carolina
&Aditi Avasthi
Embibe
Correspondence to<EMAIL_ADDRESS>
#  Towards Tractable Mathematical Reasoning: Challenges, Strategies, and
Opportunities for Solving Math Word Problems

Keyur Faldu
Embibe
&Amit Sheth
University of
South Carolina
&Prashant Kikani
Embibe
&Manas Gaur
University of
South Carolina
&Aditi Avasthi
Embibe
Correspondence to<EMAIL_ADDRESS>
Mathematical reasoning would be one of the next frontiers for artificial
intelligence to make significant progress. The ongoing surge to solve math
word problems (MWPs) and hence achieve better mathematical reasoning ability
would continue to be a key line of research in the coming time. We inspect
non-neural and neural methods to solve math word problems narrated in a
natural language. We also highlight the ability of these methods to be
generalizable, mathematically reasonable, interpretable, and explainable.
Neural approaches dominate the current state of the art, and we survey them
highlighting three strategies to MWP solving: (1) direct answer generation,
(2) expression tree generation for inferring answers, and (3) template
retrieval for answer computation. Moreover, we discuss technological
approaches, review the evolution of intuitive design choices to solve MWPs and
examine them for mathematical reasoning ability. We finally identify several
gaps that warrant the need for external knowledge and knowledge-infused
learning, among several other opportunities in solving MWPs.

Figure 1: An illustration of the mathematical reasoning process that involves
understanding of natural language and visual narratives. Essentially, the
process comprises extracting entities with explicit and implicit quantities
building a self-consistent symbolic representation invoking mathematical laws,
axioms, and symbolic rules.

##  1 Introduction

The recent success of artificial intelligence over a wide range of natural
language processing tasks like hypothesis entailment, question answering,
summarization, translation, etc., has posed the question that how intelligent
are these systems?. The General Language Understanding Evaluation (GLUE) Wang
et al. (2018a), and SuperGLUE Wang et al. (2019) benchmarks have been created
to assess their natural language understanding abilities. Soon enough, deep
learning systems have outperformed human-level performance over these
benchmarks. However it is not understood if such performance is attributed to
underlying reasoning capabilities Sap et al. (2020)Storks et al.
(2019)Khashabi et al. (2019). Discovering and learning salient heuristic
patterns to produce a likely outcome is also much better than rule-based
systems. When learning is achieved without reasoning, it will reduce to
mimicking, which would not be generalizable Bandura (2008). Reasoning ability
is at the center of performing higher-order complex tasks. Mathematical
reasoning is one of the niche abilities humans possess Lithner (2000) English
(2013), it constitutes linguistic reasoning, visual reasoning, common sense
reasoning, logical reasoning, numerical reasoning, and symbolic reasoning.
Humans do not learn mathematical reasoning just based on experience and
evidence, but it results from learning, inferring, and applying laws, axioms,
and symbolic rules. Mathematical reasoning could be one of the most important
benchmarks to test the progress made by artificial intelligence.

Mathematical reasoning ability involves understanding natural language and
visual narratives, extracting explicit and implicit quantities using common
sense and domain-specific knowledge, inferring a self-consistent symbolic
universe, and exploiting mathematical laws, axioms, and symbolic rules Amini
et al. (2019)Saxton et al. (2019). The problems requiring mathematical
reasoning ability pose a unique opportunity to build a system that requires
collaborative interactions between neural systems and symbolic systems
Kahneman (2011). AI models for natural languages understanding and visual
understanding could be developed using fast or probabilistic systems, and
corresponding mathematical laws, axioms, and rules could be executed using
slow and deterministic systems. Math word problems (MWP) are natural language
questions that require mathematical reasoning to solve them. MWP is the
natural language narrative that contains a self-consistent universe where
entities with quantitative information interact with each other under well-
defined mathematical axioms, laws, and theorems. Answering MWP would require
solving unknown quantities or sets of unknown quantities from known explicit
or implicit quantities using semantic information, commonsense world
knowledge, and mathematical domain knowledge. Thus, MWPs can be of varying
difficulty levels based on the complexity of their narratives, implicit world
and domain knowledge needed to solve the problem, and the application of
mathematical topics and their interactions. Figure 3 demonstrates couple of
examples of relatively complex MWPs.

The ability to solve MWPs in an automated fashion could lead to many
applications in the education domain for content creation and delivering
learning outcomes Donda et al. (2020) Faldu et al. (2020a). For example, it
could aid students’ learning journey by resolving their doubts on the fly;
potentially solve the problem of content deficiency for practice assessments
Dhavala et al. (2020). Particularly by complementing MWPs recommendation
systems with AI-based solution generation system Faldu et al. (2020b).

The task of solving MWPs has a long history with consistent research
attention. This article surveys the progress in solving MWPs, including
datasets, approaches, and architectures, broadly divided into non-neural and
neural techniques. We also analyze the mathematical reasoning ability of such
methods.

Figure 2: Example of math word problem and its possible problem formulations

##  2 Background and Non-Neural Approaches

The complexity of solving MWPs has been well understood for a long time,
demanding logical systems representation and machine’s ability to apply such
systems with natural language understanding and generative capabilities
Feigenbaum et al. (1963). Before the surge of neural network-based approaches
to solving MWPs, several non-neural techniques were explored since the 1960s,
classified into rule-based or pattern matching, semantic parsing, statistical,
and machine learning approaches. However, most non-neural methods focus on the
template retrieval approach or predicting the answer using rules. These
approaches are explained in the following sections.

Figure 3: Example of relatively complex MWPs from dataset MathQA Amini et al.
(2019), requiring understanding of specific words like umbra and penumbra,
applying multiple math concepts, and accessing mathematical knowledge like
math formula for area of circle, sphere, value of $\pi.$

###  2.1 Rule-based or Pattern Matching Approaches

Normalizing and transforming input narratives using heuristics, such as rules,
regular expressions, or pattern matching, was the first attempt in solving
MWPs Feigenbaum et al. (1963) Bobrow (1964) Charniak (1968). Such approaches
are very intuitive but could only handle limited scenarios, and hence, they
lack the flexibility in handling simple variations of math word problems.
Further, efforts have been made to develop schema, frame, or extract entities
and quantities associated with it to apply rules deterministically. Fletcher
(1985) Dellarosa (1986) Yuhui et al. (2010). However, such rule-based and
pattern matching algorithms would not be scalable to learn general-purpose
mathematical reasoning ability.

###  2.2 Semantic Parsing

A semantic parsing-based system attempts to learn the semantic structure of
the input problem and transforms it into mathematical expressions or a set of
equations. It represents problems in object-oriented structure, much similar
to text-SQL generation, following methods in natural language processing
Liguda and Pfeiffer (2012) Koncel-Kedziorski et al. (2015). Underlying tree
dependencies of mathematical semantics are captured by context-free grammar
rules, which is analogous to dependency parse trees from the perspective of
mathematical semantics Shi et al. (2015).

###  2.3 Statistical and Machine Learning Approaches

Prior works on Statistical and machine learning approaches developed automated
methods that are rule-based and semantic parsing, which processes training
data to derive a set of templates or learn the semantic structure from the
input. They range from text classification of MWPs with equation templates,
extracting and mapping quantities with equations, or entities extraction and
classifying their relationships Kushman et al. (2014). These methods use
machine learning algorithms like support vector machines, probabilistic
models, or margin classifiers for concept prediction, equation prediction, or
slot filling. They leverage sentence semantics or verb categorization to map
mathematical operations with segments of an MWP Hosseini et al. (2014)
Amnueypornsakul and Bhat (2014). A further set of candidate hypotheses are
reduced by handling noun slots and number slots separately Zhou et al. (2015).
Few other techniques process narratives to fill concepts-specific slots first
and derive equations from them using domain-specific rules Mitra and Baral
(2016)Roy and Roth (2018).

##  3 Mathematical Reasoning in Non-Neural Approaches

Mathematical reasoning ability is the key to solving mathematical problems.
There were few attempts made in this direction using reasoning-oriented sub-
tasks like quantity entailment for semantic parsing Roy et al. (2015),
comprehending numerical information, quantity alignment prediction for slot
filling , handling quantity slots and noun slots separately, ignoring
extraneous information present in narratives, etc Bakman (2007) Mitra and
Baral (2016) Roy and Roth (2018) Zhou et al. (2015).

Non-neural methods that consider rule-based, pattern matching, semantic
parsing, and machine learning-based approaches are limited to specific areas
of mathematics like addition-subtractions, arithmetic, linear algebra,
calculus, or quadratic equations. The performance of such methods is
constrained by the coverage and diversity of MWPs in the training corpus. The
performance of such approaches could not scale on out-of-corpus MWPs mainly
because of the relatively small-sized training corpus Koncel-Kedziorski et al.
(2016) Kushman et al. (2014). Huang et al. (2016) attempted these techniques
on a large-scale dataset Dolphin18K, having 18000 annotated linear and non-
linear MWPs. Non-neural methods performed much worse on diverse and large-
scale datasets than their reported performance on their corresponding small
and specific datasets, and their performance improved sub-linearly with more
extensive training data Huang et al. (2016). It made a solid case to build
more generic systems with better reasoning ability.

Addition-Subtraction Math Word Problems |  Subtype |  Semantic Slots
---|---|---
Sam’s dog had 8 puppies. He gave 2 to his friends. He now has 6 puppies . How many puppies did he have to start with? |  Change |  (i) start (ii) end (iii) gains (iv) losses
Tom went to 4 hockey games this year, but missed 7 . He went to 9 games last year. How many hockey games did Tom go to in all ? |  Part-Whole |  (i) whole (ii) parts
Bill has 9 marbles. Jim has 7 more marbles than Bill. How many marbles does Jim have? |  Compare |  (i) large quantity (ii) small quantity (iii) difference
Table 1: Classifying math word problems into subtypes, and semantic slots
required for solving MWPs in each subtypes.

###  3.1 Domain Knowledge or External Knowledge

Classifying MWPs into subtypes and handling them using mathematical domain
knowledge is another way to further push such systems’ capabilities. It
becomes easier to reason about a problem classified to a subtype as it reduces
the candidate math laws, axioms, and symbolic rules and contextualizes
semantics related to the specific area of math Mitra and Baral (2016) Roy and
Roth (2018)Bakman (2007) Amnueypornsakul and Bhat (2014). For example,
addition and subtraction problems can be grouped in three classes of
mathematical concepts, (i) change, (ii) part-whole and (iii) compare as shown
in table 1 Mitra and Baral (2016). Semantic parsing techniques form rules to
extract semantic information specific to these sub-types. Table 1 illustrates
subtypes of a problem and how each problem subtype would have different
semantic slots to apply math rules. The concept of part-whole has two slots,
one for the whole that accepts a single variable and the other for its parts
that accepts a set of variables of size at least two. The change concept has
four slots, namely start, end, gains, losses which respectively denote the
original value of a variable, the final value of that variable, and the set of
increments and decrements that happen to the original value of the variable.
The comparison concept has three slots, namely the large quantity, the small
quantity, and their difference Mitra and Baral (2016). Similarly, other such
mutually exclusive groups were identified by researchers to categorize MWPs,
like (i) join-separate, (ii) part-whole and (iii) compare Amnueypornsakul and
Bhat (2014); (i) change, (ii) combine (iii) compare Bakman (2007); or (i)
transfer, (ii) dimensional analysis, (iii) part-whole relation, (iv) explicit
math, Roy and Roth (2018) etc.

Commonsense and linguistic knowledge could be useful to derive the semantic
structure and represent the meaning of MWPs. Briars and Larkin (1984)
investigates how commonsense knowledge could be helpful to solve complex
problems narrating real-world situations. The knowledge that an object defined
in the MWP is a member of both a set and its superset and whether subsets can
be exchanged in the context of deriving an answer for an MWP. For example, the
math word problem shown in figure 2 illustrates objects “orange” and “apple.”
Both belong to a superset “fruits,” and they can be exchanged if the unknown
quantity in the question is agnostic to the type of fruit. Dolphin language
representation of MWPs requires extracting nodes and their relationships,
where nodes could be constants, classes, or functions. Commonsense knowledge
is used to classify entities sharing common semantic properties Shi et al.
(2015). Linguistic knowledge like verb sense and verb entailment could help
efficiently semantic parsing of MWPs narratives to extract semantic slots and
fill their values. Researchers have explored such external knowledge from
WordNet Hosseini et al. (2014) or E-HowNet Chen et al. (2005) Lin et al.
(2015).

##  4 Neural Approaches

The recent advancement in deep learning approaches has opened up new
possibilities. There is a flurry of research that aims to apply neural
networks to solve MWPs. Sequence to sequence architectures, transformers,
graph neural networks, convolutions, and attention mechanisms are a few such
architectures and techniques of neural approaches. To empower neural
approaches, larger and diverse datasets have been also created Amini et al.
(2019), Wang et al. (2017), Zhang et al. (2020), Saxton et al. (2019), Lample
and Charton (2019). Also, the potential of neural approaches solving complex
problems on calculus and integration has raised expectations on its future
promise. These approaches could be further categorized based on their problem
formulation, (i) to predict answers directly, (ii) generate intermediate math
expressions, or (iii) retrieve a template. However, it is evident that neural
networks are black boxes, and hard to interpret their functioning and explain
their decision Gaur et al. (2021). Attempts to interpret its functioning
reveal how volatile its reasoning ability is, and they lack generalizability
Patel et al. (2021).

In the following sections, we aim to categorize the current state-of-the-art
development. Then, we navigate problem formulations, datasets and analysis,
approaches and architectural design choices, data augmentation methods, and
interpretability and reasoning ability.

Dataset |  Mathematics Area |  Language |  Type |  Annotation |  Size
---|---|---|---|---|---
AI2 Hosseini et al. (2014) |  Arithmetic |  English |  Curated |  Equation / Answer |  395
IL Roy et al. (2015) |  Arithmetic |  English |  Curated |  Equation / Answer |  562
ALGES Koncel-Kedziorski et al. (2015) |  Arithmetic |  English |  Curated |  Equation / Answer |  508
AllArith Roy and Roth (2017) |  Arithmetic |  English |  Derived |  Equation / Answer |  831
Alg514 Kushman et al. (2014) |  Algebraic (linear) |  English |  Curated |  Equation/Answer |  514
Dolphin1878 Shi et al. (2015) |  Algebraic (linear) |  English |  Curated |  Equation / Answer |  1,878
DRAW Upadhyay and Chang (2015) |  Algebraic (linear) |  English |  Curated |  Equation / Answer / Template |  1,000
Dolphin18K Huang et al. (2016) |  Algebraic (linear, nonlinear) |  English |  Curated |  Equation / Answer |  18,000
MAWPS Koncel-Kedziorski et al. (2016) |  Arithmetic, Algebraic |  English |  Curated |  Equation/answer |  3,320
AQuA Ling et al. (2017) |  Arithmetic, Algebraic (linear, nonlinear) |  English |  Curated |  Rationale/MCQ-choices/Answer |  100,000
MathQA Amini et al. (2019) |  Arithmetic, Algebraic |  English |  Derived |  Equation/MCQ-choices/Answer |  37,000
MATH Hendrycks et al. (2021) |  Algebra, Number Theory, Probability, Geometry, Calculus |  English |  Curated |  Step-by-step solution / Answer |  12500
ASDiv Miao et al. (2021) |  Arithmetic, Algebraic |  English |  Derived |  Equation/Answer + Grade/Problem-type |  2,305
SVAMP Patel et al. (2021) |  Arithmetic |  English |  Derived |  Equation/Answer |  1,000
Math23K Wang et al. (2017) |  Algebraic (linear) |  Chinese |  Curated |  Equation/Answer |  23,161
HMWP Qin et al. (2020) |  Algebraic (linear + nonlinear) |  Chinese |  Curated |  Equation/Answer |  5,491
Ape210K Zhao et al. (2020) |  Algebraic (linear) |  Chinese |  Curated |  Equation/Answer |  210,488
Deepmind Mathematics Saxton et al. (2019) |  Algebra, Probability, Calculus |  English |  Synthetic |  Answer |  2,000,000
Integration & Differentiation Synthetic Dataset Lample and Charton (2019) |  Integration, Differentiation |  English |  Synthetic |  Answer |  160,000,000
AMPS Hendrycks et al. (2021) |  Algebra, Calculus, Geometry, Statistics, Number Theory |  English |  Synthetic |  Step-by-step solution / Answer |  5,000,0000
Table 2: Categorization of MWP Datasets available for training AI models based
on (a) Mathematics Area, (b) Language, (c) Type, and (d) Annotation.

###  4.1 Problem Formulation

We can categorize the problem formulation to solve MWPs in three different
ways, (i) predicting the answer directly, (ii) generating a set of equations
or mathematical expressions, and inferring answers from them by executing
them, and (iii) retrieving the templates from a pool of templates derived from
training data and augmenting numerical quantities to compute the answer. An
example is shown in Figure 2.

The first approach to predict the answers directly could demonstrate the
inherent ability of the neural model to learn complex mathematical
transformations Csáji et al. (2001); however, the black-box nature of such
models suffers from poor interpretability and explainability Gaur et al.
(2021). It has been shown that sequence to sequence (seq-to-seq) neural models
could compute free form answers for complex problems with high accuracy when
trained on large datasets Saxton et al. (2019)Lample and Charton (2019). It
has been analyzed how models behave when just numerical parameters are changed
vs. mathematical concepts in the test set. Lample and Charton (2019) achieved
near-perfect accuracy of 99% mainly attributed to the learning ability of
neural models over a huge dataset. Ran et al. (2019) proposes a numerically
aware graph neural network to directly predict the type of answer and the
actual answer.

The second approach deals with generating an expression tree and executing
them to compute the answer. Such models have relatively better
interpretability and explainability and generated expressions, which provides
scaffolding to the reasoning ability of the model. Seq-to-seq neural models
framing from LSTM, Transformers, GNN and Tree decoders have proven to useful
Wang et al. (2017) Amini et al. (2019) Xie and Sun (2019) Qin et al. (2020)
Liang et al. (2021). The key challenge for methods in this approach is the
need for expert annotated datasets, as they need expression trees as the
labels for each problem in addition to its answer Amini et al. (2019)Wang et
al. (2017)Koncel-Kedziorski et al. (2016).

The third approach derives template equations from training data, retrieves
the most similar template, and substitutes numerical parameters. This approach
suffers from generalization ability, as the set of templates are limited to
the training set. Such methods would fail to solve diverse and out-of-corpus
MWPs as they would not be able to retrieve the template needed to solve them.
It was one of the popular statistical machine learning approach Kushman et al.
(2014) Koncel-Kedziorski et al. (2015), Mitra and Baral (2016), but there are
also few other neural approaches, which have studied its value for an ensemble
setup or standalone model with larger training data Wang et al. (2017)
Robaidek et al. (2018).

Generally, these neural approaches follow encoder-decoder architecture to
predict expression trees or directly compute the answer, specifically when the
answer could be an expression in itself. We further explain different variants
of encoder-decoder architectures in section 4.3.

###  4.2 Analyzing Datasets

One of the current limitations of achieving performance similar to human-level
mathematical intelligence is the availability of labeled datasets. Table 2
categorizes various MWPs datasets along with information like dataset size,
language, type of dataset, mathematics area etc. Each dataset focus on
specific areas of Mathematics, i.e., Arithmetic, Algebra, Calculus, etc. AI2,
IL, ALGES, ALLArith, SVAMP are examples of datasets for arithmetic MWPs.
Datasets such as Alg514, Dolphin1878, DRAW, Dolphin18K, Math23K, HMWP, and
Ape210K focus on Algebraic problems. MAWPS, AQuA, MathQA, and ASDiv are
examples of datasets covering MWPs from Arithmetic and Algebra. There have
been efforts to create MWPs datasets for complex areas including Probability,
Integration and Differentiation Saxton et al. (2019) Lample and Charton
(2019). 2 covers examples of MWP datasets in English and Chinese languages.

MWPs datasets are either curated from educational resources, derived by
processing other available datasets or synthetically created. For example.,
Alg514 is created by curating problems from algerba.com. Similarly,
Dolphin1878 comprises of curated problems from both algebra.com and
answers.yahoo.com. Derived datasets have been derived by processing existing
datasets to address concerns related to training corpus size Roy and Roth
(2017), equation annotation Amini et al. (2019), lexical diversity Miao et al.
(2021), problem types Roy and Roth (2017) or adversarial reasoning ability
Patel et al. (2021). Synthetic datasets could be very large, but they lack
generalization. They are ultimately driven from deterministic sets of
templates or rules and combinations of them for specific math areas Saxton et
al. (2019) Lample and Charton (2019) Hendrycks et al. (2021). Synthetic
datasets could also be used in pre-training neural models. Curated datasets
are generally small in size, e.g., AI2, IL, Alg514, DRAW, MAWPS, etc., or
suffer from inconsistencies, e.g., AQuA or MathQA. Neural models generating a
set of equations or expression trees would need a dataset, where MWPs are
labeled with a set of equations or expression trees. Many of these curated
datasets would not have associated annotated expression trees; hence these
datasets would require manual effort to annotate. If datasets are created by
curating relevant webpages, it would also possibly need answer extraction from
free-form explanations Huang et al. (2016). Equations for MWPs could also be
extracted from their solutions on webpages using pattern matching, but which
could lead to some inconsistencies Amini et al. (2019). On the other hand, it
is very costly to augment these datasets with expert annotations. There can be
multiple possible ways to write a semantically unique equation, which demands
equation normalization to make sure labels are normalized Huang et al.
(2018a).

There is an active interest among the research community to analyze and
improve the quality of datasets. Datasets like AsDiv and SVAMP are derived to
address the quality concerns by processing existing datasets. Lexically
similar questions would lead the model to learn the heuristic patterns to
predict the answer Miao et al. (2021). Lexicon Diversity Score measures the
lexical diversity of questions provided in the dataset, and datasets with
better lexicon diversity score would be more reliable to test a model’s
performance Miao et al. (2021). Also, adversarial approaches further extend
such datasets with adversarial variants to test the mathematical reasoning
ability Patel et al. (2021).

###  4.3 Encoder Decoder Architectures

Figure 4: Popular Neural Architectures for Solving Math Word Problems

Wang et al. (2017) proposed the first encoder-decoder architecture to generate
equations or expression trees for MWPs. It used GRU based encoder and LSTM
based decoder. Since then, researchers have actively sought to explore the
optimal architecture to get better performance for predicting equations or
expression trees on diverse datasets comprising complex MWPs. As datasets are
limited in size, researchers kept iterating to find better architectures to
learn semantic structures of math word problems well. These novel neural
architectures can be grouped into four categories, (1) seq-to-seq, (2) seq-to-
tree, (3) transformer-to-tree, (4) graph-to-tree.

Sequence-to-sequence architecture is the default choice of any natural
language generative model to transform a natural language input to output.
Output sequence is either free form answer expression, equations, or traversal
sequence of expression trees Griffith and Kalita (2021) Amini et al. (2019).
Wang et al. (2017) leverages GRU-based encoder and LSTM based decoder, Amini
et al. (2019) uses attention mechanisms over RNNs, and conditions decoder with
category of MWPs. For larger synthetic datasets Lample and Charton (2019)
Saxton et al. (2019), there isn’t a need for semantically rich hybrid
architectures as seq-to-seq models could achieve good performance.
Transformers outperforms attentional LSTMs on larger datasets Saxton et al.
(2019), mainly attributed to more computations with the same numbers of
parameters, internal memory of transformers which is pre-disposed to
mathematical objects like quantities, and better gradient propagation with
shallower architectures. There has been an effort to model situations narrated
in MWPs using agents, attributes, and relations and train a seq-to-seq model
to convert relation text to equations Hong et al. (2021b).

Sequence-to-tree, transformer-to-tree, and Graph-to-tree architectures attempt
to incorporate the tree structure of the output mathematical equations or
expressions into it. The decoder in such models is a tree decoder, where it
generates the next token by taking hidden states of its parent and sibling and
its subtree (if any) into the account Xie and Sun (2019)Liu et al. (2019)Qin
et al. (2020)Liang et al. (2021) Wu et al. (2021). Attention-based
architectures leverage attention mechanisms to attend only parents and
siblings Qin et al. (2020)Liang et al. (2021). Operators would be parent
nodes, and operands would be their children nodes, which would recursively
expand further. Encoders in such architectures varied from GRU/LSTM based Xie
and Sun (2019) or transformer-based Liang et al. (2021). As tree decoders help
model learning the tree semantics out output expressions, such models perform
better than the seq-to-seq model on datasets with limited size. The expected
output could be represented as a computational tree, which is then traversed
and evaluated to compute the answer.

Graph-to-tree architecture further exploits the graph semantics present in
MWPs Ran et al. (2019) Zhang et al. (2020) Li et al. (2019). Graph semantics
captures the relationships between different entities, which can be thought of
as relationships among numerical quantities, their description, and unknown
quantities. The goal-driven tree-structured model is designed to generate
expressions using computational goals inferred from an MWP, which uses graph
transformer network as the encoder to incorporate quantity comparison graph
and quantity cell graph Zhang et al. (2020). It leverages self-attention
blocks for incorporating these instances of graphs. The numerical reasoning
module attempts to learn comparative relations between numerical quantities by
connecting them in a graph. Encoder representations are appended with
representations learned by numerical reasoning module to improve the
performance Ran et al. (2019). An MWP can be segmented into a quantity spans
and a question span. Specific self-attention head blocks which could just
attend quantity span, question span, or both could be used to learn graph
relationships between quantities and question present in an MWP Li et al.
(2019).

###  4.4 Design Choices

In this section, we aim to highlight several design decisions for the above
architectures. First, the seq-to-seq model suffers from generating spurious
numbers or predicting numbers at the wrong position. Copy and alignment
mechanisms can be used to avoid this problem Huang et al. (2018a). Second, it
has become a common practice to keep decoder vocabulary limited to numbers,
constants, operators, and other helpful information present in the question
Xie and Sun (2019) Zhang et al. (2020).

There are multiple ways of generating output expressions, and maximum
likelihood expectations would compromise learning if predicted, and expected
expressions are different but semantically the same. Reinforcement learning
would solve this problem by rewarding the learning strategy based on the final
answer Huang et al. (2018a).

Tree-based decoders have received lots of attention because of their ability
to invoke tree relationships present in mathematical expressions. Tree
decoders attend both parents and siblings to generate the next token. Bottom-
up representation of subtree of a sibling could further help to derive better
outcomes Qin et al. (2020). For efficient implementation of tree decoder, the
stack data structure can be used to store and retrieve hidden representations
of parent and sibling Liu et al. (2019). Tree regularization transforms
encoder representation and subtree representations and minimizes their L2
distance as a regularization term of the loss function Qin et al. (2020).

Graph-based encoder aims to learn different types of relationships among the
constituents of MWPs. There are several attempts to construct different types
of graphs. Ran et al. (2019) inserts a numerical reasoning module that uses
numerically aware graph neural networks, where two types of edge less-than-
equal-to and greater-than connect numerical quantities present in the
question. A situation model for algebra story problem SMART aims to build a
graph using attributed grammar, which connects nodes with its attributes using
relationships extracted from the problem Hong et al. (2021b). Self-attention
blocks of transformers could also be used to model the graph relationships
Zhang et al. (2020) Li et al. (2019). The goal-driven tree-structured model
incorporated quantity cell graph and quantity comparison graph Zhang et al.
(2020). Quantity cell graph connects numerical quantity of the input equations
with its descriptive words or entities, whereas quantity comparison graph is
similar to the concept used by Ran et al. (2019). Segmentation of question
into quantity span and question span, and using self-attention blocks to
derive global attention, quantity-related attention, quantity pair attention,
and question-related attention is another such approach to derive better
semantic representation Li et al. (2019).

Multitasking is a prevalent paradigm to train the same model for multiple
tasks. It enriches the semantic representations of models and avoids them
getting overfitted. Auxiliary tasks could also be part of such a setup.
Auxiliary tasks like Common Sense Prediction Task, Number Quantity Prediction,
and Number location prediction could be helpful Qin et al. (2021). The
Commonsense prediction task aims to predict the commonsense required to solve
an MWP, like the number of legs of a horse, or a number of days in a week,
etc. On the other hand, mask language modeling pre-training on mathematical
corpus Ape210k in a self-supervised way to help its downstream application for
math word problems Liang et al. (2021).

###  4.5 Data Augmentation and Weak Supervision

Large datasets have empowered neural models to learn complex mathematical
concepts like integration, differentiation Lample and Charton (2019) Saxton et
al. (2019), etc, but such models often overfit on smaller datasets Miao et al.
(2021) Patel et al. (2021). Data augmentation is a popular preprocessing
technique to increase the size of training data. Generally, data augmentation
creates the variant of training records by applying domain-specific
argumentation techniques. Several techniques were proposed for the data
augmentation of MWP datasets. Reverse operation-based augmentation techniques
swap an unknown quantity with a known quantity in an MWP, and their expression
trees are modified to reflect the change Liu et al. (2020). Different
traversal orders of expression trees and other preprocessing techniques like
lemmatization, POS tagging, sentence reordering, stop word removal, etc., and
their impact on the performance of models have been studied by Griffith and
Kalita (2021). Data processing and augmentation could also help to generate
adversarial datasets, which could test the reasoning ability of models Patel
et al. (2021)Miao et al. (2021).

Weak supervision is another popular technique that generates large datasets
with noisy labels. The model training process could cancel noise and learn
actual patterns. For example, generating expression trees for an MWP dataset
with question and answer could be done using weak supervision, where
expression trees may not always be accurate. Still, the answer it evaluates
would match the desired answer. Learning by fixing is a technique where
expression trees are generated iteratively by fixing operators and operands
till it evaluates to the desired answer Hong et al. (2021a). Such a technique
helps model learning the more diverse ways of solving mathematical problems.

##  5 Mathematical Reasoning in Neural Approaches

Mathematical reasoning is core to human intelligence. As explained before, it
is a complex phenomenon interfacing natural language understanding and visual
understanding to invoke mathematical transformations. Over the past few
decades, there has been a flurry of research to develop models with
mathematical reasoning ability; however, the progress is limited. Larger
datasets definitely help models learning to solve complex and niche problems
Lample and Charton (2019) Saxton et al. (2019), but a common purpose model to
solve a diverse variety of math problems is still a distinct reality Patel et
al. (2021) Miao et al. (2021). Smaller datasets overfit the models, and hence
reasoning ability is questionable. Non-neural models could not improve
performance with larger datasets Huang et al. (2016). On the other hand,
neural models achieve much better performance on the smaller datasets and
attain near-perfect performance over very large datasets; however, its
BlackBox structure hinders interpreting its reasoning ability. There is a
decent consensus among researchers to solve problems by first generating
intermediate expression trees and evaluating them using solvers to predict the
answer Wang et al. (2017) Amini et al. (2019) Xie and Sun (2019). As opposed
to directly predicting the final answer, generating intermediate expression
trees helps understand models’ reasoning ability better. Currently, the
efforts are mainly to solve MWPs of primary or secondary schools as shown in
Table 2. The state of the current art is still very far from building
mathematical intelligence that could also infer visual narratives and natural
language and use domain-specific knowledge.

Models with desired reasoning ability would not only be generalizable, but it
would also help to make systems interpretable and explainable. On the other
hand, interpretable and explainable systems are easier to comprehend, which
helps to progress further by expanding the scope of its reasoning on more
complex mathematical concepts. The field of mathematics is full of axioms,
rules, and formulas, and it would also be important to plug explicit and
definite knowledge into such systems.

###  5.1 Interpretability, and Explainability

Deep learning models directly predicting answers from an input problem lack
interpretability and explainability. As models often end up learning shallow
heuristic patterns to arrive at the answer. Huang et al. (2016), Patel et al.
(2021). That’s where it is important to generate intermediate representations
or explanations, then infer the answer using them. It not only improves
interpretability but also provides scaffolding to streamline reasoning ability
and hence generalizability. Such intermediate representations could be of
different forms, and they could be interspersed with natural language
explanations to aid the explainability of the model further.

####  5.1.1 Intermediate Representations

The semantic parsing approach attempts to derive semantic structure from the
problem narratives. Still, they are not generalizable because of the inherent
limitations of statistical and classical machine learning techniques to
represent the meaning. However, deriving intermediate representations and
inferring the solution henceforth would remain one of the niche areas to
explore in the context of neural approaches. Transforming problem narratives
to expressions or equations and then evaluating them to get an answer is one
of the most important approaches suitable for elementary level MWPs Wang et
al. (2017). There were other attempts in deriving representation language Shi
et al. (2015), using logic forms Liang et al. (2018) and intermediate meaning
representations Huang et al. (2018b), etc. It would require much more effort
to derive intermediate representation for comprehensive and complex
mathematical forms and expressions in hindsight.

####  5.1.2 Interspersed Natural language Explanations

Natural language explanations to describe the mathematical rationale would
reflect the system’s reasoning ability and also makes the system easier to
comprehend and explainable. For example, Ling et al. (2017) uses LSTM to focus
on generating answer rationale, which is a natural language explanation
interspersed with algebraic expression to arrive at a decision. AQuA dataset
contains 100,000 such algebraic problems with answer rationales. Such answer
rationales not only improve interpretability but provide scaffolding which
helps the model learn mathematical reasoning better. Language models like WT5,
which produce explanations along with prediction, could be an inspiration
Narang et al. (2020). It would be essential to extend such approaches for more
complex mathematical concepts beyond algebra. Natural language explanations
interspersed with mathematical expressions would be critical areas for the
researchers to focus on for building more comprehensive systems. Such
interspersed explanations would open up the possibility to help students
understand how to solve the problem more effectively or would foster users’
trust in the system.

###  5.2 Infusing Explicit and Definitive Knowledge

Mathematical problem solving requires identifying relevant external knowledge
and applying it to the mathematical rules to arrive at an answer. External
knowledge could be commonsense world knowledge, particular domain knowledge,
or complex mathematical formulas. There is some progress to infuse commonsense
world knowledge Liang et al. (2021) or predict it using auxiliary tasks Qin et
al. (2021), however remaining two dimensions are yet to be addressed. Such
external knowledge is incorporated using rule-based, pattern-matching
approaches in non-neural models; on the other hand, neural models leverage
data augmentation and auxiliary task training. Zhang et al. (2020) Xie and Sun
(2019) developed novel neural architectures to represent semantics captured in
the problems using graph and tree relations. These examples are of the shallow
knowledge infusion category. It would be intriguing to explore deep knowledge
infusion wherein model architecture would transform and leverage external
knowledge vector spaces during its training Wang et al. (2020) Faldu et al.
(2021). Furthermore, it would be promising to build an evaluation benchmark
for such knowledge-intensive mathematical problems to encourage and streamline
efforts to solve math problems Sheth et al. (2021).

###  5.3 Reinforcement Learning

Mathematical reasoning ability comprises applying a series of mathematical
transformations. Rather than learning mathematical transformations, the system
could focus on synthesizing the computational graph. Learning algorithms like
gradient descent works on iterative reduction of the error. The choices of
these mathematical transformations are not differential, and hence it is
unclear if its the optimal strategy. Learning policy to construct such a
computation graph using reinforcement learning could be helpful Palermo et al.
(2021) Huang et al. (2018a). Exponential search space of mathematical concepts
is another key problem where reinforcement learning could be useful Wang et
al. (2018b). Reinforcement learning reduces a problem to state transition
problem with reward function. It initializes the state from a given problem,
and based on the semantic information extracted from the problem, it derives
actions from updating the state with a reward or penalty. It learns a policy
to maximize the reward as it traverses through several intermediate states.
For an MWP, the computation graph of expression trees denotes the state, and
action denotes the choices of mathematical transformations and their mapping
with operands from the input. The evaluation of the computation graph would
give an answer, and a penalty or reward would be given based on the comparison
between the computed answer and the expected ground truth. Researchers have
also leveraged reinforcement learning on difficult math domains like automated
theorem proving Crouse et al. (2021) Kaliszyk et al. (2018). It is still an
early stage of the application of reinforcement learning for solving math word
problems. It could bring the next set of opportunities to build systems
capable of mathematical reasoning ability.

##  6 Outlook

The mathematical reasoning process involves understanding natural language and
visual narratives and extracting entities with explicit and implicit
quantities building a self-consistent symbolic representation invoking
mathematical laws, axioms, and symbolic rules.

The ability to solve MWPs in an automated fashion could lead to many
applications in the education domain for content creation and delivering
learning outcomes Donda et al. (2020) Faldu et al. (2020a).

We surveyed sets of different approaches and their claims to have reasonable
solved MWPs on specific datasets. However, we have also highlighted studies
with “concrete evidence” that existing MWP solvers tend to rely on shallow
heuristics to achieve their high performance and questions these models’
capabilities to solve even the simplest of MWPs robustly Huang et al. (2016)
Patel et al. (2021) Miao et al. (2021).

Non-neural approaches do not improve linearly with larger training data Huang
et al. (2016), on the other hand, neural approaches have shown a promise to
solve even complex MWPs trained with very large corpus Lample and Charton
(2019) Saxton et al. (2019). However, the availability of extensive training
data with diverse MWPs is a crucial challenge. The inherent limitation of
neural models in terms of interpretability and explainability could be
addressed partly by predicting expression trees instead of directly predicting
answers Gaur et al. (2021). Such expression trees could be converted into
equations, and which are evaluated to infer the final answer. Further,
expression trees also provide scaffolding to assist mathematical reasoning
ability and open up design choices like graph-based encoder and tree-based
decoder, which helps to learn from the inadequately sized training corpus.
Interspersed natural language explanations and infusing explicit knowledge
could be of significant value as it not only improves the explainability of
the system but also guides the model to traverse through intermediate states
anchoring mathematical reasoning Ling et al. (2017) Gaur et al. (2021).
Interspersed explanations not only solve the math word problems but also
engage users on how to solve them.

We highlight that solving MWPs would require the inherent ability of
mathematical reasoning, which comprises natural language understanding, image
understanding, and the ability to invoke domain knowledge of mathematical
laws, axioms, and theorems. There is a massive scope of complementing neural
approaches with external expertise/knowledge and developing design choices as
we extend the problem to complex mathematical areas.

## References

  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  _arXiv preprint arXiv:1905.13319_.
  * Amnueypornsakul and Bhat (2014) Bussaba Amnueypornsakul and Suma Bhat. 2014.  Machine-guided solution to mathematical word problems.  In _Proceedings of the 28th Pacific Asia Conference on Language, Information and Computing_ , pages 111–119.
  * Bakman (2007) Yefim Bakman. 2007.  Robust understanding of word problems with extraneous information.  _arXiv preprint math/0701393_.
  * Bandura (2008) Albert Bandura. 2008.  Observational learning.  _The international encyclopedia of communication_.
  * Bobrow (1964) Daniel G Bobrow. 1964.  Natural language input for a computer problem solving system.
  * Briars and Larkin (1984) Diane J Briars and Jill H Larkin. 1984.  An integrated model of skill in solving elementary word problems.  _Cognition and instruction_ , 1(3):245–296.
  * Charniak (1968) Eugene Charniak. 1968.  _Calculus word problems_.  Ph.D. thesis, Ph. D. thesis, Massachusetts Institute of Technology.
  * Chen et al. (2005) Keh-Jiann Chen, Shu-Ling Huang, Yueh-Yin Shih, and Yi-Jun Chen. 2005.  Extended-hownet: A representational framework for concepts.  In _Proceedings of OntoLex 2005-Ontologies and Lexical Resources_.
  * Crouse et al. (2021) Maxwell Crouse, Ibrahim Abdelaziz, Bassem Makni, Spencer Whitehead, Cristina Cornelio, Pavan Kapanipathi, Kavitha Srinivas, Veronika Thost, Michael Witbrock, and Achille Fokoue. 2021.  A deep reinforcement learning approach to first-order logic theorem proving.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 6279–6287.
  * Csáji et al. (2001) Balázs Csanád Csáji et al. 2001.  Approximation with artificial neural networks.  _Faculty of Sciences, Etvs Lornd University, Hungary_ , 24(48):7.
  * Dellarosa (1986) Denise Dellarosa. 1986.  A computer simulation of children’s arithmetic word-problem solving.  _Behavior Research Methods, Instruments, & Computers_, 18(2):147–154.
  * Dhavala et al. (2020) Soma Dhavala, Chirag Bhatia, Joy Bose, Keyur Faldu, and Aditi Avasthi. 2020.  Auto generation of diagnostic assessments and their quality evaluation.  _International Educational Data Mining Society_.
  * Donda et al. (2020) Chintan Donda, Sayan Dasgupta, Soma S Dhavala, Keyur Faldu, and Aditi Avasthi. 2020\.  A framework for predicting, interpreting, and improving learning outcomes.  _arXiv preprint arXiv:2010.02629_.
  * English (2013) Lyn D English. 2013.  _Mathematical reasoning: Analogies, metaphors, and images_.  Routledge.
  * Faldu et al. (2020a) Keyur Faldu, Aditi Avasthi, and Achint Thomas. 2020a.  Adaptive learning machine for score improvement and parts thereof.  US Patent 10,854,099.
  * Faldu et al. (2021) Keyur Faldu, Amit Sheth, Prashant Kikani, and Hemang Akabari. 2021.  Ki-bert: Infusing knowledge context for better language and domain understanding.  _arXiv preprint arXiv:2104.08145_.
  * Faldu et al. (2020b) Keyur Faldu, Achint Thomas, and Aditi Avasthi. 2020b.  System and method for recommending personalized content using contextualized knowledge base.  US Patent App. 16/586,512.
  * Feigenbaum et al. (1963) Edward A Feigenbaum, Julian Feldman, et al. 1963.  _Computers and thought_.  New York McGraw-Hill.
  * Fletcher (1985) Charles R Fletcher. 1985.  Understanding and solving arithmetic word problems: A computer simulation.  _Behavior Research Methods, Instruments, & Computers_, 17(5):565–571.
  * Gaur et al. (2021) Manas Gaur, Keyur Faldu, and Amit Sheth. 2021.  Semantics of the black-box: Can knowledge graphs help make deep learning systems more interpretable and explainable?  _IEEE Internet Computing_ , 25(1):51–59.
  * Griffith and Kalita (2021) Kaden Griffith and Jugal Kalita. 2021.  Solving arithmetic word problems with transformers and preprocessing of problem text.  _arXiv preprint arXiv:2106.00893_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_.
  * Hong et al. (2021a) Yining Hong, Qing Li, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu. 2021a.  Learning by fixing: Solving math word problems with weak supervision.  In _AAAI Conference on Artificial Intelligence_.
  * Hong et al. (2021b) Yining Hong, Qing Li, Ran Gong, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu. 2021b.  Smart: A situation model for algebra story problems via attributed grammar.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13009–13017.
  * Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014\.  Learning to solve arithmetic word problems with verb categorization.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 523–533.
  * Huang et al. (2018a) Danqing Huang, Jing Liu, Chin-Yew Lin, and Jian Yin. 2018a.  Neural math word problem solver with reinforcement learning.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 213–223.
  * Huang et al. (2016) Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016.  How well do computers solve math word problems? large-scale dataset construction and evaluation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 887–896.
  * Huang et al. (2018b) Danqing Huang, Jin-Ge Yao, Chin-Yew Lin, Qingyu Zhou, and Jian Yin. 2018b.  Using intermediate representations to solve math word problems.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 419–428.
  * Kahneman (2011) Daniel Kahneman. 2011.  _Thinking, fast and slow_.  Macmillan.
  * Kaliszyk et al. (2018) Cezary Kaliszyk, Josef Urban, Henryk Michalewski, and Mirek Olšák. 2018\.  Reinforcement learning of theorem proving.  _arXiv preprint arXiv:1805.07563_.
  * Khashabi et al. (2019) Daniel Khashabi, Erfan Sadeqi Azer, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2019.  On the capabilities and limitations of reasoning for natural language understanding.  _arXiv preprint arXiv:1901.02522_.
  * Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015.  Parsing algebraic word problems into equations.  _Transactions of the Association for Computational Linguistics_ , 3:585–597.
  * Koncel-Kedziorski et al. (2016) Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016.  Mawps: A math word problem repository.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1152–1157.
  * Kushman et al. (2014) Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014.  Learning to automatically solve algebra word problems.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 271–281.
  * Lample and Charton (2019) Guillaume Lample and François Charton. 2019.  Deep learning for symbolic mathematics.  _arXiv preprint arXiv:1912.01412_.
  * Li et al. (2019) Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. 2019.  Modeling intra-relation in math word problems with different functional multi-head attentions.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6162–6167.
  * Liang et al. (2018) Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, and Keh-Yih Su. 2018.  A meaning-based statistical english math word problem solver.  _arXiv preprint arXiv:1803.06064_.
  * Liang et al. (2021) Zhenwen Liang, Jipeng Zhang, Jie Shao, and Xiangliang Zhang. 2021.  Mwp-bert: A strong baseline for math word problems.  _arXiv preprint arXiv:2107.13435_.
  * Liguda and Pfeiffer (2012) Christian Liguda and Thies Pfeiffer. 2012.  Modeling math word problems with augmented semantic networks.  In _International Conference on Application of Natural Language to Information Systems_ , pages 247–252. Springer.
  * Lin et al. (2015) Yi-Chung Lin, Chao-Chun Liang, Kuang-Yi Hsu, Chien-Tsung Huang, Shen-Yun Miao, Wei-Yun Ma, Lun-Wei Ku, Churn-Jung Liau, and Keh-Yih Su. 2015.  Designing a tag-based statistical math word problem solver with reasoning and explanation.  In _International Journal of Computational Linguistics & Chinese Language Processing, Volume 20, Number 2, December 2015-Special Issue on Selected Papers from ROCLING XXVII_.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_.
  * Lithner (2000) Johan Lithner. 2000.  Mathematical reasoning in task solving.  _Educational studies in mathematics_ , pages 165–190.
  * Liu et al. (2020) Qianying Liu, Wenyu Guan, Sujian Li, Fei Cheng, Daisuke Kawahara, and Sadao Kurohashi. 2020.  Reverse operation based data augmentation for solving math word problems.  _arXiv preprint arXiv:2010.01556_.
  * Liu et al. (2019) Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. 2019.  Tree-structured decoding for solving math word problems.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2370–2379.
  * Miao et al. (2021) Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2021.  A diverse corpus for evaluating and developing english math word problem solvers.  _arXiv preprint arXiv:2106.15772_.
  * Mitra and Baral (2016) Arindam Mitra and Chitta Baral. 2016.  Learning to use formulas to solve simple arithmetic problems.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2144–2153.
  * Narang et al. (2020) Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020.  Wt5?! training text-to-text models to explain their predictions.  _arXiv preprint arXiv:2004.14546_.
  * Palermo et al. (2021) Joseph Palermo, Johnny Ye, and Alok Singh. 2021.  A reinforcement learning environment for mathematical reasoning via program synthesis.  _arXiv preprint arXiv:2107.07373_.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.  Are nlp models really able to solve simple math word problems?  _arXiv preprint arXiv:2103.07191_.
  * Qin et al. (2021) Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin. 2021.  Neural-symbolic solver for math word problems with auxiliary tasks.  _arXiv preprint arXiv:2107.01431_.
  * Qin et al. (2020) Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020.  Semantically-aligned universal tree-structured solver for math word problems.  _arXiv preprint arXiv:2010.06823_.
  * Ran et al. (2019) Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019.  Numnet: Machine reading comprehension with numerical reasoning.  _arXiv preprint arXiv:1910.06701_.
  * Robaidek et al. (2018) Benjamin Robaidek, Rik Koncel-Kedziorski, and Hannaneh Hajishirzi. 2018.  Data-driven methods for solving algebra word problems.  _arXiv preprint arXiv:1804.10718_.
  * Roy and Roth (2017) Subhro Roy and Dan Roth. 2017.  Unit dependency graph and its application to arithmetic word problem solving.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Roy and Roth (2018) Subhro Roy and Dan Roth. 2018.  Mapping to declarative knowledge for word problem solving.  _Transactions of the Association for Computational Linguistics_ , 6:159–172.
  * Roy et al. (2015) Subhro Roy, Tim Vieira, and Dan Roth. 2015.  Reasoning about quantities in natural language.  _Transactions of the Association for Computational Linguistics_ , 3:1–13.
  * Sap et al. (2020) Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020.  Commonsense reasoning for natural language processing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , pages 27–33.
  * Saxton et al. (2019) David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019.  Analysing mathematical reasoning abilities of neural models.  _arXiv preprint arXiv:1904.01557_.
  * Sheth et al. (2021) Amit Sheth, Manas Gaur, Kaushik Roy, and Keyur Faldu. 2021.  Knowledge-intensive language understanding for explainable ai.  _IEEE Internet Computing_ , (01):1–1.
  * Shi et al. (2015) Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015.  Automatically solving number word problems by semantic parsing and reasoning.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1132–1142.
  * Storks et al. (2019) Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019.  Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches.  _arXiv preprint arXiv:1904.01172_ , pages 1–60.
  * Upadhyay and Chang (2015) Shyam Upadhyay and Ming-Wei Chang. 2015.  Draw: A challenging and diverse algebra word problem set.  Technical report, Citeseer.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_.
  * Wang et al. (2018a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018a.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Wang et al. (2018b) Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. 2018b.  Mathdqn: Solving arithmetic word problems via deep reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Wang et al. (2020) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2020.  K-adapter: Infusing knowledge into pre-trained models with adapters.  _arXiv preprint arXiv:2002.01808_.
  * Wang et al. (2017) Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017.  Deep neural solver for math word problems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 845–854.
  * Wu et al. (2021) Qinzhuo Wu, Qi Zhang, Zhongyu Wei, and Xuan-Jing Huang. 2021.  Math word problem solving with explicit numerical values.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5859–5869.
  * Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019.  A goal-driven tree-structured neural model for math word problems.  In _IJCAI_ , pages 5299–5305.
  * Yuhui et al. (2010) Ma Yuhui, Zhou Ying, Cui Guangzuo, Ren Yun, and Huang Ronghuai. 2010.  Frame-based calculus of solving arithmetic multi-step addition and subtraction word problems.  In _2010 Second International Workshop on Education Technology and Computer Science_ , volume 2, pages 476–479. IEEE.
  * Zhang et al. (2020) Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020.  Graph-to-tree learning for solving math word problems.  Association for Computational Linguistics.
  * Zhao et al. (2020) Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020.  Ape210k: A large-scale and template-rich dataset of math word problems.  _arXiv preprint arXiv:2009.11506_.
  * Zhou et al. (2015) Lipu Zhou, Shuaixiang Dai, and Liwei Chen. 2015.  Learn to solve algebra word problems using quadratic programming.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 817–822.