# Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem
Solvers

Vivek Kumar, Rishabh Maheshwary and Vikram Pudi
Data Sciences and Analytics Center, Kohli Center on Intelligent Systems
International Institute of Information Technology, Hyderabad, India
{vivek.k<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Existing Math Word Problem (MWP) solvers have achieved high accuracy on
benchmark datasets. However, prior works have shown that such solvers do not
generalize well and rely on superficial cues to achieve high performance. In
this paper, we first conduct experiments to showcase that this behaviour is
mainly associated with the limited size and diversity present in existing MWP
datasets. Next, we propose several data augmentation techniques broadly
categorized into Substitution and Paraphrasing based methods. By deploying
these methods we increase the size of existing datasets by five folds.
Extensive experiments on two benchmark datasets across three state-of-the-art
MWP solvers shows that proposed methods increase the generalization and
robustness of existing solvers. On average, proposed methods significantly
increase the state-of-the-art results by over five percentage points on
benchmark datasets. Further, the solvers trained on the augmented dataset
performs comparatively better on the challenge test set. We also show the
effectiveness of proposed techniques through ablation studies and verify the
quality of augmented samples through human evaluation.

##  1 Introduction

A Math Word Problem (MWP) consists of natural language text which describes a
world state involving some known and unknown quantities, followed by a
question text to determine the unknown values. The task is to parse the
problem statement and generate equations that can help find the value of
unknown quantities. An example of a simple MWP is shown in Table 1. In recent
years, the challenge of solving MWP has gained much attention in the NLP
community as it needs the development of commonsense multi step reasoning with
numerical quantities. With the rise of deep learning, performance of math
solvers has also increased significantly over the years Wang et al. ; Zhang et
al. . However, recent analysis conducted in Kumar et al. (2021) and Patel et
al. (2021) show that these deep learning based solvers rely on shallow
heuristics to solve vast majority of problems. They curated adversarial
examples and SVAMP challenge set respectively to infer that MWP solvers $(1)$
do not understand the relationship between numbers and their associated
entities, $(2)$ do not focus on the question text and $(3)$ ignore word order
information. In this paper, we first conduct experiments to establish that the
above drawbacks are due to the limited size and diversity of problems present
in the existing MWP datasets. Next, we propose various augmentation methods to
create diverse and large number of training examples to mitigate these
shortcomings. Our methods are focused on: $(1)$ Increasing the number of
problems in the existing datasets and $(2)$ enhancing the diversity of the
problem set.

| Original Problem
---
Problem: Nancy grew 8 potatoes. Sandy grew 5 potatoes.
How many potatoes did they grow in total ?
True Equation: X = 8+5
| Paraphrasing Method
---
Problem: How many potatoes did they grow in all given that
nancy grew 8 potatoes and sandy grew 5 potatoes.
Equation Label: X = 8+5
| Substitution Method
---
Problem: Dwight grew 8 potatoes. Juliette grew 5 potatoes.
How many potatoes did they grow together ?
Equation Label: X = 8+5
Table 1: A MWP and its augmentation examples generated by our methods with
preserved equation labels. Blue and Violet colours denote the changes made
after the primary stage and secondary stage respectively.

Training deep neural models effectively requires large number of data points
Longpre et al. (2020). Constructing large datasets which are annotated,
labeled and have MWPs of similar difficulty level is a very expensive and
tedious task. To address these key challenges, we resort to data augmentation
techniques. Our motivation behind generating augmentations is that humans
require sufficient practice to understand MWPs. Humans learn to solve MWPs by
going through a variety of similar examples and slowly become capable enough
to tackle variations of similar difficulty levels. We aim to generate
augmentations such that sufficient linguistic variations of a similar problem
are present in the dataset. These variations will make the solver more robust
in tackling MWP, increase their reasoning ability and numerical understanding.

Data augmentation for MWPs is a challenging task as we need to preserve the
equation labels while generating new samples Kumar et al. (2021). The
generated samples should be $(1)$ semantically similar to their original
counterpart, $(2)$ must have the same numerical values and preserve
relationship with their respective entities and $(3)$ should maintain the same
sequence of events in the problem text. Existing augmentation methods Wei and
Zou  cannot be directly applied due to the above mentioned reasons. Our
methods can be broadly classified as follows:

  * •

Paraphrasing Methods: It generates variations of the question text by re-
statement such that the semantic and syntactic meaning along with the equation
labels is preserved.

  * •

Substitution Methods: These methods generate variations of the problem
statement by identifying and substituting some of the keywords such that the
augmentations are semantically and syntactically correct.

To ensure high quality augmentations 111Codebase and augmented datasets are
available at: https://github.com/kevivk/MWP-Augmentation, we propose a
selection algorithm which selects samples that have high similarity with
original problem and incur high loss values when tested on existing solvers.
This algorithm helps selecting only those samples that can make existing
solvers more robust. Further, we also verify the validity and the quality of
generated augmentations through human evaluation.

Most of the existing MWP datasets are either in languages other than English
or contain problems of varying difficulty levels Koncel-Kedziorski et al.
(2016); Wang et al. ; Huang et al. (2016); Amini et al. (2019); Miao et al. .
We focus on strengthening existing English language datasets which can
facilitate the development of better MWP solvers. We consider datasets
containing MWP that can be solved using linear equations in one variable.
These datasets include MaWPS Koncel-Kedziorski et al. (2016) and ASDiv-A Miao
et al.  both having $2,373$ and $1,213$ problems respectively. Following are
the key contributions made in this paper:

  * •

To the best of our knowledge, this is the first work that extensively
evaluates data augmentation techniques for MWP solving. This is the first
attempt to generate MWP problems automatically without manual intervention.

  * •

Accuracy of the state of the art solvers increases after training on the
proposed augmented dataset. This demonstrates the effectiveness of our
methods. To verify the validity of generated augmentations we conduct human
evaluation studies.

  * •

We increase the diversity of the training dataset through augmentations and
obtain comparatively better results than state-of-the-art solvers on the SVAMP
challenge set.

##  2 Related Work

Math Word Solvers: Many research efforts have been undertaken in the recent
past to solve the challenging MWP task. Broadly, these solvers can be
categorized into statistical learning based and deep learning based models.
Traditional approaches focused more on statistical machine learning Kushman et
al. (2014); Hosseini et al. (2014) with the aim of categorizing equations into
templates and extracting key patterns in the problem text. Recently, due to
the advent of deep learning in NLP, solvers have witnessed a considerable
increase in their performances. Wang et al.  modelled MWP task as a sequence
to sequence task and used LSTM’s Hochreiter and Schmidhuber (1997) for
learning problem representations. Chiang and Chen (2018) focused on learning
representations for operators and operands.Wang et al. (2019); Xie and Sun
(2019) used tree structures for decoding process. Zhang et al.  modelled
question as a graph to map quantities and their attributes. Existing datasets
which have been used as benchmark for english language includes MaWPS Koncel-
Kedziorski et al. (2016) and Chinese language dataset Math23K Wang et al. .
These datasets although constrained by their size deal with algebraic problems
of similar difficulty levels. Recently, ASDiv Miao et al.  has been proposed,
which has diverse problems which includes annotations for equations, problem
type and grade level. Other large datasets in English language include MathQA
Amini et al. (2019) and Dolphin18k Huang et al. (2016). Although, these
datasets have larger problem set but they are noisy and contain problems of
varied difficulty levels.

Text Data Augmentation: To effectively train deep learning models, large
datasets are required. Data augmentation is a machine learning technique that
artificially enlarges the amount of training data by means of label preserving
transformations Taylor and Nitschke (2018). Longpre et al. (2020) hypothesize
that textual data augmentation would only be helpful if the generated data
contains new linguistic patterns that are relevant to the task and have not
been seen in pre-training. In NLP, many techniques have been used for
generating augmentations, Wei and Zou  introduced noise injection, deletion,
insertion and swapping of words in text. Rizos et al.  used recurrent neural
networks and generative adversarial networks for short-text augmentation
Maheshwary et al. (2021b). Recently, hard label adversarial attack models have
also been used Maheshwary et al. (2021a). Other frequently used methods
include inducing spelling mistakes Belinkov and Bisk (2018), synonym
replacement Zhang et al. (2016), identifying close embeddings from a defined
search space Alzantot et al. (2018), round trip translations Sennrich et al.
(2016), paraphrasing techniques Kumar et al. (2019) and words predicted by
language model Kobayashi (2018) among many others. These methods are specific
to the task at hand and needs to be adapted such that the generated
augmentations bring diversity in the concerned dataset.

##  3 Proposed Augmentation Approach

Data augmentation generates new data by modifying existing data points through
transformations based on prior knowledge about the problem domain. We
introduce carefully selected transformations on well known text augmentation
techniques to develop examples suited for the task of MWP. These
transformations help in increasing the diversity and size of problem set in
existing datasets.

###  3.1 Problem Definition

A MWP is defined as an input of $n$ tokens,
$\mathcal{P}=\\{w_{1},w_{2}..w_{n}\\}$ where each token $w_{i}$ is either a
numeric value or a word from a natural language. The goal is to generate a
valid mathematical equation $\mathcal{E_{P}}$ from $\mathcal{P}$ such that the
equation consists of numbers from $\mathcal{P}$, desired numerical constants
and mathematical operators from the set $\\{/,*,+,-,=,(,)\\}$. Let
$\mathcal{F}:\mathcal{P}\rightarrow\mathcal{E_{P}}$ be an MWP solver where
$\mathcal{E_{P}}$ is the equation to problem $\mathcal{P}$. Our task is to
generate augmented problem statement $\mathcal{P}^{*}$ from the original input
$\mathcal{P}$ such that $\mathcal{P^{*}}$ is: $(1)$ semantically similar to
the initial input $\mathcal{P}$, $(2)$ preserves the sequence of events in the
problem statement, $(3)$ keeps the numerical values intact and $(4)$ the
solution equation is same as $\mathcal{E_{P}}$.

###  3.2 Deficiencies in Existing Models

As showcased by Patel et al. (2021), existing MWP solvers trained on benchmark
datasets like MaWPS and ASDiv-A focus their attention only on certain keywords
in the problem statement and do not pay much heed to the question text. We
further show that even after performing significant transformations on the
test set such as $(1)$ dropping the question text, $(2)$ randomly shuffling
the sequence of sentences, $(3)$ random word deletion, and $(4)$ random word
reordering, the solvers are still able to produce correct equations. Upon
introducing these transformations we should expect a very high drop in
accuracy values as the transformed problems are now distorted. Surprisingly,
the decrease in accuracy scores is relatively very less than expected as shown
in Table 2. We only observe a relatively moderate drop for word reordering.
From this analysis, we can say that instead of focusing on the sequence of
events, question text and semantic representation of the problem, solvers pick
word patterns and keywords from the problem statement. We hypothesize that the
drop in accuracy for word reordering experiment indicates that the solvers try
to identify a contiguous window of words having some keywords and numbers in
them, and generates equation based on these keywords. We further probe on this
hypothesis by visualizing the attention weights in the experiment section.

Dataset | Eval Type | Seq2Seq | GTS | Graph2Tree
---|---|---|---|---
MaWPS | True | 84.6 | 87.5 | 88.7
WD | 80.2 | 81.5 | 77.3
QR | 77.4 | 82.0 | 80.2
SS | 77.0 | 60.4 | 66.4
WR | 54.9 | 34.8 | 39.3
ASDiv-A | True | 70.6 | 80.3 | 82.7
WD | 60.2 | 61.3 | 56.7
QR | 58.7 | 52.4 | 54.1
SS | 56.2 | 59.3 | 60.7
WR | 47.1 | 32.3 | 34.6
Table 2:  Performance of solvers on modified test sets. True represents
unaugmented test set. WD,QR,SS,WR represent word deletion, question
reordering, sentence shuffling and word reordering respectively.

###  3.3 Augmentation Methods

A MWP can also be expressed as $\mathcal{P}=(S_{1},S_{2}..S_{k},Q)$ where $Q$
is the question and $(S_{1},S_{2}..S_{k})$ are the sentences constituting the
problem description. To mitigate the deficiencies in MWP solvers, we propose a
two stage augmentation paradigm consisting of primary and secondary stage. In
primary stage, we generate base augmentation candidates which then proceed to
the secondary stage and get modified accordingly to become potential
candidates. After identifying the potential candidates, we filter out the best
candidates using proposed candidate selection algorithm. Table 1 shows changes
in MWP after primary and secondary stage. Following are the details:

  * •

Primary Stage: In the primary stage, our focus is on inducing variations in
the question text $Q$ of a given problem statement $\mathcal{P}$. For this, we
first generate $n$ base candidates ${\\{b_{1},b_{2},...,b_{n}\\}}$ from $Q$
using $T5$ paraphrasing model Raffel et al. (2020). The key intuition behind
this step is to ensure that each augmentation of a given problem has a
different question text. This will empower the solver to learn variations from
the question text as well.

  * •

Secondary Stage: After the generation of base candidates, we implement
augmentation methods to generate potential candidates. These methods although
well known, require careful tuning to adapt for MWP generation. Table 3
showcases MWP examples and their generated augmentations. Detailed description
of these techniques follow.

Category |  Augmentation Method |  Example
---|---|---
Paraphrasing Methods |  Round trip Translation |  Original: The schools debate team had 4 boys and 6 girls on it. If they were split into groups of 2, how many groups could they make ?
Augmented: The school discussion group consisted of 4 boys and 6 girls. If
they are divided into groups of 2 . How many groups could they have created ?
Problem Reordering |  Original: Lucy has an aquarium with 5 fish . She wants to buy 1 more fish . How many fish would Lucy have then ?
Augmented: If lucy has an aquarium with 5 fish and she wants to buy 1 more
fish then how many fish would lucy have ?
Substitution Methods |  Fill Masking |  Original: There are 8 walnut trees currently in the park . Park workers will plant 3 more walnut trees today . How many walnut trees will the park have when the workers are finished ?
Augmented: There are 8 walnut trees currently in the park . Park workers will
plant 3 more walnut trees soon . How many walnut trees will the park have
after the workers are finished ?
Named-Entity Replacement |  Original: Sally found 7 seashells , Tom found 12 seashells , and Jessica found 5 seashells on the beach . How many seashells did they find together ?
Augmented: Edd found 7 seashells , Alan found 12 seashells , and Royal found 5
seashells on the beach . How many seashells were found together ?
Synonym Replacement |  Original: Katie ’s team won their dodgeball game and scored 25 points total . If Katie scored 13 of the points and everyone else scored 4 points each , how many players were on her team ?
Augmented: Katie’s group won their rumble game and scored 25 points total . If
Katie scored 13 of the points and all else scored 4 points each, How many
players was on her group ?
Table 3: Augmentation examples from all proposed methods. Coloured text
represents the changes in problem statement.

####  3.3.1 Paraphrasing Methods

Paraphrasing has proved to be an effective way of generating text
augmentations Witteveen and Andrews . It generates samples having diverse
sentence structures and word choices while preserving the semantic meaning of
the text. These additional samples guide the model to pay attention to not
only the keywords but its surroundings as well. This is particularly
beneficial for the task of MWP solving, where most of the problem statements
follow a general structure.

Problem Reordering: Given original problem statement
$\mathcal{P}=(S_{1},S_{2},...S_{k},Q)$, we alter the order of problem
statement such that $\mathcal{P^{*}}=(Q,S_{1},S_{2},...,S_{k})$. To preserve
the semantic and syntactic meaning of problem statement we use filler phrases
like ’Given that’ and ’If-then’. To make these paraphrases more fluent, we use
named entity recognition and co-reference resolution to replace the
occurrences of pronouns with their corresponding references. Please note that
this method is better than random shuffling of sentences as it preserves the
sequence of events in the problem statement.

Round Trip Translations: Round trip translations, more commonly referred as
back-translation is an interesting method to generate paraphrases. This idea
has evolved as a result of the success of machine translation models Wu et al.
(2016). In this technique, sentences are translated from their original
language to foreign languages and then translated back to the original
language. This round trip can be between multiple languages as well. The
motivation behind using this technique is to utilize the different structural
constructs and linguistic variations present in other languages.
Back-translation is known to diverge uncontrollably Tan et al. (2019) for
multiple round trips. This may lead to change in the semantics of the problem
statement. Numerical quantities are fragile to translations and their order
and representation may change. To overcome these challenges, we worked with
languages that have structural constructs similar with English. For instance,
languages like Finnish which are gender neutral, can become problematic as
they can lead to semantic variance in augmented examples. To preserve
numerical quantities, we replace them with special symbols and keep a map to
restore numerical quantities in the generated paraphrases. We have used the
following round trips:

English - Russian - English: Although Russian is linguistically different from
English, we still chose it as word order does not affect the syntactic
structure of a sentence in Russian language Voita et al. (2019). For single
round trip, we preferred Russian as it has the potential to generate different
paraphrase structures.

English - German - French - English: German and french are structurally
similar to English language Kim et al. (2019), we chose them for multiple
round trips to both maintain semantic in-variance and induce minor alterations
in the paraphrases.

####  3.3.2 Substitution Methods

In this class of methods, the focus is on generating variations of the problem
statement by identifying and substituting some of the keywords such that the
augmentations are semantically and syntactically correct, with the equation
labels preserved. Substitution is effective for MWP solving as it guides the
solvers focus away from certain keywords, allowing it to distribute its
attention and generalize better. We propose the following methods:

Fill-Masking: In this technique, we model the challenge of generating
candidates as a masked language modelling problem. Instead of randomly
choosing words for masking, we use part of speech tags to focus on nouns and
adjectives, preferably in the vicinity of numerical quantities. We replace
these identified keywords with mask tokens. These masked candidate sentences
are then passed through a masked language model Devlin et al. (2019a) and
suitable words are filled in masked positions to generate our candidate
sentences.

Synonym Replacement: In this method, after stop-word removal, we select
keywords randomly for substitution. Unlike fill-mask technique, where masked
language models were deployed, here we use Glove embeddings Pennington et al.
(2014) to find the top $k$ candidates that are close synonyms of the keywords.
To ensure syntactic correctness in candidates, we maintain the part of speech
tags for the substitute candidates. These synonyms are then used to substitute
the keywords in the problem statement and generate augmented candidates.

Named-Entity Replacement: A common occurrence in MWP is the usage of named
entities. These entities play a crucial role in stating the problem statement,
but the solution equations do not change on altering these entities. Following
this insight, we first identify the named entities222https://www.nltk.org/
such as person, place and organizations present in the problem statement. Then
we replace these named entities with their corresponding substitutes, like a
person’s name is replaced by another person’s name to generate the potential
candidates.
Table 4 reports the statistics of augmented datasets on both MaWPS and
ASDiv-A. All the techniques described in paraphrasing and substitution methods
are used for generating the potential candidates for a problem statement.
After generation of the potential candidates for augmenting a problem
statement, the best possible candidate is selected by using Algorithm 1. Key
motivation behind developing this algorithm is to select candidates on which
the solver does not perform well and which are similar to the original problem
statement.

We use negative log likelihood as the loss function $\mathcal{L}$ and
Sentence-BERT Reimers and Gurevych (2019) fine tuned on MWP equation
generation task as sentence embedding generator $\mathcal{S}$. We calculate
the similarity of each candidate embedding with the original problem
representation using cosine similarity as shown in Line $3$ of the algorithm.
Further, for each candidate sentence, we evaluate their loss values and select
the candidate with the maximum mean normalized loss and similarity score.

Dataset | Problem Size | Vocabulary Size
---|---|---
MaWPS | 2,373 | 2,632
ASDiv-A | 1,213 | 2,893
Paraphrase | 5,909 | 3,832
Substitution | 6,647 | 3,923
Combined-MaWPS | 10,634 | 5,626
Combined-ASDiv | 5,312 | 6,109
Table 4: Statistics of augmented dataset compared with MaWPS and ASDiv-A.
Combined-Dataset represents combination of Paraphrase and Substitution
methods. Algorithm 1 MWP Candidate Selection Algorithm

Requires: $\mathcal{M}$ is augmentation method, $\mathcal{S}$ is similarity
model, $\mathcal{F}$ is solver model, $\mathcal{L}$ is Loss function.
Input: Problem text $\mathcal{P}$
Output: Augmented Text $\mathcal{P^{*}}$

1:$\mathcal{E}_{P}\leftarrow\mathcal{F}(\mathcal{P})$

2:$Candidates\leftarrow\mathcal{M}\mathcal{(P)}$

3:for $C_{j}\ in\ Candidates:$ do

4: $S_{j}\leftarrow\mathcal{S}(C_{j},\mathcal{P})$

5: $L_{j}\leftarrow(\mathcal{L}(C_{j})-\mathcal{L}(P))/\mathcal{L}(P)$

6: $CandidateScore.add(S_{j}*L_{j})$

7:$\mathcal{P^{*}}=\underset{C_{j}}{\arg\max}\ CandidateScore(C_{j})$

8:end

##  4 Experiments

Datasets and Models: To showcase the effectiveness of proposed augmentation
methods, we select three state-of-the-art MWP solvers: $(1)$ _Seq2Seq_ Wang et
al.  having an LSTM encoder and an attention based decoder. $(2)$ _GTS_ Xie
and Sun (2019) having an LSTM encoder and a tree based decoder and $(3)$
_Graph2tree_ Zhang et al.  consists of a both tree based encoder and decoder.
_Seq2Seq_ serves as our base model for experimentation. Many existing datasets
are not suitable for our analysis as either they are in Chinese (Wang et al.,
) or they have problems of higher complexities (Huang et al., 2016) . We
conduct experiments across the two largest available English language datasets
satisfying our requirements: $(1)$ _MaWPS_ Koncel-Kedziorski et al. (2016)
containing $2,373$ problems $(2)$ _ASDiv-A_ Miao et al.  containing $1,213$
problems. Both datasets have MWPs with linear equation in one variable.

Experiment Setup: We train and evaluate the three solvers on both MaWPS and
ASDiv-A using five fold cross validation. Evaluation is conducted on both
original and augmented datasets. We use the same hyperparameter values as
recommended in the original implementation of these solvers. Further, each
solver has been trained from scratch and by using BERT embeddings Devlin et
al. (2019b). We also evaluate the models on _SVAMP_ Patel et al. (2021)
challenge set. This test set has been designed specifically to examine the
robustness and adaptability of the solvers. Ablation studies have been
conducted to assess the effectiveness of candidate selection algorithm and
augmentation techniques.

Dataset | Evaluation Type | Seq2Seq | GTS | G2T
---|---|---|---|---
MaWPS | True | 84.6 | 87.5 | 88.7
Paraphrasing | 88.3 | 90.4 | 92.6
Substitution | 89.2 | 89.7 | 91.7
Combined | 91.3 | 92.6 | 93.5
ASDiv-A | True | 70.6 | 80.3 | 82.7
Paraphrasing | 75.6 | 84.2 | 83.6
Substitution | 73.2 | 83.3 | 84.1
Combined | 78.2 | 85.9 | 86.3
Table 5: Result of augmentation methods. True is original dataset, Combined is
combination of paraphrasing and substitution. G2T represents Graph2Tree
solver.

###  4.1 Results and Analysis

Table 5 shows the result of proposed methods. These results have been reported
on BERT embeddings. Table 11 shows a comparison between training from scratch
and using BERT embeddings. By training these state-of-the-art models on the
augmented dataset we achieve better results for both MaWPS and ASDiv-A. On
average, we were able to increase the accuracy significantly by more than five
percentage points. Both paraphrasing and substitution methods have performed
well independently and in combination. Further, we conduct ablation studies to
analyze the performance of each augmentation method.

|  Problem 1: Ricardo was making baggies of cookies with
---
5 cookies in each bag. If he had 7 chocolate chip cookies
and 3 oatmeal cookies, how many baggies could he make ?
Solution Equation: X = (7+3)/5
Pre Augmentation Equation: X = (7/3)/3
Post Augmentation Equation: X = (7+3)/5
|  Problem 2: For halloween Destiny bought 9 pieces of candy.
---
She ate 3 pieces the first night and then her sister
gave her 2 more pieces. How many pieces of candy does
Destiny have now ?
Solution Equation: X = 9-3+2
Pre Augmentation Equation: X = ((9+3-3
Post Augmentation Equation: X = (9+3-2)
|  Problem 3 : Audrey needs 6 cartons of berries to make
---
a berry cobbler. She already has 2 cartons of strawberries
and 3 cartons of blueberries. How many more cartons of
berries should Audrey buy ?
Solution Equation: X = 6-2-3
Pre Augmentation Equation: X = (6-(2)+3)
Post Augmentation Equation: X = 6-(2+3)
Table 6: Examples illustrating equation results before and after training on
the full augmented dataset.

In Table 6 we illustrate some examples on which existing models generate
incorrect equations. However, after being trained with augmented dataset they
generate correct equations. Additionally, in Problem 2 the base model
generates syntactically incorrect solution, but post augmentation it generates
syntactically correct equation. These examples show the increased robustness
and solving abilities of solvers.

|  Problem: Gavin has 6 shirts . 3 are blue the rest are
---
green. How many green shirts does Gavin have ?
Mean attention values: 0.27 0.14 0.08
Problem: Gavin has 6 shirts . 3 are blue the rest are
green. How many green shirts does Gavin have ?
Augmented mean attention values : 0.23 0.18 0.11
|  Problem: There are 3 pencils in the drawer. Sara
---
placed 7more pencils in the drawer. How many pencils
are there in all ?
Mean attention values: 0.45 0.11 0.05
Problem: There are 3 pencils in the drawer. Sara
placed 7 more pencils in the drawer. How many pencils
are there in all ?
Augmented mean attention values : 0.31 0.16 0.09
Table 7: Examples illustrating distribution of top three attention weights
before and after training on the full augmented dataset.

Attention Visualizations: Through this investigation, we aim to ascertain our
hypothesis that to generate equations MWP solvers focus only on certain
keywords and patterns in a region. They ignore essential information like
semantics, sequence of events and content of the question text present in the
problem statement. In Table 7, we show some sample problem statements with
their attention weights. These weights are generated during the decoding
process using Luong attention mechanism Luong et al. (2015). Moreover, to
illustrate the effectiveness of our augmentation techniques, we show the
distribution of attention weights for models trained on the augmented dataset.
We can infer from the examples showcased in Table 7 that before augmentation
the focus of the solver is limited to a fixed region around numerical
quantities and it does not pay heed to the question text. However, after
training on the augmented dataset the solver has a better distribution of
attention weights, the weights are not localised and and the model is also
able to pay attention on the question text.

Ablation Studies: To assert the effectiveness of our methods, we conduct the
following ablations:
Candidate Selection Algorithm: For testing the usefulness of candidate
selection algorithm, we compare it with a random selection algorithm. In this,
we randomly select one of the possible candidates as augmented problem
statement. We evaluate the accuracy of models trained on the augmented
datasets, generated using both the algorithms. Result in Table 8, shows that
candidate selection algorithm performs better than random selection algorithm
and this demonstrates the effectiveness of our algorithm.

Augmentation Methods: To examine the effectiveness of proposed augmentation
techniques, we evaluate the models on each of the proposed techniques
independently and report the results in Table 9. Although, all the methods
contribute towards increase in accuracy but Round trip translations and
synonym replacement perform marginally better than others. This behaviour can
be linked to the structural diversity and keyword sensitivity that round trip
translations and synonym replacement bring respectively Feng et al. (2021).

Method | Eval Type | Seq2Seq | GTS | Graph2Tree
---|---|---|---|---
RSA | True | 84.6 | 87.5 | 88.7
Paraphrasing | 85.3 | 88.1 | 89.2
Substitution | 86.8 | 87.3 | 87.9
Combined | 87.0 | 89.2 | 89.5
CSA | True | 84.6 | 87.5 | 88.7
Paraphrasing | 88.3 | 90.4 | 92.6
Substitution | 89.2 | 89.7 | 91.7
Combined | 91.3 | 92.6 | 93.5
Table 8: Ablation Study for Random Selection Algorithm (RSA) and Candidate Selection Algorithm (CSA). Augmentation | Seq2Seq | GTS | Graph2Tree
---|---|---|---
True | 84.6 | 87.5 | 88.7
RRT | 86.5 | 89.1 | 91.6
PR | 85.9 | 88.4 | 90.7
FM | 84.8 | 87.2 | 89.1
SR | 85.2 | 90.1 | 91.2
NER | 86.1 | 88.3 | 89.7
Table 9: Result of Ablation study for each augmentation method. True represents unaugmented MaWPS dataset, RRT, PR, FM, SR, NER represents round trip translations, problem reordering, fill masking,synonym replacement and named entity replacement respectively. Augmentation | Seq2Seq | GTS | Graph2Tree
---|---|---|---
True | 37.5 | 39.6 | 41.2
MaWPS(P+S) | 39.2 | 40.1 | 42.3
ASDiv-A(P+S) | 37.8 | 40.4 | 42.1
Combined | 40.2 | 41.3 | 43.8
Table 10: Result of augmentations on SVAMP Challenge Set. P and S represent
paraphrasing and substitution methods. Combined represents augmented MaWPS and
ASDiv-A. True is combined MaWPS and ASDiv-A.

SVAMP Challenge Set: SVAMP Patel et al. (2021) is a manually curated challenge
test set consisting of $1,000$ math word problems. These problems have been
cherry picked from MaWPS and ASDiv-A, then altered manually to modify the
semantics of question text and generate additional equation templates. This
challenge set is suitable for evaluating a solver’s performance as it modifies
problem statements such that solver’s generalization can be checked. The
results are shown in Table 10. Although, our proposed augmented dataset has
very limited equation templates, still it performs comparatively better than
state-of-the-art models on SVAMP challenge set. This result signifies the need
for a larger and diverse dataset with enhanced variety of problems. Further,
it demonstrates the effectiveness of our method which is able to perform
better on SVAMP test set and increase model’s accuracy despite the challenges.

Augmentation Method | MaWPS | ASDiv-A
---|---|---
Scratch | BERT | Scratch | BERT
True | 77.2 | 84.6 | 53.2 | 70.6
Paraphrasing | 79.8 | 88.3 | 58.1 | 75.6
Substitution | 81.3 | 89.2 | 57.3 | 73.2
Combined | 82.7 | 91.3 | 60.4 | 78.2
Table 11: Performance comparison of baseline model trained from scratch and
trained using BERT embeddings. True represents unaugmented dataset.

BERT Embeddings: We train the solvers in two different settings, using pre-
trained BERT embeddings and training from scratch. We chose BERT specifically
as we require contextual embeddings which could be easily adapted for the task
of MWP. Moreover, existing models have also shown results using BERT and it
would be fair to compare their performances when trained using similar
embeddings. Results obtained are shown in Table 11. We observe that for
solver’s trained using BERT, accuracy is higher than models trained from
scratch.

Human Evaluation: To verify the quality of augmented examples, we conduct
human evaluation. The focus of this evaluation is: $(1)$ To check if the
augmentations will result in the same linear equation as present in the
original problem statement, $(2)$ To evaluate if the numerical values for each
augmentation example is preserved, $(3)$ Evaluate each sample in the range $0$
to $1$ for its semantic similarity with the original problem statement, $(4)$
On a scale of $1$ to $5$ rate each augmented example for its grammatical
correctness. We conduct the human evaluations on randomly shuffled subsets
consisting of around $40\%$ of the total augmented examples for both the
datasets. This process is repeated three times with different subsets, five
human evaluators evaluate each example in all subsets, and the mean results
are computed as shown in Table 12.

Evaluation Criteria | MaWPS | ASDiv-A
---|---|---
Para | Sub | Para | Sub
Preserves Equation | 92.3% | 89.5% | 93.6% | 90.1%
Preserves Numbers | 88.4% | 91.2% | 87.3% | 90.3%
Semantic Similarity | 0.96 | 0.89 | 0.91 | 0.87
Syntactic Similarity | 4.67 | 4.36 | 4.59 | 4.33
Table 12: Human Evaluation scores on augmentated dataset. Para and Sub
represents paraphrasing and substitution methods respectively.

##  5 Future Work and Conclusion

We showcase that the existing MWP solvers are not robust and do not generalize
well on even simple variations of the problem statement. In this work, we have
introduced data augmentation techniques for generation of diverse math word
problems. We were able to enhance the size of existing dataset by 5 folds and
significantly increase the performance of state-of-the-art solvers by over 5
percentage points. Future works could focus on developing techniques to
generate data artificially and making robust MWP solvers.

##  6 Acknowledgment

We thank IHub-Data, IIIT Hyderabad333https://ihub-data.iiit.ac.in/ for
financial support.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.
  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
  * Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018.  Synthetic and natural noise both break neural machine translation.
  * Chiang and Chen (2018) Ting-Rui Chiang and Yun-Nung Chen. 2018.  Semantically-aligned equation generation for solving and reasoning math word problems.  _arXiv preprint arXiv:1811.00720_.
  * Devlin et al. (2019a) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for nlp.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014\.  Learning to solve arithmetic word problems with verb categorization.  pages 523–533, Doha, Qatar. Association for Computational Linguistics.
  * Huang et al. (2016) Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016.  How well do computers solve math word problems? large-scale dataset construction and evaluation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 887–896, Berlin, Germany. Association for Computational Linguistics.
  * Kim et al. (2019) Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. 2019\.  Pivot-based transfer learning for neural machine translation between non-English languages.  pages 866–876, Hong Kong, China. Association for Computational Linguistics.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.
  * Koncel-Kedziorski et al. (2016) Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016.  MAWPS: A math word problem repository.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1152–1157, San Diego, California. Association for Computational Linguistics.
  * Kumar et al. (2019) Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. 2019.  Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation.  pages 3609–3619, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kumar et al. (2021) Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. 2021.  Adversarial examples for evaluating math word problem solvers.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2705–2712, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kushman et al. (2014) Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014.  Learning to automatically solve algebra word problems.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 271–281, Baltimore, Maryland. Association for Computational Linguistics.
  * Longpre et al. (2020) Shayne Longpre, Yu Wang, and Christopher DuBois. 2020.  How effective is task-agnostic data augmentation for pretrained transformers?
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.
  * Maheshwary et al. (2021a) Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021a.  Generating natural language attacks in a hard label black box setting.  In _AAAI_.
  * Maheshwary et al. (2021b) Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021b.  A strong baseline for query efficient attacks in a black box setting.
  * (21) Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su.  A diverse corpus for evaluating and developing English math word problem solvers.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.  Are nlp models really able to solve simple math word problems?
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.
  * (26) Georgios Rizos, K. Hemker, and Björn Schuller.  Augment to prevent: Short-text data augmentation in deep learning for hate-speech classification.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.
  * Tan et al. (2019) Hao Tan, Licheng Yu, and Mohit Bansal. 2019.  Learning to navigate unseen environments: Back translation with environmental dropout.
  * Taylor and Nitschke (2018) Luke Taylor and Geoff Nitschke. 2018.  Improving deep learning with generic data augmentation.  In _2018 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pages 1542–1547.
  * Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019.  Context-aware monolingual repair for neural machine translation.
  * Wang et al. (2019) Lei Wang, D. Zhang, Jipeng Zhang, Xing Xu, L. Gao, B. Dai, and H. Shen. 2019.  Template-based math word problem solvers with recursive neural networks.  In _AAAI_.
  * (32) Yan Wang, Xiaojiang Liu, and Shuming Shi.  Deep neural solver for math word problems.
  * (33) Jason Wei and Kai Zou.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.
  * (34) Sam Witteveen and Martin Andrews.  Paraphrasing with large language models.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Z. Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _ArXiv_ , abs/1609.08144.
  * Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019.  A goal-driven tree-structured neural model for math word problems.  In _IJCAI_ , pages 5299–5305.
  * (37) Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim.  Graph-to-tree learning for solving math word problems.
  * Zhang et al. (2016) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016.  Character-level convolutional networks for text classification.

##  7 Appendix

###  7.1 Implementation Details

For conducting our experiments we have used two Boston SYS-7048GR-TR nodes
equipped with NVIDIA GeForce GTX 1080 Ti computational GPU’s having 11GB of
GDDR5X RAM. All implementations of training and testing is coded in Python
with Pytorch framework. The number of parameters range from 20M to 130M for
different models. We use negative log likelihood as the loss criterion. Hyper-
parameter values were not modified, and we follow the recommendations of the
respective models. To reduce carbon footprint from our experiments, we run the
models only on a single fold for searching hyperparameter values. We chose the
number of base candidates after primary stage $n$ as 7. Generating
augmentation examples using Paraphrasing Methods took around 12 minutes on
average for MaWPS and 8 minutes for ASDiv-A datasets. Substitution methods
took around 5 minutes on average for both MaWPS and ASDiv-A dataset. The
experiments conducted by us are not computation heavy. Each of the state-of-
the-art models get trained within 5 hrs of time, with Graph2Tree taking the
maximum time.

###  7.2 Additional Augmented Examples

In this section, we present some additional valid as well as invalid augmented
examples. Additionally, we also show some more examples with their attention
weights. Table 13 shows some additional examples with their attention weight
distribution. These weights have been shown for the base model trained before
augmentation and after augmentation on MaWPS dataset. Table 14 illustrates
some additional problem statements for all the techniques in paraphrasing
methods and substitution methods. In Table 15, we present some invalid
augmented examples which do not satisfy our human evaluation criteria. These
examples are such that they alter the semantics of the original problem
statement.

|  Problem: A magician was selling magic card decks for 2
---
dollars each. If he started with 25 decks and by the end of
the day he had 4 left, how much money did he earn ?
Mean attention values: 0.34 0.11 0.09
Problem: A magician was selling magic card decks for 2
dollars each. If he started with 25 decks and by the end of
the day he had 4 left, how much money did he earn ?
Augmented mean attention values : 0.19 0.18 0.15
|  Problem: There are 18 pencils in the drawer and 6 pencils
---
on the desk. Dan placed 4 pencils on the desk. How many
pencils are there in total ?
Mean attention values: 0.21 0.16 0.06
Problem: There are 18 pencils in the drawer and 6 pencils
on the desk.
Dan placed 4 pencils on the desk. How many
pencils are there in total ?
Augmented mean attention values : 0.29 0.19 0.12
|  Problem: Dan has 12 violet marbles, he gave Mary 4
---
of the marbles. How many violet marbles does he
now have ?
Mean attention values: 0.23 0.21 0.17
Problem: Dan has 12 violet marbles , he gave Mary 4
of the marbles. How many violet marbles does he
now have ?
Augmented mean attention values : 0.23 0.18 0.11
|  Problem: Angela has 7 tickets . Annie gives Angela
---
5 more. How many tickets does Angela have in all ?
Mean attention values: 0.30 0.19 0.15
Problem: Angela has 7 tickets . Annie gives Angela
5 more. How many tickets does Angela have in all ?
Augmented mean attention values : 0.29 0.21 0.14
|  Problem: Maria had 5 bottles of water in her fridge.
---
If she drank 1 of them and then bought 2 more, how many
bottles would she have ?
Mean attention values: 0.48 0.14 0.04
Problem Maria had 5 bottles of water in her fridge.
If she drank 1 of them and then bought 2 more , how
many bottles would she have ?
Augmented mean attention values : 0.23 0.17 0.11
Table 13: Examples illustrating distribution of top three attention weights before and after training on the full augmented dataset. Category |  Augmentation Method |  Example
---|---|---
Paraphrasing Methods |  Round trip Translation |  Original: Alyssa’s dog had puppies. She gave 2 to her friends. She now has 3 puppies. How many puppies did she have to start with ?
Augmented: Alyssa’s dog had puppies . She gave her friends 2 . She now has 3
puppies . How many puppies did she start ?
Problem Reordering |  Original: Rachel was organizing her book case making sure each of the shelves had exactly 3 books on it. If she had 4 shelves of mystery books and 2 shelves of picture books , how many books did she have total ?
Augmented:How many books did she have given that rachel was organizing her
book case making sure each of the shelves had exactly 3 books on it and she
had 4 shelves of mystery books and 2 shelves of picture books .
Substitution Methods |  Fill Masking |  Original: A cell phone company has a total of 1000 customers across the world . If 740 of its customers live in the United States , how many of its customers live in other countries ?
Augmented: A mobile phone firm has a network of 1000 customers across the
world . If 740 of its customers live in the US, How many customers live in
other locations?
Named-Entity Replacement |  Original: Daniel had some noodles. He gave 20 noodles to William. Now Daniel only has 11 noodles. How many noodles did Daniel have to begin with ?
Augmented: Matt had some noodles. He gave 20 noodles to Zeal. Now Matt only
has 11 noodles. How many noodles did Matt have initially ? Edd found 7
seashells , Alan found 12 seashells , and Royal found 5 seashells on the beach
. How many seashells were found together ?
Synonym Replacement |  Original: There are 5 rulers in the drawer. Tim took 3 rulers from the drawer. How many rulers are now in the drawer ?
Augmented: There are 5 consonants in the drawer. Tim went 3 consonants from
the drawer. How many other consonants are in the drawer now ?
Table 14: Valid Augmentation examples from all proposed methods. Coloured text represents the changes in problem statement. Category |  Augmentation Method |  Example
---|---|---
Paraphrasing Methods |  Round trip Translation |  Original: Kimberly went to the store 6 times last month . She buys 9 peanuts each time she goes to the store . How many peanuts did Kimberly buy last month ?
Augmented: Kimberly travelled to club six times last month. She buys 9 peanuts
every time she goes to the club . How many peanuts did Kimberly buy last year
?
Problem Reordering |  Original: Fred has 10 blue marbles . Fred has number1 times more blue marbles than Tim . How many blue marbles does Tim have ?
Augmented: If fred has 10 blue marbles and fred has number1 more blue marbles
than Tim then how many blue marbles does tim have ?
Substitution Methods |  Fill Masking |  Original: Sarah had 7 homework problems . She finished 2 of them but still had 3 pages of problems to do . If each page has the same number of problems on it , how many problems are on each page ?
Augmented: Sarah had 7 of them . She had 2 of them but still had 3 more of
them to do . If each more has the same number of them on it, How many them are
on each more ?
Named-Entity Replacement |  Original: Beverly had 10 dimes in his bank . His sister Maria borrowed 2 of his dimes . How many dimes does Beverly have now ?
Augmented: Silva had 10 dimes in his bank . His sister Jeanie borrowed a pair
of his dimes . How many dimes does Jeanie have now ?
Synonym Replacement |  Original: Shawn’s team won their dodgeball game and scored 25 points total . If Shawn scored 13 of the points and everyone else scored 4 points each , how many players were on his team ?
Augmented: Shawn’s group won their rumble game and scored 25 points total . If
Shawn scored 13 of the points and everyone else scored quarter points each,
how many people were there ?
Table 15: Invalid Augmentation examples from all proposed methods. Coloured
text represents the changes in problem statement.


# Empirical fits to inclusive electron-carbon scattering data obtained by
deep-learning methods

Beata E. Kowal<EMAIL_ADDRESS>Krzysztof M. Graczyk
<EMAIL_ADDRESS>Artur M. Ankowski
Rwik Dharmapal Banerjee  Hemant Prasad  Jan T. Sobczyk  Institute of
Theoretical Physics, University of Wrocław, plac Maxa Borna 9, 50-204,
Wrocław, Poland

###### Abstract

Employing the neural network framework, we obtain empirical fits to the
electron-scattering cross section for carbon over a broad kinematic region,
extending from the quasielastic peak, through resonance excitation, to the
onset of deep-inelastic scattering. We consider two different methods of
obtaining such model-independent parametrizations and the corresponding
uncertainties: based on the NNPDF approach [J. High Energy Phys. 2002, 062],
and on the Monte Carlo dropout. In our analysis, the $\chi^{2}$ function
defines the loss function, including point-to-point uncertainties and
considering the systematic normalization uncertainties for each independent
set of measurements. Our statistical approaches lead to fits of comparable
quality and similar uncertainties of the order of $7$% and $12$% for the first
and the second approaches, respectively. To test these models, we compare
their predictions to a test dataset, excluded from the training process, a
dataset lying beyond the covered kinematic region, and theoretical predictions
obtained within the spectral function approach. The predictions of both models
agree with experimental measurements and the theoretical predictions. However,
the first statistical approach shows better interpolation and extrapolation
abilities than the one based on the dropout algorithm.

##  I Introduction

The stringent precision requirements of the long-baseline neutrino-oscillation
program create an urgent need to significantly improve the accuracy of
available estimates of the cross sections for neutrino interactions with
atomic nuclei [1, 2]. The complexity of this problem stems from multiple
factors: the process of scattering involves multicomponent vector and axial
currents, different interaction mechanisms can produce the same final states,
experimental data with the desired precision are only available for nuclear
cross sections, and flux-averaging greatly diminishes differences between
different interaction channels.

A widely recognized way to address those difficulties is to leverage the
similarities of electron and neutrino interactions [3]. In electron
scattering, only electromagnetic currents contribute, cross sections are
higher by several orders of magnitude, beams are monoenergetic and adjustable,
and precise data can be collected not only for complex nuclei, but also for
deuterium and hydrogen. These advantages allow nuclear effects to be studied
in detail using electron-scattering data.

Provided that the cross sections for neutrinos and electrons are calculated
consistently and are sufficiently constrained by precise electron data, the
only significant source of uncertainties in the neutrino predictions would be
related to the axial contributions, to be determined from the near-detector
measurements.

In parallel to the efforts of developing models of nuclear response, it would
be useful to explore novel techniques for obtaining model-independent
estimates of nuclear cross sections. Neural networks (NNs) are ideally
equipped to provide such predictions. Indeed, NNs are known for their
excellent adaptive abilities [4], which enabled obtaining unbiased
parametrizations of nuclear cross sections [5], parton distribution functions
[6, 7], vector [8, 9, 10, 11] and axial [12] nucleon form factors, deeply
virtual Compton form factors [13], and nuclear charge radii [14].

In this article, we explore methods of obtaining such parametrizations of
electron-scattering cross sections, independent of nuclear model assumptions,
and pay special attention to estimating their uncertainties. For an analysis
of this type, the abundance of available experimental data is critical.

We consider the carbon target, for which experimental cross sections have been
collected in various laboratories worldwide over the last five decades and
span the broadest kinematic region [15]. These features are of paramount
importance to maximize the predictive abilities of the resulting NN models in
the broad kinematic region of relevance for neutrino-oscillation experiments.

Using NNs to parametrize electron-scattering data, we obtain both the best
fits and an estimate of the associated uncertainties. In this analysis, we
consider two different methods.

In the first one, we adopt the approach developed by the Neural Network Parton
Distribution (NNPDF) group [6]. The experimental data are used to generate
clone datasets. For each clone, a NN fit is obtained. The model’s prediction
is obtained by averaging all the fits, and the square root of the variance
gives its uncertainty.

In the second approach, we employ the Monte Carlo (MC) dropout technique [16],
in which NN units are temporarily removed at random to prevent overfitting.
This procedure is repeatedly used during the training and inference stages,
resulting in random responses of the NN. These responses are then averaged
over to obtain the NN prediction, and their distribution provides the
uncertainty estimate.

Our analysis pays special attention to treating systematic uncertainties
related to data normalization, as previously done in Refs. [9, 10, 11]. We
introduce them as a penalty term in the loss function, given by the $\chi^{2}$
function. We use an NN architecture with ten hidden layers and address the
vanishing gradient problem (VGP) by implementing batch normalization layers
[17]. Our models are validated by comparisons to the test dataset and with
predictions of the spectral function approach [18]. Eventually, we compare the
model’s prediction with higher energetic data from [19].

A similar analysis of electron-scattering data was performed by Al Hammal et
al. [5], considering nuclear targets ranging from 4He to 59Ni. Their loss
function did not account for experimental uncertainties, and was introduced as
a relative absolute error. However, an effort was made to estimate the
uncertainties due to the model parameter variations. The authors of Ref. [5]
adopted a network with $35$ hidden layers, and relied on the residual
mechanism for the VGP. To test their model, Al Hammal et al. compared its
results with the predictions of GiBUU [20] and the SuSAv2 approach [21].

Our paper is organized as follows. In Sec. II, we discuss the experimental
data underlying our analysis and provide the rationale for the imposed
kinematic cuts. In Sec III, we introduce our statistical model, providing
details of the loss function, the methods for estimating uncertainties, and
the network architectures. In Sec. IV, we report and analyze our numerical
results. Finally, we state our conclusions in Sec. V.

##  II Electron-scattering data

For an NN to successfully learn electron-scattering cross sections, it is
essential to provide it with an abundance of experimental data. In this
analysis, we consider the carbon target, as it has been studied extensively in
past experiments; see Ref. [15] for the recent review.

Figure 1: Kinematic domain covered by the experimental data [22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32] considered in our analysis, shown in the planes of
(top) energy transfer and four-momentum transfer, and (bottom) energy transfer
and cosine of the scattering angle.

The available data [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33] are the
double-differential cross sections ${d^{2}\sigma}/{d\omega d\Omega}$ for
inclusive electron scattering—in which only the final kinematics of the
electron is measured—at various beam energies $E$ and scattering angles
$\theta$, given as a function of the energy $\omega$ that the interaction
transferred to the nucleus.

We consider a broad kinematic region, in which different interaction
mechanisms are known to play an important role, starting from quasielastic
scattering at low energy transfers, to pion production through resonance
excitation in the intermediate regime, and to the onset of deep-inelastic
scattering at the highest $\omega$ values.

At the lowest energy transfers, elastic scattering of the whole nucleus may
occur, and inelastic interactions with the nucleus may lead to an excitation
of the giant dipole resonance or a discrete nuclear state. Due to the scarcity
of data in this region, accounting for these processes is not currently
feasible in our approach. We remove their contributions by applying a cut,

$\omega>\omega_{\text{min}}=\frac{M_{A}E_{x}+0.5E_{x}^{2}+E^{2}(1-\cos\theta)}{M_{A}+E(1-\cos\theta)},$
(1)

where $\omega_{\text{min}}$ is the energy transfer required to excite the
nucleus of the mass $M_{A}$ with the energy $E_{x}$; $\theta$ is the
scattering angle. Typically, $\omega_{\text{min}}\approx E_{x}+t_{A}$, with
$t_{A}$ being the nuclear recoil energy. In our analysis, the $E_{x}$ value is
set to $26$ MeV [18].

We also do not include the measurements reported by Gomez et al. [19], who
studied how the EMC effect depends on the nuclear mass number. Those seven
points, each collected for a different $(E,\cos\theta)$ pair, provide useful
constraints on theoretical models, but they cannot be readily accommodated
within our machine-learning approach. We make use of those data to investigate
the extrapolation properties of our models.

The kinematic region spanned by the data used in our analysis is presented in
Fig. 1, projected in the $(\omega,Q^{2})$ and $(\omega,\cos\theta)$ planes,
$Q^{2}$ being the four-momentum transfer. While at low $Q^{2}$ and at low
$\theta$, there is an abundance of the results; this is not the case in
general. Even for carbon, the best studied nuclear target, there are still
large swaths with very scarce information on the cross sections. This issue is
challenging to machine-learning models based on the data alone.

Table 1: Summary of the data used in this analysis. The numbers of points refer to the data surviving the cut (1). Reference | Abbrev. | Norm. | Number
---|---|---|---
uncert. | of points
Arrington et al. [22] | Arri1995 | $4.0\%$ | 56
Arrington et al. [23] | Arri1998 | $4.0\%$ | 398
Bagdasaryan et al. [24] | Bagd1988 | $10.0\%$ | 125
Baran et al. [25] | Bara1988 | $3.7\%$ | 259
Barreau et al. [26] | Barr1983 | $2.0\%$ | 1243
Dai et al. [27] | Dai2018 | $2.2\%$ | 177
Day et al. [28] | Day1993 | $3.4\%$ | 316
Fomin et al. [29] | Fomi2010 | $4.0\%$ | 359
O’Connell et al. [30] | O’Con1987 | $5.0\%$ | 51
Sealock et al. [31] | Seal1989 | $2.5\%$ | 250
Whitney et al. [32] | Whit1974 | $3.0\%$ | 31
Total |  |  | 3265

In Table 1, we summarize the data and provide the normalization uncertainties
(‘Norm. uncert.’) assumed in our analysis.

##  III Neural network framework

###  III.1 Likelihood analysis

As discussed in the previous section, in this analysis, we consider $11$
independent cross-section datasets. We denote the $k$th dataset containing
$N_{k}$ points as

$\mathcal{D}_{k}=\\{(E_{k}^{i},\theta_{k}^{i},\omega_{k}^{i},d\sigma_{k}^{i},\Delta
d\sigma_{k}^{i}):\;\;i=1,\dots,N_{k}\\},$ (2)

where $d\sigma_{k}^{i}$ and $\Delta d\sigma_{k}^{i}$ are the $i$-th
measurement in $k$-th dataset and its corresponding uncertainty, $N_{k}$
number of data points in $k$-th dataset. The full dataset reads
$\mathcal{D}=\mathcal{D}_{1}\cup\dots\cup\mathcal{D}_{11}$.

Every data set has been collected independently by a given experiment. Each
dataset can be characterized by overall statistical uncertainty. As
demonstrated in [34] and [35], special treatment of normalization
uncertainties improves the global analysis of the scattering data. Indeed,
taking into account the normalization uncertainties allowed us to get
consistent statistical fits for elastic electron-proton [35] scattering data
as well as charged current single pion production [36] and quasielastic [12]
neutrino-deuteron scattering data. In the present paper, we follow similar
philosophy. Let us denote the overall statistical uncertainty for $k$th data
set by $\Delta\lambda_{k}$. Then, to fit the data, we introduce the following
loss function [9]:

$\chi_{\text{tot}}=\sum_{k=1}^{11}\left[\chi_{k}^{2}(\lambda_{k})+\frac{1}{2}\left(\frac{1-\lambda_{k}}{\Delta\lambda_{k}}\right)^{2}\right],$
(3)

where

$\chi_{k}^{2}(\lambda_{k})=\frac{1}{2}\sum_{i=1}^{N_{k}}\left(\frac{d\sigma_{k}^{i}-\lambda_{k}d\sigma_{i}^{\text{fit}}(E_{k}^{i},\theta_{k}^{i})}{\Delta
d\sigma_{k}^{i}}\right)^{2},$ (4)

and $\lambda_{k}$ is a subject of optimization.

In principle, $\lambda_{k}$’s can be included in the set of model parameters
and then optimized by the training algorithm; however, we follow the algorithm
proposed in Refs. [9, 10, 11] to estimate these parameters.

Figure 2: The neural network architecture for model A. The light gray box
corresponds to the hidden layer of fully connected neurons with ReLU
activation function. The dark gray box denotes the batch normalization layer.
In the output, there is the sigmoid function. In the architecture for model B,
not depicted, each fully connected layer is followed by the dropout layer. In
both models A and B, the network has ten layers of hidden units, consisting of
$300$ units each.

###  III.2 Neural network model

We shall fit the cross-section data using the NN parameterization. We shall
obtain a function that, for input energy, scattering angle, and energy
transfer, gives the value of differential cross-section for quasielastic
electron-carbon scattering. NNs are known for their adaptive ability, but they
are also known for their limitations. Indeed, to perform the optimization, one
should rescale the input and/or output data appropriately. In our case, we
consider the following rescaling of the data:

$d\sigma\rightarrow\left(\frac{10^{9}}{137^{2}E\cos(\theta/2)}\frac{\cos(\theta/2)^{2}}{4E^{2}\sin(\theta/2)^{4}}\right)^{-1}d\sigma,$
(5)

where the beam energy, $E$, is given in GeV and the cross section, $d\sigma$,
in nb/(sr GeV).

Figure 3: Illustration of the algorithm used in model A to estimate the fit’s
uncertainty. Clone datasets are generated from the original data according to
the normal distribution (not depicted), and fits are performed 50 times (green
lines). The cross section’s estimate corresponds to the mean of the fits
(black line), and the uncertainty corresponds to their standard deviation (not
shown). The data are taken from Ref. [26].

To accelerate the optimization process and help NN catch proper normalization,
we extend the number of input variables from three to five
$(E,\omega,\theta,\cos\theta,Q^{2})$. The last two are the functions of the
first three, but keeping such input accelerates the optimization. The same
procedure was employed by NNPDF group [6] to fit parton distribution functions
and by Al Hammal et al. [5] to fit electron-nucleus scattering data.

We consider two types of NN architecture. The first one, statistical model A,
consists of the input layer, followed by ten blocks and an output layer. The
network structure is shown in Fig. 2. The input is five-dimensional, and the
output is one-dimensional. Each block has a hidden layer with $300$ units
fully connected with the previous module and batch-normalization layer. We
keep ReLU activation functions in each module. The output activation function
is given by a sigmoid function. The batch-normalization layer [17] was
essential to obtain successful fits. This type of layer helps to maintain a
proper normalization of the outputs that, as a result, prevents the vanishing
of the gradients and stopping the optimization process - a problem typical for
multi-layer NNs. Additionally, the batch-normalization layer naturally
regularizes the model parameters. Hence, such a model should not tend to over-
fit the data.

The second network, statistical model B, has a similar structure, but each
module consists of the fully connected layer followed by a batch-normalization
layer and the dropout layer. The dropout layer was invented by Srivastava et
al. [16] to regularize the network parameters and, prevent the model from
over-fitting and, as a consequence, increase the predictive abilities of the
networks. However, we keep dropout layers for Monte Carlo dropout analysis;
see the following sections.

###  III.3 Uncertainty estimation

We considered two approaches to estimate the uncertainties of the network
predictions. In model A, we adopt the method proposed in [6, 7]. The idea is
to use experimental measurements to produce the so-called clone datasets and
fit every set. A given new clone point is obtained from sampling from a normal
distribution. Namely, if the original data point is characterized by central
value $d\sigma_{k}^{i}$ and uncertainty $\Delta d\sigma_{k}^{i}$, the clone
datum reads

$d\sigma_{k}^{\text{clone, {\hbox{i}}}}=d\sigma_{k}^{i}+r\Delta
d\sigma_{k}^{i},$ (6)

where $r$ is drawn from the normal distribution of mean zero and variance $1$.
In contrast to [7], we treat the normalization uncertainties as described in
the previous sections. But similarly, as in [7], we consider $\chi^{2}$ as a
loss to optimize the network’s parameters.

We split the full data set into the training and test data sets in proportions
$9$ to $1$. For the training data, $50$ clone data sets are created. Then, we
obtain the network fit for every clone. The mean gives the model’s response
over predictions of all $50$ fits, while the corresponding square root of
variance gives uncertainty. In Fig. 3 we illustrate how the algorithm works.
The figure shows the experimental data together with $50$ fits to the clone
datasets.

In the second method, model B, we utilize the Monte Carlo dropout technique
[16]. The idea is to keep the dropout layer after every convolutional layer.
This layer switches off each unit in the network with user-defined probability
$p$. Initially, this method was introduced to regularize the training process
to prevent overfitting and improve the generalization ability of the model.
However, when one keeps the dropout layer active in the inference mode, the
network’s prediction becomes random. Then one computes dozens of times (in our
case $50$) the network’s response to obtain the network prediction and takes
the average [37]. The uncertainty, in this case, is given by corresponding
variance. In this method, the result may depend on $p$. In the pre-analyses,
we tested several values of $p$, and we found $p=0.04$ to be the most optimal.
In the MC dropout analysis, we split the data into training and test data sets
in proportions $9:1$.

Figure 4: Double-differential cross section ${d^{2}\sigma}/{d\omega d\Omega}$ for inclusive electron scattering on carbon. We compare the predictions of model A to the experimental data from Refs. [26, 31, 28, 23]. The shaded areas denote the $1\sigma$ uncertainties. The panels are labeled with the beam energy and scattering angle values. The red (blue) points represent the training (test) dataset. The predictions are not rescaled according to the determined normalization parameters. Figure 5: Same as Fig. 4 but for model B. Table 2: Values of the normalization parameters $\lambda$ obtained in models A and B.  Abbrev. | Norm. | model A | model B
---|---|---|---
uncert.
Arri1995 | $4.0\%$ | 1.01 | 1.05
Arri1998 | $4.0\%$ | 1.00 | 1.05
Bagd1988 | $10.0\%$ | 1.03 | 1.01
Bara1988 | $3.7\%$ | 1.01 | 0.99
Barr1983 | $2.0\%$ | 0.99 | 0.94
Dai2018 | $2.2\%$ | 1.00 | 0.96
Day1993 | $3.4\%$ | 0.99 | 1.03
Fomi2010 | $4.0\%$ | 1.01 | 1.04
O’Con1987 | $5.0\%$ | 1.02 | 1.07
Seal1989 | $2.5\%$ | 1.02 | 0.99
Whit1974 | $3.0\%$ | 0.93 | 0.94

##  IV Numerical results

We performed both analyses employing the Jax package [38], and cross-checked
the obtained results by using the Keras package [39]. We needed about 60 000
epochs in each analysis to obtain the NN fit. To optimize the models, we used
the AdamW algorithm [40]. The models were regularized by keeping the weight
decay set to 0.004. The training was done in the minibatch configuration with
five batches. The best fit corresponds to the model parameters configuration
for which loss has the lowest value during the training.

In Figs. 4 and 5, we present comparisons of the predictions of the model A and
B to six datasets [26, 31, 28, 23]. The data span a broad kinematic region,
corresponding to energy transfers between $\sim$30 MeV and $\sim$1.9 GeV, and
momentum transfers $|\mathbf{q}|$ between $\sim$0.14 and $\sim$2.7 GeV.

The first six panels of Figs. 4 and 5 sample the $|\mathbf{q}|$ values below 1
GeV, where the cross section is dominated by quasielastic scattering and the
$\Delta$-resonance excitation, and nuclear effects are the most pronounced.
Model A reproduces the data with excellent accuracy, both at low and high
scattering angles, although the contributions of the mechanism of interactions
involving two-body currents are expected to be very different in these
regimes. It outperforms model B, for which the agreement is very good
nevertheless.

The last three panels of Figs. 4 and 5 probe the transition of the cross
section from the region where excitation of higher resonances plays an
important role to where it can be described in terms of deep-inelastic
scattering. Despite of the complexity of the interaction mechanisms at these
kinematics, model A is able to reproduce the data with remarkable accuracy.
For model B, the agreement is good, especially keeping in mind that the
normalization parameters are not included in the results presented in Figs. 4
and 5.

Figure 6: Histograms of the normalized residuals,
$(d\sigma_{i}-d\sigma_{i}^{\text{fit}})/d\sigma_{i}$, for (top) model A and
(bottom) model B. All the considered data are included. Figure 7: Histograms
of the standard deviation normalized by the mean for the predictions of (top)
model A and (bottom) model B. All the considered data are included.

As discussed in Sec. III, the normalization parameters are incorporated in our
fits to the data, giving additional freedom to the parameter space. Table 2
shows that their values typically agree with the unity to about $2\%$ for
model A and $5$% for model B.

Interestingly, in both our analyses, the normalization parameter for the data
reported by Whitney et al. [32] is low. It suggests that there might be a
tension at the level of 5% – 6% between the normalization of this dataset and
those of the other measurements, with the results of Ref. [32] being
overestimated.

We refer the reader to Supplemental Material for comparisons of the
predictions of both the models to all the measurements considered in this
analysis, including the dataset of Whitney et al.

Figure 8: Interpolation tests of the obtained fits for beam energy 0.6 GeV and
scattering angles 30°, 45°, and 60°. We compare the spectral function
calculations [18] of the quasielastic cross section, depicted by the solid red
lines, with the predictions of model A (model B), represented by the solid
(dashed) blue lines. The green (orange) areas correspond to the $1\sigma$
uncertainties.

The quality of fits is summarized in Fig. 6, where we present the histograms
of the residuals normalized by the true values,
$(d\sigma_{i}-d\sigma_{i}^{\text{fit}})/d\sigma_{i}$. The results turn out to
have a shape similar to that of a Gaussian distribution.

In Fig. 7, we also present the histograms of the standard deviation normalized
by the mean predicted value. The mean uncertainties are about $7$% and $12$%
for models A and B, respectively.

So far we have discussed the performance of the obtained fits in describing
the available data. It is interesting to check how well they interpolate the
cross section within the domain covered by the training data, and extrapolate
beyond it.

To explore the abilities of the models to interpolate, in Fig. 8 we compare
their predictions with the spectral function calculations of Ref. [18], which
include only quasielastic scattering. For these tests, we select the beam
energy of 600 MeV—relevant for neutrino-oscillation experiments such as T2K
and the Short Baseline Neutrino program—and the scattering angles of 30°, 45°,
and 60°.

The amount of data collected at similar kinematic settings is the largest for
$\theta=60\degree$, and the smallest for $\theta=30\degree$. As a consequence,
the predictions of the two models and their uncertainties are the most similar
for $\theta=60\degree$. For the lower scattering angles, the uncertainties
apparently increase in model A, but not in model B. At 45°, one can clearly
see a sudden jump in the uncertainties of model A at $\omega\simeq 0.55$ GeV,
reflecting the behavior of the underlying data [26].

The predicted positions of the quasielastic peak turn out to agree very well
between the predictions of model A and the spectral function approach. This is
also the case for model B when $\theta=60\degree$. However, when we lower the
scattering angle, an increasing difference can be observed.

The magnitude of the cross section predicted by the fits at $\theta=60\degree$
seems consistent with the theoretical model, having in mind a large
contribution of interaction mechanisms other than quasielastic scattering
induced by one-body currents. At $\theta=45\degree$, the theoretical
calculation agrees with model A (model B) at the $1\sigma$ $(2\sigma)$ level.
The differences increase at 30°, showing both the limitations of the obtained
fits, and the urgent need for more experimental data to model electroweak
interactions with the precision required by the neutrino-oscillation program.

In view of the obtained results, we conclude that model A is better at
propagating the uncertainties of the underlying experimental data than model
B.

To test the extrapolation properties of both the statistical models, we
compare their predictions for the cross section in the deep-inelastic regime,
at the kinematic settings of the measurements by Gomez et al. [19]. As shown
in Fig. 9, the predictions of model A manage to reproduce these data at the
1–2$\sigma$ level. For model B, we find that it overestimates them more
significantly.

In Fig. 9, the beam energy is fixed to 12.1 GeV, and scattering angles vary
between $12.82\degree$ and $15.85\degree$, with the corresponding momentum
transfers $4.4\leq|\mathbf{q}|\leq 7.0$ GeV. Only 2 datasets used in the
training probe similar kinematics—the measurements performed by Fomin et al.
[29] for beam energy 5.8 GeV, and scattering angles $40\degree$ and
$50\degree$, with $3.9\leq|\mathbf{q}|\leq 5.3$ GeV.

We expect that the accuracy of extrapolation would improve, should more data
be available for training.

Figure 9: Extrapolation tests of model A against the measurements by Gomez et
al. [19] in the deep-inelastic regime, not included in the training. The
highlighted areas represent the $1\sigma$ uncertainties.

##  V Summary

We develop a deep neural network framework to parametrize the electron-
scattering cross section for carbon over a broad kinematic region, extending
from the quasielastic peak, through resonance excitation, to the onset of
deep-inelastic scattering. Our results do not depend on theoretical
assumptions but rely exclusively on experimental measurements. We pay special
attention to estimating the uncertainties of the network’s predictions, which
reflect experimental uncertainties and are a measure of consistency between
different datasets.

We obtain two statistical models. Model A is based on an ensemble of $50$
neural networks, which fit clone datasets. Model B uses a single neural
network with dropout layers. Both approaches manage to reproduce the training
data accurately, with a preference for model A. Additionally, model A turns
out to be able to estimate the cross section outside the kinematic domain of
the training data.

In addition to their importance for the studies of inclusive electron
scattering on nuclei, our empirical fits can be readily included in the NuWro
Monte Carlo generator and pave the way toward the development of a more
accurate description of electroweak cross sections. In future analyses, we
plan to adapt and extend the methods developed in this article for modeling
neutrino interactions.

To facilitate their usage, the resulting fits are released to the public
through the GitHub repository [41].

###### Acknowledgements.

This research is partly (K.M.G., A.M.A., J.T.S.) or fully (B.E.K., R.D.B.,
H.P.) supported by the National Science Centre under grant UMO-2021/41/B/ST2/
02778.

## References

  * Abi  _et al._ [2020] B. Abi _et al._ (DUNE Collaboration), JINST 15 (08), T08008.
  * Abe  _et al._ [2015] K. Abe _et al._ (Hyper-Kamiokande Proto-Collaboration), PTEP 2015, 053C02 (2015).
  * Ankowski  _et al._ [2023] A. M. Ankowski _et al._ , J. Phys. G 50, 120501 (2023).
  * Cybenko [1989] G. Cybenko, Math Control, Signal 2, 303 (1989).
  * Al Hammal  _et al._ [2023] O. Al Hammal, M. Martini, J. Frontera-Pons, T. H. Nguyen, and R. Pérez-Ramos, Phys. Rev. C 107, 065501 (2023), arXiv:2305.08217 [nucl-th] .
  * Forte  _et al._ [2002] S. Forte, L. Garrido, J. I. Latorre, and A. Piccione, JHEP 05, 062, arXiv:hep-ph/0204232 .
  * Del Debbio  _et al._ [2005] L. Del Debbio, S. Forte, J. I. Latorre, A. Piccione, and J. Rojo (NNPDF), JHEP 03, 080, arXiv:hep-ph/0501067 .
  * Graczyk  _et al._ [2010] K. M. Graczyk, P. Plonski, and R. Sulej, JHEP 1009, 053, arXiv:1006.0342 [hep-ph] .
  * Graczyk [2011] K. M. Graczyk, Phys.Rev. C84, 034314 (2011), arXiv:1106.1204 [hep-ph] .
  * Graczyk and Juszczak [2014] K. M. Graczyk and C. Juszczak, Phys. Rev. C 90, 054334 (2014), arXiv:1408.0150 [hep-ph] .
  * Graczyk and Juszczak [2015] K. M. Graczyk and C. Juszczak, J. Phys. G 42, 034019 (2015), arXiv:1409.5244 [hep-ph] .
  * Alvarez-Ruso  _et al._ [2019] L. Alvarez-Ruso, K. M. Graczyk, and E. Saul-Sala, Phys. Rev. C 99, 025204 (2019), arXiv:1805.00905 [hep-ph] .
  * Kumericki  _et al._ [2011] K. Kumericki, D. Mueller, and A. Schafer, JHEP 07, 073, arXiv:1106.2808 [hep-ph] .
  * Dong  _et al._ [2022] X.-X. Dong, R. An, J.-X. Lu, and L.-S. Geng, Phys. Rev. C 105, 014308 (2022), arXiv:2109.09626 [nucl-th] .
  * Ankowski and Friedland [2020] A. M. Ankowski and A. Friedland, Phys. Rev. D 102, 053001 (2020).
  * Srivastava  _et al._ [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Journal of Machine Learning Research 15, 1929 (2014).
  * Ioffe and Szegedy [2015] S. Ioffe and C. Szegedy, in  _International conference on machine learning_ (Proc. Machine Learning Res., 2015) pp. 448–456.
  * Ankowski  _et al._ [2015] A. M. Ankowski, O. Benhar, and M. Sakuda, Phys. Rev. D 91, 033005 (2015).
  * Gomez  _et al._ [1994] J. Gomez _et al._ , Phys. Rev. D 49, 4348 (1994).
  * Mosel and Gallmeister [2019] U. Mosel and K. Gallmeister, Phys. Rev. C 99, 064605 (2019), arXiv:1811.10637 [nucl-ex] .
  * Amaro  _et al._ [2020] J. E. Amaro, M. B. Barbaro, J. A. Caballero, R. González-Jiménez, G. D. Megias, and I. Ruiz Simo, J. Phys. G 47, 124001 (2020), arXiv:1912.10612 [nucl-th] .
  * Arrington  _et al._ [1996] J. Arrington _et al._ , Phys. Rev. C 53, 2248 (1996).
  * Arrington  _et al._ [1999] J. Arrington _et al._ , Phys. Rev. Lett. 82, 2056 (1999).
  * Bagdasaryan  _et al._ [1988] D. S. Bagdasaryan, M. K. Boyadzhian, G. B. Kazarian, K. P. A. Kechian, E. R. Markarian, G. G. Mkrtchian, O. P. Petrosian, I. A. Troshenkova, and V. O. Tatevosian (1988), yERPHI-1077-40-88 (unpublished).
  * Baran  _et al._ [1988] D. Baran _et al._ , Phys. Rev. Lett. 61, 400 (1988).
  * Barreau  _et al._ [1983] P. Barreau _et al._ , Nucl. Phys. A402, 515 (1983).
  * Dai  _et al._ [2018] H. Dai _et al._ , Phys. Rev. C 98, 014617 (2018).
  * Day  _et al._ [1993] D. B. Day _et al._ , Phys. Rev. C 48, 1849 (1993).
  * Fomin  _et al._ [2010] N. Fomin _et al._ , Phys. Rev. Lett. 105, 212502 (2010).
  * O’Connell  _et al._ [1987] J. S. O’Connell _et al._ , Phys. Rev. C 35, 1063 (1987).
  * Sealock  _et al._ [1989] R. M. Sealock _et al._ , Phys. Rev. Lett. 62, 1350 (1989).
  * Whitney  _et al._ [1974] R. R. Whitney, I. Sick, J. R. Ficenec, R. D. Kephart, and W. P. Trower, Phys. Rev. C 9, 2230 (1974).
  * Benhar  _et al._ [2006] O. Benhar, D. Day, and I. Sick, arXiv:nucl-ex/0603032 (2006), http://discovery.phys.virginia.edu/research/groups/qes-archive/.
  * Arrington [2003] J. Arrington, Phys. Rev. C 68, 034325 (2003), arXiv:nucl-ex/0305009 .
  * Alberico  _et al._ [2009] W. M. Alberico, S. M. Bilenky, C. Giunti, and K. M. Graczyk, Phys. Rev. C79, 065204 (2009), arXiv:0812.3539 [hep-ph] .
  * Graczyk  _et al._ [2009] K. M. Graczyk, D. Kielczewska, P. Przewlocki, and J. T. Sobczyk, Phys. Rev. D80, 093001 (2009), arXiv:0908.2175 [hep-ph] .
  * Gal and Ghahramani [2015] Y. Gal and Z. Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning (2015), arXiv:1506.02142 [stat.ML] .
  * Bradbury  _et al._ [2018] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, JAX: composable transformations of Python+NumPy programs (2018).
  * Chollet  _et al._ [2015] F. Chollet _et al._ , Keras, https://keras.io (2015).
  * Loshchilov and Hutter [2019] I. Loshchilov and F. Hutter, Decoupled weight decay regularization (2019), arXiv:1711.05101 [cs.LG] .
  * [41] https://github.com/bekowal/CarbonElectronNeuralNetwork.