# Dynamic Data Selection for Curriculum Learning via Ability Estimation

John P. Lalor1, Hong Yu2,3
1Department of IT, Analytics, and Operations, University of Notre Dame
2 Department of Computer Science, University of Massachusetts Lowell
3 College of Information and Computer Sciences, University of Massachusetts
Amherst
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work performed while at UMass Amherst.

###### Abstract

Curriculum learning methods typically rely on heuristics to estimate the
difficulty of training examples or the ability of the model. In this work, we
propose replacing difficulty heuristics with learned difficulty parameters. We
also propose Dynamic Data selection for Curriculum Learning via Ability
Estimation (DDaCLAE), a strategy that probes model ability at each training
epoch to select the best training examples at that point. We show that models
using learned difficulty and/or ability outperform heuristic-based curriculum
learning models on the GLUE classification tasks.

##  1 Introduction

Curriculum learning trains a model by using easy examples first and gradually
adding more difficult examples. It can speed up learning and improve
generalization in supervised learning models Bengio et al. (2009); Amiri et
al. (2017); Platanios et al. (2019). A major drawback of existing curriculum
learning techniques is that they rely on heuristics to measure the difficulty
of data, and either ignore the competency of the model during training or rely
on heuristics there as well. For example, sentence length is often used as a
proxy for difficulty in NLP tasks Bengio et al. (2009); Platanios et al.
(2019). Such heuristics can be useful but have limitations. First, the
heuristic chosen may not actually be a proxy for difficulty. Depending on the
task, long sequences could signal easier or harder examples, or have no signal
for difficulty. Second, a model’s notion of difficulty may not align with the
heuristic imposed by a human developing the model. It may be that examples
that appear difficult for the human are in fact easy for the model to learn.

Competency was recently introduced as a mechanism to determine when new
examples should be added to the training data Platanios et al. (2019).
However, in that work competency is a monotonically increasing function of a
pre-determined initial value. Once set, competency is not evaluated during
training. Ideally, model competency should be measured at each training epoch,
so that the training data could be appropriately matched with the model at a
given point in the training. If a model is improving, then more difficult
training data can be added at the next epoch. But if performance declines,
then those difficult examples can be removed, and a smaller, easier training
set can be used in the next epoch.

In this study, we propose to estimate both the difficulty of examples and the
ability of deep learning models as latent variables based on model performance
using Item Response Theory (IRT), a well-studied methodology in psychometrics
for test set construction and subject evaluation Baker and Kim (2004). IRT
models estimate latent parameters such as difficulty for examples and a latent
ability parameter for individuals (“subjects”). IRT models are learned by
administering a test to a large number of subjects, collecting and grading
their responses, and using the subject-response matrix to estimate the latent
traits of the data. These learned parameters can be used to estimate the
ability of future subjects, based on their graded responses to the examples.

IRT has not seen wide adoption in the machine learning community, primarily
due to the fact that fitting IRT models requires a large amount of human
annotated data for each example. However, recent work has shown that IRT
models can be fit using machine-generated data instead of human-generated data
as input Lalor et al. (2019).

Because IRT learns example difficulty and subject ability together, in this
work we propose replacing heuristics for learned parameters in curriculum
learning. First, we experiment with replacing a typical difficulty heuristic
(sentence length) with learned difficulty parameters. Second, we propose
Dynamic Data selection for Curriculum Learning via Ability Estimation
(DDaCLAE), a novel curriculum learning framework that uses the estimated
ability of a model during the training process to dynamically identify
appropriate training data. At each training epoch, the latent ability of the
model is estimated using output labels generated at the current epoch. Once
ability is known, only training data that the model has a reasonable chance of
labeling correctly is included in training. As the model improves, the
estimated ability will improve, and more training examples will be added.

To the best of our knowledge, this is the first work to learn a model
competency during training that is directly comparable to the difficulty of
the examples. Our study will test the following three hypotheses: H1: Using
learned latent difficulties instead of difficulty heuristics leads to better
held-out test set performance for models trained using curriculum learning,
H2: A dynamic data selection curriculum learning strategy that probes model
ability during training leads to better held-out test set performance than a
static curriculum learning strategy that does not take model ability into
account, H3: Dynamic curriculum learning is more efficient than static
curriculum learning, leading to faster convergence. We test our hypotheses on
the GLUE classification data sets Wang et al. (2019).

Our contributions are as follows: (i) we demonstrate that for curriculum
learning, learned difficulty outperforms traditional difficulty heuristics,
(ii) we introduce a novel curriculum learning framework which automatically
selects training data based on the estimated ability of the model, and (iii)
we show that training using DDaCLAE leads to better performance than both
traditional curriculum learning methods and a fully supervised competitive
baseline. Our findings support the overall curriculum learning framework, and
demonstrate that learning difficulty and ability lead to further performance
improvements beyond common heuristics.111Code implementing our experiments and
learned difficulty parameters for the GLUE data sets are available at
https://jplalor.github.io/irt.

##  2 Methods

###  2.1 Curriculum Learning

Training ExamplesLearner$t_{e}=f(e)$ (a)

Training ExamplesLearner$t_{e}=f(\hat{\theta}_{e})$$\hat{\theta}_{e}$ (b)

Figure 1: (1(a)) Traditional curriculum learning, where examples are added at
each epoch according to a static monotonically-increasing schedule
($t_{e}=f(e)$). (1(b)) DDaCLAE estimates ability at each epoch
($\hat{\theta_{e}}$) to dynamically select appropriate training data
($t_{e}=f(\hat{\theta_{e}})$).

In a traditional curriculum learning framework, training data examples are
ordered according to some notion of difficulty, and the training set shown to
the learner is augmented at a set pace with more and more difficult examples
over time (Fig. 1(a)).

Typically, the model’s current performance is not taken into account. Recent
work has incorporated a notion of competency to curriculum learning Platanios
et al. (2019). In that work the authors structure the rate at which training
examples are added based on an assumption that model competency is modeled by
either a linear or root function of the training epoch.222The prior work
proposed other functions as well, but found that the linear and root functions
performed best. However, there are two issues with such an approach. First,
this notion of competency is artificially rigid. If a model’s competency
improves quickly, data cannot be added more quickly because the rate is
predetermined. On the other hand, if a model is slow to improve, it may
struggle because data is being added too quickly. Second, the formulation of
competency proposed by the authors reduces to a competency-free curriculum
learning strategy with a tuneable parameter for speed inclusion. Once this
parameter is set, there is no check of model ability during training to assess
competency and update training data. In this work our goal is to replace
curriculum learning heuristics with difficulty and competency parameters
learned directly using IRT (Fig. 1(b)).

###  2.2 Item Response Theory

IRT methods learn latent parameters of test set examples (called “items” in
the IRT literature) and latent ability parameters of individual “subjects”. We
refer to “items” as “examples” and “subjects” as “models” respectively for
clarity and consistency with the curriculum learning literature.

For a model $j$ and an example $i$, the probability that $j$ labels $i$
correctly ($z_{ij}=1$) is a function of the latent parameters of $j$ and $i$.
The one-parameter logistic (1PL) model, or Rasch model, assumes that the
probability of labeling an example correctly is a function of a single latent
difficulty parameter of the example, $b_{i}$ and a latent ability parameter of
the model, $\theta_{j}$ Rasch (1960); Baker and Kim (2004):

$p(z_{ij}=1|\theta_{j},b_{i})=\frac{1}{1+e^{-(\theta_{j}-b_{i})}}$ (1)

The probability that model $j$ will label item $i$ incorrectly ($z_{ij}=0$)
is:

$p(z_{ij}=0|\theta_{j},b_{i})=1-p(z_{ij}=1|\theta_{j},b_{i})$ (2)

With a 1PL model, there is an intuitive relationship between difficulty and
ability. An example’s difficulty value $b$ can be thought of as the point on
the ability scale where a model has a 50% chance of labeling an example
correctly. Put another way, a model has a 50% chance of labeling an example
correctly when model ability is equal to example difficulty
($\theta_{j}=b_{i}$, see Fig. 2).

$-4$$-2$$0$$2$$4$$0$$0.2$$0.4$$0.6$$0.8$$1$$\theta$$p(z=1|\theta,b)$ Figure 2:
Plot of $p(z_{ij}=1|\theta_{j},b_{i})$ as a function of $\theta$ for an
example with difficulty $b=0$. Models with ability $\theta\geq 0$ (right of
dashed line) have greater than 50% chance of labeling the example correctly.

Fitting a 1PL model requires a set of $I$ examples
$\\{i_{0},i_{1},\dots,i_{I}\\}$, a set of $J$ models
$\\{j_{0},j_{1},\dots,j_{J}\\}$, and the binary graded responses
$Z=\\{\forall_{i\in I}\forall_{j\in J}:z_{ij}\\}$ of the models to each of the
examples. The likelihood of a data set of response patterns $Z$ given the
parameters $\Theta$ and $B$ is:

$\displaystyle p(Z|\Theta,B)$
$\displaystyle=\prod_{j=1}^{J}\prod_{i=1}^{I}p(Z_{ij}=z_{ij}|\theta_{j},b_{i})$
(3)

where $z_{ij}=1$ if individual $j$ answers item $i$ correctly and $z_{ij}=0$
if they do not.

###  2.3 IRT with Artificial Crowds

A major bottleneck of using IRT methods on machine learning data sets is the
fact that each human subject would have to label all of (or most of) the
examples in order to have enough response patterns to estimate the latent
parameters. Having humans annotate all of the examples in a large data set is
impractical, but recent work has shown that the human subjects can be replaced
with an ensemble of machine learning models Lalor et al. (2019). The response
patterns from this “artificial crowd” can be used to estimate latent
parameters by fitting IRT models using variational inference (VI-IRT) Natesan
et al. (2016); Lalor et al. (2019).

VI-IRT approximates the joint posterior distribution $\pi(\Theta,B|Z)$ by the
variational distribution:

$\displaystyle q(\Theta,B)$
$\displaystyle=\prod_{j=1}^{J}\pi^{\theta}_{j}(\theta_{j})\prod_{i=1}^{I}\pi^{b}_{i}(b_{i})$
(4)

where $\pi^{\theta}_{j}()$ and $\pi^{b}_{i}()$ denote Gaussian densities for
different parameters. Parameter means and variances are determined by
minimizing the KL-Divergence between $q(\Theta,B)$ and $\pi(\Theta,B|Z)$.

In selecting priors for VI-IRT we follow the results of prior work where
hierarchical priors were used Natesan et al. (2016); Lalor et al. (2019). The
hierarchical model assumes that ability and difficulty means are sampled from
a vague Gaussian prior, and ability and difficulty variances are sampled from
an inverse Gamma distribution:

$\displaystyle\theta_{j}\ |\ m_{\theta},u_{\theta}$ $\displaystyle\sim
N(m_{\theta},u^{-1}_{\theta})$ $\displaystyle b_{i}\ |\ m_{b},u_{b}$
$\displaystyle\sim N(m_{b},u^{-1}_{b})$ $\displaystyle m_{\theta},m_{b}$
$\displaystyle\sim N(0,10^{6})$ $\displaystyle u_{\theta},u_{b}$
$\displaystyle\sim\Gamma(1,1)$

#### Estimating Model Ability

Estimating the ability of a model at a point in time is done with a “scoring”
function. When example difficulties are known, model ability is estimated by
maximizing the likelihood of the data given the response patterns and the
example difficulties to obtain the ability estimate. All that is required is a
single forward pass of the model on the data, as is typically done with a test
or validation set.

$\displaystyle Z_{j}$ $\displaystyle=\forall_{y\in
Y}\mathbf{I}[y_{i}=\hat{y_{i}}]$ (5) $\displaystyle\hat{\theta}_{j}$
$\displaystyle=\operatorname*{arg\,max}_{\theta_{j}}\prod_{i=1}^{I}p(z_{ij}=y_{ij}|\theta_{j})$
(6)

###  2.4 Dynamic Data selection for Curriculum Learning via Ability Estimation

We propose DDaCLAE, where training examples are selected dynamically at each
training epoch based on the estimated ability of the model at that epoch. With
DDaCLAE, model ability can be estimated according to a well-studied
psychometric framework as opposed to heuristics. The estimated ability of the
model at a given epoch $e$, $\hat{\theta}_{e}$, is on the same scale as the
difficulty parameters of the data, so there is a principled approach for
selecting data at any given training epoch.

Algorithm 1 describes the training procedure. The first step of DDaCLAE is to
estimate the ability of the model using the scoring function (§2.3, Alg. 1
line 2). To do this we use the full training set, but crucially, only to get
response data, not to update parameters (i.e., no backward pass). We do not
use a held out development set for estimating ability because we do not want
the development set to influence training. In our experiments the development
set is only used for early stopping. Model outputs are obtained for the
training set, and graded as correct or incorrect as compared to the gold
standard label (Alg. 1 line 8). This response pattern is then used to estimate
model ability at the current epoch ($\hat{\theta}_{e}$, Alg. 1 line 9).

Once ability is estimated, data selection is done by comparing estimated
ability to the examples’ difficulty parameters. Each example in the training
examples has an estimated difficulty parameter ($b_{x}$). If the difficulty of
an example is less than or equal to the estimated ability, then the example is
included in training for this epoch. Examples where the difficulty is greater
than estimated ability are not included (Alg. 1 line4). Finally, the model is
trained with the training data subset (Alg. 1 line 5).

With DDaCLAE, the training data size does not have to be monotonically
increasing. DDaCLAE adds or removes training data based not on a fixed step
schedule but rather by probing the model at each epoch and using the estimated
ability to match data to the model (Figure 1). This way if a model has a high
estimated ability early in training, then more data can be added to the
training set more quickly, and learning isn’t artificially slowed down due to
the curriculum schedule. If a model’s performance suffers when adding data too
quickly, then this will be reflected in lower ability estimates, which leads
to less data selected in the next epoch.

Algorithm 1 DDaCLAE

Input: Data (X, Y), model $\phi$, difficulties D, num_epochs
Output: Learned model $\phi$

1:for $e$ in num_epochs do

2: $\hat{Y}=\phi(X)$

3: $\hat{\theta}_{e}=score(Y,\hat{Y},D)$

4: $X_{e},Y_{e}=\\{(x,y):b_{x}\leq\hat{\theta}_{e}\\}$

5: $train(\phi,X_{e},Y_{e})$

6:end for

7:procedure score($Y,\hat{Y},D$)

8: $Z=\forall_{y\in Y}\mathbf{I}[y_{i}=\hat{y_{i}}]$

9: $\hat{\theta}_{e}=\operatorname*{arg\,max}_{\theta}p(Z|\theta,b)$

10: return $\hat{\theta_{e}}$

11:end procedure

##  3 Data and Experiments

###  3.1 Data

For our experiments we consider the GLUE English-language classification tasks
Wang et al. (2019): MNLI Williams et al. (2018),
QQP,333https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs QNLI
Rajpurkar et al. (2016); Wang et al. (2019), SST-2 Socher et al. (2013), MRPC
Dolan and Brockett (2005), and RTE Bentivogli et al. (2009). Data set summary
statistics are provided at Table 1. We exclude the WNLI data set.444See
https://gluebenchmark.com/faq note 12.

Because test set labels for our tasks are only available via the GLUE
evaluation server, we use the held-out development sets to measure
performance. For training, we do a 90%-10% split of the training data, and use
the 10% split as our held out development set for early stopping. We can then
use the full development set as our test set to evaluate performance without
making multiple submissions to the GLUE server.

Data set | Train | Dev | Test
---|---|---|---
MNLI | 353k | 39k | 9.8k
MRPC | 3.3k | 366 | 409
QNLI | 94k | 10k | 5.5k
QQP | 327k | 36k | 40k
RTE | 2.2k | 249 | 278
SST-2 | 61k | 6.7k | 873
Table 1: Statistics for our experiments. Note that these values differ from
the GLUE server data (see §3.1).

###  3.2 Generating Response Patterns

To learn the difficulty parameters of the data we require a data set of
response patterns. Gathering enough labels for each example to fit an IRT
model would be prohibitively expensive for human annotators. In addition, the
annotation quality may be suspect due to the humans labeling tens of thousands
of examples. Therefore we used artificial crowds to generate our response
patterns. Prior work has shown that this is an effective way to generate a set
of response patterns for fitting IRT models to machine learning data Lalor et
al. (2019).

Briefly, for each data set an ensemble of neural network models are trained
with different subsets of the training data. Training data is subsampled and
corrupted via label flipping so that performance across models in the ensemble
is varied. Each trained model then labels all of the examples
(train/validation/test), which are graded against the gold-standard label. The
output response patterns are used to fit an IRT model for the data (§2.2).

###  3.3 Experiments

To demonstrate the effectiveness of DDaCLAE we compare against a fully
supervised baseline as well as a competence-based method (CB) that uses a
fixed, monotonically-increasing schedule to add examples during training
Platanios et al. (2019). All experiments described were run five times.
Average performance and 95% CI are reported below.

For each task, we trained two standard model architecture for a set number of
epochs: $\text{BERT}_{base}$ and LSTM. We use the $\text{BERT}_{\text{base}}$
model Devlin et al. (2018) as implemented by
HuggingFace.555https://github.com/huggingface/pytorch-transformers Each model
was trained for 10 epochs, with a learning rate of 2e-5 and a batch size of 8.
Dropout for all fine-tuning layers was set to $0.1$. We used gradient norm
clipping at $1$ to avoid exploding gradients.

The LSTM model consists of a 300D LSTM sequence-embedding layer Hochreiter and
Schmidhuber (1997) (one or two LSTMs for single- and two-sentence tasks,
respectively). The sentence encodings are then concatenated and passed through
three tanh layers. Finally, the output is passed to a softmax classifier layer
to output class probabilities. The models were implemented in DyNet Neubig et
al. (2017). Models were trained with SGD for 100 epochs with a learning rate
of 0.1, and dev set accuracy was used for early stopping.

Training data available to the model at each epoch varied according to the
curriculum applied:

  1. 1.

Fully Supervised: At each epoch, the model has access to the full training set

  2. 2.

CB-Linear: The proportion of training examples to include at time $t$ is

$c_{linear}(t)\overset{\Delta}{=}\text{min}(1,t\frac{1-c_{0}}{T}+c_{0})$

  3. 3.

CB-Root: The proportion of training examples to include at time $t$ is

$c_{sqrt}(t)\overset{\Delta}{=}\text{min}(1,\sqrt{t\frac{1-c_{0}^{2}}{T}+c_{0}^{2}})$

  4. 4.

DDaCLAE: At each epoch, model ability is estimated ($\hat{\theta}_{e}$, see
§2.4) and all examples where $b_{x}\leq\hat{\theta}_{e}$ are included

| MNLI | MRPC | QNLI | QQP | RTE | SST2
---|---|---|---|---|---|---
Fully Supervised | 72.8 [$\pm$13.1] | 76.76 [$\pm$4.19] | 88.32 [$\pm$1.86] | 84.87 [$\pm$7.08] | 59.35 [$\pm$3.55] | 87.55 [$\pm$4.77]
CB Lin. ($d_{length}$) | 72.37 [$\pm$12.9] | 74.95 [$\pm$3.65] | 81.63 [$\pm$10.5] | 90.07 [$\pm$0.22] | 58.77 [$\pm$2.55] | 73.78 [$\pm$12.2]
CB Lin. ($d_{irt}$) $\clubsuit$ | 82.36 [$\pm$0.28] | 84.85 [$\pm$1.24] | 87.9 [$\pm$1.14] | 90.13 [$\pm$0.06] | 58.27 [$\pm$3.46] | 90.05 [$\pm$0.26]
CB Root ($d_{length}$) | 82.17 [$\pm$0.17] | 77.11 [$\pm$3.37] | 81.74 [$\pm$10.6] | 84.79 [$\pm$6.97] | 57.26 [$\pm$2.82] | 76.93 [$\pm$11]
CB Root ($d_{irt}$) $\clubsuit$ |  82.46 [$\pm$0.12] |  84.95 [$\pm$0.96] |  88.48 [$\pm$1.16] |  90.27 [$\pm$0.05] |  62.67 [$\pm$1.4] |  90.73 [$\pm$0.22]
DDaCLAE $\clubsuit$ | 81.65 [$\pm$0.37] | 81.37 [$\pm$2.19] | 89.33 [$\pm$0.75] | 90.15 [$\pm$0.13] | 59.71 [$\pm$2.63] | 90.99 [$\pm$0.37]
Table 2: BERT dev set accuracy results, including 95% CI, for each task under consideration. During training, 10% of the training set was held out and used for early stopping. Highest overall accuracy is bolded. Highest accuracy among competence-based methods is underlined. $\clubsuit$ indicates newly proposed methods. | MNLI | MRPC | QNLI | QQP | RTE | SST2
---|---|---|---|---|---|---
Fully Supervised | 64.96 [$\pm$0.18] | 69.04 [$\pm$0.71] | 62.72 [$\pm$0.48] | 81 [$\pm$0.12] | 50.12 [$\pm$0.68] | 84.73 [$\pm$0.34]
CB Lin. ($d_{length}$) | 63.79 [$\pm$0.23] | 68.38 [$\pm$0.55] | 58.97 [$\pm$0.84] | 79.99 [$\pm$0.21] | 49.96 [$\pm$1.2] | 83.97 [$\pm$0.39]
CB Lin. ($d_{irt}$) $\clubsuit$ | 65.18 [$\pm$0.14] |  70.18 [$\pm$0.5] | 61.21 [$\pm$0.5] | 81.68 [$\pm$0.11] | 50.45 [$\pm$1.06] | 84.71 [$\pm$0.35]
CB Root ($d_{length}$) | 64.05 [$\pm$0.39] | 69.36 [$\pm$0.35] | 60.04 [$\pm$0.73] | 79.23 [$\pm$0.55] | 50.32 [$\pm$1.62] | 83.74 [$\pm$0.43]
CB Root ($d_{irt}$) $\clubsuit$ |  65.37 [$\pm$0.16] | 69.98 [$\pm$0.52] |  61.87 [$\pm$0.27] |  81.91 [$\pm$0.08] |  50.93 [$\pm$0.69] |  84.74 [$\pm$0.12]
DDaCLAE $\clubsuit$ | 65.93 [$\pm$0.24] | 70.38 [$\pm$0.62] | 63.01 [$\pm$0.4] | 81.75 [$\pm$0.11] | 52.35 [$\pm$1.2] | 85.7 [$\pm$0.19]
Table 3: LSTM dev set accuracy results, including 95% CI, for each task under
consideration. During training, 10% of the training set was held out and used
for early stopping. Highest overall accuracy is bolded. Highest accuracy among
competence-based methods is underlined. $\clubsuit$ indicates newly proposed
methods.

For the competence-based methods, $t$ is the current time-step in training,
$T$ is the point where the model is fully competent, $c_{0}$ is the initial
competency. We set $c_{0}=0.01$ as per the original paper and set $T$ to be
equal to $\frac{numepochs}{2}$ Platanios et al. (2019). The competence-based
models reach “competency” halfway through training and train with the full
training set for the second half.

To determine the effectiveness of difficulty as estimated by IRT methods, we
experiment with two versions of the competency-based models in our NLP tasks:
(1) $d_{length}$: using sentence length as a heuristic for difficulty, as in
the prior work Platanios et al. (2019). For sentence-pair tasks such as MNLI
we use the length of the first sentence for $d_{length}$. (2) $d_{irt}$:
difficulty as estimated by fitting an IRT model using the artificial crowd
(§2.2). $d_{length}$ is one of two common heuristics used for difficulty in
prior work. Word rarity, where the negative log likelihood of the sentence is
computed, has also been proposed as a heuristic Platanios et al. (2019).
Recent empirical evaluations have shown that word rarity and sentence length
perform similarly as heuristics, and we therefore use sentence length as our
heuristic for comparison Platanios et al. (2019).

It is worth noting here that neither CB-Linear nor CB-Root actually measure
competency of the model at any point. Instead it is assumed that the model
becomes more and more competent over time, whereas with DDaCLAE model
competency is probed at each training epoch and training data is selected
based on this competency.

##### Estimating Ability

For DDaCLAE, there is a potentially significant cost associated with
estimating $\theta_{e}$. Estimating $\theta_{e}$ requires an additional
forward pass through the training data set to gather the labels for scoring as
well as MLE estimation (2.3). For large data sets this can effectively double
the number of forward passes during training. To alleviate the extra cost, we
sample from the training set before our first epoch, and use this down-sampled
subset as our ability estimation set. As most examples have difficulty values
between $-3$ and $3$, the full training set isn’t necessary for estimating
$\theta_{e}$. For our experiments we sampled $1000$ examples for ability
estimation, significantly reducing the cost. Identifying the optimal number of
examples needed to estimate ability is left for future work.

##  4 Results

###  4.1 Analysis of Difficulty Heuristic

To explore the discrepancy between common difficulty heuristics and a learned
difficulty parameter, we calculated the Spearman rank-order correlation
between difficulty using the sentence-length heuristic, $d_{length}$, and
difficulty as estimated by IRT, $d_{irt}$ (Table 4). In most cases,
correlation is around $0$, indicating no (or minimal) correlation between the
two values. In fact for some tasks the correlation is negative (e.g., $-0.19$
for MRPC). For SST-2, there is a moderate positive correlation, indicating
that some short examples are easier for the task of sentiment analysis.
However, the lack of (or negative) correlation in other tasks indicates that
sentence length, a common heuristic for difficulty in curriculum learning
work, is not aligned with the more theoretically-grounded difficulty estimates
of IRT.

| SST-2 | MRPC | QNLI | QQP | RTE | MNLI
---|---|---|---|---|---|---
$\rho$ | 0.21 | -0.19 | 0.01 | -0.09 | -0.02 | 0.03
Table 4: Spearman rank-order correlations between length difficulty heuristic
and IRT-estimated difficulty.

###  4.2 Replacing Difficulty Heuristics (H1)

By replacing difficulty heuristics with learned difficulty parameters in a
static curriculum learning framework, we see that performance is improved for
all GLUE classification tasks, with both BERT and LSTM models (Tables 2 and
3). Using learned difficulty parameters outperforms both the fully supervised
baseline and the equivalent curriculum learning strategy with difficulty
heuristics (e.g., comparing CB Root ($d_{irt}$) with CB Root ($d_{length}$)).

This result confirms our first hypothesis (H1) and demonstrates that learning
difficulty parameters for data leads to more effective curriculum learning
models. The models are able to leverage a theoretically-based difficulty
metric instead of a heuristic such as sentence length.

| MNLI | MRPC | QNLI | QQP | RTE | SST2
---|---|---|---|---|---|---
Fully Supervised | 1.8 [$\pm$0.26] | 5.2 [$\pm$1.51] | 3.4 [$\pm$0.78] | 3.6 [$\pm$1.14] | 3.2 [$\pm$1.12] | 2.6 [$\pm$0.78]
CB Lin. ($d_{length}$) |  6 [$\pm$0.41] |  6.6 [$\pm$1.78] | 5.4 [$\pm$1.47] | 7.8 [$\pm$0.49] | 8.4 [$\pm$0.67] | 5.6 [$\pm$1.73]
CB Lin. ($d_{irt}$) $\clubsuit$ |  6 [$\pm$0] | 7.6 [$\pm$1.28] | 5.6 [$\pm$0.32] | 8 [$\pm$0.41] | 6.2 [$\pm$1.51] | 8 [$\pm$0.83]
CB Root ($d_{length}$) |  6 [$\pm$0.72] | 7 [$\pm$1.8] | 4.4 [$\pm$1.14] | 7.4 [$\pm$1.28] |  5 [$\pm$1.89] |  5.2 [$\pm$1.57]
CB Root ($d_{irt}$) $\clubsuit$ | 6.2 [$\pm$0.26] | 7 [$\pm$1.17] | 6 [$\pm$0.72] | 7.8 [$\pm$0.64] | 8 [$\pm$0.72] | 7.8 [$\pm$0.96]
DDaCLAE $\clubsuit$ | 8.4 [$\pm$0.98] | 9.4 [$\pm$0.32] |  2 [$\pm$0.41] |  5.8 [$\pm$0.76] | 6.2 [$\pm$1.57] | 5.8 [$\pm$0.76]
Table 5: Mean number of training epochs to convergence (lower is better) for BERT models, with 95% CI. The best overall is bolded, and the best among the CB methods is underlined. $\clubsuit$ indicates newly proposed methods. | MNLI | MRPC | QNLI | QQP | RTE | SST2
---|---|---|---|---|---|---
Fully Supervised | 17.25 [$\pm$1.7] | 14.17 [$\pm$4.13] | 19.17 [$\pm$5.03] | 78.6 [$\pm$15.2] | 58 [$\pm$6.18] | 43.67 [$\pm$8.68]
CB Lin. ($d_{length}$) | 37 [$\pm$0.72] | 59.2 [$\pm$3.39] | 85.6 [$\pm$16.54] | 80 [$\pm$10.54] | 53.8 [$\pm$10.93] | 73.6 [$\pm$6.48]
CB Lin. ($d_{irt}$) $\clubsuit$ | 39 [$\pm$0.96] | 37.33 [$\pm$3.89] | 55.08 [$\pm$8.15] | 86.1 [$\pm$5.25] | 63.92 [$\pm$9.44] | 77.33 [$\pm$4.44]
CB Root ($d_{length}$) | 30.67 [$\pm$1.1] | 39.4 [$\pm$1.41] | 69.2 [$\pm$21.02] | 77.5 [$\pm$13.1] | 43.6 [$\pm$11.41] | 63 [$\pm$1.09]
CB Root ($d_{irt}$) $\clubsuit$ | 29.5 [$\pm$0.8] |  30.5 [$\pm$4.93] | 37.75 [$\pm$6.86] | 80.7 [$\pm$7.11] | 56.17 [$\pm$10.19] | 68.42 [$\pm$3.85]
DDaCLAE $\clubsuit$ |  20.2 [$\pm$1.33] | 46.33 [$\pm$13.1] |  23.8 [$\pm$2] |  68.6 [$\pm$14.8] |  26.17 [$\pm$14.29] |  50.33 [$\pm$6.68]
Table 6: Mean number of training epochs to convergence (lower is better) for
LSTM models, with 95% CI. The best overall is bolded, and the best among the
CB methods is underlined. $\clubsuit$ indicates newly proposed methods.

###  4.3 Dynamic Curriculum Learning (H2)

For the $\text{BERT}_{base}$ model, DDaCLAE outperforms the fully supervised
baseline as well as all other curriculum learning methods on 2 of the 6 tasks
(QNLI and SST-2, Table 2). However, we find that DDaCLAE does not lead to
further performance improvements on the other tasks for $\text{BERT}_{base}$.

DDaCLAE does give the best performance for 5 of the 6 GLUE tasks (all except
QQP) when used to train the LSTM model (Table 3). This could be due to the
fact that training the $\text{BERT}_{base}$ models is a fine-tuning procedure
against the already pre-trained models. Therefore there is not much room for
performance improvement switching from a static to a dynamic curriculum
learning model. On the other hand, the LSTM models are all randomly
initialized, and therefore require a full training procedure. In scenarios
like this, DDaCLAE is an effective procedure to improve performance. With
DDaCLAE, the model is trained using data that is most appropriate for its
current ability. Examples that are too hard are not included too early.

One potential issue with DDaCLAE is the chance of a high variance model, due
to the additional step of estimating model ability during training. However we
find that variance in terms of output performance is quite low for both
$\text{BERT}_{base}$ and LSTM models trained with DDaCLAE.

###  4.4 Training Efficiency (H3)

In addition to test-set performance, we analyzed the efficiency of the
curriculum learning training methods. For each experiment, we calculated the
average number of training epochs required to reach the point of early
stopping (based on held out dev set accuracy). For $\text{BERT}_{base}$, fully
supervised training is almost always the most efficient (Table 5). This should
not be surprising, as the model is already pre-trained, and fine-tuning only
requires a small number of passes over the task data.

For training the LSTM model, efficiency results are more mixed (Table 6). In
most cases the fully-supervised training is again most efficient, however
DDaCLAE does not incur significant efficiency costs. For QQP and RTE, DDaCLAE
is the most efficient training paradigm. For MNLI, QNLI, and SST-2, DDaCLAE
efficiency is within the 95% CI of the baseline results. Recall that for the
LSTM model DDaCLAE is the most effective in terms of test set accuracy as
well, so we can say that the improved test set performance does not come at
the cost of training time efficiency.

###  4.5 Distribution of Difficulty

Figure 3 shows percentage plots of estimated difficulty for two of the GLUE
classification tasks, QNLI and MRPC. As the plots show, the distribution in
difficulty varies between the tasks. For MRPC, there are more difficult
examples, percentage-wise, than in the QNLI data set. This reflects the
current state of the GLUE leaderboard, where the top-performing model
accuracies are 97.8% and 92.6% on QNLI and MRPC, respectively.666Scores as of
the time of this submission. This is also reflected in our results, where
model performance is higher for QNLI than MRPC (Table 2). Knowing the
distribution of difficulty in a data set is useful information for model
development and evaluation strategies. In the case of curriculum learning we
leverage this learned difficulty to train our models.

(a) QNLI

(b) MRPC

Figure 3: Percentage plot of difficulty values in QNLI and MRPC data sets.
Please note that x-axes are intentionally scaled to reflect the range for a
given data set, and are not consistent between plots. Additional plots for
other tasks included in the supplemental material.

###  4.6 Additional Training Time Considerations

Estimating model ability every training epoch with DDaCLAE can potentially
increase training time significantly. If at each training epoch there is a
need to run a full MLE optimization, the cost in terms of time could
significantly outweigh performance improvements. To mitigate this we randomly
sample examples for ability estimation.

Comparing training with DDaCLAE to training a fully-supervised baseline, the
average impact on training time ranges from an additional few minutes for
smaller data sets (e.g., MRPC) to an additional few hours for the larger data
sets (e.g., MNLI). This impact grows with the data set size because when
estimating ability, all of the training examples are used to generate a
response pattern, then a subset of $1000$ are selected for estimating ability.
Future implementations can sample the training data before gathering response
patterns, or pre-select a subset with varying difficulty parameters and to use
as a static “probe set” to estimate ability at each epoch.

##  5 Related work

Curriculum learning is a well-studied area of machine learning Bengio et al.
(2009). The primary focus has been on developing new heuristics to identify
easy and difficult examples in order to build a curriculum. Originally,
curriculum learning methods were evaluated on toy data sets with heuristic
measures of difficulty Bengio et al. (2009). For example, on a shapes data
set, shapes with more sides were considered difficult. Similarly, longer
sentences were considered difficult. Word rarity has also been proposed as a
heuristic for difficulty, with similar results to sentence length Platanios et
al. (2019). Recent work on automating curriculum learning strategies use
multi-arm bandits to minimize regret with respect to curriculum selection
Graves et al. (2017). In that work the authors again rely on proxies for
progress (loss-driven and complexity-driven). Loss-driven proxies are
inherently model-specific, in that the difficulty of an example is determined
by a specific model’s performance on the example. Using a global difficulty
such as one learned using IRT methods allows for an interpretable difficulty
metric that applies across models. The complexity-driven proxies proposed are
specific to neural networks, while DDaCLAEis a generic algorithm for dynamic
curriculum learning.

Spaced repetition strategies (SR) can be effective for improving model
performance Amiri et al. (2017); Amiri (2019). Instead of using a traditional
curriculum learning setup, spaced repetition bins examples based on estimated
difficulty, and shows bins to the model at differing intervals so that harder
examples are seen more frequently than easier ones. This method has been shown
to be effective for human learning, and results demonstrate effectiveness on
NLP tasks as well. Similarly to traditional curriculum learning, SR uses
heuristics for difficulty and rigid schedulers to determine when examples
should be re-introduced to the learner.

Self-paced learning (SPL) is another related strategy for example ordering
during training Kumar et al. (2010). In our work the difficulty values of
examples are global and static, where for SPL examples do not have set
difficulty values; instead, groups of examples are considered in sets. In
addition, model competency is not considered in SPL, it is assumed that
competency improves as more difficult examples are added.

There has been recent work investigating the theory behind curriculum learning
Weinshall et al. (2018); Hacohen and Weinshall (2019), particularly around
trying to define an ideal curriculum. The authors explicitly identify the two
key aspects of curriculum learning, namely “sorting by difficulty” and
“pacing.” Curriculum learning theoretically leads to a steeper optimization
landscape (i.e., faster learning) while keeping the same global minimum of the
task without curriculum learning. In that work there is still a reliance on
“pacing functions” as opposed to an actual assessment of model ability at a
point in time. This work may open interesting new areas of theoretical study
linking difficulty and ability in curriculum learning.

Theoretical results Hacohen and Weinshall (2019) have also demonstrated a key
distinction between curriculum learning and similar methods such as self-paced
learning Kumar et al. (2010), hard example mining Shrivastava et al. (2016),
and boosting Freund and Schapire (1997): namely that the former considers
difficulty with respect to the final hypothesis space (i.e., a model trained
on the full data set) while the later methods consider ranking examples
according to how difficult the current model determines them to be. DDaCLAE
bridges a gap between these methods by probing model ability at the current
point in training and using this ability to identify appropriate training
examples in terms of global difficulty.

In addition, a key component of most prior work in curriculum learning is the
notion of balance. When defining a curriculum, it is often the case that
proportions are maintained between classes. That is, difficulty itself is not
the only factor when building the curriculum. Instead, the easiest examples
for each class are added so that the model is proportionally exposed to the
data consistent with the full training set. DDaCLAE does not consider class
labels when selecting examples for training. It is important to note here that
labels are used when learning difficulties, estimating ability, and actually
updating parameters during training. They are not used to balance the
curriculum. In this way DDaCLAE is more closely aligned with a pure curriculum
learning strategy that considers only the easiness/hardness of an example
during training. This is an added benefit to DDaCLAE as there is no need for
class label accounting during training.

##  6 Conclusion

This work validates and supports the existing literature on curriculum
learning. Our results confirm that curriculum learning methods for supervised
learning can lead to faster convergence or better local minima, as measured by
test set performance Bengio et al. (2009). We have shown that by replacing a
heuristic for difficulty with a theoretically-based, learned difficulty value
for training examples, static curriculum learning strategies can be improved.
We have also proposed DDaCLAE, the first curriculum learning method to
dynamically probe a model during training to estimate model ability at a point
in time. Knowing the model’s ability allows for data to be selected for
training that is appropriate for the model and is not rigidly tied to a
heuristic schedule. DDaCLAE trains more effective models in most cases that we
considered, particularly for randomly initialized LSTM models.

Based on our experiments, we report mixed results on our stated hypotheses.
Replacing heuristics with learned difficulty values leads to improved
performance when training models with curriculum learning, supporting H1.
DDaCLAE does outperform other training setups when used to train LSTM models.
Results are mixed when used to fine-tune the $\text{BERT}_{base}$ model.
Therefore H2 is partially supported. We see similarly mixed results when
evaluating efficiency. With $\text{BERT}_{base}$ fine-tuning, fully supervised
fine-tuning is usually the most efficient, as the number of fine-tuning epochs
needed is already very low. For LSTM, DDaCLAE is more efficient than the other
curriculum learning strategies, and is the most efficient overall for two of
the six tasks. H3 is partially supported by these results.

Even though it is dynamic, DDaCLAE employs a simple schedule: only include
examples where $b_{x}\leq\hat{\theta}_{e}$. However, being able to estimate
ability on the fly with DDaCLAE opens up an exciting new research direction:
what is the best way to build a curriculum, knowing example difficulty and
model ability (e.g., the 85% rule of Wilson et al., 2019)?

## Acknowledgements

The authors would like to thank Hao Wu and Hadi Amiri for their helpful
conversations with regards to this work. We thank the anonymous reviewers for
their comments and suggestions. This work was supported in part by LM012817
from the National Institutes of Health. This work was also supported in part
by the Center for Intelligent Information Retrieval at UMass Amherst, and the
Center for Research Computing and Mendoza College of Business at the
University of Notre Dame. The contents of this paper do not represent the
views of CIIR, NIH, VA, Notre Dame, University of Massachusetts Lowell or the
United States Government.

## References

  * Amiri (2019) Hadi Amiri. 2019.  Neural Self-Training through Spaced Repetition.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 21–31.
  * Amiri et al. (2017) Hadi Amiri, Timothy Miller, and Guergana Savova. 2017.  Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2401–2410.
  * Baker and Kim (2004) Frank B. Baker and Seock-Ho Kim. 2004.  _Item Response Theory: Parameter Estimation Techniques, Second Edition_.  CRC Press.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 41–48. ACM.
  * Bentivogli et al. (2009) Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009.  The fifth PASCAL recognizing textual entailment challenge.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805 [cs]_.  ArXiv: 1810.04805.
  * Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the International Workshop on Paraphrasing_.
  * Freund and Schapire (1997) Yoav Freund and Robert E Schapire. 1997.  A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting.  _J. Comput. Syst. Sci._ , 55(1):119–139.
  * Graves et al. (2017) Alex Graves, Marc G Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. 2017.  Automated curriculum learning for neural networks.  In _International Conference on Machine Learning_ , pages 1311–1320.
  * Hacohen and Weinshall (2019) Guy Hacohen and Daphna Weinshall. 2019.  On The Power of Curriculum Learning in Training Deep Networks.  In _ICML 2019_.  ArXiv: 1904.03626.
  * Hochreiter and Schmidhuber (1997) S Hochreiter and J Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780.
  * Kumar et al. (2010) M. P. Kumar, Benjamin Packer, and Daphne Koller. 2010.  Self-Paced Learning for Latent Variable Models.  In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, _Advances in Neural Information Processing Systems 23_ , pages 1189–1197. Curran Associates, Inc.
  * Lalor et al. (2019) John P. Lalor, Hao Wu, and Hong Yu. 2019.  Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing_ , volume 2019. Association for Computational Linguistics.
  * Natesan et al. (2016) Prathiba Natesan, Ratna Nandakumar, Tom Minka, and Jonathan D. Rubright. 2016.  Bayesian Prior Choice in IRT Estimation Using MCMC and Variational Bayes.  _Frontiers in Psychology_ , 7.
  * Neubig et al. (2017) Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017.  Dynet: The dynamic neural network toolkit.  _arXiv preprint arXiv:1701.03980_.
  * Platanios et al. (2019) Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019.  Competence-based Curriculum Learning for Neural Machine Translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1162–1172, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of EMNLP_ , pages 2383–2392. Association for Computational Linguistics.
  * Rasch (1960) Georg Rasch. 1960.  _Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests_.  Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche, Oxford, England.
  * Shrivastava et al. (2016) Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. 2016.  Training region-based object detectors with online hard example mining.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 761–769.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, D. Christopher Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In the Proceedings of ICLR.
  * Weinshall et al. (2018) Daphna Weinshall, Gad Cohen, and Dan Amir. 2018.  Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks.  In _ICML 2018_.  ArXiv: 1802.03796.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of NAACL-HLT_.
  * Wilson et al. (2019) Robert C. Wilson, Amitai Shenhav, Mark Straccia, and Jonathan D. Cohen. 2019.  The eighty five percent rule for optimal learning.  _Nature Communications_.


Student gender modulates the intersection of calculus proficiency and calculus
self-efficacy in an introductory electricity and magnetism course

Christopher J. Fischer*, Jennifer Delgado, Sarah LeGresley, Jessy Changstrom

Department of Physics and Astronomy, University of Kansas, 1251 Wescoe Hall
Drive, Lawrence, KS, 66049

*<EMAIL_ADDRESS>
## Abstract

We assessed changes in calculus proficiency and calculus self-efficacy in a
second semester course of introductory physics focused on electricity and
magnetism. While all students demonstrated an increase in calculus
proficiency, including a possible improvement in calculus transfer to physics,
women displayed larger gains than men. Conversely, men showed larger gains in
calculus self-efficacy. When combined, these data suggest that student
identity modulates the correlation between a student’s calculus abilities and
their perception or self-evaluation of those abilities. These data highlight a
potential contributing factor to gender-related differences in physics self-
efficacy as well as the complexity of addressing those differences.

##  1 Introduction

The ability to use mathematics is an essential skill common to all science and
engineering disciplines. Because of this and due to economic and logistic
constraints, courses in introductory mathematics at most colleges and
universities are shared by many degree programs and centrally taught in a
department of mathematics. Unfortunately, this structure can result in the
mathematics being decontextualized, thereby placing on students the burden of
transferring their skills in solving mathematical problems to other STEM
areas. However, while students are generally expected to be capable of this
transference, evidence suggests that it actually poses a large barrier to
student learning in physics [4, 6, 29, 25]. Indeed, the emphasis on applied
mathematics in introductory physics courses is believed to be responsible for
the observed positive correlation between mathematics ability and student
performance in these courses [16, 5, 17, 3]. Although we and others have
suggested how calculus proficiency in introductory physics and possibly
mathematics transfer to physics may be improved [19, 18, 39, 36], a
quantitative analysis of how curriculum changes and/or interventions in
introductory physics affect student ability to solve calculus problems in the
context of physics has not yet been completed. As part of an ongoing effort to
address this need, we report here our analysis of proficiency with integral
calculus in the context of introductory physics, covering electricity and
magnetism, combined with an independent assessment of calculus self-efficacy.
We show that while a positive correlation between student self-assessment of
calculus ability and student proficiency with calculus exists for men, a
negative correlation between these traits is present for women, even though
women perform equal to or better than men on assessments of calculus
proficiency. These results thus build off our previous work indicating an
identity-modulated correlation between physics self-efficacy and calculus
proficiency [9].

##  2 Background

The suggested schedule for most calculus-based STEM (e.g., physical sciences
and engineering) degree programs at the University of Kansas has students
enrolling in General Physics II (the second semester of calculus-based
introductory physics covering electricity and magnetism) during their third
semester at the same time they are enrolled in the third semester of calculus;
General Physics I is taken alongside the second semester of calculus. The
coordinating scheduling of calculus and introductory physics courses allows us
to task students with using calculus continually in their physics courses [19,
10, 11] and thus to assess how well students transfer their calculus knowledge
and skills to physics. When characterizing and analyzing this knowledge
transfer, we use the model proposed by Rebello et al. [29], which builds upon
earlier work by Gagné [13] and Reddish [30]. In this context, we make a
distinction between horizontal transfer, which describes the use of new
information in existing mental models and vertical transfer in which an
existing mental model does not exist, thus requiring the creation of new
mental models [6, 9].

Self-efficacy is a measure of one’s belief to succeed [33, 27, 8]. As such, it
can affect many dimensions of learning including motivation [28], interest
[42], engagement [34], and how difficult tasks are perceived [40]. Not
surprisingly, self-efficacy influences both performance and retention in STEM
courses [33] and women often have lower self-efficacy in physics than men
despite performing similarly or better than men in their physics courses [27,
23, 8].

###  2.1 Research Questions

This work seeks to address the following research questions (RQ) in the
context of our calculus-based introductory physics courses:

> RQ1. How does the emphasis on calculus in introductory physics courses
> affect student ability with pure and applied calculus?

> RQ2. How is student self-efficacy for using calculus in introductory physics
> correlated with student proficiency for pure and applied calculus?

##  3 Methodology

Students in General Physics II at the University of Kansas are tasked with
using integral calculus to calculate the electric and magnetic fields and
potentials of distributions of electric charge, electric and magnetic flux,
and the electric and magnetic forces and potential energies associated with
the interactions of charged systems. Our survey for assessing calculus
proficiency in this course builds upon a previously implemented assessment [9]
and consists of the following six questions:

C1. Complete the following integral $\int\limits_{0}^{6}x^{2}dx$

C2. Complete the following integral
$\int\limits_{0}^{1}\frac{2x}{\left(x^{2}+1\right)^{2}}dx$

C3. The population of a town, denoted by the variable $P$, increases at a rate
described by the following equation: $\frac{dP}{dt}=20+0.6t^{2}$. The variable
$t$ is the time. What is the population of the time at $t=5$ if the population
at $t=0$ is 0.

C4. The electric potential energy $\left(U_{e}\right)$ of a point particle
with electric charge $q$ in the presence of an external electric potential
$\left(V_{e}\right)$ is $U_{e}=qV_{e}$. The uniformly charged straight and
rigid rod shown in the figure is in the presence of the following external
electric potential: $V_{e}=\left(4\mathrm{\frac{J}{Cm^{2}}}\right)x^{2}$. The
variable $x$ in this equation denotes the position along the $x$-axis as
defined in the figure. The rod has a length of 2 m and a constant electric
charge density of $3\mathrm{\frac{C}{m}}$. Determine the electric potential
energy associated with the interaction between the rod and the external
electric potential.

C5. The electric potential $\left(V_{e}\right)$ a distance $r$ away from a
point particle with electric charge $q$ is described by the following
equation: $V_{e}=\frac{q}{4\pi\epsilon r}$. The straight and rigid rod shown
in the figure to the right has a length of 4 m and a non-uniform linear
electric charge density described by the following equation: $\left(2\pi\times
10^{-9}\mathrm{\frac{C}{m^{2}}}\right)x$. The variable $x$ in this equation
denotes position along the $x$-axis as defined in the figure. Determine the
electric potential of this rod at the position $P$ that is a distance 3 m
above one end of the rod as shown in the figure. The variable
$\epsilon=1\times 10^{-11}\mathrm{\frac{C}{Vm}}$.

C6. The moment of inertia $\left(I\right)$ of a point particle with mass $m$
located a distance $x$ away from an axis of rotation is $I=mx^{2}$. The
straight and rigid rod shown in the figure to the right has a length of 2 m
and non-uniform mass density described by the following equation:
$\left(5\mathrm{\frac{kg}{m^{3}}}\right)x^{2}$. The variable $x$ in this
equation denotes position along the $x$-axis as defined in the figure.
Determine the moment of inertia of this rod for rotations around an axis
located at $x=0\mathrm{m}$.

We denote questions C1, C2, and C3 on this survey as “pure” calculus
questions. Conversely, we denote questions C4, C5, and C6 as “applied”
calculus questions as they require students to use calculus within a physics
context. We also note that the integration required to solve C4 and C6 is
equivalent to the integration required to solve C1. Similarly, the integration
required to solve C5 is equivalent to the integration required to solve C2.
Finally, since students at our institution have not used calculus to determine
moments of inertia in General Physics I or General Physics II [9], solving C6
requires represents a new application of integral calculus in the context of
physics. However, by prompting students with the equation for the moment of
inertia of a point particle, we hoped that student would recognize that the
integration necessary for the solution is similar to the integration necessary
to solve C1 and C4.

Our assessment of calculus self-efficacy [9] consists of the three questions
shown in Figure 1. The calculus self-efficacy survey and calculus proficiency
survey were administered at the beginning (survey 1), in the middle (survey
2), and at the end (survey 3) of the semester in General Physics II during the
Summer 2021 and Fall 2021 semesters. The responses to survey questions were
graded as favorable, neutral, or unfavorable, and given a corresponding
numeric score of 3, 2, or 1; for the calculus proficiency survey, a correct
answer was scored favorable and an incorrect answer was scored unfavorable.

Figure 1: The calculus self-efficacy questions used in this study.

The total enrollment in the courses where the surveys were deployed was 362
students, however, not all students consented to their survey results being
included in the analysis presented here. We further restricted our data set to
only those students who completed all self-efficacy surveys and all calculus
proficiency surveys so that our analysis was conducted on matched students
only. This leaves a total of 293 students in General Physics II (205 men and
88 women) included in the results presented here. Please note that since we
are relying upon institutional data for creating these cohorts of students,
these divisions do not therefore necessarily reflect how students identify
personally.

##  4 Results

###  4.1 Validation of the Survey

In our first analysis of the survey results, we calculated the discrimination
and difficulty of each calculus proficiency question [2, 32]; note that
difficulty is defined as the fraction of students answering the question
correctly [2, 32]. As shown in Figure 2, a smaller fraction of students
answered the “applied” calculus questions (C4, C5, and C6) correctly than
answered the “pure” calculus questions (C1, C2, and C3) correctly. This is
consistent with the expectation that “applied” calculus questions would be
more difficult for students, perhaps owing to difficulties of using calculus
in a physics context (i.e., with calculus transfer to physics). All questions
showed a discrimination above 0.35, suggesting that these questions provide
moderate to high discrimination [32].

Figure 2: The discrimination and difficulty of the “pure” and “applied”
calculus questions.

To establish the separability of three different categories of questions on
the assessment - pure calculus questions, applied calculus questions, and
calculus self-efficacy questions - we performed an exploratory factor analysis
on the items in the survey. A principal components analysis with Varimax
rotation method was used, and the initial eigenvalues indicated that the first
three components (all with eigenvalues greater than 1) explained a total of
51% of the variance. The factor loadings determined from these principal
components are shown in Table 1. Factor 1 is associated with calculus self-
efficacy questions (S1, S2, and S3), factor 2 is associated with the “pure
calculus” questions (C1, C2, and C3), and factor 3 is associated with the
“applied calculus” questions (C4, C5, and C6). Thus, the data supported the
existence of three separate scales and items loaded on the scales as intended.

When the fourth principal component is included in the analysis, the total
variance explained increases to 62%. Factor 1 and factor 2 generated by this
analysis are identical in composition to those determined using only the first
three eigenvectors. The newly determined factor 3 is associated with question
C3 and question C6, and the new factor 4 is associated with question C5. We
ascribe this splitting into two factors to describe the “applied calculus”
questions with the fact question C5 had the lowest correct response rate of
any of the questions (Figure 2). Regardless, this analysis is also consistent
with the presence of three separate scales - pure calculus questions, applied
calculus questions, and calculus self-efficacy questions - existing within the
survey.

Table 1: Factor loadings for each question on the assessment. Loadings greater than 0.2 are highlighted in gray. Factor | Question
---|---
C1 | C2 | C3 | C4 | C5 | C6 | S1 | S2 | S3
F1 (calculus self-efficacy) | -0.052 | 0.003 | -0.037 | -0.003 | 0.042 | -0.025 |  gray!250.416 |  gray!250.451 |  gray!250.462
F2 (pure calculus proficiency) |  gray!250.513 |  gray!250.466 |  gray!250.405 | -0.114 | -0.103 | 0.066 | -0.013 | -0.026 | -0.065
F3 (applied calculus proficiency) | -0.102 | -0.070 | 0.123 |  gray!250.514 |  gray!250.408 |  gray!250.524 | -0.112 | 0.01 | 0.051

###  4.2 Survey Results

The results for the calculus proficiency survey and calculus self-efficacy
survey are shown in Figure 3. In this figure, the average self-efficacy score
is calculated by averaging the scores on the three calculus self-efficacy
questions (Figure 1). On average, students scoring higher on the calculus
proficiency survey earned higher grades in the course, consistent with
previous work showing a correlation between mathematics ability and student
performance in physics courses [16, 5, 17, 3]. In addition, these data also
indicate that students score higher on the “pure” calculus questions than on
the “applied” calculus questions. While this result may not be surprising for
the first survey, and perhaps the second survey, as the students are still in
the process of learning the associated physics content, it is nevertheless
noteworthy that this trend persists on the third survey taken at the end of
the semester.

Figure 3: Responses to calculus proficiency survey (left panels) and calculus
self-efficacy survey (right panel) disaggregated by course grade. 190 students
earned an A (purple), 70 students earned a B (orange), and 33 students earned
a C, D, or F (green). Error bars denote the 68% confidence interval of the
standard error.

As the data shown in Figure 3 demonstrate that scores on the calculus
proficiency survey and calculus self-efficacy survey were dependent on course
grade, we next sought to determine how student identity (i.e., student gender)
modulates the interaction between calculus proficiency and calculus self-
efficacy by focusing on students earning the same course grade. The average
responses to the surveys by students earning a grade of A in General Physics
II (129 men and 61 women) are shown in Figure 4. These data show that women
score higher than men on the “pure” calculus questions on the calculus
proficiency survey, with differences between 1 and 2 pooled standard errors,
and that the difference between the scores of women and men increases
throughout the semester. Similarly, while men score higher than women on the
“applied” calculus questions on the calculus proficiency survey at the
beginning of the semester, by the end of the semester, women score higher than
men on those questions too. Figure 4 also shows the differences between the
average calculus self-efficacy scores for men and women earning an A in
General Physics II. While the average calculus self-efficacy score is higher
for women than for men at the beginning of the semester, by the end of the
semester, the situation has reversed. When the data for calculus self-efficacy
and calculus proficiency are combined, we see that as the semester progresses
women score ever better than men on calculus proficiency while scoring ever
worse than men on the calculus self-efficacy. Thus, these data indicate an
inverse relationship between the perception of calculus proficiency and the
demonstration of calculus proficiency.

It is interesting to note that while the scores on the “applied” calculus
questions increase throughout the semester, the scores on the “pure” calculus
questions decrease. Furthermore, as indicated by the Z-statistics shown in
Figure 4, these changes are not of equal magnitude for men and women. For the
“pure” calculus questions, the average score for women decreased from $(89\pm
2)\%$ to $(84\pm 3)\%$, corresponding to a Z-statistic of -1.09, while the
average score for men decreased from $(83\pm 2)\%$ to $(75\pm 3)\%$,
corresponding to a Z-statistic of -2.31; the errors indicate the 68%
confidence interval for the standard error. In contrast, for the “applied”
calculus questions, the average score for women increased from $(13\pm 2)\%$
to $(36\pm 4)\%$, corresponding to a Z-statistic of 4.74, while the average
score for men increased from $(20\pm 2)\%$ to $(27\pm 3)\%$, corresponding to
a Z-statistic of 1.92; the errors indicate the 68% confidence interval for the
standard error. Women thus displayed a larger improvement in scores on the
“applied” calculus questions and a smaller decrease in scores on the “pure”
calculus questions when compared to men.

Figure 4: Responses to surveys for women (blue) and men (red) earning an A in
General Physics II. The top panels are the percentage of correct answers for
“pure” calculus (circles) and “applied” calculus questions (squares). The
bottom panels are the average score on the calculus self-efficacy questions.
All uncertainties are the 68% confidence interval of the standard error. The
corresponding Z-statistic for the difference between the distributions for men
and women are shown in the right panels. All uncertainties are the 68%
confidence interval of the standard error.

To investigate this relationship further, we calculated the correlation
between the average self-efficacy scores, average “pure” calculus scores, and
average “applied” calculus scores for men and women, separately, for students
earning an A in the course. These correlation coefficients are shown in Figure
5 and indicate that the inverse relationship (i.e., negative correlation)
between calculus proficiency and calculus self-efficacy exist within the
population of women students, but not within the population of men students.

Figure 5: Correlation coefficients between average self-efficacy scores,
average “pure” calculus scores, and average “applied” calculus scores for men
(red) and women (blue) earning an A in the course.

###  4.3 Principal Component Analysis

Similar conclusions are found through a principal component analysis [20] of
these data. The coefficients of contributions of each survey question to the
first three of these principal components and corresponding probability
contributions are shown in Table 2; these principal components were calculated
using all matched student responses (i.e., for all students, regardless of
grade earned in the course). The average values of the first three of these
principal components for men and women earning an A in General Physics II are
shown in Figure 6. The first principal component, Z1, is dominated by the
calculus self-efficacy questions and the “pure” calculus questions with the
coefficients of all questions having the same sign (Table 2). The values of Z1
are consistently higher for women than for men (Figure 6) as the difference in
calculus self-efficacy between men and women is smaller than the difference in
calculus proficiency between men and women. In other words, the value of Z1 is
consistently larger for women than for men since decreases in calculus self-
efficacy for women occurring throughout the semester are compensated by
concomitant smaller increases in calculus proficiency by men.

As shown in Table 2, the coefficients for the contributions of the calculus
questions to the second principal component, Z2, are all positive while the
coefficients for the contributions from the calculus self-efficacy questions
to Z2 are all negative. Thus, the second principal component of these data
represents the anti-correlation between student proficiency with calculus and
their calculus self-efficacy. As shown in Figure 6, the values of this
principal component for men and women diverge over the course of the semester
with the means of the populations separated by more than two pulled standard
errors by the third survey. In light of the results shown in Figure 4, we
understand the origin of this divergence between men and women resulting from
women scoring higher than men on the calculus proficiency survey and scoring
less favorable than men on the calculus self-efficacy survey.

The third principal component, Z3, is dominated by questions from the calculus
proficiency survey with “applied” calculus questions having positive
coefficients and “pure” calculus questions having negative coefficients (Table
2). Thus, a more positive value of Z3 corresponds to answering the “applied”
calculus questions more favorably, the “pure” calculus questions less
favorably, or some combination of both. As shown in Figure 6, the values of Z3
are initially higher for men than for women, but the difference between men
and women decreases steadily throughout the semester as the value of Z3 for
women increases more than the value of Z3 for men. These changes in Z3 thus
reflect the fact that women are answering the “pure” calculus questions and
the “applied” questions more favorably than men throughout the semester, with
the gap between men and women increasing over the semester (Figure 4).

Figure 6: The first (Z1), second (Z2), and third (Z3) principal components of
the combined data set of pure math questions, applied math questions, and math
self-efficacy questions for students earning an A in General Physics II. The
left panels show the mean and standard error (68% confidence interval) of the
values of these principal components for women (blue) and men (red). The right
panels show the corresponding Z-statistic for the difference between the
values of the principal components for men and women.

Table 2: The first three principal components for all students in General
Physics II with negative coefficients highlighted in gray. These principal
components account for 21.7% (Z1), 16.3% (Z2), and 13.3% (Z3) of the total
variance in the data. For Z2, the signs of the coefficients for calculus self-
efficacy questions (S1, S2, and S3) are opposite the signs of the “pure”
calculus questions (C1, C2, and C3) and the “applied” calculus questions (C4,
C5, and C6). For Z3, the signs of the coefficients for the “pure” calculus
questions are opposite the signs for the “applied” calculus questions.

Z1 Z2 Z3 Question Coefficient Probability Cumulative Question Coefficient
Probability Cumulative Question Coefficient Probability Cumulative Probability
Probability Probability S2 0.443 0.196 0.196 C6 0.424 0.180 0.180 C4 0.504
0.254 0.254 S3 0.438 0.192 0.388 S1 gray!25-0.416 0.173 0.352 C1 gray!25-0.446
0.199 0.454 C2 0.368 0.134 0.524 S3 gray!25-0.370 0.137 0.489 C5 0.419 0.175
0.629 S1 0.365 0.133 0.657 S2 gray!25-0.366 0.134 0.623 C6 0.386 0.149 0.778
C3 0.360 0.129 0.787 C3 0.357 0.127 0.751 C2 gray!25-0.375 0.140 0.918 C1
0.338 0.114 0.901 C4 0.291 0.085 0.835 S3 0.186 0.034 0.952 C6 0.258 0.066
0.968 C1 0.278 0.077 0.913 C3 gray!25-0.181 0.033 0.985 C5 0.132 0.017 0.985
C2 0.228 0.052 0.964 S2 0.122 0.015 1.000 C4 0.123 0.015 1.000 C6 0.189 0.036
1.000 S1 0.005 $<0.001$ 1.000

###  4.4 Network Analysis

The inverse relationship between performance on the “pure” calculus questions
and the “applied” calculus questions on the calculus proficiency survey is
also clear from network analysis [7] of the combined results from the calculus
proficiency survey and the calculus self-efficacy survey. These networks were
subsequently sparsified using previously published methods [12, 7] and the
NetworkX package [14] for Python [38] to elucidate the backbone of the
corresponding networks. We employed a significance level [12, 7] of 0.3 for
these sparisifications.

As shown in Figure 7, the sparsified networks for all students earning an A in
General Physics II show a positive correlation exists between responses to the
“applied” calculus question (C4, C5, and C6) with the majority of students
answering those questions incorrectly. Responses to these questions are
inversely correlated with responses to the “pure” calculus questions (C1, C2,
and C3) and the calculus self-efficacy questions (S1, S3, and S3). This
inverse correlation between the “pure” calculus questions and the “applied”
calculus questions is also manifest in the third principal component of the
data for General Physics II (Table 2). The modular communities [26],
determined using the greedy modularity maximization algorithm within the
NetworkX package [14], are the same for both networks. The three communities
within each network consist of the “pure” calculus questions (C1, C2, and C3),
the “applied” calculus questions (C4, C5, and C6), and the calculus self-
efficacy questions (S1, S2, and S3). This result is consistent with the
exploratory factor analysis discussed above and provides further support that
three separate scales - pure calculus questions, applied calculus questions,
and calculus self-efficacy questions - exist within the survey.

Figure 7: The locally adaptive network sparsifications for students earning an
A in General Physics II using a significance level of 0.3. The questions on
surveys are represented by the nodes of these graphs and the thickness of the
edge denotes the weight of the connection [7]. A red edge denotes a positive
correlation between responses to the connected questions, with a majority of
the students answering both questions favorably. A blue edge denotes a
positive correlation between responses to the connected questions, with a
majority of the students answering both questions unfavorably. A grey edge
denotes an inverse correlation between the responses to the questions.

##  5 Discussion

The increase in correct responses to the “applied” calculus questions over the
course of the semester is consistent with the answer to RQ1 being that the
emphasis on calculus in this course may be helping students with their ability
with “applied” calculus. Furthermore, the differences observed in student
proficiency answering “pure” calculus and “applied” calculus questions shown
in Figure 3 and Figure 2 is consistent with student difficulty in
mathematization within physics and/or mathematics transfer to physics. In that
context, the results presented here are consistent with previous work
demonstrating that such transfer is difficult [4, 6, 29, 25], while also
indicating that student identity (i.e., gender) may influence transfer
proficiency.

In response to RQ2, the data presented here demonstrate that student
perception of calculus ability (i.e., calculus self-efficacy) and student
proficiency with calculus is inversely correlated for women, but not for men.
This relationship stems, at least in part, from the fact that while men
experience very little change in their calculus self-efficacy over the course
of the semester (a Z-statistic of 0.39 for the change from survey 1 to survey
3 for students earning an A in the course), women experience a significant
decrease in their calculus self-efficacy over the same period (a Z-statistic
of -1.72 for the change from survey 1 to survey 3 for students earning an A in
the course). Thus, even though proficiency with the “applied” calculus
questions increases more for women (Z-statistic of 4.74 for the change from
survey 1 to survey 3) than for men (Z-statistic of 1.92 for the change from
survey 1 to survey 2), women nevertheless perceive their proficiency to
decrease. Thus, while an emphasis on calculus in this course may help students
improve their proficiency with applied calculus, this emphasis does not appear
to impact the calculus self-efficacy of men and negatively impacts the
calculus self-efficacy of women.

###  5.1 Correlation With Physics Self-Efficacy

Our observation of a gender imbalance in calculus self-efficacy is consistent
with the results of previously reported studies of mathematics self-efficacy
[15, 31]. Furthermore, the disconnect between calculus self-efficacy and
calculus proficiency is reminiscent of previous studies showing a gender-
dependent disconnect between physics self-efficacy and physics proficiency
(using course grade as a proxy for proficiency)[27, 23, 8]. For example,
Marshman et al. showed that women had lower physics self-efficacy compared to
that of similarly performing men [23].

To explore this further, we compared the changes in physics self-efficacy to
changes in calculus self-efficacy. Physics self-efficacy was assessed using
the instrument previously developed by Marshman et al. [23]. This multiple
choice survey consists of six questions, each of which involved 4-point Likert
scales, with higher scores indicating higher levels of self-efficacy. A single
physics self-efficacy score for each student is determined by averaging the
scores for the six questions. Students completed the assessment at the
beginning, in the middle, and at the end of the semester concomitant with
completing the other surveys described above.

Figure 8: Left Panel: The normalized gain in physics self-efficacy (circles)
and calculus self-efficacy (squares) for students earning an A in the course.
Right Panel: Correlation between physics self-efficacy and calculus self-
efficacy for students earning an A in the course. The three data points for
each cohort correspond to the three deployments of the surveys.

The normalized gains in the calculus-self efficacy and physics self-efficacy
of students earning an A in the course are shown in Figure 8. For women, the
changes in physics self-efficacy and calculus self-efficacy track with one
another, possibly suggesting that whatever experiences and/or influences are
affected one trait are also affecting the other. The situation is different
for men, however, where the trajectories of changes in physics self-efficacy
and calculus self-efficacy do not overlap. Furthermore, since the changes in
physics self-efficacy for men are well aligned with the changes in physics
self-efficacy and calculus self-efficacy for women, these data suggest that
the trends in calculus self-efficacy for men are the outlier. One possible
explanation for this result is that men react to difficulties using calculus
in the course differently from how women do. For example, men may not
associate making mistakes using calculus with a lack of proficiency. Indeed,
their self-efficacy may not decrease if they perceive that other students are
also struggling to use calculus since they could associate the making of
mistakes as being common in physics courses and thus somehow not a reflection
of their intrinsic proficiency with calculus. That women would react
differently to making mistakes using calculus in a physics course could be
related to how their perception of success and belonging in that course are
influenced by being a member of an underrepresented group in the course and
the implicit and/or explicit stereotype threats associated with that status
[21, 22].

Our observation that the correlation between physics self-efficacy and
calculus self-efficacy is weaker for men than for women (Figure 8), may also
suggest that men perceive the role of calculus in the physics course
differently than women do. If so, perhaps such differing views of the role of
calculus in physics might be associated not only with the differences in
correlation shown in Figure 8, but also with differences in improvement in
applied calculus skills (i.e., calculus transfer to physics) during the
semester.

###  5.2 Limitations

While the conclusion that these data demonstrate gender-mediated correlations
between calculus proficiency and calculus self-efficacy, it is important to
note the limitations of this work. First, the surveys employed in this project
were all administered online owing partly to limitations associated with the
COVID-19 pandemic. It is possible that students were not engaged with the
surveys, particularly the calculus proficiency survey, as earnestly online as
they would have been in-person. Related to that is the possibility that the
preparation obtained by the students in this study in their previously taken
calculus courses was affected by those courses being taught online or with a
modified curriculum due to pandemic-associated accommodations.

Second, that fewer than 40% of the students were answering the “applied”
calculus questions correctly on the calculus proficiency survey (Figure 3 and
Figure 2) may suggest that some of these questions are too difficult or
otherwise not able to provide the dynamic range necessary to evaluate student
proficiency with using calculus in a physics context.

###  5.3 Future Work

The next step in this project is the further refinement of the calculus
proficiency survey. In addition to potential modification of the questions
asked, we would also welcome collaborating with other institutions as that
would provide an opportunity to collect data from a larger population of
students and potentially a wider distribution of demographics, as well as
students completing different physics curricula where expectations for
calculus usage may be different. Such data would also allow us to examine how
other dimensions of student identity intersect with calculus proficiency and
calculus self-efficacy, as well as the correlation between physics self-
efficacy and calculus self-efficacy.

We also intend to coordinate with instructors in other departments to deploy
similar surveys of calculus self-efficacy, discipline-specific calculus
proficiency, and discipline self-efficacy in their courses. Data collected
from such surveys can help us identify the extend to which the results
presented here are specific to introductory physics. We similarly need to
develop and deploy a calculus proficiency survey for General Physics I so that
we can identify how early this divergence between calculus self-efficacy and
calculus proficiency occurs.

##  6 Conclusion

Since applied mathematics is an important component of all STEM disciplines,
there is a clear need to promote student competency with mathematical skills
across all STEM curricula. Introductory physics courses are the ideal venue
for improving fluency with applied mathematics as their curricula emphasize
quantitative problem solving skills. Providing opportunities to improve
calculus fluency and transfer skills will likely have the largest benefit for
students with weaker secondary mathematics preparation [19], such as students
from traditionally underserved populations [35, 1], who are also at a higher
risk of dropping out of STEM degree programs [24, 37, 41]. Improving calculus
proficiency and transfer ability thus has the potential to improve the
retention of these students in these programs and thereby increase the
diversity within STEM disciplines and careers. Similarly, understanding the
origins of differences in calculus self-efficacy and the interplay between
those differences and variation in calculus proficiency will assist in the
development and assessment of curriculum changes and other interventions
targeting improvements in these attributes. This has the potential to help
reduce the imbalances in physics self-efficacy [27, 23], leading to an
increased participation and retention of traditionally underrepresented
students within STEM and, by extension, in the STEM workforce.

## References

  * [1] F. Atuahene and T. A. Russell.  Mathematics readiness of first-year university students.  Journal of Developmental Education, 39(3):12, 2016.
  * [2] F. B. Baker.  The basics of item response theory.  ERIC, 2001.
  * [3] S. E. Brekke.  Some factors affecting student performance in physics.  Spectrum, 01 1994.
  * [4] S. Britton.  Are students able to transfer mathematical knowledge.  In Proceedings of the 2nd International Conference on the Teaching of Mathematics, New York: Wiley. http://www. math. uoc. gr/ ictm2/Proceedings/ICTM2_Presentations_by_Author. html# B (accessed 5 July 2013). Citeseer, 2002.
  * [5] H. D. Cohen, D. F. Hillman, and R. M. Agne.  Cognitive level and college physics achievement.  American Journal of Physics, 46(10):1026–1029, 1978.
  * [6] L. Cui.  Assessing college students’ retention and transfer from calculus to physics.  PhD thesis, Kansas State University, 2006.
  * [7] R. P. Dalka, D. Sachmpazidi, C. Henderson, and J. P. Zwolak.  Network analysis approach to likert-style surveys.  arXiv preprint arXiv:2202.12281, 2022.
  * [8] T. Espinosa, K. Miller, I. Araujo, and E. Mazur.  Reducing the gender gap in students’ physics self-efficacy in a team-and project-based introductory physics class.  Physical Review Physics Education Research, 15(1):010132, 2019.
  * [9] C. Fischer, J. Delgado, and S. LeGresley.  Interplay between physics self-efficacy, calculus transfer ability, and gender.  arXiv preprint arXiv:2110.07364, 2021.
  * [10] C. J. Fischer.  The Energy of Physics, Part I: Classical Mechanics and Thermodynamics. Second Edition.  Cognella, Inc., 2019.
  * [11] C. J. Fischer.  The Energy of Physics, Part II: Electricity and Magnetism. Second Edition.  Cognella, Inc., 2019.
  * [12] N. J. Foti, J. M. Hughes, and D. N. Rockmore.  Nonparametric sparsification of complex multiscale networks.  PloS one, 6(2):e16431, 2011.
  * [13] R. M. Gagné.  The conditions of learning 1965 new york holt.  Rinehart and Winston, 308.
  * [14] A. A. Hagberg, D. A. Schult, and P. J. Swart.  Exploring network structure, dynamics, and function using networkx.  In G. Varoquaux, T. Vaught, and J. Millman, editors, Proceedings of the 7th Python in Science Conference, pages 11 – 15, Pasadena, CA USA, 2008.
  * [15] C. Hill, C. Corbett, and A. St Rose.  Why so few? Women in science, technology, engineering, and mathematics. ERIC, 2010.
  * [16] H. Hudson and W. McIntire.  Correlation between mathematical skills and success in physics.  American Journal of Physics, 45(5):470–471, 1977.
  * [17] H. Hudson and R. M. Rottmann.  Correlation between performance in physics and prior mathematics knowledge.  Journal of Research in Science Teaching, 18(4):291–294, 1981.
  * [18] R. Y. Kezerashvili, C. Cabo, and D. K. Mynbaev.  The transfer of knowledge from physics and mathematics to engineering applications.  arXiv preprint arXiv:0708.2577, 2007.
  * [19] S. E. LeGresley, J. A. Delgado, C. R. Bruner, M. J. Murray, and C. J. Fischer.  Calculus-enhanced energy-first curriculum for introductory physics improves student performance locally and in downstream courses.  Phys. Rev. Phys. Educ. Res., 15:020126, September 2019.
  * [20] B. F. Manly.  Multivariate statistical methods: a primer. Second Edition.  Chapman and Hall/CRC, 1994.
  * [21] G. C. Marchand and G. Taasoobshirazi.  Stereotype threat and women’s performance in physics.  International Journal of Science Education, 35(18):3050–3061, 2013.
  * [22] A. Maries, N. I. Karim, and C. Singh.  Is agreeing with a gender stereotype correlated with the performance of female students in introductory physics?  Physical Review Physics Education Research, 14(2):020119, 2018.
  * [23] E. M. Marshman, Z. Y. Kalender, T. Nokes-Malach, C. Schunn, and C. Singh.  Female students with a’s have similar physics self-efficacy as male students with c’s in introductory courses: A cause for alarm?  Physical Review Physics Education Research, 14(2):020123, 2018.
  * [24] C. Moller-Wong and A. Eide.  An Engineering Student Retention Study.  Journal of Engineering Education, 86(1):7–15, January 1997.
  * [25] P. New, S. Britton, M. Sharma, and A. Brew.  Researching the transferability of mathematical skill.  In Proceedings of the australian conference on science and mathematics education (formerly uniserve science conference), volume 7, 2012.
  * [26] M. E. Newman and M. Girvan.  Finding and evaluating community structure in networks.  Physical review E, 69(2):026113, 2004.
  * [27] J. M. Nissen and J. T. Shemwell.  Gender, experience, and self-efficacy in introductory physics.  Physical Review Physics Education Research, 12(2):020105, 2016.
  * [28] M. Prat-Sala and P. Redford.  The interplay between motivation, self-efficacy, and approaches to studying.  British Journal of Educational Psychology, 80(2):283–305, 2010.
  * [29] N. Rebello, L. Cui, A. Bennett, D. Zollman, and D. Ozimek.  Transfer of learning in problem solving in the context of mathematics and physics.  In Learning to Solve Complex Scientific Problems, pages 223–246. Taylor and Francis Group, 2007.
  * [30] E. F. Redish.  A theoretical framework for physics education research: Modeling student thinking.  arXiv preprint physics/0411149, 2004.
  * [31] D. Reilly, D. L. Neumann, and G. Andrews.  Investigating gender differences in mathematics and science: Results from the 2011 trends in mathematics and science survey.  Research in Science Education, 49(1):25–50, 2019.
  * [32] N. J. Salkind.  Tests & measurement for people who (think they) hate tests & measurement.  Sage Publications, 2017.
  * [33] V. Sawtelle, E. Brewe, and L. H. Kramer.  Exploring the relationship between self-efficacy and retention in introductory physics.  Journal of research in science teaching, 49(9):1096–1121, 2012.
  * [34] D. H. Schunk and F. Pajares.  The development of academic self-efficacy.  In Development of achievement motivation, pages 15–31. Elsevier, 2002.
  * [35] R. Tsoi-A and F. Bryant.  College preparation for african american students: Gaps in the high school educational experience.  2015\.
  * [36] S. Turşucu, J. Spandaw, S. Flipse, and M. J. de Vries.  Teachers’ beliefs about improving transfer of algebraic skills from mathematics into physics in senior pre-university education.  International Journal of Science Education, 39(5):587–604, 2017.
  * [37] W. Tyson, R. Lee, K. M. Borman, and M. A. Hanson.  Science, Technology, Engineering, and Mathematics (STEM) Pathways: High School Science and Math Coursework and Postsecondary Degree Attainment.  Journal of Education for Students Placed at Risk (JESPAR), 12(3):243–270, October 2007.
  * [38] G. Van Rossum and F. L. Drake Jr.  Python reference manual.  Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  * [39] J. F. Wagner, C. A. Manogue, and J. R. Thompson.  Representation issues: Using mathematics in upper-division physics.  In AIP Conference Proceedings, volume 1413, pages 89–92. American Institute of Physics, 2012.
  * [40] H. M. Watt.  The role of motivation in gendered educational and occupational trajectories related to maths.  Educational Research and Evaluation, 12(4):305–322, 2006.
  * [41] G. Zhang, T. J. Anderson, M. W. Ohland, and B. R. Thorndyke.  Identifying Factors Influencing Engineering Student Graduation: A Longitudinal and Cross-Institutional Study.  Journal of Engineering Education, 93(4):313–320, October 2004.
  * [42] B. J. Zimmerman.  Self-efficacy: An essential motive to learn.  Contemporary educational psychology, 25(1):82–91, 2000.


Maximum Likelihood Estimation of Latent Variable Structural Equation Models: A Neural Network Approach

Mehrzad Saremi<EMAIL_ADDRESS><EMAIL_ADDRESS>
Graduate from the Department of Artificial Intelligence

Amirkabir University of Technology

Tehran, Iran

We propose a graphical structure for structural equation models that is stable under marginalization under linearity and Gaussianity assumptions. We show that computing the maximum likelihood estimation of this model is equivalent to training a neural network. We implement a GPU-based algorithm that computes the maximum likelihood estimation of these models.

Neural Network, Maximum Likelihood Estimation, Graphical Causal Model, Structural Equation Model

§ MOTIVATION

Structural Equation Models (SEMs), also known as Bayesian Networks, are widely used in probabilistic reasoning and causal inference. The classes of SEMs range from simple Directed Acyclic Graph-based (DAG) Bayesian networks through marginalized (hyper-)graphs like Acyclic Directed Mixed Graphs (ADMGs) [Richardson and Spirtes, 2002] and marginal DAGs (mDAGs) [Evans, 2016] to graphical
models with cycles and confounding latent variables [Forré and Mooij, 2017]. Many algorithms for finding the fittest parameters in these models have been proposed. Prominent examples are [Savalei and Rosseel, 2022] and [Drton et al., 2019].

ADMGs have long been shown not to be stable under marginalization [Evans, 2016]. Moreover, they do not have straightforward causal interpretations (see [Fox et al., 2015]). As a remedy for the former, mDAGs and their cyclic version, Hyper-Edged Directed Graphs (HEDGes) [Forré and Mooij, 2017], have been proposed. Despite mDAGs' nice statistical properties [Shpitser et al., 2014], they fail to capture margins of Bayesian networks when additional constraints are required.

In this article, we introduce a graphical structure that is more general than ADMGs or mDAGs, called Ancestral Marginal Acyclic Structural Equation Models (AMASEM). mDAGs are constructed by removing the nodes that have parents (exogenization; Def. <ref>–<ref>) and removing the latent random variables whose visible children are the subset of the children of any other latent random variable (coalescence; Def. <ref>–<ref>). We construct AMASEMs by letting exogenization but avoiding coalescence. We show that the existing graphical structures are not rich enough to capture the margins of linear Gaussian Bayesian networks and that AMASEMs are capable of doing so.

As the main contribution, we show that under the Gaussianity and linearity assumptions, the parametrization of an AMASEM is the dual problem of training a feedforward neural network. Neural networks are trained using a mechanism called backpropagation.
Hinton [Rumelhart et al., 1986, Plaut and Hinton, 1987] popularized the term backpropagation as a means for neural networks to learn the conditional distribution of data. We show that by training a neural network using only a single datum (the covariance matrix of the observed variables) one can parameterize an AMASEM. In particular, we compute the Maximum Likelihood Estimation (MLE) of AMASEMs using neural networks. To this end, we introduce intermediate structures that are called SEMNAN structures, which are both a neural network and an AMASEM. We propose an algorithm that computes the MLE of the original AMASEM by training SEMNANs.

§.§ Outline

In <ref>, we will introduce the new structures and bring examples to emphasize the difference between our new structure and the existing ones. In <ref>, we will advance the details of The SEMNAN Solver. We conclude our work in <ref>.

§ MODELS

§.§ Basic Definitions

In the following, we use a cursive typeface for random variables (e.g. $\varV$) and a boldface cursive font for random vectors (e.g. $\vecvarV$). We reserve the italic typeface (e.g. $W$) for matrices of non-stochastic variables (see Appendix <ref> for our notations).

* Consider a random variable $\varV$ over the (co-domain) measure space $\tuple{\spaceOf{\varV}, \sigmaAlgebraOf{\spaceOf{\varV}}}$ with strictly positive probability $\Prob[]$. Define an arbitrary measure space $\tuple{\Omega, \Sigma}$ and a Markov kernel $\kappa_{\Sigma,\,\varV} : \Sigma \times \spaceOf{\varV} \to \intClose{0}{1}$. This gives a new variable $\varU$ over $\tuple{\spaceOf{\varU}, \sigmaAlgebraOf{\spaceOf{\varU}}} = \tuple{\Omega, \Sigma}$ with probability $\Prob^*{\varU \in U,\,\varV \in V} = \int_{v \in V}\kappa\!\parens{U,\,v} d\Prob{\varV = v}$, where $\Prob[]^*$ measures $\sigmaAlgebraOf{\spaceOf{\varU}} \tensorProd \sigmaAlgebraOf{\spaceOf{\varV}}$. Since $\kappa_{\Sigma,\,\varV}$ is defined and not measured, we call it an a priori relation between $\varV$ and $\varU$.

* If for all $\pair{U,\,v} \in \sigmaAlgebraOf{\spaceOf{\vecvarU}} \times \spaceOf{\varV}$ we have $\kappa_{\Sigma,\,\varV}\parens{U,\,v} = \indicator_{U}{v}$ then $\kappa_{\Sigma,\,\varV}$ is the identity a priori relation.

We denote the reference [Kripke, 2008] of a random variable $\varV$ (i.e. the phenomenon that $\varV$ refers to) using $\refOf{\varV}$. We write $\refOf{\varX} \equiv \refOf{\varY}$ iff. the relationship between $\varX$ and $\varY$ is identity a priori (we use $\equiv$ to indicate that two generic objects are identical).

Definition <ref> introduces our new structures. We define the existing structures in terms of our new structures in Definition <ref> (compare with Appendix <ref>).