# Investigating Deep Learning Model Calibration for Classification Problems in
Mechanics

Saeed Mohammadzadeh
Division of Systems Engineering
Boston University
Boston, MA 02215
<EMAIL_ADDRESS>
&Peerasait Prachaseree
Department of Mechanical Engineering
Boston University
Boston, MA 02215
<EMAIL_ADDRESS>
&Emma Lejeune
Department of Mechanical Engineering
Boston University
Boston, MA 02215
<EMAIL_ADDRESS>

###### Abstract

Recently, there has been a growing interest in applying machine learning
methods to problems in engineering mechanics. In particular, there has been
significant interest in applying deep learning techniques to predicting the
mechanical behavior of heterogeneous materials and structures. Researchers
have shown that deep learning methods are able to effectively predict
mechanical behavior with low error for systems ranging from engineered
composites, to geometrically complex metamaterials, to heterogeneous
biological tissue. However, there has been comparatively little attention paid
to deep learning model calibration, i.e., the match between predicted
probabilities of outcomes and the true probabilities of outcomes. In this
work, we perform a comprehensive investigation into machine learning model
calibration across $7$ open access engineering mechanics datasets that cover
three distinct types of mechanical problems. Specifically, we evaluate both
model and model calibration error for multiple machine learning methods, and
investigate the influence of ensemble averaging and post hoc model calibration
via temperature scaling. Overall, we find that ensemble averaging of deep
neural networks is both an effective and consistent tool for improving model
calibration, while temperature scaling has comparatively limited benefits.
Looking forward, we anticipate that this investigation will lay the foundation
for future work in developing mechanics specific approaches to deep learning
model calibration.

##  1 Introduction

Over the past decade, there have been unprecedented advances in applying
machine learning techniques to problems in mechanics. Researchers have used
machine learning approaches to enable design optimization [18, 20, 22, 66,
71], inverse analysis [5, 28, 72], real-time predictions [26, 29, 83], and
multi-scale modeling [2, 30, 43, 70, 80] among many other applications. There
has also been a growing interest in using machine learning approaches for
uncertainty quantification for constitutive modeling [27, 69] and multi-
fidelity surrogate modeling [21, 54, 15]. For these applications, which are
ubiquitous in engineering design and real world decision making, it is
important to investigate machine learning model calibration, i.e., the match
between predicted probabilities of outcomes and the true probabilities of
outcomes [17, 19, 45, 48, 49, 82], in addition to model error. This is a
particularly important consideration for deep learning models, which tend to
have low model error with no associated promise of being well calibrated [19].
In the computational mechanics community, there is a rich history of studying
uncertainty quantification and model calibration [6, 58, 73]. Here, our goal
is to add additional context specific to deep learning based classification
problems. In Fig. 1, we illustrate the concept of model calibration for binary
classification problems. In Fig. 1b, we specifically highlight the
relationship between model error and model calibration error.

Figure 1: Conceptual illustration of model calibration for binary
classification problems. Panel (a) depicts a standard binary classification
task where the model outputs probabilities for both “Class 0” and “Class 1.”
Panel (b) illustrates the combined implications of both “model error” and
“model calibration error” via a toy example. In this graph, the bottom left
quadrant represents high accuracy and good calibration and the top right
quadrant represents low accuracy and poor calibration. In the circular test
cases, the top half of each circle represents the true label, and the bottom
half shows the predicted probability for class $1$. Overall, low model
calibration error is achieved when the network has high confidence for correct
predictions and low confidence for incorrect predictions.

In the machine learning community, there is growing interest in improving
model calibration without compromising predictive accuracy [19, 45]. For deep
learning in particular, where model calibration remains poorly understood,
there is a concurrent focus on evaluating model calibration and on improving
model calibration. Critically, there is growing attention to empirical
evaluation of established model calibration metrics (e.g., expected
calibration error defined in Section 2.5.3) across different deep learning
model architectures [19, 45]. For example, researchers have empirically
investigated the relationship between model error and model calibration error
across multiple architectures on the ImageNet dataset for image classification
[45, 9]. Simultaneously, developing effective metrics for evaluating model
calibration remains an active area of research [45, 48, 50, 51, 84](discussed
further in Section 2.5.4).

Beyond evaluating deep learning model calibration as an emergent property,
there has also been significant interest in developing methods specifically
for improving model calibration [19, 36, 55, 60, 82, 84]. Notably, interest in
this topic predates the prevalent current interest in deep learning [17, 49].
For example, Platt Scaling, a post hoc calibration method where scores of the
trained model are additionally trained through logistic regression, was
initially developed for support vector machines [55]. Broadly speaking, there
are multiple post hoc calibration methods that rely on an additional held out
training dataset (e.g., Platt Scaling [55], Isotonic Regression [82]) that are
applicable to improving machine learning model calibration. In the context of
deep learning, deep ensembles [36], temperature scaling [19], and combinations
of deep ensembles and temperature scaling [60, 84] are straightforward and
commonly implemented strategies for improving deep model calibration that we
will investigate here. As our ability to train large deep learning based
models with low error becomes more commonplace [13, 19], working towards
better model calibration is a natural next step.

In this work, our goal is to perform a comprehensive investigation into deep
learning model calibration for classification problems in mechanics.  The
structure of our investigation is informed by two high level objectives.
First, because deep learning model performance is dataset dependant, it is our
goal to design and implement a mechanics-specific challenge for assessing
different approaches to model calibration. Namely, we want to create a multi-
faceted framework to apply broad advances in machine learning to the mechanics
domain. Second, we want to conduct a study that can be directly leveraged by
others. This means that we not only want our findings to be of clear utility
to others, but also that we want our framework to be directly accessible for
others to build on it to assess alternative methods. This structure is
directly informed by similar investigations into deep learning methods
conducted by others outside the field of mechanics [33, 45, 10, 44].

Following these high level goals, the foundation for our investigation is $7$
previously published datasets that span three distinct mechanical problems,
detailed in Section 2.1. Necessitated by the diversity in these three
mechanical problems, we train distinct problem-specific deep learning models
on these datasets, detailed in Section 2.2. And, across all datasets, we
explore the influence of ensemble averaging, detailed in Section 2.3, and
temperature scaling, detailed in Section 2.4, on model calibration. In Section
3, we present the main findings from our investigation as plots of machine
learning model error with respect to machine learning model calibration error.
From this investigation, we find that ensemble averaging of deep neural
networks is both an effective and consistent tool for improving model
calibration for problems in mechanics, while temperature scaling has
comparatively limited benefits. To our knowledge, this is the largest
investigation of deep learning model calibration on open access mechanics
datasets to date. It is our hope that the results of this investigation are
both informative to others, and will lay the foundation for further
exploration of this important topic.

##  2 Methods

In this Section, we will begin by introducing the datasets used in this
investigation. Please note that all datasets used in this study have been
previously published by our group under Creative Commons Attribution-
ShareAlike 4.0 International licenses, and are thus freely available for
others to use in follow-up studies to this work. Then, in Section 2.2, we will
describe the machine learning models investigated in this work. In Section 2.3
we will describe our implementation of ensemble averaging, and in Section 2.4
we will describe our implementation of temperature scaling. Finally, in
Section 2.5, we will specify the error and calibration metrics used to report
results in Section 3.

###  2.1 Benchmark Datasets Used in this Study

In this investigation, our goal is to comprehensively evaluate model
calibration on a diverse set of mechanics-based classification datasets. To
this end, we will conduct our analysis on $7$ open access datasets across
three types of mechanical problems. In Section 2.1.1, we provide background
details on the “Buckling Instability Classification” (BIC) dataset and sub-
datasets [38], in Section 2.1.2, we provide background details on the
“Asymmetric Buckling Columns” (ABC) dataset and sub-datasets [56], and in
Section 2.1.3 we provide details on the “Mechanical MNIST – Crack Path”
dataset [46]. We note briefly that all datasets are derived from simulations
conducted via the open source finite element analysis software FEniCS [3, 41],
and the structures in the ABC dataset are generated through Gmsh [16].
Overall, these datasets cover both a range of mechanical mechanisms (i.e.,
both geometric and material nonlinearity), and rely on a range of deep
learning techniques (i.e., standard neural networks [40], graph neural
networks [57], and convolutional networks [47]).  Specific details for
accessing each dataset and the additional background information required to
recreate each dataset are provided in Section 5.

Figure 2: Schematic illustration of the $7$ datasets used in this study: the
BIC dataset (which contains BIC 1, BIC 2, and BIC 3) [38], the ABC dataset
(which contains ABC 1, ABC 2, and ABC 3) [56], and the Mechanical MNIST –
Crack Path dataset [46].

####  2.1.1 Buckling Instability Classification (BIC)

The BIC dataset was disseminated in conjunction with our previous publication
exploring multiple straightforward approaches to classification problems in
mechanics [40]. The BIC dataset contains three sub-datasets: “BIC 1”, “BIC 2”,
and “BIC 3” that differ only in their input parameter distribution. In all
cases, the input is the material property distribution of a heterogeneous
column that is then subject to a fixed level of applied compressive
displacement. The output then corresponds to class “Stable” or “Unstable”
based on the results of a Finite Element Analysis (FEA) simulation. For all
sub-datasets, the input property distribution of each sample is represented by
a $16\times 1$ vector. For BIC 1, there are two possible discrete modulus $E$
values: $E=1$, and $E=4$. For BIC 2, there are three possible discrete values:
$E=1$, $E=4$, and $E=7$. For BIC 3, the modulus varies continuously to three
degrees of precision in the range $E=[1,8]$. Further details regarding data
curation and our FEA implementation are available in our previous publication
[40]. For the work presented in this manuscript, we used $10,000$ samples for
machine learning model training and $1,000$ samples for post hoc model
calibration across all three sub-datasets, and $6,553$ samples for BIC 1 and
$10,000$ samples for BIC 2 and BIC 3 for model testing. Briefly, we note that
in Section 3.1.2, we also specifically investigate training set sizes of
$200$, $500$, $1000$, $2000$, $5000$, $10000$ samples. For the BIC 1 dataset,
the ratio between “Stable” and “Unstable” samples is $0.28$, for BIC 2 it is
$0.40$, and for BIC 3 it is $0.26$.

####  2.1.2 Asymmetric Buckling Columns (ABC)

As a follow up to the BIC dataset, we introduced the ABC dataset [56] in our
previous work in conjunction with an exploration of geometric deep learning
for mechanics-specific classification problems [57]. Similar to BIC, the ABC
dataset contains three subdatasets, where each subdataset corresponds to a
different algorithm for generating the geometry of the input domain. For ABC
1, columns are generated by vertically stacking rectangular blocks of randomly
varying widths. For ABC 2, columns are generated by randomly overlaying rings
of identical inner and outer radii. For ABC 3, columns are generated by
overlaying and trimming rings of varying inner and outer radii (i.e., varying
size and thickness). For all sub-datasets, the columns are subjected to fixed-
fixed boundary conditions and are compressed until the onset of buckling. Each
input geometry in the ABC dataset is then classified as buckling “left” or
“right.” Further details of data curation, and FEA implementation are
available in our previous publication [57]. For the work presented in this
manuscript, we used $20,000$ samples for machine learning model training,
$1,000$ samples for post hoc model calibration, and $2,500$ samples for model
testing for each of the three sub-datasets. For all three ABC sub-datasets,
the classes are balanced.

####  2.1.3 Mechanical MNIST – Crack Path

The Mechanical MNIST dataset collection [39, 34, 81] is a collection of
benchmark datasets initially conceptualized as mechanics-relevant drop-in
replacements for the popular MNIST dataset [37]. For the datasets in the
Mechanical MNIST collection, input bitmaps dictate heterogeneous material
properties, and outputs are defined as curated results from FEA simulations.
The “Mechanical MNIST – Crack Path” dataset is an example from the collection
where the input bitmap distribution is a heterogeneous pattern of inclusions
derived from the Fashion MNIST dataset [78], and the main output is a damage
field predicted by a linear elastic phase-field fracture simulation [77, 76].
For this manuscript, we will focus exclusively on the $64\times 64$ input
bitmap and a downsampled $64\times 64$ output crack path. Notably, each pixel
in this downsampled crack path is in either the “damaged” (true) class or the
“undamaged” (false) class, thus conceptualizing the Mechanical MNIST – Crack
Path dataset as a binary classification problem similar to the BIC and ABC
datasets described previously. Further details of data curation and our FEA
implementation are available in our previous publication [47]. In the original
dataset there are $60000$ samples in the training set and $10000$ samples in
the test set. For the work in this manuscript, we used the first $10,000$
samples from the training set for machine learning model training, the next
$1,000$ samples from the training set for post hoc model calibration, and
$10,000$ samples from the test set for model testing. For this dataset,
classes are heavily imbalanced, where the “damaged” class corresponds to
$2.88\%$ of pixels.

###  2.2 Machine Learning Models Investigated

The main focus of this work is on deep learning model calibration. However, in
Section 3.1, we provide baseline comparisons to Gaussian Process
Classification and Support Vector Classification to add additional context to
our results. Here we briefly summarize these methods along with the Neural
Network based approaches used for prediction.

####  2.2.1 Gaussian Process Classification

Gaussian Processes are commonly used in machine learning literature for both
classification and regression tasks when uncertainty quantification is
critical [7]. The Gaussian Process Classification [75], which we use in this
work, is a generalization of the linear logistic regression model where the
linear latent function is replaced by a Gaussian Process. To train a Gaussian
Process Classification in the context of a machine learning problem, the user
must define a kernel function that will determine the form of the covariance
matrix. For further details on Gaussian Process methods in machine learning,
we refer the reader to the literature [75]. In this work, we used scikit-learn
[53] to train Gaussian Process Classification with a Radial Basis Function
kernel [12] on the BIC 1, BIC 2, and BIC 3 datasets. The performance of
Gaussian Process Classification on these data is shown in Section 3.1. The
code for this model is included in the GitHub Repository for this publication.

####  2.2.2 Support Vector Classification

Support Vector Machines are a commonly used machine learning algorithm for
classification problems [23]. The Support Vector Machine was initially
developed for binary classification problems and later on extended to deal
with multi-class classification [74] and regression tasks [11]. Here, we will
focus on “Support Vector Classification” for binary classification. In brief,
the general idea behind Support Vector Classification is to transform the data
in a high dimensional space and identify a hyperplane that most accurately
separates the classes. Similar to Gaussian Process Classification, choice of
kernel function impacts Support Vector Classification performance in the
context of machine learning. In this work, we use scikit-learn [53] to train
Support Vector Classifications with a Radial Basis Function kernel and no
additional regularization on the BIC 1, BIC 2, and BIC 3 datasets. The
performance of Support Vector Classifications on these data is shown in
Section 3.1. Critically, we note that scikit-learn uses Platt scaling,
described in [55], coupled with five-fold cross-validation to obtain
probabilistic outputs from otherwise non-probabilistic Support Vector
Classification scores. The code for this model is included in the GitHub
Repository for this publication.

####  2.2.3 Fully Connected Neural Network

Fully Connected Neural Networks are a well-established method for both
regression and classification tasks. These networks commonly consist of an
input layer, an output layer, and a series of fully connected hidden layers.
Each layer applies a linear transformation followed by a non-linear activation
function such as Rectified Linear Units (ReLU) [1] on its input vector. These
layers are designed to eventually transform a given input vector into an
output vector of the desired size. For the BIC datasets, we use Fully
Connected Neural Networks (simply referred to as neural networks) with a $16$
node input layer, three $200$ node hidden layers, and a $2$ node output layer
(see Appendix C). The $2$ output nodes are the logits indicating to which
class “stable” or “unstable” a sample belongs. To reduce overfitting during
training, we add batch normalization [24] before applying each ReLU activation
function and use dropout [68] with the rate of $0.5$ before the second and
third hidden layers and output layer. We use the PyTorch library [52] for our
implementation, and train each network for $50$ epochs using the Adam
optimizer [32] with an initial learning rate of $0.001$, which is dropped to
$0.0005$ after $25$ epochs. We train this model on the BIC 1, BIC 2, and BIC 3
datasets, and present the results in both Section 3.1 and Section 3.2. The
code for this model is included in the GitHub Repository for this publication.

####  2.2.4 Graph Neural Network

By design, the ABC datasets contain complex geometries that are attractive to
represent as spatial graphs rather than as “image-like” arrays. In our
previous work, we identified a set of best performing models and data
representation approaches for each ABC subdataset that we directly build on
for this work [57]. In brief, each ABC input geometry is first represented as
a spatial graph. In our previous work, we identified a high performing
strategy for spatial graph representation where spatial graphs are constructed
via discretizing the structure into nodes and then performing a ball query to
form edges. Based on our prior investigation, ABC 1 had a “medium” node
density ($\approx 306$ nodes per structure) with a ball radius of $40\%$ of
the column width, and ABC 2 and ABC 3 have a “dense” node density ($\approx
566$ and $\approx 768$ nodes per structure respectively) with a ball radius of
$30\%$ of the column width [57]. Given this spatial graph representation, we
used PointNet++ layers [59] as spatial graph convolution layers coupled with
batch normalization [24], followed by skip connections and a linear classifier
to construct our machine learning model (see Appendix C). To improve model
performance, we also augment our dataset by flipping the columns along the $x$
axis, $y$ axis, and both axes while changing labels as needed. All models are
implemented with the Pytorch Geometric library [14] and trained using the Adam
optimizer [32] for $50$ epochs. We present the results from this model on the
ABC datasets in Section 3.2. The code for the model is published on GitHub
alongside our previous publication.

####  2.2.5 UNet Neural Network

To complement the models described in Section 2.2.1 \- 2.2.4 which are trained
to predict a single quantity of interest, we train a deep neural network on
the Mechanical MNIST – Crack Path dataset that is designed to predict full-
field quantities of interest, specifically the whole domain damage field. In
our previous work [47], we used a modified version of the UNet model [63], the
MultiRes-WNet, combined with a convolutional autoencoder for an end-to-end
prediction of $256\times 256$ images of the damage field from $64\times 64$
material distribution input images. Here, we regenerated lower resolution
output damage fields directly from our FEA results as $64\times 64$ arrays and
used a standard UNet with three downsampling and upsampling steps [67]. The
outputs of the model are logits in the form of two-channel images that can be
transformed into probabilities by applying a softmax function to each pixel
(see Appendix C). We briefly note that we trained the network by minimizing
the Dice-loss [25]. We use the PyTorch library [52] for the UNet model
implementation, and train each network for $50$ epochs using the Adam
Optimizer [32]. We present the results from this model on the Mechanical MNIST
– Crack Path dataset in Section 3.2. The code for this model is included in
the GitHub Repository for this publication.

###  2.3 Ensemble Methods

For the Neural Network approaches introduced in Section 2.2.3-2.2.5, the
behavior of each trained neural network will vary based on the random weight
initialization. Thus, it is possible to train multiple neural networks and
subsequently combine them into an ensemble [8, 36]. Here, we take a
straightforward approach and individually train $10$ models with different
initialization seeds before aggregating the predictions using soft voting,
also referred to as unweighted model averaging [36]. In soft voting, the
predicted probability for each class is averaged over all models and the label
with the highest probability then becomes the final class prediction. The goal
of ensemble averaging is to increase the overall prediction accuracy.
Additionally, if the neural networks are trained with proper scoring rules
like cross entropy, ensemble averaging may also lead to averaged probabilities
that are well calibrated [36]. One major goal of this work is to critically
evaluate the efficacy of neural network ensemble averaging for deep learning
approaches to classification problems in mechanics.

###  2.4 Post Hoc Calibration via Temperature Scaling

Post hoc calibration of neural networks using a held-out calibration dataset
is a popular approach with both parametric (e.g., Platt scaling [55],
temperature scaling, matrix scaling [19]) and non-parametric (e.g., Bayesian
Binning [48], isotonic regression [82]) implementations. Motivated by its
popularity in the literature, we choose temperature scaling as a standard post
hoc calibration technique. Specifically, given a trained classifier, we divide
the logits vector $\mathbf{z}$ by a single variable $T$ called the
temperature. The optimal temperature is obtained by minimizing the Negative
Log Likelihood (NLL) on the held out calibration set. The NLL is written as:

$\displaystyle\min_{T}$
$\displaystyle\sum_{i=1}^{N_{c}}\text{NLL}\bigg{(}\sigma({\mathbf{z}_{i}\,/\,T})\,,\,y_{i}\bigg{)}$
(1) s.t. $\displaystyle T>0$

where $\sigma(\mathbf{x})$ is the softmax function, $y$ is the true labels,
$N_{c}$ is the number of sample points in the calibration set, and
$\mathbf{z}$ is the previously defined logits vector. Notably, temperature
scaling can be applied either before or after ensemble averaging [60].

In Section 3.2, we report the results of applying post hoc calibration methods
on our datasets. For clarity, the methods investigated are defined as follows:

  * •

Method I: Individual neural network without post hoc calibration.

  * •

Method I-C: Individual neural network with post hoc calibration via
temperature scaling.

  * •

Method E-M1: Ensemble neural network without post hoc calibration.

  * •

Method E-M2: Ensemble neural network with post hoc calibration via temperature
scaling applied before ensemble averaging.

  * •

Method E-M3: Ensemble neural network with post hoc calibration via temperature
scaling applied after ensemble averaging.

Fig. 6 in Section 3.2 and Fig. 8 in Appendix B directly reference these
definitions. Our code for implementing temperature scaling is included in the
GitHub Repository for this publication.

###  2.5 Error and Calibration Metrics Reported in this Investigation

In this work, all supervised learning tasks are binary classification
problems. For the datasets without severely imbalanced class labels (i.e. BIC
and ABC) we report classification accuracy as our error metric, while for the
severely imbalanced class label (i.e. damaged pixels in the Mechanical MNIST
Crack Path dataset) we report the $F_{1}$ score as our error metric. To
measure calibration, we report the Expected Calibration Error (ECE). In all
cases, we report these metrics on our held out test datasets. Details on
metrics for model error and model calibration error are as follows.

####  2.5.1 Classification Error Definition for BIC and ABC

We evaluate model performance for the BIC and ABC datasets via traditional
classification error. Specifically, we define classification error as the
fraction of wrong predicted labels with respect to the total number of labels.
Mathematically, this is written as:

$\text{Error}(y,\,\hat{y})=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}(\hat{y}_{i}\neq
y_{i})$ (2)

where $N$ is the number of labels to evaluate, $y$ and $\hat{y}$ are the true
and predicted labels respectively, and $\mathbf{1}(\hat{y}\neq y)$ represents
the 0-1 loss function [65].

####  2.5.2 Classification Error Definition for Mechanical MNIST Crack Path

Following our previous work, the damage field for each sample in the
Mechanical MNIST dataset is treated as a binary matrix (image) where $1$
represents a damaged sub-region (pixel) and $0$ represents an undamaged sub-
region (pixels) [47]. Because the damaged region represents a crack path, the
relative prevalence of damaged pixels is small ($2.88\%$) leading to severe
class imbalance. As such, the classification error as defined in eqn. 2 will
automatically appear as a small value due to the high percentage of true
negative pixels in each prediction. To better evaluate model performance with
these imbalanced labels, we report the error as the Sørensen-Dice index, often
referred to as the $F_{1}$ score, defined as:

$F_{1}=\frac{2\,\text{{\color[rgb]{0,0,0}True
Positive}}}{2\,\text{{\color[rgb]{0,0,0}True
Positive}}+\text{{\color[rgb]{0,0,0}False
Positive}}+\text{{\color[rgb]{0,0,0}False Negative}}}$ (3)

where True Positive, False Positive, and False Negative denote the number of
correctly predicted damaged pixels, incorrectly predicted undamaged pixels,
and incorrectly predicted damaged pixels respectively. We note that that
$F_{1}$ score defined in eqn. 3 is not influenced by the number of true
negative pixels (correctly predicted undamaged pixels), making it an easier-
to-interpret metric of model predictive performance for the Mechanical MNIST –
Crack Path dataset. In Fig. 6, where we report $F_{1}$ score, we plot error as
$1-F_{1}$ on the y axis to maintain visual consistency.

Figure 3: Representative reliability diagram for an individual neural network
trained on the BIC 3 dataset. To promote completeness while preserving data
visualization clarity, we report additional reliability diagrams in Appendix B
as a supplement to Section 3.

####  2.5.3 Expected Calibration Error (ECE)

In this work, we use the Expected Calibration Error (ECE) to evaluate model
calibration. As stated in Section 1, calibration refers to the match between
predicted probabilities of outcomes and the true probabilities of outcomes.
For example, when a model is perfectly calibrated, $100$ predictions with
$80\%$ confidence should be correct $80/100$ times. While there are many
potential metrics used to evaluate calibration [19, 45, 48, 49, 50, 51, 84],
the ECE is one of the most prevalent in the literature and one of the most
interpretable. Namely, the ECE is connected to the reliability diagram, a
common approach to visualizing model calibration illustrated in Fig. 3 [19,
48, 49]. Reliability diagrams are constructed in two steps. First, sample
prediction confidences (i.e., confidence that a given sample is in a chosen
class) are binned. In this work, we use $10$ equally spaced bins to construct
all reliability diagrams. Then, the average bin confidence is compared to the
true fraction of samples with the chosen class. The ECE is then computed as
the weighted average of the gap between perfect calibration and model
confidence within each bin. Mathematically, this is defined as:

$\text{ECE}=\sum_{i=1}^{B}\frac{n_{i}}{N}\,|\,F_{i}-C_{i}\,|$ (4)

where $B$ is the number of bins, $n_{i}$ is the number of samples in each bin,
$N$ is the total number of samples, $F_{i}$ is the frequency of the chosen
class in the bin, and $C_{i}$ is the average confidence that the sample is in
the chosen class in the bin. Following this definition, a lower ECE
corresponds to a better calibrated model, and a model with ECE$=0$ corresponds
to a perfectly calibrated model. As indicated in Fig. 3, the reliability curve
bins in a perfectly calibrated model will follow the diagonal $y=x$.

####  2.5.4 Note on the Limitations of Expected Calibration Error (ECE)

As stated previously, we report ECE as our main model calibration metric
because it is both prevalent in the literature and relatively interpretable.
However, it is not without limitations. For example, ECE is known to be
sensitive to the selection of binning scheme [50, 51, 84]. More specifically,
ECE values can depend on the bin size as well as the number of samples in each
bin. Multiple modifications to the definition of ECE defined in Section 2.5.3
such as adaptive binning schemes [50], using the $\ell_{2}$ norm instead of
the $\ell_{1}$ norm to compute the ECE [45, 50], and kernel-density based
methods [84], have been proposed in the literature. Aside from the ECE, other
methods that are related to reliability diagrams like the Maximum Calibration
Error [48] have been proposed. Alternatively, proper scoring rules [17] with
roots in statistical analysis like the Negative Log-Likelihood (NLL) and Brier
Score [45, 51] have been used to evaluate model calibration. However, it is
not clear if these proposed methods are significantly better than the standard
method for computing ECE, and these new metrics potentially lose some of the
clear relationship to the interpretable reliability diagram. Looking forward,
we anticipate that the framework we establish in this paper could also be used
to investigate the behavior of these alternative metrics. However, this is
beyond the scope of our current work.

Figure 4: Error vs ECE plots for Gaussian Process Classification (GPC),
Support Vector Classification (SVC), 10 individual neural networks (NN-I), and
an ensemble of 10 neural networks (NN-E) trained on the BIC 1, BIC 2, and BIC
3 datasets. Lower ECE and error indicate better performance (bottom left
corner, represented by the green background gradient). Figure 7 in Appendix B
contains reliability diagrams that supplement these results.

##  3 Results and Discussion

In Section 2, we introduced $7$ datasets (BIC 1, BIC 2, BIC 3, ABC 1, ABC 2,
ABC 3, and Mechanical MNIST – Crack Path), described multiple machine learning
models for making predictions with these datasets, procedures for ensemble
averaging and temperature scaling, and metrics for evaluating model error and
calibration. Here, we will begin in Section 3.1 by comparatively evaluating
different machine learning methods on the BIC datasets. Then, in Section 3.2,
we will investigate multiple strategies for improving model calibration across
the BIC, ABC, and Mechanical MNIST – Crack Path datasets. Throughout this
Section, we present the results of our study following the format introduced
in Fig. LABEL:fig:inro where each model is represented as a marker on a
prediction error vs. model calibration error axis.

###  3.1 Evaluating Model Calibration

As discussed in Section 1, large deep neural networks are prone to being
poorly calibrated [19]. In response to this claim, our first goal is to
evaluate this statement in the context of our mechanics-specific datasets and
establish baseline results that will contextualize future endeavors for
improving model calibration.

####  3.1.1 Ensemble Averaging Improves the Performance of Deep Neural
Networks Across the BIC 1, BIC 2, and BIC 3 Datasets

In Fig. 4, we plot model error vs. model expected calibration error (ECE) for
multiple machine learning models trained on the BIC 1, BIC 2, and BIC 3
datasets. Note that each marker corresponds to the test performance of a
trained machine learning model. To add context to our investigation of deep
learning model calibration, we show the performance of Gaussian Process
Classification and Support Vector Classification with Platt scaling alongside
the performance of neural networks. Consistent with results from our prior
publication [40], model performance across different BIC sub-datasets varies
due to the different input parameter space for each BIC dataset. A machine
learning problem with larger input parameter space (i.e. BIC 3) will typically
result in a higher error than a machine learning problem with smaller input
parameter space (i.e. BIC 1). In Fig. 4, we see that individual neural
networks have a large ECE range, and tend to be poorly calibrated in
comparison to the Gaussian Process Classification and Support Vector
Classification models. However, ensemble neural networks have both lower error
and lower ECE, and perform similarly to their Gaussian Process Classification
and Support Vector Classification counterparts. This context is important
because it justifies the choice of ensemble neural networks as an approach to
designing well calibrated deep learning based model frameworks. In Appendix B,
Fig. 7, we show supplementary reliability diagrams that further support these
results.

Figure 5: Error vs ECE plots with varied training set sizes: a) Gaussian
Process Classification (GPC), Support Vector Classification (SVC), and an
ensemble of $10$ neural networks (NN-E); b) $10$ individual neural networks
(NN-I) and mean values of these $10$ networks (NN-I (Mean)). All models are
trained on multiple subsets of the BIC $1$, BIC $2$, and BIC $3$ datasets with
varying training set sizes ($200$, $500$, $1,000$, $2,000$, $5,000$, and
$10,000$ samples). Lower ECE and error indicate better performance (bottom
left corner, represented by the green background gradient). The Pearson
correlation coefficients for error and ECE on BIC $1$, BIC $2$, and BIC $3$,
respectively, are as follows. SVC: ($0.899$, $0.887$, $0.917$); GPC: ($0.691$,
$0.967$, $0.995$); NN-E: ($0.981$, $0.731$, $0.983$); NN-I: ($0.850$, $0.612$,
$0.659$); NN-I (Mean): ($0.969$, $0.656$, $0.936$).

####  3.1.2 Increasing the Training Set Size Leads to Both Lower Error and
Lower Expected Calibration Error (ECE) in Ensemble Neural Networks Across BIC
1, BIC 2, and BIC 3

In Fig. 5, we plot model error vs. model expected calibration error (ECE) for
multiple model types with different training set sizes trained on the BIC 1,
BIC 2, and BIC 3 datasets. In Fig. 5, each marker corresponds to the test
performance of a trained machine learning model, and marker size indicates the
size of the training dataset. Overall, Fig. 5 illustrates two important
results. First, it is clear from the distribution of results across all
machine learning models and datasets that model error and ECE are at most
weakly correlated (the overall correlation coefficient across all models is
$0.516$). Namely, low model error does not necessarily indicate low ECE, and
high model error does not necessarily indicate high ECE. Second, if we examine
individual model types (i.e., Gaussian Process Classification, Support Vector
Classification, or ensemble neural networks) and individual datasets (i.e.,
BIC 1, BIC 2, BIC 3) increasing the training set size consistently lowers both
model error and model ECE. For example, for ensemble neural networks the
correlation coefficients relating error and ECE are $0.981$, $0.731$, and
$0.983$ for BIC 1, BIC 2, and BIC 3 respectively. Critically, for these
examples, increasing the training set size improves both model error and model
calibration which is an important observation because it is an actionable
strategy for improving both dimensions of performance.

###  3.2 Improving Model Calibration

In Sections 2.3 and 2.4, we introduced two strategies for improving model
calibration. The first, ensemble averaging, relies on training multiple neural
networks on the same data using different random weight initialization. The
second, post hoc calibration via temperature scaling, relies on reserving
additional data for calibration. In this Section, we will compare the
performance of individual neural networks (I), individual neural networks with
post hoc calibrated via temperature scaling (I-C), ensemble averaging (E-M1)
and two different methods for ensemble averaging combined with post hoc
calibrated via temperature scaling (E-M2, and E-M3, see Section 2.4 for
definitions).  Similar to machine learning literature, we will investigate all
approaches across the BIC, ABC, and Mechanical MNIST – Crack Path datasets in
order to identify outcomes that are potentially consistent across diverse
types of mechanical data.  We note that, as expected, Fig. 5 shows that error
generally decreases as the training set size increases, but eventually reaches
a point of “diminishing returns”, where additional training data only
marginally improves the accuracy. This result is consistent with the standard
observation in machine learning and highlights the importance of selecting an
appropriate training set size.

Figure 6: Visualization of the influence of ensemble averaging and post hoc
model calibration on error and ECE for (a) the BIC datasets, (b) the ABC
datasets, and (c) the Mechanical MNIST Crack Path dataset. Color indicates the
(sub)dataset, and marker style specifies the ensemble averaging and
calibration approach, see Section 2.4. Lower ECE and error indicate better
performance (bottom left corner, represented by the green background
gradient). Fig. 8 in Appendix B contains reliability diagrams that supplement
these results.

####  3.2.1 Ensemble Averaging Improves Deep Learning Model Calibration Across
All Datasets

Consistent with the results presented in Section 3.1.2, we find that ensemble
averaging without post hoc calibration (E-M1) improves error and ECE across
all datasets. We note briefly that in Fig. 6a, which represents the BIC
datasets, the I and E-M1 datapoints are repeated from Fig. 4, and in Fig. 6b,
which represents the ABC datasets, the I and E-M1 datapoints are repeated from
our previous publication [57]. Across all datasets shown in Fig. 6a-c, direct
comparison of the I and E-M1 points convincingly indicates that ensemble
averaging holds up as a strategy for improving model performance and
calibration. As outlined in section 2.3, we would like to emphasize that
during the training of each individual network, the weights were randomly
initialized. The impact of this random initialization can be observed in
figure 6. However, it is worth noting that the use of ensemble averaging has
the potential to mitigate the effects of such randomness and effectively
nullify any potential impact of random weight initialization. These positive
results, all without post hoc calibration, serve as a baseline for evaluating
the efficacy of methods designed specifically to improve calibration,
discussed in the next Section.

####  3.2.2 Temperature Scaling After Ensemble Averaging Offers Limited
Improvement to Deep Learning Model Calibration

In Fig. 6, we also show the results of three approaches to performing post hoc
model calibration via temperature scaling (I-C, E-M2, and E-M3), where the
individual models (I) and models with straightforward ensemble averaging
(E-M1) serve as the baseline. From Fig. 6, we see that unlike the results of
applying ensemble averaging, the results of applying temperature scaling are
much less consistent.  In contrast with machine learning literature [19], when
applying temperature scaling to individual neural networks (i.e., comparing
the I models to the I-C models), there were limited and inconsistent benefits.
For example, in Fig. 6b, temperature scaling leads to limited decreases in ECE
for individual networks, while in Fig. 6c temperature scaling leads to a
modest increase in ECE. And, in comparing individual models with temperature
scaling (I-C) to ensemble models without temperature scaling (E-M1), we note
that the E-M1 models more consistently lead to both lower ECE and lower error.

When comparing ensemble averaging combined with post hoc calibration via
temperature scaling (E-M2 and E-M3) to the baseline of straightforward
ensemble averaging (E-M1), we find similar inconsistent results. Specifically,
ensemble averaging combined with temperature scaling (E-M2, E-M3) lead to
inconsistent performance improvements in comparison to ensemble averaging
alone (E-M1). And, temperature scaling prior to ensemble averaging (E-M2) led
to strikingly worse performance compared to ensemble averaging alone (E-M1)
for the Mechanical MNIST – Crack Path dataset, illustrated in Fig. 6c.
Notably, this dramatic difference may be due to the severe class imbalance
present in the Mechanical MNIST – Crack Path dataset. Overall, we found that
for these datasets ensemble averaging offers much more consistent performance
improvements than post hoc model calibration via temperature scaling. The
supplementary reliability diagrams shown in Appendix B, Fig. 8 also support
these results. In addition, it is worth mentioning that results shown in Fig.
5, where increasing the training set size led to improvements in both model
error and model ECE for ensemble neural networks, also indicate that
increasing the initial training set size may be a better use of data resources
than post hoc calibration via temperature scaling. However, we acknowledge
that this statement may vary based on specific desired outcomes and data
resources.

Overall, we note that our findings are based on _empirical_ evidence obtained
through analyzing the $7$ datasets introduced in this manuscript. Broadly
speaking, the trends observed in this study are not necessarily guaranteed to
be consistent with either previous or forthcoming literature that is based on
different data. For example, others have empirically shown that temperature
scaling applied after ensemble averaging (M3) can consistently improve model
calibration compared to ensemble averaging alone (M1) [60].  However, there is
no real consensus on the best method to calibrate deep learning models, and
calibration strategies to date appear dependent on the type of dataset and
deep learning architecture used [19, 36, 45, 51, 60, 84]. As such,  rather
than taking the results of this investigation at face value, we hope that our
work (1) highlights the need for more research towards understanding deep
neural network calibration, and (2) emphasizes the need for additional domain
specific open access datasets for systematically exploring the efficacy of
deep learning approaches.

##  4 Conclusion

To the author’s knowledge, this is the largest investigation to date of deep
learning model calibration for classification problems in mechanics. From this
investigation, we found four key results. First, we found that ensemble neural
networks perform comparably to Gaussian Process Classification and Support
Vector Classification with Platt scaling in terms of model error and model
expected calibration error (ECE) for all three BIC datasets. Second, we found
that increasing the training set size decreases both model error and model ECE
for all three BIC datasets. Third, we found that ensemble averaging
consistently improves both model error and model ECE for all $7$ datasets.
Fourth, we found that temperature scaling offers limited benefits in
comparison to ensemble averaging for all $7$ datasets. Overall, we believe
that this work demonstrates the utility of large scale studies of machine
learning methods applied to problems in mechanics.

Looking forward, we anticipate several major areas of future investigation by
both us and others. First, these datasets can be used to investigate
alternative approaches to simultaneously improving both model error and ECE.
For example, there are multiple approaches to post hoc model calibration
beyond temperature scaling that are amenable to similar investigation [35, 48,
61]. Alternatively, Bayesian methods [31, 42, 85] and evidential deep learning
models [4, 64] aim to output calibrated predictions without any additional
post hoc training. And, building on exciting recent work [62, 79], we
anticipate that there are rich possibilities for physics-informed approaches
to this problem. Second, these datasets can be used to investigate alternative
approaches to evaluating model calibration. As stated in Section 2.5.4,
developing more effective metrics remains an open area of research and one
that deserves attention in a mechanics-specific context. Third, there is a
need to extend this study to additional open-access mechanics-based datasets
from diverse sources. Ultimately, we acknowledge that the current study is
limited to data generated through FEA, which introduces a potential bias. This
highlights the need for further research to explore the problem with data from
other sources, such as experimental testing or molecular dynamic simulations,
which may contain stochastic behavior. Additionally, our study is limited to
simulated data since, to our knowledge, there are currently no open access
experimental mechanics datasets that are both readily accessible and
sufficiently large to include in this study. Overall, our hope is that this
work both offers a starting point for researchers beginning work with deep
learning model calibration, and motivates future mechanics-specific advances
in deep learning model calibration. Because all datasets and codes associated
with this manuscript are available under open-source licenses, others can
readily build on our work and make direct comparisons to alternative methods.

##  5 Additional Information

All datasets used in this investigation have been previously published in
conjunction with prior manuscripts from our group [40, 56, 47]. Each dataset
contains both the metadata to interpret files and the code needed to reproduce
all results. In all cases, data is shared under a CC BY-SA 4.0 License through
the OpenBU Institutional Repository and code is shared under a MIT License
through GitHub. The datasets used are as follows:

  * •

Buckling Instability Classification (BIC) [38]: rectangular columns with
heterogeneous material properties are subject to a fixed level of applied
displacement and classified as either stable or unstable (i.e., buckled).

BIC contains three independent sub-datasets where all three $16\times 1$ input
patterns are sampled from different distributions:

    * BIC 1: $16\times 1$ pattern from distribution 1, sampling 2 discrete values (input) $\mapsto$ stable vs. unstable (output)

    * BIC 2: $16\times 1$ pattern from distribution 2, sampling 3 discrete values (input) $\mapsto$ stable vs. unstable (output)

    * BIC 3: $16\times 1$ pattern from distribution 3, sampling continuous range (input) $\mapsto$ stable vs. unstable (output)

  * •

Asymmetric Buckling Columns (ABC) [56]: heterogeneously architected and
asymmetric columns with homogeneous material properties are subject to a fixed
level of applied displacement and classified as either left buckling or right
buckling.

ABC contains three independent sub-datasets where all three input domain
architecture types are generated through different procedural approaches:

    * ABC 1: spatial graph that represents domains from domain type 1, block stacking (input) $\mapsto$ left vs. right (output)

    * ABC 2: spatial graph that represents domains from domain type 2, uniform rings (input) $\mapsto$ left vs. right (output)

    * ABC 3: spatial graph that represents domains from domain type 3, clipped non-uniform rings (input) $\mapsto$ left vs. right (output)

  * •

Mechanical MNIST – Crack Path [46]: two-dimensional square domains with
heterogeneous material properties and a defined initial crack are subject to a
fixed level of applied displacement. Under these loading conditions, a crack
propagates throughout the domain with the crack path dictated by the
heterogenous material property distribution.

Mechanical MNIST – Crack Path is a single dataset:

    * Mechanical MNIST – Crack Path: $64\times 64$ material property array (input) $\mapsto$ $64\times 64$ damage field (output)

All datasets are schematically illustrated in Fig. 2 for a total of $7$
independently trained and tested cases [38, 56, 46].

The code to reproduce the novel computational results presented in this paper
is available through GitHub (https://github.com/saeedmhz/model-calibration)
shared under a MIT License.

##  6 Acknowledgements

We would like to thank the staff of the Boston University Research Computing
Services and the OpenBU Institutional Repository (in particular Eleni Castro)
for their invaluable assistance with generating and disseminating the datasets
used in this paper. This work was made possible through start up funds from
the Boston University Department of Mechanical Engineering, the David R.
Dalton Career Development Professorship, the Hariri Institute Junior Faculty
Fellowship, the Haythornthwaite Research Initiation Grant, the National
Science Foundation Grant CMMI-2127864, the American Heart Association Career
Development Award 856354, and the Office of Naval Research Grant
N00014-22-1-2066.

##  Appendix A List of Abbreviations

  * FEA \- Finite Element Analysis

  * BIC \- Buckling Instability Classification

  * ABC \- Asymmetric Buckling Columns

  * NLL \- Negative Log Likelihood

  * ECE \- Expected Calibration Error

  * GPC \- Gaussian Process Classification

  * SVC \- Support Vector Classification

  * NN-I \- Individual neural network (see figure 4 and 5)

  * NN-E \- Ensemble neural network (see figure 4 and 5)

  * Method (I) \- Individual neural network without post hoc calibration (see figure 6)

  * Method (I-C) \- Individual neural network with post hoc calibration via temperature scaling (see figure 6)

  * Method (E-M1) \- Ensemble neural network without post hoc calibration (see figure 6)

  * Method (E-M2) \- Ensemble neural network with post hoc calibration via temperature scaling applied before ensemble averaging (see figure 6)

  * Method (E-M3) \- Ensemble neural network with post hoc calibration via temperature scaling applied after en- semble averaging (see figure 6)

##  Appendix B Supplementary Reliability Diagrams

In Section 3, we present the core results of this investigation as plots of
Error vs. Expected Calibration Error (ECE). As introduced in Section 2.5.3 and
illustrated in Fig. 3, ECE is connected to the reliability diagram, a common
strategy for visualizing model calibration. Here we provide supplementary
reliability diagrams to accompany Fig. 4 and Fig. 6. In Fig. 7, we show $12$
calibration curves that correspond to the results shown in Fig. 4.
Specifically, for BIC 1, BIC 2, and BIC 3 we plot reliability diagrams for
Gaussian Process Classification, Support Vector Classification with Platt
scaling, a representative Individual Neural Network, and an Ensemble Neural
Network. In Fig. 8, we show $9$ calibration curves that correspond to the
results shown in Fig. 6. Specifically, for BIC 3, ABC 3, and Mechanical MNIST
Crack Path we plot reliability diagrams for straightforward ensemble averaging
(E-M1), post hoc calibration temperature scaling followed by ensemble
averaging (E-M2), and ensemble averaging followed by post hoc calibration
temperature scaling (E-M3). In addition, we plot histograms of the
representative distribution of model confidence for E-M1 for BIC 3, ABC 3, and
Mechanical MNIST Crack Path. These histograms not only indicate the bins that
will have the heaviest weight, but also illustrate the severe class imbalance
present in the Mechanical MNIST Crack Path dataset.

Beyond overall ECE, which is a weighted average of the calibration error in
each bin, the reliability diagram allows us to visualize the Maximum
Calibration Error and the regions where the different models tend to be over
and under confident. In addition, they provide a visualization of the outcomes
of post hoc model calibration via temperature scaling. These diagrams provide
additional contextual information that helps address some of the limitations
of ECE as a metric, discussed in Section 2.5.4. Overall, we recommend
visualizing the reliability diagram in addition to computing ECE prior to
deploying a given model.

Figure 7: Reliability diagrams as a supplement to Fig. 4. Figure 8:
Reliability diagrams and confidence distribution histograms as a supplement to
Fig. 6.

##  Appendix C Details of the Neural Networks

To supplement the description of the neural networks we used in this work,
introduced in Sections 2.2.3, 2.2.4, and 2.2.5, we have included network
schematics for our implementations of each network in Fig. 9.

Figure 9: This figure supplements Sections 2.2.3, 2.2.4, and 2.2.5 by
schematically illustrating the different neural network architectures. The
fully connected neural networks consists of three linear layers, each with
$200$ nodes, and utilizes dropout with a probability of $0.5$ during training.
Our previous work [57] provides further details on the graph neural network
that we used. Fiinally, the UNet network employs “Conv-blocks,” which repeat a
2D convolutional layer followed by a 2D batch normalization and a ReLU
activation function twice. The convolutional layers have a filter size of $3$
and a stride and padding of $1$, while the maxpool layers use a filter size of
$2$. The transposed convolutional (“UpConv”) layers employ a filter and stride
of $2$.

## References

  * [1] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375, 2018.
  * [2] Mark Alber, Adrian Buganza Tepole, William R Cannon, Suvranu De, Salvador Dura-Bernal, Krishna Garikipati, George Karniadakis, William W Lytton, Paris Perdikaris, Linda Petzold, et al.  Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences.  NPJ digital medicine, 2(1):1–11, 2019.
  * [3] Martin Alnæs, Jan Blechta, Johan Hake, August Johansson, Benjamin Kehlet, Anders Logg, Chris Richardson, Johannes Ring, Marie E Rognes, and Garth N Wells.  The fenics project version 1.5.  Archive of Numerical Software, 3(100), 2015.
  * [4] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus.  Deep evidential regression.  Advances in Neural Information Processing Systems, 33:14927–14937, 2020.
  * [5] Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe.  Analyzing inverse problems with invertible neural networks.  arXiv preprint arXiv:1808.04730, 2018.
  * [6] Paul D Arendt, Daniel W Apley, and Wei Chen.  Quantification of model uncertainty: Calibration, model discrepancy, and identifiability.  2012\.
  * [7] Albert P Bartók, James Kermode, et al.  Improved uncertainty quantification for gaussian process regression based interatomic potentials.  arXiv preprint arXiv:2206.08744, 2022.
  * [8] Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber.  Multi-column deep neural networks for image classification.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3642–3649. IEEE, 2012.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [10] Hung P Do, Yi Guo, Andrew J Yoon, and Krishna S Nayak.  Accuracy, uncertainty, and adaptability of automatic myocardial asl segmentation using deep cnn.  Magnetic resonance in medicine, 83(5):1863–1874, 2020.
  * [11] Harris Drucker, Christopher J Burges, Linda Kaufman, Alex Smola, and Vladimir Vapnik.  Support vector regression machines.  Advances in neural information processing systems, 9, 1996.
  * [12] David Duvenaud.  The kernel cookbook: Advice on covariance functions.  URL https://www. cs. toronto. edu/~ duvenaud/cookbook, 2014.
  * [13] Azeddine Elhassouny and Florentin Smarandache.  Trends in deep convolutional neural networks architectures: A review.  In 2019 International Conference of Computer Science and Renewable Energies (ICCSRE), pages 1–8. IEEE, 2019.
  * [14] Matthias Fey and Jan E. Lenssen.  Fast graph representation learning with PyTorch Geometric.  In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  * [15] Lia Gander, Simone Pezzuto, Ali Gharaviri, Rolf Krause, Paris Perdikaris, and Francisco Sahli Costabal.  Fast characterization of inducible regions of atrial fibrillation models with multi-fidelity gaussian process classification.  Frontiers in Physiology, page 260, 2022.
  * [16] Christophe Geuzaine and Jean-François Remacle.  Gmsh: A 3-d finite element mesh generator with built-in pre-and post-processing facilities.  International journal for numerical methods in engineering, 79(11):1309–1331, 2009.
  * [17] Tilmann Gneiting and Adrian E Raftery.  Strictly proper scoring rules, prediction, and estimation.  Journal of the American statistical Association, 102(477):359–378, 2007.
  * [18] Aldair E Gongora, Kelsey L Snapp, Richard Pang, Thomas M Tiano, Kristofer G Reyes, Emily Whiting, Timothy J Lawton, Elise F Morgan, and Keith A Brown.  Designing lattices for impact protection using transfer learning.  Matter, 5(9):2829–2846, 2022.
  * [19] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In International conference on machine learning, pages 1321–1330. PMLR, 2017.
  * [20] Kai Guo, Zhenze Yang, Chi-Hua Yu, and Markus J Buehler.  Artificial intelligence and machine learning in design of mechanical materials.  Materials Horizons, 8(4):1153–1172, 2021.
  * [21] Tianhong Han, Kaleem S Ahmed, Arun K Gosain, Adrian Buganza Tepole, and Taeksang Lee.  Multi-fidelity gaussian process surrogate modeling of pediatric tissue expansion.  Journal of Biomechanical Engineering, 144(12):121005, 2022.
  * [22] Paul Z Hanakata, Ekin D Cubuk, David K Campbell, and Harold S Park.  Forward and inverse design of kirigami via supervised autoencoder.  Physical Review Research, 2(4):042006, 2020.
  * [23] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf.  Support vector machines.  IEEE Intelligent Systems and their applications, 13(4):18–28, 1998\.
  * [24] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [25] Shruti Jadon.  A survey of loss functions for semantic segmentation.  In 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pages 1–7. IEEE, 2020.
  * [26] Zeqing Jin, Zhizhou Zhang, and Grace X Gu.  Automated real-time detection and prediction of interlayer imperfections in additive manufacturing processes using artificial intelligence.  Advanced Intelligent Systems, 2(1):1900130, 2020.
  * [27] Akshay Joshi, Prakash Thakolkaran, Yiwen Zheng, Maxime Escande, Moritz Flaschel, Laura De Lorenzis, and Siddhant Kumar.  Bayesian-euclid: Discovering hyperelastic material laws with uncertainties.  Computer Methods in Applied Mechanics and Engineering, 398:115225, 2022.
  * [28] Sotirios Kakaletsis, Emma Lejeune, and Manuel K Rausch.  Can machine learning accelerate soft material parameter identification from complex mechanical test data?  Biomechanics and Modeling in Mechanobiology, pages 1–14, 2022.
  * [29] Michael G Kapteyn, Jacob VR Pretorius, and Karen E Willcox.  A probabilistic graphical model foundation for enabling predictive digital twins at scale.  Nature Computational Science, 1(5):337–347, 2021.
  * [30] K Karapiperis, L Stainier, M Ortiz, and JE Andrade.  Data-driven multiscale modeling in mechanics.  Journal of the Mechanics and Physics of Solids, 147:104239, 2021\.
  * [31] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  Advances in neural information processing systems, 30, 2017.
  * [32] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [33] Georgios Kissas, Jacob H Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pappas, and Paris Perdikaris.  Learning operators with coupled attention.  Journal of Machine Learning Research, 23(215):1–63, 2022.
  * [34] Hiba Kobeissi and Emma Lejeune.  Mechanical mnist - cahn-hilliard, 2022.
  * [35] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In International conference on machine learning, pages 2796–2804. PMLR, 2018.
  * [36] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  Advances in neural information processing systems, 30, 2017.
  * [37] Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  2010\.
  * [38] Emma Lejeune.  Buckling instability classification (BIC).  https://open.bu.edu/handle/2144/40085, 2020.
  * [39] Emma Lejeune.  Mechanical MNIST: A benchmark dataset for mechanical metamodels.  Extreme Mechanics Letters, 36:100659, 2020.
  * [40] Emma Lejeune.  Geometric stability classification: datasets, metamodels, and adversarial attacks.  Computer-Aided Design, 131:102948, 2021.
  * [41] Anders Logg, Kent-Andre Mardal, and Garth Wells.  Automated solution of differential equations by the finite element method: The FEniCS book, volume 84.  Springer Science & Business Media, 2012.
  * [42] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson.  A simple baseline for bayesian uncertainty in deep learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [43] Andrew Mann and Surya R Kalidindi.  Development of a robust cnn model for capturing microstructure-property linkages and building property closures supporting material design.  Virtual Materials Design, page 879614107, 2022.
  * [44] Alireza Mehrtash, William M Wells, Clare M Tempany, Purang Abolmaesumi, and Tina Kapur.  Confidence calibration and predictive uncertainty estimation for deep medical image segmentation.  IEEE transactions on medical imaging, 39(12):3868–3878, 2020.
  * [45] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic.  Revisiting the calibration of modern neural networks.  Advances in Neural Information Processing Systems, 34:15682–15694, 2021.
  * [46] Saeed Mohammadzadeh and Emma Lejeune.  Mechanical mnist crack path.  https://open.bu.edu/handle/2144/42757, 2021.
  * [47] Saeed Mohammadzadeh and Emma Lejeune.  Predicting mechanically driven full-field quantities of interest with deep learning-based metamodels.  Extreme Mechanics Letters, 50:101566, 2022.
  * [48] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015\.
  * [49] Alexandru Niculescu-Mizil and Rich Caruana.  Predicting good probabilities with supervised learning.  In Proceedings of the 22nd international conference on Machine learning, pages 625–632, 2005.
  * [50] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran.  Measuring calibration in deep learning.  In CVPR Workshops, volume 2, 2019.
  * [51] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  Advances in neural information processing systems, 32, 2019.
  * [52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [53] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [54] Paris Perdikaris, Daniele Venturi, Johannes O Royset, and George Em Karniadakis.  Multi-fidelity modelling via recursive co-kriging and gaussian–markov random fields.  Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2179):20150018, 2015.
  * [55] John Platt et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  Advances in large margin classifiers, 10(3):61–74, 1999.
  * [56] Peerasait Prachaseree and Emma Lejeune.  Asymmetric buckling columns (ABC).  https://open.bu.edu/handle/2144/43730, 2022.
  * [57] Peerasait Prachaseree and Emma Lejeune.  Learning mechanically driven emergent behavior with message passing neural networks.  Computers & Structures, 270:106825, 2022.
  * [58] Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis.  Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons.  arXiv preprint arXiv:2201.07766, 2022.
  * [59] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  arXiv preprint arXiv:1706.02413, 2017.
  * [60] Rahul Rahaman and Alexandre H Thiery.  Uncertainty quantification and deep ensembles.  arXiv preprint arXiv:2007.08792, 2020.
  * [61] Amir Rahimi, Kartik Gupta, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley.  Post-hoc calibration of neural networks.  arXiv preprint arXiv:2006.12807, 2020.
  * [62] Maziar Raissi, Paris Perdikaris, and George E Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational physics, 378:686–707, 2019.
  * [63] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [64] Murat Sensoy, Lance Kaplan, and Melih Kandemir.  Evidential deep learning to quantify classification uncertainty.  Advances in neural information processing systems, 31, 2018.
  * [65] Shai Shalev-Shwartz and Shai Ben-David.  Understanding machine learning: From theory to algorithms.  Cambridge university press, 2014.
  * [66] Dongil Shin, Andrea Cupertino, Matthijs HJ de Jong, Peter G Steeneken, Miguel A Bessa, and Richard A Norte.  Spiderweb nanomechanical resonators via bayesian optimization: inspired by nature and guided by machine learning.  Advanced Materials, 34(3):2106248, 2022.
  * [67] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhaktuni.  U-net and its variants for medical image segmentation: A review of theory and applications.  Ieee Access, 9:82031–82057, 2021.
  * [68] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [69] Xiao Sun, Bahador Bahmani, Nikolaos N Vlassis, WaiChing Sun, and Yanxun Xu.  Data-driven discovery of interpretable causal relations for deep learning material laws with uncertainty propagation.  Granular Matter, 24(1):1–32, 2022.
  * [70] Nikolaos N Vlassis, Ran Ma, and WaiChing Sun.  Geometric deep learning for computational mechanics part i: anisotropic hyperelasticity.  Computer Methods in Applied Mechanics and Engineering, 371:113299, 2020.
  * [71] Liwei Wang, Yu-Chin Chan, Faez Ahmed, Zhao Liu, Ping Zhu, and Wei Chen.  Deep generative modeling for mechanistic-based learning and design of metamaterial systems.  Computer Methods in Applied Mechanics and Engineering, 372:113377, 2020.
  * [72] Zhenlin Wang, Xun Huan, and Krishna Garikipati.  Variational system identification of the partial differential equations governing the physics of pattern-formation: Inference under varying fidelity and noise.  Computer Methods in Applied Mechanics and Engineering, 356:44–74, 2019.
  * [73] Zhenlin Wang, Bowei Wu, Krishna Garikipati, and Xun Huan.  A perspective on regression and bayesian approaches for system identification of pattern formation dynamics.  Theoretical and Applied Mechanics Letters, 10(3):188–194, 2020\.
  * [74] Jason Weston and Chris Watkins.  Multi-class support vector machines.  Technical report, Citeseer, 1998.
  * [75] Christopher KI Williams and Carl Edward Rasmussen.  Gaussian processes for machine learning, volume 2.  MIT press Cambridge, MA, 2006.
  * [76] Jian-Ying Wu.  A unified phase-field theory for the mechanics of damage and quasi-brittle failure.  Journal of the Mechanics and Physics of Solids, 103:72–99, 2017\.
  * [77] Jian-Ying Wu, Vinh Phu Nguyen, Chi Thanh Nguyen, Danas Sutula, Sina Sinaie, and Stéphane PA Bordas.  Phase-field modeling of fracture.  Advances in Applied Mechanics, 53:1–183, 2020.
  * [78] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  * [79] Yibo Yang and Paris Perdikaris.  Adversarial uncertainty quantification in physics-informed neural networks.  Journal of Computational Physics, 394:136–152, 2019.
  * [80] Minglang Yin, Enrui Zhang, Yue Yu, and George Em Karniadakis.  Interfacing finite elements with deep neural operators for fast multiscale modeling of mechanics problems.  Computer Methods in Applied Mechanics and Engineering, 402:115027, 2022.
  * [81] Lingxiao Yuan, S. Park, Harold, and Emma Lejeune.  Mechanical mnist – distribution shift, 2022.
  * [82] Bianca Zadrozny and Charles Elkan.  Transforming classifier scores into accurate multiclass probability estimates.  In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002.
  * [83] Mehrshad Zandigohar, Deniz Erdoğmuş, and Gunar Schirner.  Netcut: Real-time dnn inference using layer removal.  In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1845–1850. IEEE, 2021.
  * [84] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han.  Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning.  In International conference on machine learning, pages 11117–11128. PMLR, 2020.
  * [85] Xiaoxuan Zhang and Krishna Garikipati.  Bayesian neural networks for weak solution of pdes with uncertainty quantification.  arXiv preprint arXiv:2101.04879, 2021.