# Self-supervised feature distillation and design of experiments for efficient
training of micromechanical deep learning surrogates

Patxi Fernandez-Zelaia<EMAIL_ADDRESS>Jason Mayeur<EMAIL_ADDRESS>Jiahao Cheng<EMAIL_ADDRESS>Yousub Lee<EMAIL_ADDRESS>Kevin Knipe
<EMAIL_ADDRESS>Kai Kadau<EMAIL_ADDRESS>Manufacturing Science Division, Oak Ridge National Laboratory, Oak Ridge, TN,
United States  Computational Sciences & Engineering Division, Oak Ridge
National Laboratory, Oak Ridge, TN, United States  Siemens Energy, Orlando,
FL, United States

###### Abstract

Machine learning surrogate emulators are needed in engineering design and
optimization tasks to rapidly emulate computationally expensive physics-based
models. In micromechanics problems the local full-field response variables are
desired at microstructural length scales. While there has been a great deal of
work on establishing architectures for these tasks there has been relatively
little work on establishing microstructural experimental design strategies.
This work demonstrates that intelligent selection of microstructural volume
elements for subsequent physics simulations enables the establishment of more
accurate surrogate models. There exist two key challenges towards establishing
a suitable framework: (1) microstructural feature quantification and (2)
establishment of a criteria which encourages construction of a diverse
training data set. Three feature extraction strategies are used as well as
three design criteria. A novel contrastive feature extraction approach is
established for automated self-supervised extraction of microstructural
summary statistics. Results indicate that for the problem considered up to a
8% improvement in surrogate performance may be achieved using the proposed
design and training strategy. Trends indicate this approach may be even more
beneficial when scaled towards larger problems. These results demonstrate that
the selection of an efficient experimental design is an important
consideration when establishing machine learning based surrogate models.

###### keywords:

machine learning , experimental design , ICME , surrogate modeling ,
micromechanics

††journal: Computational Materials Science††Notice of Copyright. This
manuscript has been authored by UT-Battelle, LLC under Contract No. DE-
AC05-00OR22725 with the U.S. Department of Energy. The United States
Government retains and the publisher, by accepting the article for
publication, acknowledges that the United States Government retains a non-
exclusive, paid-up, irrevocable, world-wide license to publish or reproduce
the published form of this manuscript, or allow others to do so, for United
States Government purposes. The Department of Energy will provide public
access to these results of federally sponsored research in accordance with the
DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

##  1 Introduction

Numerical physics-based models are essential tools needed to study many
complex physical systems. However, due to the computational burden associated
with discretizing and solving the governing laws, using these models to
optimize and design engineering systems is difficult. Driven in part by the
materials genome initiative integrated computational materials engineering
(ICME) tools have become ubiquitous in various fields [1]. While classically
focused on physics-based simulations and statistical approaches the ICME
paradigm has benefited greatly from recent advances in machine learning (ML)
and the development of open source frameworks. For instance, original methods
for binary phase image generation relied on pixel-wise simulated annealing
approaches [2]; presently ML-based generative diffusion models can produce
much more realistic and complex structures [3, 4, 5, 6]. Google and Microsoft
have even recently released materials focused generative models indicating
their interest in ICME approaches [7, 8].

In the statistics community classic surrogate reduced order models have been
typically used to emulate parameterized numerical codes with scalar valued
outputs [9, 10, 11]. Gaussian processes (GPs) are favored in the field of
computer experiments because, for deterministic simulations, GPs can be
formulated to behave as interpolators; the function will pass exactly through
the training data without over fitting. Modern GP implementations have been
formulated which can capture more complex input and output structures [12,
13]. Additionally there are composite approaches combining GPs with moderm ML
approaches [14, 15]. Fundamentally GPs can be thought of as making a
prediction at an input $\bm{x}$ by performing a weighted average of “near-by”
training examples. This measure of closeness requires calculation of a
distance metric (and learning of spatial correlation functions). For
parametric inputs this is a rather simple Euclidean distance for low-
dimensional data. Parametric definition of a complex system is made possible
via careful use of relevant summary statistics [16]. However, for many
engineering and science problems parametric description of the problem may not
be feasible. For instance, emulating finite element (FE) models with
discretized complex geometries, which cannot be easily captured via GPs, can
however be emulated using ML approaches [17, 18]. Similarly, in many mechanics
problems the input to a numerical code may not only be fundamental scalar
properties (stiffness, strength) but input structural representations. For
instance, shown in Fig. 1, microstructural volume elements (MVEs) contain
$32^{3}$ voxels with the local state in each being represented by three Euler
angles. This corresponds to $32,768\cdot 3$ total “features” which need to be
considered in computing the necessary GP distance measure. Furthermore, in
certain settings the full-field (voxel-wise) response is needed e.g. at least
$32,768$ total output values. Hence, for these complex structural problems,
direct application of classic GPs is not well suited. Instead, convolutional
neural networks (CNNs) have been shown to be well suited towards emulating
these micromechanical localization problems [19, 20, 21, 22, 23, 24]. This is
because the convolution operation is well suited for processing spatial
structure and CNN architectures may be easily designed for multiple output
predictions. Furthermore there is a clear link to Green’s functions in
continuum theory which use analytical convolution operations to predict the
response of heterogenous mediums [20]. This theoretical basis provides good
justification for using data-driven CNN networks.

Development of effective statistical or ML surrogate models not only requires
a suitable model form but also careful construction of an appropriate
experimental design. In physical experiments designs are often focused on
eliminating confounding or biasing factors, mitigating against the effects of
experimental noise, and extracting trends e.g. estimating the sensitivity of
the response variable to input factors [25]. In the statistics community
designs for surrogate model development favor space-filling designs [26].
Designs here refer to the collection of training examples
$\left\\{\bm{x}_{1},\ldots,\bm{x}_{N}\right\\}$ used to evaluate the physics
model and train the surrogate. Consider that points close to one another are
assumed to have similar responses. In fact, for deterministic simulations, in
the limit as two points overlap the responses are identical. Hence, space-
filling designs are constructed by optimizing a specified criteria which
encourages the establishment of a desirable distribution of points in space.
The criteria always requires definition of a distance metric as it is
necessary to avoid clustering. There exists a number of different kinds of
space-filling design criteria each consisting of trade offs between design
quality and optimization complexity [26].

Development of designs for the micromechanics problem, shown in Fig. 1, is
extremely challenging due to the need to define a distance metric for pair-
wise comparison of structures. In certain materials problems this is made
possible via the use of domain-science knowledge to define statistical
descriptors. For instance in composite structures features may include phase
volume fraction, particle size, particle shape, etc. [27]. For polycrystalline
systems diversity of crystallographic texture is critically important [28]. A
recent work has established a homogenization framework which predicts
aggregate constitutive properties using single-crystalline responses [29].
Prior localization works have engineered MVEs to effectively train surrogates
[20]; single-voxel grains for fine scale responses, multi-voxel grains for
larger scales, and single crystals with embedded single-voxel speckles to
capture “delta” localization phenomena. There are a number of works that have
demonstrated that diversity of training data in atomistic problems is of
paramount importance as well [30, 31]. In one work an entropy-based criteria
was used to increase the volume of the training data to mitigate against
extrapolation [30]. Critically, however, there are no existing works that
quantitatively demonstrate the importance of the experimental design in
training full-field micromechanical ML surrogate models.

Figure 1: Overall approach is to identify most unique and informative MVEs for
subsequent physics evaluation and surrogate training. Hypothesis is that more
efficient training may be performed if MVEs are chosen using an appropriate
design criteria.

This work hypothesizes that intelligent selection of MVEs for subsequent
physics simulations will enable the establishment of more accurate ML
surrogate models. The overall strategy, shown in Fig. 1, is similar to that of
traditional computer experiments; distribute settings over a diverse space,
run simulations, and train a surrogate model. The main challenge, however, is
that it is not immediately straight forward how to quantify the MVE inputs.
Hence, three approaches are used to distill the MVEs to a lower dimensional
representation: (1) a variational autoencoder (VAE) latent description (2)
self-supervised derived microstructural statistics and (3) domain science
inspired microstructural statistics. Furthermore, due to differences in
information contained in each descriptor three space-filling design criteria
are parametrically tested to evaluate their performance. When compared against
a random selection of MVE training examples results indicate that up to a 8%
boost in surrogate performance, shown to be statistically significant, may be
achieved via the proposed training strategy. We observed that this benefit
increases with increasing data set size, and hence, this approach may be even
more beneficial when scaled towards larger problems. Interestingly, it is
observed that while the mean performance of the surrogate is only marginally
sensitive to the microstructure, there are significantly more poorly
performing outliers for textured large grain MVEs. We suspect that this may
simply be due to smaller grain size instances being more representative and,
hence, containing more information. These results demonstrate that
experimental design for micromechanical problems is imperative. In addition,
our self-supervised approach for distilling microstructure statistics may be
suitable for other tasks where image similarity metrics are important.

##  2 Methods

It is hypothesized here that careful selection of MVEs, for subsequent physics
simulation, will enable training of a more accurate micromechanical deep
learning surrogate when compared against random selection. In the active
learning context it is assumed that a trained ML model is already available
and activations from neurons may be used as features when constructing the
sequential design e.g. identifying subsequent examples for labeling, physics
simulation, etc. [32]. This approach is feasible as targeting novel
activations when constructing the sequential design will indeed ensure that
the model will “see” novel examples during subsequent training. However, there
are a few key challenges which arise which may be problematic for the
materials problem posed here. In ML the active learning sequential design task
is often referred to as core-set and the design is optimized using variants of
the maximin distance criteria. In the referenced work the VGG-16 architecture
is used on CIFAR-10 and SVHN data sets (60,000 and 600,000 images in each,
respectively). These authors note that activations from the final fully
connected layer are used which is 1,000-dimensional. At such high dimensions
the search space becomes extremely large and constructing a design is
challenging. For natural images this may be less of a concern as data sets are
rather large (60,000 and 600,000 in this case, some image data sets 1M+). In
our context, with only a total of 6,825 MVEs, searching a 1000-dimensional
space would be extremely challenging. Furthermore, the challenge addressed
here is focused on the initial design of experiments and, hence, there is no
ML available from which features may be extracted.

Despite the initial design challenge being fundamentally different than the
active learning problem the two approaches share similar strategies; identify
salient features, use a design criteria to identify a diverse design, perform
physics simulations (or perform labeling in the natural image context), and
train the ML model. In a CNN activations can be related to localized features.
For instance, a candidate example which exhibits distinctly different
activations than examples already in the training data set may contain unique
edges, corners, shapes, textures, etc.. Alternatively, unique activations may
correspond to similar features but at different spatial locations in the
input. In the micromechanical surrogate model this seems appealing; a data set
containing various triple junctions of different orientation relationships
will be desirable towards training a generalizable model. Conversely, the same
triple joint embedded in two volumes of differing overall crystallographic
texture may behave differently. Therefore, both statistical features which
describe aggregate qualities of the microstructure, and localized features
which describe spatial arrangement of structure, are likely to provide value.

Three key design criteria will be used for assessing the diversity of training
ensembles. In the context of computer experiments space-filling designs are
commonly used to ensure good “spread” in the $d$-dimensional model input space
[26]. The challenge in identifying a design is in minimizing the corresponding
criteria; consider a design of $N$ $d$-dimensional points contains $N\cdot d$
total values which must be identified. Furthermore, the curse of
dimensionality drives search spaces to be ever larger with increasing $d$.
Three key space-filling design criteria will be considered in this work; a
greedy maxi-min design [26], a greedy maximum projection (maxPro) design [33],
and a data twinning approach [34].

###  2.1 Physics simulation

The local stress response for input MVEs was simulated using open source
software PRISMS-plasticity [35]. In this exploratory work the material was
assumed to behave elastically so that a sufficiently large data set could be
quickly generated for testing the stated hypothesis. While this work is
specifically focused on the simplified elastic constitutive response the main
contribution here is demonstrating the importance of the initial experimental
design; this should extend to any mechanical constitutive model. Elastic
constants corresponding to a Ni-based superalloy were utilized with
$C_{11}=199\,GPa$, $C_{12}=128\,GPa$, and $C_{44}=99\,GPa$ [36]. MVEs were
loaded in uniaxial tension with a $50\,MPa$ traction applied on the top $z=1$
plane and periodic boundary conditions elsewhere. A $32\times 32\times 32$
element mesh was used with linear interpolation shape functions. Simulations
were run over all candidate MVEs to assemble a large data set. Experimental
designs were tested by sub-sampling from this data set prior to training the
surrogate models.

###  2.2 Microstructural features

The microstructural MVEs used in this work were generated using open source
software Neper [37]. Each example is $32^{3}$ with grain sizes varying from 4
to 16 voxels. A total of 6,825 examples were generated. 1,200 of these (12
grain sizes, 100 total random seeds) were generated with uniformly random
crystallographic texture. The remaining MVEs were generated with a randomly
selected $(hkl)$ fiber texture in a randomly selected $<uvw>$ direction.
Furthermore, to avoid the presence of solely sharp textures, orientations were
subsequently diffused by randomly applying rotations with Euler angles $\sim
Unif(-10^{\circ},10^{\circ})$. This randomization ensures that both sharp and
diffuse textures are present in the generated data set.

Three procedures for quantifying microstructural features will be considered
in this work: latent space features from a VAE, features from a novel self-
supervised network, and classical microstructural descriptors. For the
classical microstructural descriptors the grain size, prescribed during
generation, is combined with volume-averaged crystallographic information. The
latter is captured using generalized spherical harmonics (GSH) [38]. The
orientation distribution function at each spatial location, $\bm{x}$, can be
described by a basis expansion

$\displaystyle f_{\bm{x}}\left(\bm{g}\right)=\sum_{\mu,n,l}F_{l\bm{x}}^{\mu
n}\dot{\dot{T}}_{l}^{\mu n}\left(\bm{g}\right),$ (1)

where $\bm{g}$ are the Euler angles, $\mu,n,l$ are indices for multiple sums,
and $F_{l\bm{x}}^{\mu n}$ are the GSH coefficients at $\bm{x}$. An expansion
consisting of nine total terms was used which has been shown to be sufficient
for similar quantitative tasks for FCC materials [20]. $\dot{\dot{T}}_{l}^{\mu
n}$ are the corresponding GSH basis functions. Both basis weights and
functions are complex valued. The orientation distribution function over a
volume can be obtained by simply computing the mean over all spatial locations
of the individual basis coefficients. As this representation is complex in
nature, real and imaginary components are taken and concatenated together with
grain size information to define a “classic” descriptor
$\bm{z}\in\mathcal{R}^{18}$.

VAEs construct feature vectors via a non-linear dimensionality reduction
mapping [39]. Shown in Fig 2 is a schematic of inputs, outputs, and the latent
representation. The network consists of an encoder, which maps input $\bm{x}$
to a latent $\bm{z}$, and a decoder which maps back to the original modality
$\hat{\bm{x}}$. The network is trained in an unsupervised fashion to minimize
the discrepancy between input and outputs e.g. minimize the reconstruction
loss. While similar to autoencoders VAEs also include additional regularizing
constraints on the latent space. First stochasticity is introduced in the
model by having the encoder predict both a mean latent representation and a
corresponding variance measure. Next the “reparameterization trick” is used to
sample from this latent distribution; white noise is scaled by the networks
variance prediction and added to the predicted mean. This perturbed latent
vector is mapped back to $\hat{\bm{x}}$ and the mean and variance parameters
contribute to an additional Kullback–Leibler (KL) divergence loss term that
encourages the latent space to be a standard normal distribution. This effort
is needed to ensure that the latent space is continuous. Consider that small
latent perturbations in an AE may produce a nonsensical output e.g. values may
not be interpolated in the latent space. Since the VAE uses a distribution to
represent the latent space it intrinsically includes small perturbations in
the training procedure e.g. the $\bm{x}\rightarrow\bm{z}$ mapping never
produces the same $\bm{z}$ but nonetheless close $\bm{z}$’s. This ensures that
the latent is continuous and may indeed be interpolated. This is important
since this work seeks to use distance measures for experimental design and,
hence, it is important that the latent space be continuous in some fashion.
Furthermore, a variation of the VAE, $\beta$-VAE, was used to balance the KL
and reconstruction loss terms [40]. For extracting features a fairly large
latent space was necessary to produce reasonable reconstructions e.g.
$\bm{x}\in\mathcal{R}^{512}$. Note that the original dimensionality of each
MVE is $32^{3}\cdot 3=98,304$ and so this still represents a compression ratio
of $\sim 200$. The high dimensionality of the latent space may be valuable in
terms of quantifying localized spatial features, however, this may present
problems when constructing designs (high dimensional optimization problem).

Figure 2: VAE schematic for extracting localized MVE features. Figure 3: Self-
supervised approach for 3D MVE feature extraction and network training.
Subsampling of MVEs enables self-supervised learning and, critically,
encourages the learning of statistical descriptors.

Finally a self-supervised alternative is proposed for automatically extracting
relevant microstructural statistics from MVEs. Materials at most length scales
are inherently stochastic. Consider, for instance, that two micrographs from
the same material will have different appearance but statistically may be
identical. A VAE may not capture this subtlety as it is trained to optimize a
reconstruction and, hence, explicitly captures localized features. As an
example consider that two nearly identical micrographs, offset by a small
horizontal translation, will produce two different $\bm{z}$’s when passed
through the encoder network. Hence, there is a need to establish an automated
way to infer summary statistics which can enable pair-wise similarity
measurement across stochastic MVEs. Interestingly, with a few exceptions,
there are very few works in the materials community focused on using ML for
these kinds of tasks [41]. Perhaps this is because most ML vision models are
focused on natural images, where localization is important, and, hence, there
are fewer models suitable for the stochastic materials problem. Perhaps the
closest related task occurs in the field of texture analysis (not
crystallographic texture but image texture) [42]. Interestingly there is a
similar implementation, focused on the extraction of statistical features from
a CNN, for remote sensing imagery [43]. In biological applications there are
also extensive applications of “twin” networks for measuring similarity
between micrographs [44]. In summary, there is a critical need for specialized
ML methods applicable and tailored for extraction of statistical features from
stochastic materials data.

Contrastive learning seeks to establish ML models via training on pair-wise
similarity measures [45, 46, 47, 48, 49]. This is valuable for the
experimental design problem as all design criteria rely on distance metrics. A
schematic of the self-supervised contrastive learning procedure used in this
work is shown in Fig 3. During training a $32^{3}$ MVE is sub-sampled to
produce two $16^{3}$ MVEs; an anchor ($\bm{x}_{a}$) and a positive
($\bm{x}_{p}$) example. Then a negative ($\bm{x}_{n}$) $16^{3}$ MVE example is
randomly cropped from another random training example. Since the anchor and
positive examples come from the same parent MVE their statistics should be
similar. This of course assumes stationary behavior and that the $16^{3}$ MVE
is representative. Likewise the statistics of the anchor and negative example
will, on average, be different. Statistics are generated via training of a
single NN which maps each example into latent space vector. In the latent
space a contrastive loss is defined which encourages the network to place the
anchor and positive examples close together and the negative example far
apart. The training loss can be defined as [49],

$\displaystyle\mathcal{L}=\max(0,d(\bm{x}_{a},\bm{x}_{p})-d(\bm{x}_{a},\bm{x}_{n})+1/2),$
(2)

where $d(\cdot,\cdot)$ is the Euclidean distance. The constant $1/2$ is
referred to as the margin and controls the spacing between points and, hence,
the overall scale of the latent space. This loss drives the distance between
anchor and positive examples to be as small as possible and, conversely, the
distance between the anchor and negative example to be large.

The specific network architecture used for extracting microstructural
statistics is shown in Fig. 4. Using domain knowledge, that both local spatial
information and volume averaged texture information are important, the network
has been tailored to capture these two considerations explicitly. In parallel
the network (1) performs a series of CNN operations to construct spatial
feature maps and (2) uses an MLP to map the input Euler angle representation
to “texture features”. Spatial averaging is performed on the texture features
to produce a global crystallographic feature vector. Similarly, feature map
statistics are produced by computing the mean and variance over all spatial
indices. These three vectors (mean of spatial-orientation feature maps,
variance of spatial-orientation feature maps, and volume averaged orientation
features) are concatenated and passed through a MLP to mix the information
prior to outputting a $\bm{z}\in\mathcal{R}^{16}$ latent representation.
Originally we did not consider the spatial mean/variance pooling operations
but found that model performance improved doing so. Furthermore, we originally
did not include a separate texture-only information stream but found that this
also improved performance. It is suspected that this is because the network is
not tasked with “doing everything” at once and introduction of this domain
knowledge makes training easier.

Figure 4: Novelty of the feature extraction procedure is that it intrinsically
operates on image statistics via construction of the network. Mean and
variance of spatial-orientation feature maps are combined with volume averaged
orientation features prior to passing through the final MLP. This encourages
the network to separately construct orientation and spatial-orientation
statistics prior to mixing in the final MLP.

###  2.3 Design of experiments

Three key space-filling designs will be considered in this work: maximin
distance design [26], maximum projection design [33], and a data twinning
design [34]. The former two are solely space-filling designs seeking to spread
points uniformly in space with some subtle differences. The latter criteria is
unique in that it seeks to balance a space-filling objective while also
optimizing an distributional objective. The distributional objective seeks to
identify design points from a candidate data set which collectively emulate
the probability distribution representative of the original data set e.g. the
empirical distribution.

Figure 5: Maximum projection design criteria ensures good spreading in all
possible subspace projections. This ensures that even when unknown unimportant
features are present the resulting design still exhibits desirable space-
filling properties in the effective lower dimensional space.

The FE simulations performed on MVEs are deterministic in nature therefore it
is assumed that similar MVEs will produce similar responses. The idea of
space-filling designs is to therefore spread points apart in the input space
as efficiently as possible. Close points are undesirable as they may produce
similar simulation results therefore wasting computational resources. The
maximin distance design is tasked with identifying a design which maximizes
the minimum pair-wise distance across all points [50, 26]. Specifically,

$\displaystyle\max_{\mathcal{D}}\min_{i,j}d(\bm{x}_{i},\bm{x}_{j}),$ (3)

where $\mathcal{D}$ is the constructed design (collection of $\bm{x}$’s) and
$d$ is the Euclidean distance. Here we adopt a conditional maximin (cMm)
approach where a greedy algorithm is used to select points from a candidate
set one at a time. First an initial random point is selected followed by
addition of sequential points which step-by-step minimize the objective
function. This is sub-optimal but is relatively simple to implement. Note that
due to its construction this criteria tends to push points towards corners in
the design space.

One possible deficiency of the maximin design is that it does not account for
projections of the design onto subspaces e.g. x-y-z onto the x-y plane. In
experimental design there is a concept of effect sparsity which assumes that
some features or settings may be unimportant [25]. For instance, consider a
maximin design in three dimensions where, unknown to the scientist or
engineer, the third variable is unimportant. The three dimensional problem
collapses to a two dimensional problem and there is no guarantee that design
points are space-filling in this projection e.g. there may be overlap. This
deficiency was addressed the maximum projection (maxPro) design [33],

$\displaystyle\min_{\mathcal{D}}\sum_{i}^{n-1}\sum_{j=i+1}^{n}\frac{1}{\prod_{l=1}^{p}|d^{k}(\bm{x}_{il},\bm{x}_{jl})|^{2}},$
(4)

where $p$ is the dimensionality of the variable. This objective function
includes a product across all dimension-wise distances which penalizes overlap
in all possible projections of the data. An example three dimensional design
and projection schematic is shown in Fig. 5.

Figure 6: Example designs created from a three dimensional candidate data set
consisting of 1000 points from $Unif(-5,5)$ and 500 points from
$\mathcal{N}(0,1)$. The three dimensional design and one two-dimensional
projection is shown.

Finally the third design criteria considered is a data twinning approach [34,
51]. This approach seeks to subsample the candidate data set and identify
points which ensures both good coverage of the input space and emulation of
the parent data set density. The optimization problem is beyond the scope of
this work and readers are referred to the original works. In the context of
the present problem data twinning may be useful if (1) the data set is non-
uniform e.g. there are more examples of one crystallographic texture than
another and (2) if the neural network can be tailored to perform better in
these possible high density regions. The latter point is important because the
surrogates will be evaluated against a validation dataset; if the validation
data set is biased towards certain microstructures then it is beneficial to
tailor the training data set and model to be accurate on those structures.
This latter point is highly problem specific and even philosophical. The end
user must decide if they prefer uniform coverage or preference for more likely
structures. Furthermore, it is not clear if the second point is true for the
micromechanical surrogate model considered here. GPs behave as interpolators
and, hence, having a higher density of points in high density regions will
selectively improve model performance near this space. CNNs may not
necessarily behave this way. Nonetheless this design criteria will be
considered for evaluation.

A three dimensional example illustrating these design strategies is shown in
Fig. 6. Three data set sizes are considered to capture each design criteria’s
sensitivity to size. Three dimensional scatter plots and a two dimensional x-y
projection are shown to demonstrate the effects of the effect sparsity
principle. Visually it is clear that random does a rather poor job generally.
Large samples are needed to eventually obtain a uniform distribution in both
three dimensions and the subspace projection. While cMm does efficiently fill
the 3D space nicely there is significant overlap in points when projected to
the x-y plane. This is most pronounced in the 10-point design where the
criteria pushes nearly all points into corners. Consider that this problem
compounds even further in higher dimensions; there are $2^{d}$ corners in a
$d$-dimensional space. The maxPro designs perform uniformly well in all cases
with particularly good performance in the x-y projection. Finally, twinning
performs as expected by balancing spreading of points along with capturing the
distribution of the original candidate data set.

###  2.4 Surrogate model architecture

A U-net architecture was used to emulate the micromechanics FE model and
predict all six components of the stress tensor. We found that predicting all
six components was more effective than simply predicting the von Mises stress.
This is possibly be due to correlations between the outputs which provides the
network with additional information and constraint during training. Inputs to
the model are three-dimensional microstructural representations represented by
a $[32,32,32,3]$ array with orientation information encoded via Euler angles.
The U-net architecture consists of three resolution depths
($32^{3},16^{3},8^{3}$) with each resolution using ($64,128,256$) filters. At
each resolution, during both down-sampling and up-sampling, there are four
layers of three-dimensional CNNs producing feature maps. Residual connections
are used throughout to enable gradient flow [52]. Batch normalization and
dropout layers ($p=0.05$) are also used throughout. A MLP with dimensions
$[256,64,16,6]$ is used to map feature channels at the end of the U-net to the
six output stress components. Finally, a $L_{2}$ penalty weight of 0.0001 was
used on all weights and biases throughout the model. We found this to be
imperative to avoid over-fitting especially for small data set sizes.

The model was implemented in Tensorflow and trained using an Adam optimizer
with default parameters, a learning rate of $10^{-3}$, and batch size of 64
[53]. Training was performed on 80GB Nvidia A100 GPUs for a total of 200
epochs.

##  3 Results

###  3.1 Microstructural representation

In Fig. 7 grain size histograms are shown for designs constructed using the
three microstructure feature extraction methods (VAE, contrastive,
microstructural) and three design criteria (cMm, maxPro, twin). Interestingly,
across all designs contrastive features produce nearly uniform grain size
distributions. cMm and maxPro designs with microstructural features seem to be
prefer smaller grain sizes and neglect MVEs with preferred crystallographic
texture. Conversley, cMm and maxPro designs on the VAE features are heavily
weighted to neglect small grain size MVEs. We suspect that this disparity may
be due to the dimensionality of the problem; contrastive features are
16-dimensional, microstructural 18-dimensional, and VAE features
512-dimensional. The latter was essential as the VAE is voxel-by-voxel trained
to reconstruct an input MVE from the latent space. Good reconstruction
performance, which needs to capture localized features throughout the volume,
necessitated a high dimensional latent space. Hence, for designs such as cMm
and maxPro, which seek to “push” points away from another, larger
dimensionality representations will have a tendency to push points towards
boundaries. In Fig. 8 several two-dimensional projections of the VAE latent
space are shown. Markers are colored according to their grain size. Visually
it appears that small grain MVEs are represented closer to the original in the
latent space. This explains why both cMm and maxPro designs, which are
designed to “push” points towards boundaries, would under-represent fine grain
sizes. The twin design, however, balances both a space-filling objective and a
distributional objective so that the design exhibits similar statistical
properties as the full data set. Hence, the twin design criteria does not
neglect fine grain sizes and proportionally represents untextured/textured
examples.

Figure 7: Distribution of grain sizes selected by each combination of design
(cMm, maxPro, twin) and microstructural descriptors (contrastive, VAE,
microstructural). Figure 8: Latent space projections produced by the VAE. All
data is shown but then data corresponding to uniformly textured material is
filled with colors corresponding to grain size (centroid also includes labels
4, 10, 16).

An important diagnostic to assess the VAE’s behavior is to test the continuity
of the latent space. Recall that in a VAE there is both a reconstruction loss
and a KL term which encourages the latent space to be Gaussian and continuous.
This is important since designs are being constructed using latent distances
and, hence, obtaining a latent space where distances are meaningful is
critical. Shown in Fig. 9 are two dimensional reconstructions from the latent
space. The first column corresponds to a reconstruction of examples from the
validation data set. Each column after that is a reconstruction obtained from
a linear mapping of the latent vector
$\left\\{1.2\cdot\bm{z},\ldots,2.0\cdot\bm{z}\right\\}$. Each row is a
different example. These figures confirm that indeed the VAE latent space is
continuous with localized features being continuously manipulatable via
perturbation of the latent representation.

Figure 9: First column represents a two dimensional slice of a reconstructed
$\hat{\bm{x}}$ from the validation data set. All other columns are
interpolations from the latent space corresponding to $\alpha\cdot\bm{z}$
where $\alpha$ was varied from 1.2 to 2 linearly. These results indicate that
the latent representation is indeed continuous and captures localized
microstructural features.

The contrastive latent space, shown in Fig. 10, is by comparison drastically
different. All data is shown but untextured examples are colored according to
grain size. Visually it is clear that the latent space representation exhibits
strong structural patterns with respect to both crystallographic texture and
grain size; some regions are exclusively for untextured and there is a
continuous gradation of grain size. The distance matrix for all untextured
examples, shown in Fig. 11, further demonstrates the networks ability to
discriminate across grain sizes. Note that for large grain sizes, about 12
voxels and above, the network cannot discriminate. We suspect that this is
because the $16^{3}$ cropped example fed to the contrastive network is no
longer representative for relatively large grain sizes. For small grain sizes,
for instance the block corresponding to all 4-on-4 comparisons, distances are
small indicating that the network clusters all examples together suggesting
that it is indeed learning statistical descriptors.

Figure 10: Latent space projections of the self-supervised microstructure
statistics model. All data is shown but then data corresponding to uniformly
textured material is filled with colors corresponding to grain size (centroid
also includes labels 4, 10, 16). Clear structure is observed in the latent
space. Figure 11: Distance matrix considering only uniformly random textured
MVEs and comparing across grain sizes from 4 to 16 voxels.

The crystallographic texture sensitivity of the contrastive features is shown
in Fig. 12. Here random examples are sampled from the validation data set and
then three of the closest and three of the most distant MVEs are identified
using latent contrastive distances. Only data corresponding to examples with a
grain size of 4 voxels was considered. The same exercise is performed using
only GSH coefficients for comparison. Results indicate that the contrastive
features do indeed capture texture similarity accurately. In a few instances
there are visual anomalies but it is important to consider that the GSH
features only capture texture whereas the contrastive features capture both
texture and structural information. Nonetheless, these results indicate that
the established self-supervised feature extraction network does capture both
spatial and crystallographic features automatically.

Figure 12: $(100)$ pole figures corresponding to: random example from the data
set, the most similar instances, and most dissimilar instances. Similarity is
measured using both the contrastive latent vector and GSH features. Identical
color bar limits are used throughout.

###  3.2 Surrogate training results

Summary results for the MVE-feature/design parametric study are shown in Fig.
13. Across all possible combinations the VAE-maxPro combination scored the
highest improvement at 8.8% for the case where only 25% of the total data set
size was used. cMm with contrastive features appears to be systematically most
robust. Interestingly the performance of the VAE-maxPro combination
deteriorates completely when 50% of the total data set size was used.
Similarly, the contrastive/maxPro combination also suffers a significant
deterioration at 50% data set use, however, microstructural features still
enjoy a moderate boost. We suspect that this may be due to the difficulty in
optimizing designs. Recall that the design criteria is represented by a scalar
valued objective function; constructing the design requires solving a high
dimensional optimization problem. The maxPro objective function, Eqn. 4, is
rather challenging to optimize as it contains $n^{2}-n$ terms with each term a
product of $p$ terms. In the VAE case $p=512$ and for 50% ($n=3413$) there are
over 11-million terms. Note that some of this complexity is reduced by using a
greedy optimization strategy but, nonetheless, the optimization problem
remains challenging. However, the contrastive ($p=16$) and microstructural
($p=18$) roughly are of the same dimension and yet only the contrastive
features deteriorate at 50% data set size using the maxPro criteria. It may be
possible that this is because the contrastive structural-orientation features
are coupled whereas for the constructed microstructural feature vector they
are not. The maxPro criteria considers distances in projected spaces which
will make optimization more challenging if features are entangled. The
decoupled (grain size and crystallographic texture) microstructural features
are less susceptible to this effect.

Somewhat remarkably the data twinning design performed nearly identically
across all considered MVE features with a nearly constant 5% boost. This may
be because the design criteria has a balanced objective function which also
considers the underlying data distribution. If the problem dimensionality is
large, and the desired design size small, then cMm and maxPro can have a
tendency to push points towards extreme “corners” producing designs which
perform poorer than random designs. It is suspected that the data twinning
designs mitigate against this by penalizing designs which do not emulate the
original data distribution. This also may be why a slight decline in
performance is observed at 50%; eventually random sampling will begin to
represent the underlying data distribution and, hence, the twinning design’s
benefit will decay.

Finally, consider that for all designs-feature combinations, with the
exception of maxPro/cMm and microstructural features, there is no
statistically significant boost at 10% of the total data set size. However,
for cMm/maxPro designs and microstructural features there is a statistically
significant decline in performance relative to random designs. Previously it
was hypothesized that for large data set sizes the entangling of features may
make optimization of the design criteria more difficult. In the case of small
data set sizes we argue that the converse may hold true; entanglement aids in
avoiding “corner biased” designs. When features are independent the pace-
filling criteria may bias the data set towards certain grain size and texture
corners of the 18-dimensional microstructural feature space. Increases in the
data set size remedy this behavior and result in monotonically increasing
performance.

Figure 13: Parametric results for all considered microstructural features and
design criteria. Validation loss improvement is compared against models
trained from randomly selected training designs. For each feature/design
combination 10 designs were used to train 10 models. Error bars correspond to
one standard deviation computed via bootstrapping.

Validation loss curves for a few select feature and design combinations are
shown in Fig. 14. Note that the loss here corresponds to the mean of centered
and normalized components of the stress tensor e.g. the value is a non-
dimensional quantity. Visually it is clear that indeed at certain data set
sizes there appears a significant improvement in surrogate model performance
as summarized in Fig. 13. The loss curves at 10% data set size reveal that the
loss curves appear to be somewhat unstable exhibiting a great deal of
variance. This reveals that for the specific architecture used the 10% (about
600 simulations) size may be at the very limit of the models training
stability which may explain some of the previously discussed anomalous results
(worse than random design results at 10%).

Figure 14: Validation loss curves for a few selected designs and features. All
ten curves shown to demonstrate repeatability.

##  4 Discussion

A number of validation example MVEs, FE results, and surrogate results are
shown in Fig. 15. For visual comparison surrogate fields are shown as absolute
percent error relative to the FE results. While there are subtle differences
in responses across the various designs these results do not immediately shed
any light on the impact of features and the design of surrogate performance.
To further explore any potential insights the best 18 and worst 18 performance
MVEs from a contrastive cMm with 25% data are shown in Fig. 16. Remarkably,
the best and worst performing structures all visually appear to be extremely
similar.

Figure 15: Random examples from validation data: IPF map, $\sigma_{33}$ of the
stress tensor, and absolute error maps for four designs (using contrastive
features and 25% of the dataset). Examples are selected from one of the ten
surrogate realizations with validation loss closest to the mean validation
loss. Figure 16: Example microstructures with best (top) and worst (bottom)
volume averaged error. Provided number is the volume averaged $\sigma_{33}$
standard error.

Surrogate model performance trends as a function of microstructural features
are shown in Fig. 17. The mean standard error 1 and 99-percentile bounds are
shown for all grain sizes and textured/untextured structures. It is clear that
for all design criteria there are far more extreme value prediction MVEs
exhibiting both very large and small errors at large grain sizes. This
indicates that the “tails” of the surrogates’ performance are much wider for
large grain sizes. It also appears that this effect is even more pronounced
for textured materials. A possible explanation is that for small grain size
the MVEs are behaving as representative volume elements (RVEs) which capture
aggregate material behavior well. Furthermore, RVEs may intrinsically contain
more information; a single volume contains hundreds of pairs of small grain
neighbor combinations. In large grain size MVEs each example will contain far
fewer of these neighbor pairs. This is important because the ML surrogate must
be exposed to a diversity of structures to effectively learn key physical
relationships. Texture adds another degree of complexity that results in more
extreme value performance responses. Not only do spatial neighborhood
relationships need to be learned but also the response as a function of
specific aggregate crystallographic textures.

Figure 17: 1 and 99 percentiles and mean volume-averaged standard error for
various designs using contrastive features and 25% of the data set. Data is
shown as a function of grain size and texture/untextured. Results indicate
that while mean behavior is similar across all microstructures large grain
textured examples tend to exhibit a fatter tail of predictive capacity.

The presented results demonstrate that micromechanical surrogate models
trained using well designed MVEs can enjoy a moderate boost in overall
performance when compared against randomly selected training MVEs. However,
the parametric study over several MVE feature descriptors and design criteria
revealed that net performance is dependent on many factors and these are
difficult to assess a-priori. For the elastic constitutive model considered
here the “cost” of this uncertainty is rather low because evaluation of the FE
model is computationally rather inexpensive. However, for more advanced
inelastic constitutive models significant computational cost is incurred when
evaluating the model and, hence, it is imperative to identify designs that
will not result in worse than random designs.

A key consideration when constructing designs is the optimization complexity
for a specific data set size and problem dimensionality. Results indicated
that while maxPro performed well for 25% data set sizes for both VAE and
contrastive features, performance deteriorated completely at 50% data set size
resulting in worse or no performance over random designs. Our hypothesis is
that this is because construction of a design requires solving an optimization
problem and, hence, the bottleneck in the process is the design optimization
step. In the VAE case both the relatively small data set size and the
dimensionality of the problem ($p=512$) cause issues. For problems with much
larger data set sizes some of these issues may be alleviated.

While the cMm design is inferior to maxPro with respect to projection quality
the cMm objective is much more easy to optimize. Furthermore, optimization is
even more feasible for small MVE latent state representations. This is likely
why the cMm-contrastive approach yielded consistently good results
(monotonically increasing performance) with increasing data set size. While
the microstructural-cMm combination also yielded monotonically increasing
performance in the case where surrogates were trained used 10% of the data set
size these designs yielded worse than random performance. It is suspected that
this may be because of the decoupled nature of the microstructural features
(grain size and volume averaged texture). Contrastive features are capable of
capturing richer coupled richer features which may offer opportunity for
ensuring diversity. Finally, the data twinning approach, while robust against
poorer than random performance, only provides moderate boosts for moderate
data set sizes. The robustness is believed to be due to the tendency to
construct designs to emulate the original data set’s distribution and, hence,
mitigate against designs dominated by anomalies e.g. clustering in high
dimensional corners. However, this comes at the cost of locating points close
to one another which minimizes overall diversity.

Based on these results, for the specific surrogate architecture and
micromechanical model considered, the most promising feature/design
combination is the contrastive conditional maximin design. While a rather
moderate 8% boost in validation performance was achieved this may be even
better for other domain problems and larger data set sizes. Consider a case
where instead of 6,825 MVE candidate structures are available there are
instead 68,250 and a total budget of 20,000 simulations to be performed. The
larger candidate set size would provide more opportunity to diversify the
design and a more expansive data set would allow for more efficient packing of
the feature space. Furthermore, based on the results from Fig. 17 it is clear
that more examples are needed for the less representative large grain textured
MVEs. Finally, it should be noted much of these results

##  5 Conclusions

The proposed feature-extraction and experimental design strategy for
establishing micromechanics surrogates has been demonstrated to be effective.
The strategy consists of two key components: (1) a microstructural feature for
computing a distance or similarity metric and (2) a design strategy for
distributing points in the microstructural feature space. A parametric study
was performed over three design strategies and three microstructural features.
Results show that for the considered problem a reduction up to 8% of the
validation loss is achieved when compared to random selection of training
examples. Trends indicate that for bigger data sets the benefits may be even
larger. This is rationalized by the notion that high-dimensional spaces are
difficult to fill, and hence, larger designs will facilitate more efficient
sampling of the microstructural feature space. This in turn ensures a more
diverse training data set which greatly improves model performance. In
addition to demonstrating this result this work also establishes a novel self-
supervised contrastive feature extraction methodology for computing
microstructural summary statistics.

##  6 Data availability

Data will be made available from the authors upon reasonable request.

## Acknowledgements

Research was sponsored by the US Department of Energy, Office of Energy
Efficiency and Renewable Energy (EERE), Advanced Manufacturing Office, and
Advanced Materials and Manufacturing Technologies Office (AMMTO) under
contract DE-AC05-00OR22725 with UT-Battelle LLC and performed in partiality at
the Oak Ridge National Laboratory’s Manufacturing Demonstration Facility, an
Office of Energy Efficiency and Renewable Energy user facility. All the
authors would like to acknowledge the support of the HPC4Mtls program.

## References

  * De Pablo et al. [2014] Juan J De Pablo, Barbara Jones, Cora Lind Kovacs, Vidvuds Ozolins, and Arthur P Ramirez.  The materials genome initiative, the interplay of experiment, theory and computation.  _Current Opinion in Solid State and Materials Science_ , 18(2):99–117, 2014.
  * Torquato and Haslach Jr [2002] Salvatore Torquato and Henry W Haslach Jr.  Random heterogeneous materials: microstructure and macroscopic properties.  _Appl. Mech. Rev._ , 55(4):B62–B63, 2002.
  * Fernandez-Zelaia et al. [2024] Patxi Fernandez-Zelaia, Jiahao Cheng, Jason Mayeur, Amir Koushyar Ziabari, and Michael M Kirka.  Digital polycrystalline microstructure generation using diffusion probabilistic models.  _Materialia_ , 33:101976, 2024.
  * Buzzy et al. [2024] Michael O Buzzy, Andreas E Robertson, and Surya R Kalidindi.  Statistically conditioned polycrystal generation using denoising diffusion models.  _Acta Materialia_ , page 119746, 2024.
  * Robertson et al. [2023] Andreas E Robertson, Conlain Kelly, Michael Buzzy, and Surya R Kalidindi.  Local–global decompositions for conditional microstructure generation.  _Acta Materialia_ , 253:118966, 2023.
  * Lee and Yun [2023] Kang-Hyun Lee and Gun Jin Yun.  Microstructure reconstruction using diffusion-based generative models.  _Mechanics of Advanced Materials and Structures_ , pages 1–19, 2023.
  * Zeni et al. [2023] Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabbé, Lixin Sun, Jake Smith, et al.  Mattergen: a generative model for inorganic materials design.  _arXiv preprint arXiv:2312.03687_ , 2023.
  * Yang et al. [2023] Mengjiao Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Dogus Cubuk.  Scalable diffusion for materials generation.  _arXiv preprint arXiv:2311.09235_ , 2023.
  * Sacks et al. [1989] Jerome Sacks, Susannah B Schiller, and William J Welch.  Designs for computer experiments.  _Technometrics_ , 31(1):41–47, 1989.
  * Kennedy and O’Hagan [2001] Marc C Kennedy and Anthony O’Hagan.  Bayesian calibration of computer models.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 63(3):425–464, 2001.
  * Gramacy [2020] Robert B Gramacy.  _Surrogates: Gaussian process modeling, design, and optimization for the applied sciences_.  Chapman and Hall/CRC, 2020.
  * Mak et al. [2018] Simon Mak, Chih-Li Sung, Xingjian Wang, Shiang-Ting Yeh, Yu-Hung Chang, V Roshan Joseph, Vigor Yang, and CF Jeff Wu.  An efficient surrogate model for emulation and physics extraction of large eddy simulations.  _Journal of the American Statistical Association_ , 113(524):1443–1456, 2018.
  * Chen et al. [2021] Jialei Chen, Simon Mak, V Roshan Joseph, and Chuck Zhang.  Function-on-function kriging, with applications to three-dimensional printing of aortic tissues.  _Technometrics_ , 63(3):384–395, 2021.
  * Damianou and Lawrence [2013] Andreas Damianou and Neil D Lawrence.  Deep gaussian processes.  In _Artificial intelligence and statistics_ , pages 207–215. PMLR, 2013.
  * Dai et al. [2015] Zhenwen Dai, Andreas Damianou, Javier González, and Neil Lawrence.  Variational auto-encoded deep gaussian processes.  _arXiv preprint arXiv:1511.06455_ , 2015.
  * Tran and Wildey [2021] Anh Tran and Tim Wildey.  Solving stochastic inverse problems for property–structure linkages using data-consistent inversion and machine learning.  _JOM_ , 73(1):72–89, 2021.
  * Liang et al. [2018] Liang Liang, Minliang Liu, Caitlin Martin, and Wei Sun.  A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis.  _Journal of The Royal Society Interface_ , 15(138):20170844, 2018.
  * He et al. [2023] Junyan He, Seid Koric, Shashank Kushwaha, Jaewan Park, Diab Abueidda, and Iwona Jasiuk.  Novel deeponet architecture to predict stresses in elastoplastic structures with variable complex geometries and loads.  _Computer Methods in Applied Mechanics and Engineering_ , 415:116277, 2023.
  * Montes de Oca Zapiain et al. [2022a] David Montes de Oca Zapiain, Apaar Shanker, and Surya R Kalidindi.  Convolutional neural networks for the localization of plastic velocity gradient tensor in polycrystalline microstructures.  _Journal of Engineering Materials and Technology_ , 144(1):011004, 2022a.
  * Yabansu et al. [2014] Yuksel C Yabansu, Dipen K Patel, and Surya R Kalidindi.  Calibrated localization relationships for elastic response of polycrystalline aggregates.  _Acta Materialia_ , 81:151–160, 2014.
  * Khorrami et al. [2023] Mohammad S Khorrami, Jaber R Mianroodi, Nima H Siboni, Pawan Goyal, Bob Svendsen, Peter Benner, and Dierk Raabe.  An artificial neural network for surrogate modeling of stress fields in viscoplastic polycrystalline materials.  _npj Computational Materials_ , 9(1):37, 2023.
  * Wang et al. [2021] Yinan Wang, Diane Oyen, Weihong Guo, Anishi Mehta, Cory Braker Scott, Nishant Panda, M Giselle Fernández-Godino, Gowri Srinivasan, and Xiaowei Yue.  Stressnet-deep learning to predict stress with fracture propagation in brittle materials.  _Npj Materials Degradation_ , 5(1):6, 2021.
  * Pokharel et al. [2021] Reeju Pokharel, Anup Pandey, and Alexander Scheinker.  Physics-informed data-driven surrogate modeling for full-field 3d microstructure and micromechanical field evolution of polycrystalline materials.  _JOM_ , 73(11):3371–3382, 2021.
  * Pandey and Pokharel [2021] Anup Pandey and Reeju Pokharel.  Machine learning based surrogate modeling approach for mapping crystal deformation in three dimensions.  _Scripta Materialia_ , 193:1–5, 2021.
  * Wu and Hamada [2011] CF Jeff Wu and Michael S Hamada.  _Experiments: planning, analysis, and optimization_.  John Wiley & Sons, 2011.
  * Joseph [2016] V Roshan Joseph.  Space-filling designs for computer experiments: A review.  _Quality Engineering_ , 28(1):28–35, 2016.
  * Bessa et al. [2017] Miguel A Bessa, Ramin Bostanabad, Zeliang Liu, Anqi Hu, Daniel W Apley, Catherine Brinson, Wei Chen, and Wing Kam Liu.  A framework for data-driven analysis of materials under uncertainty: Countering the curse of dimensionality.  _Computer Methods in Applied Mechanics and Engineering_ , 320:633–667, 2017.
  * Paulson et al. [2017] Noah H Paulson, Matthew W Priddy, David L McDowell, and Surya R Kalidindi.  Reduced-order structure-property linkages for polycrystalline microstructures based on 2-point statistics.  _Acta Materialia_ , 129:428–438, 2017.
  * He et al. [2024] Junyan He, Deepankar Pal, Ali Najafi, Diab Abueidda, Seid Koric, and Iwona Jasiuk.  Material-response-informed deeponet and its application to polycrystal stress-strain prediction in crystal plasticity.  _arXiv preprint arXiv:2401.09977_ , 2024.
  * Montes de Oca Zapiain et al. [2022b] David Montes de Oca Zapiain, Mitchell A Wood, Nicholas Lubbers, Carlos Z Pereyra, Aidan P Thompson, and Danny Perez.  Training data selection for accuracy and transferability of interatomic potentials.  _npj Computational Materials_ , 8(1):189, 2022b.
  * Barneschi et al. [2024] Leonardo Barneschi, Leonardo Rotondi, and Daniele Padula.  Molecular geometry impact on deep learning predictions of inverted singlet–triplet gaps.  _The Journal of Physical Chemistry A_ , 2024.
  * Sener and Savarese [2017] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_ , 2017.
  * Joseph et al. [2015] V Roshan Joseph, Evren Gul, and Shan Ba.  Maximum projection designs for computer experiments.  _Biometrika_ , 102(2):371–380, 2015.
  * Vakayil and Joseph [2022] Akhil Vakayil and V Roshan Joseph.  Data twinning.  _Statistical Analysis and Data Mining: The ASA Data Science Journal_ , 15(5):598–610, 2022.
  * Yaghoobi et al. [2019] Mohammadreza Yaghoobi, Sriram Ganesan, Srihari Sundar, Aaditya Lakshmanan, Shiva Rudraraju, John E Allison, and Veera Sundararaghavan.  Prisms-plasticity: An open-source crystal plasticity finite element software.  _Computational Materials Science_ , 169:109078, 2019.
  * Fernandez-Zelaia et al. [2022] Patxi Fernandez-Zelaia, Yousub Lee, Sebastien Dryepondt, and Michael M Kirka.  Creep anisotropy modeling and uncertainty quantification of an additively manufactured ni-based superalloy.  _International Journal of Plasticity_ , 151:103177, 2022.
  * Quey and Kasemer [2022] Romain Quey and Matthew Kasemer.  The neper/fepx project: free/open-source polycrystal generation, deformation simulation, and post-processing.  In _IOP Conference Series: Materials Science and Engineering_ , volume 1249, page 012021. IOP Publishing, 2022.
  * Bunge [2013] H-J Bunge.  _Texture analysis in materials science: mathematical methods_.  Elsevier, 2013.
  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep learning_.  MIT press, 2016.
  * Higgins et al. [2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  _ICLR (Poster)_ , 3, 2017.
  * Hashemi et al. [2024] Sayed Sajad Hashemi, Michael Guerzhoy, and Noah H Paulson.  Toward learning latent-variable representations of microstructures by optimizing in spatial statistics space.  _arXiv preprint arXiv:2402.11103_ , 2024.
  * Larroza et al. [2016] Andrés Larroza, Vicente Bodí, David Moratal, et al.  Texture analysis in magnetic resonance imaging: review and considerations for future applications.  _Assessment of cellular and organ function and dysfunction using direct and derived MRI methodologies_ , pages 75–106, 2016.
  * Liu et al. [2019] Xinlong Liu, Chu He, Qingyi Zhang, and Mingsheng Liao.  Statistical convolutional neural network for land-cover classification from sar images.  _IEEE Geoscience and Remote Sensing Letters_ , 17(9):1548–1552, 2019.
  * Presberger et al. [2024] Jannik Presberger, Rashmiparvathi Keshara, David Stein, Yung Hae Kim, Anne Grapin-Botton, and Bjoern Andres.  Correlation clustering of organoid images.  _arXiv preprint arXiv:2403.13376_ , 2024.
  * Chopra et al. [2005] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , volume 1, pages 539–546. IEEE, 2005.
  * Goldberger et al. [2004] Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov.  Neighbourhood components analysis.  _Advances in neural information processing systems_ , 17, 2004.
  * Bromley et al. [1993] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  Signature verification using a” siamese” time delay neural network.  _Advances in neural information processing systems_ , 6, 1993.
  * Balestriero and LeCun [2022] Randall Balestriero and Yann LeCun.  Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods.  _Advances in Neural Information Processing Systems_ , 35:26671–26685, 2022.
  * Hadsell et al. [2006] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06)_ , volume 2, pages 1735–1742. IEEE, 2006.
  * Johnson et al. [1990] Mark E Johnson, Leslie M Moore, and Donald Ylvisaker.  Minimax and maximin distance designs.  _Journal of statistical planning and inference_ , 26(2):131–148, 1990.
  * Mak and Joseph [2018] Simon Mak and V Roshan Joseph.  Support points.  2018\.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Abadi et al. [2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: A system for large-scale machine learning.  In _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, pages 265–283, 2016.


-titleHadron Collider Physics Symposium 2011 11institutetext: Laboratoire de l’Accélérateur Linéaire, Orsay, France

# Photon polarisation in $b\rightarrow s\gamma$ using $B\rightarrow$ K∗e+e- at
LHCb

Michelle Nicol (on behalf of the LHCb collaboration) 11<EMAIL_ADDRESS>
###### Abstract

The $b\rightarrow s\gamma$ transition proceeds through flavour changing
neutral currents, and thus is particularly sensitive to the effects of new
physics. An overview of the method to measure the photon polarisation at the
LHCb experiment via an angular analysis of $B\rightarrow K^{*}e^{+}e^{-}$ at
low $q^{2}$ is presented. The status of the $B\rightarrow K^{*}\mu^{+}\mu^{-}$
analysis with 309 pb-1 of $pp$ collisions at $\sqrt{s}$=7 TeV at LHCb is also
given.

##  1 Introduction

Although the branching ratio of $b\rightarrow s\gamma$ has been measured to be
consistent with Standard Model (SM) predictions, new physics could still be
present and detectable through the analysis of details of the decay process.
In particular, the photon from the b is predominantly left handed in the SM,
whereas additional right handed currents can arise in certain new physics
models, such as the Left-Right symmetric models, or in some supersymmetric
models nonSM . Access to the polarisation information is available via an
angular analysis of $B\rightarrow K^{*}e^{+}e^{-}$.

Hadronic form factors render theoretical prediction over the whole $q^{2}$
(the dilepton invariant mass squared) range difficult. However, it has been
shown that these uncertainties are controllable at low $q^{2}$, where the
photon term dominates, and certain asymmetries providing information on the
photon polarisation can be formed. RefJ

##  2 $B\rightarrow K^{*}\mu^{+}\mu^{-}$ status at LHCb

With 309 pb-1 of $pp$ collisions at $\sqrt{s}$=7 TeV, collected in three
months during the first half of 2011, the forward backward asymmetry of the
dilepton system, $A_{FB}$ has been measured muon  using $B\rightarrow
K^{*}\mu^{+}\mu^{-}$ events, (as is shown in Fig. 1), along with $F_{L}$, the
K∗ longitudinal polarisation (Fig. 1); an input required for the photon
polarisation measurement. These observables have been measured as being in
good agreement with SM predictions, SM , implying a SM like Wilson Coefficient
$C_{7}$, but still allowing for the existence of $C_{7}^{{}^{\prime}}$ (right
handed currents). As stressed above, the measurement is most sensitive at low
$q^{2}$. It would therefore be preferable to perform the analysis using
electrons. However, experimentally it is more challenging to observe electrons
than muons, primarily due to the fact that muons provide a very clean
signature to trigger on. With 309 pb-1 of LHCb data, $B\rightarrow
K^{*}\mu^{+}\mu^{-}$ in the $q^{2}$ range 0-2 GeV has been observed, as is
shown, along with other $q^{2}$ ranges, in Fig. 2. With the rest of the 2011
data, one can expect to see a $B\rightarrow K^{*}e^{+}e^{-}$ signal.

Figure 1: $A_{FB}$ and FL as a function of $q^{2}$, as measured at LHCb with
$B\rightarrow K^{*}\mu^{+}\mu^{-}$ muon . The SM predictions are given by the
cyan (light) band, and this prediction integrated in the $q^{2}$ bins is
indicated by the purple (dark) regions.

Figure 2: The mass distributions of $B\rightarrow K^{*}\mu^{+}\mu^{-}$ in six
$q^{2}$ bins. The solid line shows a fit with a double-Gaussian signal
component (thin-green line) and an exponential background component (dashed-
red line).

Figure 3: Definition of the angles $\phi$, $\theta_{K}$ and $\theta_{L}$ in
the decay $B\rightarrow K^{*}e^{+}e^{-}$.

##  3 Analysis formalism

$B\rightarrow K^{*}e^{+}e^{-}$ can be uniquely described by four variables:
$q^{2}$ and three angular variables, $\theta_{L}$, $\theta_{K}$ and $\phi$,
(the definitions of which can be seen in Fig. 3). Following the formalism as
described in krug , the differential decay distribution can be written in
terms of these variables as:

$\displaystyle\frac{d\Gamma}{dq^{2}d\cos\Theta_{l}d\cos\Theta_{K}d\phi}=$
$\displaystyle\frac{9}{32\pi}[I_{1}\left(\cos\Theta_{K}\right)+I_{2}\left(\cos\Theta_{K}\right)\cos
2\Theta_{l}+$ $\displaystyle
I_{3}\left(\cos\Theta_{K}\right)\sin^{2}\Theta_{l}\cos
2\phi+I_{4}\left(\cos\Theta_{K}\right)\sin 2\Theta_{l}\cos\phi+$
$\displaystyle
I_{5}\left(\cos\Theta_{K}\right)\sin\Theta_{l}\cos\phi+I_{6}\left(\cos\Theta_{K}\right)\cos\Theta_{l}+$
$\displaystyle
I_{7}\left(\cos\Theta_{K}\right)\sin\Theta_{l}\sin\phi+I_{8}\left(\cos\Theta_{K}\right)\sin
2\Theta_{l}\sin\phi\ +$ $\displaystyle
I_{9}\left(\cos\Theta_{K}\right)\sin^{2}\Theta_{l}\sin 2\phi]$ (1)

When measuring this rate at LHCb, the 3D angular acceptance,
$\varepsilon\left(\cos\Theta_{l},\cos\Theta_{K},\phi\right)$ must also be
taken into account. It is assumed to be factorisable as the products of
$\varepsilon_{1}$, the acceptance as a function of $\phi$, and
$\varepsilon_{D}$, the acceptance as a function of $\cos\Theta_{K}$ and
$\cos\Theta_{L}$. Furthermore, assuming that $\varepsilon_{1}$ is an even
function, Equation 3 can be simplified by performing the $\phi$ transformation
that if $\phi$ $>$0, then $\phi$=$\phi$+$\pi$. A similar transformation can be
performed for $\cos\Theta_{L}$. Equation 3 can then be written as:

$\displaystyle\frac{d\Gamma}{dq^{2}d\cos\Theta_{l}d\cos\Theta_{K}d\phi}=$
$\displaystyle\frac{9}{32\pi}[I_{1}\left(\cos\Theta_{K}\right)+I_{2}\left(\cos\Theta_{K}\right)\cos
2\Theta_{l}+$ $\displaystyle
I_{3}\left(\cos\Theta_{K}\right)\sin^{2}\Theta_{l}\cos
2\phi+I_{9}\left(\cos\Theta_{K}\right)\sin^{2}\Theta_{l}\sin 2\phi]$
$\displaystyle\times\varepsilon_{D}\left(\cos\Theta_{l},\cos\Theta_{K}\right)$
(2)

In order to minimize theoretical uncertainties, it is desirable to measure
ratios of the amplitudes. Neglecting the lepton mass, the remaining I terms in
equation 3 can be written in terms of three such parameters,
$\mathrm{F_{L},A_{T}^{(2)},A_{Im}}$:

$\begin{split}\mathrm{F_{L}}&=\frac{\mathrm{\left|A_{0}\right|^{2}}}{\mathrm{\left|A_{0}\right|^{2}}+\left|A_{\bot}\right|^{2}+\left|A_{\|}\right|^{2}}\\\
\mathrm{A_{T}^{(2)}}&=\frac{\mathrm{\left|A_{\bot}\right|^{2}-\left|A_{\|}\right|^{2}}}{\mathrm{\left|A_{\bot}\right|^{2}+\left|A_{\|}\right|^{2}}}\\\
\mathrm{A_{Im}}&=\frac{\mathrm{\Im(A^{*}_{\bot L}A_{\bot L})-\Im(A^{*}_{\bot
R}A_{\bot
R})}}{\mathrm{\left|A_{0}\right|^{2}}+\left|A_{\bot}\right|^{2}+\left|A_{\|}\right|^{2}}\end{split}$
(3)

When expressed in terms of the helicity amplitudes, for small real values of
$\frac{A_{R}}{A_{L}}$, one obtains
$A_{T}^{(2)}\approx-2\frac{A_{right}}{A_{left}}$.

##  4 $B\rightarrow$ K∗e+e- Monte Carlo studies

Although work is ongoing on the analysis of the $B\rightarrow$ K∗e+e- data,
and yield predictions from Monte Carlo (MC) have been validated using the
control channel $B\rightarrow$ K∗J$/\Psi$ with J$/\Psi\rightarrow$(e+e-),
there is not yet, at the time of this conference, enough data to perform the
analysis or test the fitting procedure. Toy MC studies have therefore been
carried out for this purpose schune . 190k signal events were generated using
EvtGen, and separated into files containing 250 events: the predicted yields
from MC studies with 2fb-1 at a centre of mass energy of 14 TeV, excluding
effects from LHCb’s high level trigger. By performing the fit on each file, it
is shown that with 200-250 signal events and a signal to background ratio of
the order of 1, a precision of 0.2 is attainable on $\mathrm{A_{T}^{2}}$,
equivalent to an accuracy on the fraction of wrongly polarised photons of 0.1.
An example of one of the fits for one toy MC study can be seen in Fig. 4. The
analysis also demonstrates that the measurements are not sensitive to the
knowledge of the angular acceptance, and hence shall not be systematics
limited.

Figure 4: Example of the fit of $\cos\Theta_{L}$, $\cos\Theta_{K}$ and $\phi$
for one toy MC study containing 250 signal events and no background events.

## References

  * (1) for example E. Lunghi and J. Matias, J. High Enerfy Physics, 04, (2007) 058
  * (2) Y. Grossman and D. Pirjol, J. High Enerfy Physics, 06, (2009) 029
  * (3) LHCB-CONF-2011-039
  * (4) C. Bobeth, G. Hiller and D. van Dyk, J. High Enerfy Physics, 07, (2010) 098
  * (5) CF. Kruger and J. Matias, Physics Rev, D71, (2005) 094009
  * (6) J. LeFrançois, M.H. Schune, LHCb-2009-008